Merge fix/827-worker-deadline

docs/VirtualTags.md

@@ -58,6 +58,27 @@ processes:
   `script timed out after …` failure and the run token is cancelled inside the worker; at `T + G`
   (`Scripting:Worker:HardGrace`, default `max(1 s, T)`) the worker is **killed** (`Process.Kill(entireProcessTree)`,
   plus a worker-side fail-fast watchdog) — the thread, its memory and its ALC go with it.
+- **`T` starts when the run starts (lmxopcua#827).** As in process (where the guard's clock starts when the thread
+  pool picks the run up), `T` starts when a worker evaluation thread starts the run — not at submit — so the pipe
+  round trip and the time a starved worker takes to **start** the run never count against the script. From the start
+  on it is wall-clock time, exactly as in process: a GC pause, a CPU-starved or suspended worker **during** the run
+  still counts against `T` (not separated, and not tested beyond that equivalence). The worker runs that clock and
+  reports the crossing (`EvalResults.soft_expired_eval_ids`); the host resumes its caller on the result dispatcher
+  (never on the pipe reader), sends the cancel, and arms the kill `G` after the notice.
+  - **Transport deadline.** What the host still times on its own is `T + G` from submit (restarted at a queued run's
+    start): no result and no soft notice by then is the runner's fault — the tag publishes **`BadResourceUnavailable`**
+    (`"script worker unresponsive"`), nothing is struck, the worker is not killed, and the seam re-evaluates when the
+    worker answers. A re-trigger while that run is still unanswered gets the same answer, not
+    `previous run still running`. If the worker then reports the script over `T` after all (slow **and** late — a
+    notice delayed past `T + G` lands here too), the run still gets its cancel and counts as a timeout. Counted as
+    `otopcua.script.ipc.timeouts{kind, answer = stalled | retrigger}`.
+  - **Progress ceiling.** The host does not depend on the worker's notices to end a run: while evaluations are
+    outstanding, a worker that keeps heartbeating but produces no result, start or soft notice for
+    `3 × the largest outstanding T + G + 3 heartbeat intervals` is killed (`killed_unresponsive`, no culprit,
+    attributed like a heartbeat kill). A worker that stops heartbeating is killed by the silence rule as before.
+  - **Negotiated.** The clock is agreed in the handshake (`Hello`/`Ready.worker_soft_clock`): a worker that does not
+    echo it keeps the host clock from submit (the pre-#827 contract), so a mismatched host/worker pair degrades to the
+    old behaviour instead of losing its deadlines.
 - **Attribution and quarantine** live in the host and survive restarts: a deadline kill strikes exactly its key
   (other evaluations in flight are victims, not strikes); a crash with no deadline crossed makes every key in flight
   a suspect that must complete `Scripting:Worker:IsolationProbationRuns` (20) clean runs in the isolation worker,
@@ -108,7 +129,7 @@ processes:
   "every process with this name", never a live host's worker (mxaccessgw's `OrphanWorkerTerminator` lesson).
 - **Observability.** On `OtOpcUaTelemetry.Meter`: `otopcua.script.runner.state` (0 ready / 1 degraded / 2 circuit
   open), `otopcua.script.worker.up` / `.inflight` / `.queue_depth` / `.heap_bytes` (tag `shard`; queue and heap from
-  the worker heartbeat), `otopcua.script.worker.starts` / `.exits` (`reason` adds `idle`), `otopcua.script.ipc.roundtrip`
+  the worker heartbeat), `otopcua.script.worker.starts` / `.exits` (`reason` adds `idle`), `otopcua.script.ipc.roundtrip`, `otopcua.script.ipc.timeouts` (transport deadline, lmxopcua#827)
   and `otopcua.script.eval.duration` (µs, tag `kind`; no per-key tag), `otopcua.script.compile` (`cache` = `hit` |
   `miss` | `rejected` | `uncached` | `negative`) and `otopcua.script.isolation.suspects`. The `script-runner` health check
   is on the **ready** tier and is at worst **Degraded** (still 200): a sick runner never fails readiness, so a pair
@@ -168,7 +189,8 @@ what is published** — and each compiled evaluation is then mirrored to the `Sc
 
 - **Compared:** value (structural for arrays), exact CLR type, status (Ok vs failed) and failure class (`compile`,
   `budget` = timed out / killed / refused / quarantined, `threw` with its message, `unsupported_type`). A shadow outcome
-  that is the runner's own fault (worker restarted, none ready, worker busy) is **not compared**
+  that is the runner's own fault (worker restarted, none ready, worker busy, worker unresponsive — a stalled or starved
+  worker past the transport deadline, lmxopcua#827) is **not compared**
   (`result = shadow_unavailable`). An input or result the protocol cannot carry (design §14 #11) counts as a mismatch
   with reason `unsupported_type` — out of process that evaluation would publish Bad.
 - **Never reaches a published value or readiness.** The mirror copies the inputs and queues the shadow run on the pool;