Commit Graph

3 Commits

Author SHA1 Message Date
Joseph Doherty 03ac57d5d6 fix(scripting): #789 review — reader faults end the worker, write stalls kill it, Cancel off the worker reader
- Host reader: the frame read and its handling are caught separately. Only the
  read's I/O exceptions mean the pipe ended; anything HandleFrame's inline
  completion work throws now kills that worker as ProtocolFault (outstanding
  runs -> worker restarted / BadResourceUnavailable) instead of escaping a
  dedicated thread (process crash) or being swallowed as a clean end (an
  InvalidOperationException used to be, leaving the worker unread until a Crash
  2 s later).
- Worker reader: the same catch-all around HandleFrame (exit ProtocolFault).
- Worker: Cancel() runs on the pool, so a script's blocking
  ctx.CancellationToken callback cannot stop the worker reading its pipe.
- Host write-stall check: ScriptWorkerFrameWriter.WriteInProgressFor; the
  heartbeat monitor kills a worker whose pipe write has been blocked longer than
  max(10 s, silence limit) as KilledUnresponsive, which fails the write and
  releases the producer (seam submit, deploy load, Cancel). Fixes the wrong
  'heartbeat monitor ends it' comment on the pipe buffer quota.
- StopAsync sends Shutdown off the caller so its 1 s budget holds on a full pipe.
- WindowsBlockingPipeStream refuses the async surface (Stream's base versions
  serialise a write behind a waiting read).
- Tests: PipeFaultTests (throwing completion callback -> ProtocolFault and a
  BadResourceUnavailable sibling, shard serves again; a worker that stops reading
  but heartbeats is killed within the bound and the blocked producer returns,
  via the Hello.test_stall_reader_on_diag seam); BlockingPipeStreamTests dispose
  with a write blocked on a full pipe, and with a read and a write both pending;
  Category=ScriptWorker on the protocol, handshake and pipe test classes.
  Mutation checks: no stall kill -> stall test red; catch-all without the fault
  -> callback test red (Crash).

Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
2026-11-11 20:26:02 -05:00
Joseph Doherty 8582ce91bc perf(scripting): cut out-of-process runner host CPU; compile timeout from compile start; quiet start-failure logs (lmxopcua#789, #790, #788)
v2-ci / unit-tests (push) Has been cancelled
#789 (partial, further tuning deferred): combining inline frame writer, buffered
frame reader, one timer per evaluation for T and T+G, slot release on the reader
before the caller resumes, a non-spinning per-connection result dispatcher, and
blocking pipe I/O on Unix (both ends) so the socket never enters the socket
engine; TerminateJobObject kill on Windows. macOS host CPU at 10k evals/s K=2:
228-240 % -> 79 % (InProcess 69 %, IPC share +10 vs budget 25); windev 153 % ->
123 % (+115, still a miss). Harness reports the "vs InProcess" share, warm-cache
worker RSS, and times kill latency from the soft-timeout answer.

#790: the worker sends LoadStarted; the compile timeout is armed then, so a load
queued behind other compiles is never killed or struck.

#788: one Error with stack per down period, rate-limited Warnings with attempt
count, one Information on recovery.

Design doc §8.2 notes + §12.2 record.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
2026-11-10 04:25:08 -05:00
Joseph Doherty 69a6e0b4c8 feat(scripting): out-of-process pre-emptible script runner behind Scripting:Runner (lmxopcua#780 P1)
v2-ci / unit-tests (push) Waiting to run
Opt-in (Scripting:Runner=OutOfProcess; InProcess stays the default and its code path is unchanged). Implements
the P1 slice of docs/plans/2026-11-09-out-of-process-script-runner-design.md with the owner's §14 decisions:

- Options: Scripting:Runner (InProcess | Shadow | OutOfProcess), Scripting:Runner:Seams, Scripting:Worker:*
  (Shards 2, HardGrace max(1 s,T), HeapLimitBytes 512 MiB, QuarantineAfterKills 2, UnavailableGrace 5 s,
  IsolationProbationRuns 20, PeCacheDir validated absolute + outside the content root (fail closed),
  ExecutablePath). Shadow is not implemented yet: runs InProcess with a startup Warning.
- Core.Scripting: IScriptRunner seam + InProcessScriptRunner (today's guard, the parity baseline).
- Core.Scripting.Protocol: protobuf envelopes, 4-byte LE framing (zero/oversize/truncated/garbage refused),
  Variant carrying the exact CLR type; unsupported types are explicit failures.
- ScriptWorker app: compiles in the worker, P-thread scheduler, host-driven soft cancel, fail-fast
  watchdog, heartbeats, ctx.Logger forwarding, exits on pipe EOF / parent gone, forbidden-assembly
  self-check; Akka and Grpc.Core.Api excluded from its folder. Shipped in script-worker/ (Build + Publish).
- Core.Scripting.OutOfProcess: K shards + isolation worker, soft T / hard T+G kill, attribution
  (deadline kill = one key; crash = suspects -> isolation; isolation crash = strike), kill-strike
  quarantine, backoff 250 ms -> 30 s, fail closed, cleared allow-listed env, nonce + pid handshake on a
  current-user pipe, Windows Job Object.
- VirtualTag + Calculation seams use the runner when selected; a runner-side failure (worker restarted /
  runner unavailable) publishes BadResourceUnavailable (0x80040000) — ScadaBridge#112. Script faults keep
  BadInternalError. The scripted-alarm seam stays in process (alarms last).

Tests: new ZB.MOM.WW.OtOpcUa.ScriptWorker.Tests (Category=ScriptWorker, real processes; green on macOS and
on windev/Windows), seam tests in Host/Runtime/Calculation suites; mutation-checked.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
2026-11-09 23:33:08 -05:00