2cf7c1d2
11 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Joseph Doherty
|
2a8e2981fe |
feat(scripting): ScriptWorker isolation on Windows: restricted worker identity, pipe ACL, firewall + PeCacheDir installer (lmxopcua#780)
v2-ci / unit-tests (push) Has been cancelled
Owner decisions §14 #5/#6 as built (design §12.5): - Scripting:Worker:Identity (default unset, Windows only, fail-closed validation): each worker starts under a restricted token derived from the host's. Every privilege but SeChangeNotify is removed; the restricting SIDs are NT SERVICE\OtOpcUaScriptWorker's SID + Everyone + Users + RESTRICTED; the token and default DACLs are host + SYSTEM + identity + RESTRICTED. The worker is created suspended, put in its Job Object before it runs, inherits only its stdio pipes, gets an ACL'd work dir as TEMP, and runs with DOTNET_EnableDiagnostics=0. The identity is granted on the host's window station and desktop, because USER32 init made coreclr.dll fail with ERROR_DLL_INIT_FAILED on windev. The logon SID is not used because the host's process DACL gives it PROCESS_VM_READ. - The worker pipe on Windows always has an explicit protected DACL: host account + identity only. There is no owner ACE, so an elevated host's Administrators gets no access. - The worker treats access denied on its parent-pid poll as alive; the Job Object covers that case. - scripts/install/Install-ScriptWorkerIsolation.ps1 + ScriptWorkerIsolation.psm1 are idempotent and support -WhatIf, -Uninstall and -PlanOnly. They create the disabled anchor service, the PeCacheDir ACL (protected, no Users, key and markers host-only, pe\ Modify for the identity), read+execute on script-worker\, a ReadData deny on the install root, the outbound firewall block on the worker exe, and optionally the host-service env. - Tests: the dry-run harness scripts/install/tests/Test-ScriptWorkerIsolation.ps1 (46 assertions, mutation-checked); ScriptWorkerIdentityTests; WorkerIsolationTests, which cover the pipe ACL, token access and a real restricted worker running a script, all green on windev. Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV |
||
|
Joseph Doherty
|
0e96dd08a2 |
fix(scripting): deny System.AppContext and other process-wide mutators in the script sandbox (lmxopcua#792)
v2-ci / unit-tests (push) Waiting to run
ForbiddenTypeAnalyzer now denies System.AppContext, System.Console, GCSettings, ProfileOptimization,
EncodingProvider/CodePagesEncodingProvider, TypeDescriptor and CryptoConfig as types, and the ambient
CultureInfo.{DefaultThreadCurrentCulture,DefaultThreadCurrentUICulture,CurrentCulture,CurrentUICulture}
and TimeZoneInfo.ClearCachedData members (member deny-list now covers properties and member-access reads).
Shadow-mode tests diverge via culture-sensitive string.Compare (worker is InvariantGlobalization) instead of
AppContext.BaseDirectory, with a premise test. Mutation-proved: 21 new cases fail with the analyzer reverted.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
|
||
|
Joseph Doherty
|
b1eedceacf |
docs(scripting): §12.4 — Shadow mode as built, its tests, and the docker-dev drill with alarms (lmxopcua#780 P2)
v2-ci / unit-tests (push) Has been cancelled
Shadow on central-1/2 with all three seams: zero mismatches on ordinary VTs / Calc / alarms, counted mismatches on exactly the divergent (value) and decimal (unsupported_type) scripts, kill -9 of the shadow workers left every published value Good. OutOfProcess with ScriptedAlarm listed: alarms raise and clear, a runaway predicate is killed twice and quarantined (BadInternalError) with its siblings Active and Good, kill -9 under an in-flight predicate freezes it Active + BadResourceUnavailable and it re-evaluates in isolation. Answers A-7 (Modbus UInt16 boxes as ushort) and A-8 (a decimal VT publishes BadNotSupported in process). Rig restored to revision e221b179. Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV |
||
|
Joseph Doherty
|
0796928504 |
feat(scripting): batch submit, exact per-item deadlines, scripted-alarm seam on the out-of-process runner (lmxopcua#780)
v2-ci / unit-tests (push) Waiting to run
- IScriptRunner.RunBatch: one EvalBatch frame per worker for every evaluation ready to run; per-item, streamed results. The default (InProcessScriptRunner) is RunAsync per item. Used by the Calculation pump (dequeue all, split at a repeated tag, publish in dequeue order), the VirtualTag seam (ScriptBatchCoalescer, no timer) and the scripted-alarm engine (whole dirty set). In process the pump and the engine stay one at a time, unchanged. - Exact per-item deadlines: the worker hands work to an idle evaluation thread or defers it (EvalResults deferred_eval_ids); the host stops that run's clock and restarts T then G when it starts (started_eval_ids). A run queued behind a runaway is neither timed out nor struck and is a victim, not the culprit, at the kill. - Scripted-alarm seam behind Scripting:Runner:Seams; unset Seams now means VirtualTag + Calculation, so alarms run out of process only when listed. Load still compiles each predicate (to PE bytes only) so a broken one fails the load; passes batch and apply in the serial order under the gate. Script faults freeze the condition BadInternalError; runner faults freeze it BadResourceUnavailable and re-evaluate on recovery; a dead worker never clears an alarm. IScriptRunnerProvider carries the runner to DriverHostActor. - Perf harness: batch throughput (RunBatch vs RunAsync x n) and alarm-pass latency (in process / batched / serial); design §12.3 records macOS + windev. On windev the serial out-of-process alarm pass misses 250 ms from ~500 alarms; batched p99 <= 53 ms at 1,000. Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV |
||
|
Joseph Doherty
|
8582ce91bc |
perf(scripting): cut out-of-process runner host CPU; compile timeout from compile start; quiet start-failure logs (lmxopcua#789, #790, #788)
v2-ci / unit-tests (push) Has been cancelled
#789 (partial, further tuning deferred): combining inline frame writer, buffered frame reader, one timer per evaluation for T and T+G, slot release on the reader before the caller resumes, a non-spinning per-connection result dispatcher, and blocking pipe I/O on Unix (both ends) so the socket never enters the socket engine; TerminateJobObject kill on Windows. macOS host CPU at 10k evals/s K=2: 228-240 % -> 79 % (InProcess 69 %, IPC share +10 vs budget 25); windev 153 % -> 123 % (+115, still a miss). Harness reports the "vs InProcess" share, warm-cache worker RSS, and times kill latency from the soft-timeout answer. #790: the worker sends LoadStarted; the compile timeout is armed then, so a load queued behind other compiles is never killed or struck. #788: one Error with stack per down period, rate-limited Warnings with attempt count, one Information on recovery. Design doc §8.2 notes + §12.2 record. Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV |
||
|
Joseph Doherty
|
1ead8f458a |
feat(scripting): self-contained script worker publish, docker-dev drill, §8 perf harness (lmxopcua#780 P1)
v2-ci / unit-tests (push) Waiting to run
- A self-contained Host publish (-r win-x64 --self-contained, the Windows installs) now publishes the ScriptWorker self-contained for the same RID into script-worker/. The launcher starts its apphost directly when runtimeconfig.json lists includedFrameworks. A framework-dependent worker (docker-dev) keeps running through the dotnet muxer. - docker-dev/docker-compose.script-runner.yml: OutOfProcess for the central pair's VT + Calc seams. Live drill recorded in design §12.1 (runaway kill + quarantine, kill -9 recovery, open circuit with /health/ready still 200, metrics). - ScriptRunnerPerfTests (Category=Perf, opt-in via OTOPCUA_SCRIPT_PERF=1): InProcess vs OutOfProcess at K=1/2/4. The accepted 5k evals/s/node gate passes on macOS and windev. The host-CPU, RSS and Windows latency budget misses are tracked as #789. - Fix: the single-flight slot is released before the awaiting caller resumes. Before, a caller that started its next run right after the previous one returned could be refused with "previous evaluation still running", which the seams publish as Bad. - Fix: the worker compiles on dedicated threads, not the thread pool. A burst of cold loads starved the heartbeat, and the host killed the shard (found on windev). Load-timeout queueing is #790. Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV |
||
|
Joseph Doherty
|
f50b29abae |
feat(scripting): PE cache, quarantine-race fix, recovery, readiness for the out-of-process runner (lmxopcua#780 P1)
v2-ci / unit-tests (push) Failing after 1h12m10s
Second P1 slice of docs/plans/2026-11-09-out-of-process-script-runner-design.md; still opt-in behind Scripting:Runner=OutOfProcess (InProcess unchanged). - Compiled-PE cache (design §7, §14 #8): Scripting:Worker:PeCacheDir holds <dir>/pe/<id>.pe entries authenticated with HMAC-SHA256 under a per-node random key (<dir>/pe-cache.key, created by the host, 0600/0700 on Unix), handed to the worker in Hello. Entry id = SHA-256(compile fingerprint + kind + source); the fingerprint covers the Roslyn version, the Core.Scripting build (analyzer, wrapper, stack guard), the sandbox reference set and the runtime. Edited, truncated, other-key or renamed entries are rejected, deleted and recompiled; the worker never loads unverified bytes. Pruned by recency at startup (30 d / 50 000). ScriptEvaluator splits into CompileToImage + FromImage. Measured on 40 scripts, restart to every first result: macOS 820 ms no cache / 840 ms empty / 85 ms warm; windev (Windows) 9.5 s / 9.4 s / 1.0 s. - Quarantine race: WorkerConnection runs exit attribution synchronously BEFORE completing the evaluations/loads it held (and before Exited), and the shard passes it straight to the runner — a kill strike is in place before the killed key's slot is released. - Supervision: a dead worker restarts by itself after its backoff; Unavailable outcomes carry ScriptRunOutcome.Recovery (-> VirtualTagEvalResult.RetryWhen) and the VT actor / Calculation driver re-evaluate once against current inputs when it completes. Isolation worker stops after Scripting:Worker:IsolationIdleTimeout (10 min). Keys compiling when a worker dies are suspects too (isolation finds a poison compile and quarantines it). Startup orphan sweep by launch marker: kills only a process whose marker's host is gone and whose pid + start time + image match. - Crash-loop circuit (Scripting:Worker:CircuitOpenAfter, 2 min): runner Unavailable, one Error, evaluations answered at once while shards keep retrying at the cap. - Observability (§10): otopcua.script.runner.state, worker.up/inflight/queue_depth/heap_bytes (shard), ipc.roundtrip + eval.duration (kind), compile{cache}, isolation.suspects, exits reason=idle. `script-runner` health check on the ready tier, at worst Degraded (§14 #12: degraded but still ready). Left on #780: self-contained worker publish, installer (account, pipe ACL, firewall), docker-dev live drill, scripted-alarm seam, the §4.5 per-seam batch submit, the §8 perf harness. Tests: SupervisionTests, PeCacheStoreTests/PeCacheWorkerTests (real workers), ScriptRunnerHealthCheckTests, VT + Calc recovery seam tests; 14 mutations each turned their test red. Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV |
||
|
Joseph Doherty
|
69a6e0b4c8 |
feat(scripting): out-of-process pre-emptible script runner behind Scripting:Runner (lmxopcua#780 P1)
v2-ci / unit-tests (push) Waiting to run
Opt-in (Scripting:Runner=OutOfProcess; InProcess stays the default and its code path is unchanged). Implements the P1 slice of docs/plans/2026-11-09-out-of-process-script-runner-design.md with the owner's §14 decisions: - Options: Scripting:Runner (InProcess | Shadow | OutOfProcess), Scripting:Runner:Seams, Scripting:Worker:* (Shards 2, HardGrace max(1 s,T), HeapLimitBytes 512 MiB, QuarantineAfterKills 2, UnavailableGrace 5 s, IsolationProbationRuns 20, PeCacheDir validated absolute + outside the content root (fail closed), ExecutablePath). Shadow is not implemented yet: runs InProcess with a startup Warning. - Core.Scripting: IScriptRunner seam + InProcessScriptRunner (today's guard, the parity baseline). - Core.Scripting.Protocol: protobuf envelopes, 4-byte LE framing (zero/oversize/truncated/garbage refused), Variant carrying the exact CLR type; unsupported types are explicit failures. - ScriptWorker app: compiles in the worker, P-thread scheduler, host-driven soft cancel, fail-fast watchdog, heartbeats, ctx.Logger forwarding, exits on pipe EOF / parent gone, forbidden-assembly self-check; Akka and Grpc.Core.Api excluded from its folder. Shipped in script-worker/ (Build + Publish). - Core.Scripting.OutOfProcess: K shards + isolation worker, soft T / hard T+G kill, attribution (deadline kill = one key; crash = suspects -> isolation; isolation crash = strike), kill-strike quarantine, backoff 250 ms -> 30 s, fail closed, cleared allow-listed env, nonce + pid handshake on a current-user pipe, Windows Job Object. - VirtualTag + Calculation seams use the runner when selected; a runner-side failure (worker restarted / runner unavailable) publishes BadResourceUnavailable (0x80040000) — ScadaBridge#112. Script faults keep BadInternalError. The scripted-alarm seam stays in process (alarms last). Tests: new ZB.MOM.WW.OtOpcUa.ScriptWorker.Tests (Category=ScriptWorker, real processes; green on macOS and on windev/Windows), seam tests in Host/Runtime/Calculation suites; mutation-checked. Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV |
||
|
Joseph Doherty
|
9e8b5b0ea7 |
fix(scripting): stop unbounded script recursion from killing the server (lmxopcua#779)
Reproduced first, out of process (ScriptRecursionContainmentTests re-launches the xunit v3 test assembly to run one recursive script through ScriptEvaluator + ScriptRunGuard, so the runner never executes one): before this change every shape — local function, mutual local functions, self-referencing Func<>, anonymous self-application, recursion through LINQ, an expression-bodied void lambda, an anonymous method — ended the child with exit 134 (SIGABRT, "Stack overflow."). A StackOverflowException cannot be caught, so in production one script takes the whole node down. Fix: ScriptEvaluator.Compile now rewrites the validated wrapper (ScriptStackGuardInjector, after ForbiddenTypeAnalyzer) so the script body and every lambda, anonymous method and local function starts with ScriptStackGuard.Check(), which throws ScriptStackExhaustedException when RuntimeHelpers.TryEnsureSufficientExecutionStack reports the stack nearly spent. The script then fails like any throwing script (Bad, script log) and the process lives. Why a runtime check, not an analyzer rule: recursion is not decidable from syntax (anonymous self-application never names itself), and bounded recursion must keep working. Expression-bodied lambdas/local functions become blocks (`return e;` or `e;`, chosen from the bound symbol, so the same overload binds); expression-tree and async lambdas are left alone. Limits (documented on ScriptStackGuard and in docs/VirtualTags.md): depth depends on the thread's stack size; a recursion that never re-enters script code is not seen; a script may catch the exception, which only unwinds it. Tests: ScriptRecursionContainmentTests (7 shapes, child process each; all 7 fail with exit 134 when the rewrite is disabled), ScriptStackGuardInjectionTests (bounded recursion and every rewritten lambda/local-function shape keep their value; the expression-tree case fails if those lambdas are rewritten). Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV |
||
|
Joseph Doherty
|
a84d38707b |
fix(virtual-tags): evaluate scripts off the Akka dispatcher thread (lmxopcua#779)
VirtualTagActor called the blocking IVirtualTagEvaluator.Evaluate, and RoslynVirtualTagEvaluator blocked on the run guard with .GetAwaiter().GetResult(), so every script run parked a dispatcher thread for up to its 2 s budget (a first-use Roslyn compile did too). This is the #750 P0 refactor for virtual tags. - IVirtualTagEvaluator.EvaluateAsync (default: a completed task over the snapshot overload, so the null evaluator and fakes are unchanged). - RoslynVirtualTagEvaluator implements it without blocking: passthrough and compile failures complete synchronously, a first-use compile goes to the pool, the run is awaited. The blocking overload now delegates to it. - VirtualTagActor: a completed task is applied in the same turn, as before; an incomplete one switches to an "evaluating" behaviour that stashes every dependency/reassert message until a private completion message arrives, then applies the result and unstashes. Each trigger is still evaluated against its own inputs, results are applied/published in trigger order and the dedup/Bad-transition state sees the same sequence. The dependency map is not touched while a run reads it. Calculation and scripted-alarm serialisation is NOT relaxed: neither blocks a thread, and running independent scripts in parallel is not semantics-neutral (pool consumption bound, cross-tag publish order, timer/pump exclusion for Calc; state-machine/persist boundary, transition timestamps and the store-failure path for alarms). Recorded in docs/VirtualTags.md and the #750 design's P0 entry. Tests: VirtualTagActorAsyncEvaluationTests — on a one-thread dispatcher a slow script no longer starves a sibling actor and the slow actor answers Identify (fails when the actor calls the blocking Evaluate); triggers during a run are evaluated afterwards in order against their own inputs; a reassert during a run is applied after it; a faulted evaluation takes the evaluator-threw path (the ordering/reassert tests fail with the stash removed). RoslynVirtualTagEvaluatorAsyncTests — a slow script and a cold compile return an incomplete task (fail without the override); cheap outcomes and timeouts match the blocking overload. Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV |
||
|
Joseph Doherty
|
9e4a26f888 |
docs(scripting): design an out-of-process, pre-emptible script runner (lmxopcua#750)
v2-ci / unit-tests (push) Has been cancelled
Design only. Documents why today's in-process ScriptRunGuard can abandon but not stop a runaway script (file:line evidence), and proposes a sharded worker pool with an isolation worker under a ScriptRunnerActor, a named-pipe + protobuf frame protocol carrying full input snapshots, soft/hard deadlines with process kill, kill attribution and quarantine, worker sandboxing (scrubbed env, pinned assemblies, Job Object, heap limit), an HMAC'd content-addressed PE cache, performance budgets, failure-quality semantics, a flag-gated InProcess/Shadow/OutOfProcess rollout, a test plan, open decisions and an assumptions register. Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV |