dohertj2/lmxopcua
1
0
Fork 0
Code Issues 26 Pull Requests 1 Actions 2 Packages Projects Releases Wiki Activity

Implement the out-of-process script runner (design #750) — code on master; QA shadow week (#793, #794) + P3 flip remain #780

New Issue
Open
opened 2026-11-09 17:05:01 -05:00 by dohertj2 · 8 comments
dohertj2 commented 2026-11-09 17:05:01 -05:00
Owner
Copy Link
Copy Source

Implement the out-of-process, pre-emptible script runner designed in #750 (docs/plans/2026-11-09-out-of-process-script-runner-design.md), behind Scripting:Runner (InProcess default / Shadow / OutOfProcess), phases P1–P4 with alarms last. Blocked on the 14 owner decisions in the design (pool shape, kill grace, the "runner unavailable" OPC UA status code — a ScadaBridge-visible contract change, fail-closed start, worker account, network blocking, compile location, cache, encoding, rollout order, unsupported types, readiness when the breaker trips, keeping in-process, and the scale target).

Implement the out-of-process, pre-emptible script runner designed in #750 (docs/plans/2026-11-09-out-of-process-script-runner-design.md), behind `Scripting:Runner` (InProcess default / Shadow / OutOfProcess), phases P1–P4 with alarms last. Blocked on the 14 owner decisions in the design (pool shape, kill grace, the "runner unavailable" OPC UA status code — a ScadaBridge-visible contract change, fail-closed start, worker account, network blocking, compile location, cache, encoding, rollout order, unsupported types, readiness when the breaker trips, keeping in-process, and the scale target).
dohertj2 added the type:featurepriority:p3blocked labels 2026-11-09 17:05:01 -05:00
dohertj2 commented 2026-11-09 22:13:24 -05:00
Author
Owner
Copy Link
Copy Source

Owner decision 2026-11-09: accept all 14 ★ recommendations in docs/plans/2026-11-09-out-of-process-script-runner-design.md §14 as written — K=2 shards + isolation worker; G=max(1 s,T); BadResourceUnavailable for runner-unavailable/worker-restarted (ScadaBridge-visible contract change — coordinate via a cross-repo ScadaBridge issue before the OutOfProcess flip); fail closed; dedicated low-privilege virtual account + pipe ACL; firewall outbound block (analyzer-only on Linux); compile in the worker; node-local HMAC'd PE cache; protobuf IPC; alarms last; explicit UNSUPPORTED_TYPE; degraded-but-ready; keep InProcess as default/rollback; accept the 5k evals/s/node guess as the gate. Unblocked: P1→P4 may proceed.

**Owner decision 2026-11-09:** accept all 14 ★ recommendations in docs/plans/2026-11-09-out-of-process-script-runner-design.md §14 as written — K=2 shards + isolation worker; G=max(1 s,T); **BadResourceUnavailable** for runner-unavailable/worker-restarted (ScadaBridge-visible contract change — coordinate via a cross-repo ScadaBridge issue before the OutOfProcess flip); fail closed; dedicated low-privilege virtual account + pipe ACL; firewall outbound block (analyzer-only on Linux); compile in the worker; node-local HMAC'd PE cache; protobuf IPC; alarms last; explicit UNSUPPORTED_TYPE; degraded-but-ready; keep InProcess as default/rollback; accept the 5k evals/s/node guess as the gate. Unblocked: P1→P4 may proceed.
dohertj2 removed the blocked label 2026-11-09 22:13:48 -05:00
dohertj2 commented 2026-11-09 23:38:29 -05:00
Author
Owner
Copy Link
Copy Source

P1 slice landed on master @ 69a6e0b4 (opt-in; Scripting:Runner=InProcess stays the default and its code path is unchanged: the seams only take the new branch when a runner is injected).

What is in

  • Options: Scripting:Runner (InProcess / Shadow / OutOfProcess), Scripting:Runner:Seams, Scripting:Worker:{Shards,HardGrace,HeapLimitBytes,QuarantineAfterKills,UnavailableGrace,IsolationProbationRuns,PeCacheDir,ExecutablePath}. These are bound and clamped, with a startup Warning per clamp. A relative PeCacheDir, or one inside the content root, stops startup. A misspelled mode fails startup. Shadow currently runs InProcess with a Warning (that is P2).
  • IScriptRunner + InProcessScriptRunner (the existing guard, used as the parity baseline) in Core.Scripting.
  • Core.Scripting.Protocol: protobuf envelopes with 4-byte little-endian length framing, and a Variant that carries the exact CLR type. An unsupported input or result type is an explicit unsupported type: … failure (§14 #11).
  • ZB.MOM.WW.OtOpcUa.ScriptWorker app:
    • compiles in the worker (§14 #7) and runs on a P-thread scheduler;
    • cancels the soft deadline only when the host asks, so ordering matches the in-process path;
    • fail-fast watchdog, heartbeats and ctx.Logger forwarding;
    • exits on pipe EOF or when its parent process disappears;
    • checks itself for forbidden assemblies. Akka.dll and Grpc.Core.Api.dll are not even in its folder.
    • The Host Build and Publish copy it into script-worker/.
  • Core.Scripting.OutOfProcess.OutOfProcessScriptRunner:
    • K=2 shards placed by a stable FNV hash, plus an on-demand isolation worker that runs one evaluation at a time;
    • soft deadline T, then a kill at T+G with G=max(1 s,T);
    • attribution: a deadline kill strikes exactly one key; a crash makes the keys in flight suspects, which move to isolation; a crash in isolation is a strike;
    • quarantine after 2 kill strikes in an hour, plus the existing timeout and allocation rules, all held host-side;
    • restart backoff from 250 ms to 30 s with ±20 % jitter;
    • fails closed (§14 #4);
    • worker environment is cleared and rebuilt from an allow-list; the handshake checks nonce and pid on a current-user pipe; Windows workers run in a Job Object (kill-on-close, one process, commit backstop).
  • The VirtualTag and Calculation seams use the runner when it is selected. A runner-side failure publishes BadResourceUnavailable (0x80040000) with the last value. Reasons are script worker restarted or script runner unavailable. Script faults keep BadInternalError. ScadaBridge consumer issue: dohertj2/ScadaBridge#112 (cross-repo, followup, p3). It must close before any node ScadaBridge reads is flipped (P3).

Evidence

  • dotnet build ZB.MOM.WW.OtOpcUa.slnx -c Release: 0 warnings, 0 errors (also --no-incremental).
  • New tests/Server/ZB.MOM.WW.OtOpcUa.ScriptWorker.Tests (Category=ScriptWorker, real worker processes): 89/89 on macOS and 89/89 on windev (Windows). Cases:
    • while(true){} killed at T+G, the host burns no CPU on it afterwards, the sibling keeps working, quarantine after 2 kills, lifted by a source change or a redeploy;
    • a cooperative script ends at T with no kill;
    • kill -9 leaves the in-flight script a victim (Unavailable), then it becomes a suspect and recovers in isolation;
    • isolation crashes lead to quarantine;
    • a deadline kill blames only its culprit;
    • an allocation bomb is contained (host heap unchanged);
    • a missing worker reports Unavailable and never runs in-process;
    • pipe EOF: the worker exits by itself with code 3;
    • the environment carries no ServerHistorian__ApiKey or ConnectionStrings (checked against a positive control);
    • no host stacks loaded;
    • logs are forwarded; an alarm predicate runs;
    • parity with the in-process runner for 12 script shapes plus (T)ctx.GetTag(x).Value for 13 types (pins A-7);
    • frame codec rejection cases plus a 3000-case fuzz;
    • handshake: wrong nonce or wrong pid refused;
    • end to end: a real CalculationDriver on the real worker publishes BadInternalError for a script fault, BadResourceUnavailable when the worker dies, then Good again.
  • AKKA_TEST_TIMEFACTOR=3:
    • Core.Scripting.Tests 221 passed / 1 skipped;
    • Host.Tests 263;
    • Runtime.Tests 990 passed / 3 skipped;
    • Calculation.Tests 82;
    • VirtualTags 16, ScriptedAlarms 106;
    • OpcUaServer.IntegrationTests 132/132 (re-run after rebasing on f5c30ed0).
    • Two timing tests failed once each under a loaded sequential run and passed on every re-run: ScriptRunGuardTests.Three_consecutive_timeouts… and DependencyMuxReplayTests.Retention…. Neither is in code this change touches.
  • Mutation checks. Each mutation below turned its test red, and was reverted:
    • hard-deadline kill disabled;
    • environment not cleared;
    • nonce not checked;
    • worker ignoring EOF (both the read path and the write path);
    • short encoded as int;
    • no isolation strike;
    • suspects not isolated;
    • Akka deployed beside the worker;
    • victim reported as a script fault;
    • VirtualTagActor and CalculationDriver dropping the status code;
    • soft cancel not sent.

Deviations from the design

  • Supervision is a plain DI-owned object (WorkerShard / OutOfProcessScriptRunner), not the ScriptRunnerActor / ScriptWorkerActor pair.
  • Scripting:Runner:Seams is bound explicitly, because Scripting:Runner is itself a value.
  • Seams still submit one evaluation at a time; the batch-submit part of P0 is open.

P1 remaining (keep open)

  1. The node-local HMAC'd PE cache (§7). PeCacheDir is validated but not used yet, so every worker restart recompiles.
  2. A self-contained worker publish for the Windows installs. Today the worker runs through the dotnet muxer, which suits the framework-dependent docker-dev image.
  3. Installer work: the low-privilege worker account and pipe ACL (§14 #5) and the firewall outbound rule (§14 #6).
  4. Enable OutOfProcess on the docker-dev rig and run the live drill (§13 live gates).
  5. Observability: the otopcua.script.runner.state gauge, the readiness component (degraded but still ready, §14 #12), and the rest of the §10 metrics. Only worker.starts and worker.exits exist so far.
  6. Re-evaluate victims when their shard returns; stop the isolation worker after it has been idle; run the orphan sweep by nonce file; bisect load-phase crashes.
  7. Move the scripted-alarm seam onto the runner, with batch submit (alarms go last).
  8. Build the §8 perf harness (Category=Perf) and measure against the accepted 5k evals/s/node gate.
  9. Race fixed in the tests: the exit handler records a kill strike asynchronously to Exited, so a trigger that lands in the restart gap can run once more before the quarantine is in place.

Then P2 (Shadow), P3 (flip, after ScadaBridge#112) and P4.

**P1 slice landed on `master` @ `69a6e0b4`** (opt-in; `Scripting:Runner=InProcess` stays the default and its code path is unchanged: the seams only take the new branch when a runner is injected). **What is in** - Options: `Scripting:Runner` (InProcess / Shadow / OutOfProcess), `Scripting:Runner:Seams`, `Scripting:Worker:{Shards,HardGrace,HeapLimitBytes,QuarantineAfterKills,UnavailableGrace,IsolationProbationRuns,PeCacheDir,ExecutablePath}`. These are bound and clamped, with a startup Warning per clamp. A relative `PeCacheDir`, or one inside the content root, stops startup. A misspelled mode fails startup. `Shadow` currently runs InProcess with a Warning (that is P2). - `IScriptRunner` + `InProcessScriptRunner` (the existing guard, used as the parity baseline) in Core.Scripting. - `Core.Scripting.Protocol`: protobuf envelopes with 4-byte little-endian length framing, and a `Variant` that carries the exact CLR type. An unsupported input or result type is an explicit `unsupported type: …` failure (§14 #11). - `ZB.MOM.WW.OtOpcUa.ScriptWorker` app: - compiles in the worker (§14 #7) and runs on a P-thread scheduler; - cancels the soft deadline only when the host asks, so ordering matches the in-process path; - fail-fast watchdog, heartbeats and `ctx.Logger` forwarding; - exits on pipe EOF or when its parent process disappears; - checks itself for forbidden assemblies. Akka.dll and Grpc.Core.Api.dll are not even in its folder. - The Host Build and Publish copy it into `script-worker/`. - `Core.Scripting.OutOfProcess.OutOfProcessScriptRunner`: - K=2 shards placed by a stable FNV hash, plus an on-demand isolation worker that runs one evaluation at a time; - soft deadline T, then a kill at T+G with G=max(1 s,T); - attribution: a deadline kill strikes exactly one key; a crash makes the keys in flight suspects, which move to isolation; a crash in isolation is a strike; - quarantine after 2 kill strikes in an hour, plus the existing timeout and allocation rules, all held host-side; - restart backoff from 250 ms to 30 s with ±20 % jitter; - fails closed (§14 #4); - worker environment is cleared and rebuilt from an allow-list; the handshake checks nonce and pid on a current-user pipe; Windows workers run in a Job Object (kill-on-close, one process, commit backstop). - The VirtualTag and Calculation seams use the runner when it is selected. A runner-side failure publishes **`BadResourceUnavailable` (0x80040000)** with the last value. Reasons are `script worker restarted` or `script runner unavailable`. Script faults keep `BadInternalError`. ScadaBridge consumer issue: **dohertj2/ScadaBridge#112** (cross-repo, followup, p3). It must close before any node ScadaBridge reads is flipped (P3). **Evidence** - `dotnet build ZB.MOM.WW.OtOpcUa.slnx -c Release`: 0 warnings, 0 errors (also `--no-incremental`). - New `tests/Server/ZB.MOM.WW.OtOpcUa.ScriptWorker.Tests` (`Category=ScriptWorker`, real worker processes): **89/89 on macOS and 89/89 on windev (Windows)**. Cases: - `while(true){}` killed at T+G, the host burns no CPU on it afterwards, the sibling keeps working, quarantine after 2 kills, lifted by a source change or a redeploy; - a cooperative script ends at T with no kill; - `kill -9` leaves the in-flight script a victim (Unavailable), then it becomes a suspect and recovers in isolation; - isolation crashes lead to quarantine; - a deadline kill blames only its culprit; - an allocation bomb is contained (host heap unchanged); - a missing worker reports Unavailable and never runs in-process; - pipe EOF: the worker exits by itself with code 3; - the environment carries no `ServerHistorian__ApiKey` or ConnectionStrings (checked against a positive control); - no host stacks loaded; - logs are forwarded; an alarm predicate runs; - parity with the in-process runner for 12 script shapes plus `(T)ctx.GetTag(x).Value` for 13 types (pins A-7); - frame codec rejection cases plus a 3000-case fuzz; - handshake: wrong nonce or wrong pid refused; - end to end: a real CalculationDriver on the real worker publishes BadInternalError for a script fault, BadResourceUnavailable when the worker dies, then Good again. - `AKKA_TEST_TIMEFACTOR=3`: - Core.Scripting.Tests 221 passed / 1 skipped; - Host.Tests 263; - Runtime.Tests 990 passed / 3 skipped; - Calculation.Tests 82; - VirtualTags 16, ScriptedAlarms 106; - **OpcUaServer.IntegrationTests 132/132** (re-run after rebasing on `f5c30ed0`). - Two timing tests failed once each under a loaded sequential run and passed on every re-run: `ScriptRunGuardTests.Three_consecutive_timeouts…` and `DependencyMuxReplayTests.Retention…`. Neither is in code this change touches. - Mutation checks. Each mutation below turned its test red, and was reverted: - hard-deadline kill disabled; - environment not cleared; - nonce not checked; - worker ignoring EOF (both the read path and the write path); - `short` encoded as `int`; - no isolation strike; - suspects not isolated; - Akka deployed beside the worker; - victim reported as a script fault; - VirtualTagActor and CalculationDriver dropping the status code; - soft cancel not sent. **Deviations from the design** - Supervision is a plain DI-owned object (`WorkerShard` / `OutOfProcessScriptRunner`), not the `ScriptRunnerActor` / `ScriptWorkerActor` pair. - `Scripting:Runner:Seams` is bound explicitly, because `Scripting:Runner` is itself a value. - Seams still submit one evaluation at a time; the batch-submit part of P0 is open. **P1 remaining (keep open)** 1. The node-local HMAC'd PE cache (§7). `PeCacheDir` is validated but not used yet, so every worker restart recompiles. 2. A self-contained worker publish for the Windows installs. Today the worker runs through the dotnet muxer, which suits the framework-dependent docker-dev image. 3. Installer work: the low-privilege worker account and pipe ACL (§14 #5) and the firewall outbound rule (§14 #6). 4. Enable OutOfProcess on the docker-dev rig and run the live drill (§13 live gates). 5. Observability: the `otopcua.script.runner.state` gauge, the readiness component (degraded but still ready, §14 #12), and the rest of the §10 metrics. Only `worker.starts` and `worker.exits` exist so far. 6. Re-evaluate victims when their shard returns; stop the isolation worker after it has been idle; run the orphan sweep by nonce file; bisect load-phase crashes. 7. Move the scripted-alarm seam onto the runner, with batch submit (alarms go last). 8. Build the §8 perf harness (`Category=Perf`) and measure against the accepted 5k evals/s/node gate. 9. Race fixed in the tests: the exit handler records a kill strike asynchronously to `Exited`, so a trigger that lands in the restart gap can run once more before the quarantine is in place. Then P2 (Shadow), P3 (flip, after ScadaBridge#112) and P4.
dohertj2 commented 2026-11-10 00:20:56 -05:00
Author
Owner
Copy Link
Copy Source

Second P1 slice landed on master @ f50b29ab (still opt-in behind Scripting:Runner=OutOfProcess; InProcess unchanged). Code only: no installer work, nothing deployed.

What is in (items from the P1-remaining list in the previous comment)

  1. HMAC'd compiled-PE cache (§7, ★8), at Scripting:Worker:PeCacheDir, optional. The host creates a per-node random key at <dir>/pe-cache.key; on Unix the key file is 0600 and the directory 0700, and on Windows it relies on the directory ACL (installer). The key reaches the worker only in Hello. Entries live at <dir>/pe/<id>.pe, and id = SHA-256(compile fingerprint + kind + source). The fingerprint covers the Roslyn version, the Core.Scripting build (analyzer, wrapper, stack guard), the sandbox reference set and the runtime, so a compiler or analyzer change misses every old entry. The HMAC covers magic + id + PE. An entry that has been edited, truncated, written under another key or copied to another id is rejected, deleted and recompiled. Compile errors are never cached. An unusable directory gives a Warning and a runner without a cache. Deviation: pruning is by recency at startup (30 d / 50 000 entries), not by generation, because this runner has no generation object.
    Measured (40 scripts, worker restart to every first result):
    Host No cache Empty cache Warm cache
    macOS 820 ms 840 ms 85 ms
    windev (Windows) 9.5 s 9.4 s 1.0 s
  2. Quarantine race fixed. WorkerConnection now runs exit attribution synchronously, before it completes any evaluation or load it held and before Exited. The shard passes that callback straight to the runner. A killed key is therefore quarantined before its single-flight slot is released.
  3. Observability + readiness (§10, ★12):
    • gauges otopcua.script.runner.state (0 ready / 1 degraded / 2 circuit open), worker.up, worker.inflight, worker.queue_depth and worker.heap_bytes (tagged shard, from heartbeats), and isolation.suspects;
    • histograms ipc.roundtrip and eval.duration (µs, tagged kind, no per-key tag);
    • counter compile{cache=hit|miss|rejected|uncached|negative}, and exits gains reason=idle;
    • the script-runner health check is on the ready tier and is at worst Degraded (still 200). It is never Unhealthy.
    • New crash-loop circuit, Scripting:Worker:CircuitOpenAfter (default 2 min): when it opens, one Error is logged and evaluations are answered at once while the shards keep retrying at the backoff cap.
  4. Supervisor gaps:
    • Victims are re-evaluated when their shard comes back. A dead worker restarts by itself after its backoff. An Unavailable outcome carries ScriptRunOutcome.Recovery → VirtualTagEvalResult.RetryWhen, and the VirtualTagActor and CalculationDriver re-evaluate once against their current inputs when it completes.
    • Idle isolation worker stop, Scripting:Worker:IsolationIdleTimeout (default 10 min). It is an Idle exit: no backoff and not counted as down.
    • Orphan sweep at host startup. Each launched worker gets a marker file recording its pid, start time and image, plus the host's pid and start time. At startup the sweep kills a process only when the marker's host is gone and that process's pid, start time and image all match. It never kills a live host's worker or a reused pid.
    • Culprit-finding for a crash during script load. Keys that were still compiling when a worker died become suspects. They then compile alone in the isolation worker, where a further death strikes and quarantines them ("its compile crashed the isolation worker"). This is isolation per key, not the §5.3.5 three-strikes bisection, because loads here are per key.
  5. Batch submit (P0 §4.5): not done. It means a batch API on IScriptRunner plus rework of the Calc pump and the alarm engine, and the alarm seam is not on the runner yet. The connection writer already merges concurrent submits into one EvalBatch.

Evidence

  • dotnet build ZB.MOM.WW.OtOpcUa.slnx -c Release --no-incremental: 0 warnings, 0 errors.

  • AKKA_TEST_TIMEFACTOR=3, all on macOS:

    Suite Result
    Core.Scripting 221 passed / 1 skipped
    ScriptedAlarms 106
    VirtualTags 16
    Calculation 83
    Host 269
    Runtime 991 passed / 3 skipped
    ScriptWorker 122/122
    OpcUaServer.IntegrationTests 132/132

    ScriptWorker.Tests is also 122/122 on windev (Windows).

  • New tests: SupervisionTests (real workers), PeCacheStoreTests, PeCacheWorkerTests (cold/warm measurement and a forged entry that returns 666 and never runs), ScriptRunnerHealthCheckTests, and recovery tests for the VirtualTag actor and the Calculation driver.

  • 14 mutations, each turned its test red and was reverted:

    • attribution moved after completion;
    • the old async shard wiring;
    • no proactive restart;
    • no idle stop;
    • no sweep at startup;
    • no host-alive check;
    • no start-time check;
    • loading keys not counted in attribution;
    • no circuit fast path;
    • worker ignores the cache;
    • no HMAC check;
    • VirtualTag actor does not re-arm;
    • Calc does not re-arm;
    • an open circuit reported Unhealthy.

Still open on #780 (keep open)

  • A self-contained worker publish for the Windows installs.
  • Installer work: the low-privilege worker account, the pipe ACL, the firewall outbound rule, and the PeCacheDir ACL on Windows.
  • Enable on docker-dev and run the live drill (§13).
  • The scripted-alarm seam, with its batch submit.
  • The §4.5 batch submit for Calc.
  • The §8 perf harness (Category=Perf).

Then P2 (Shadow), P3 (flip, after ScadaBridge#112) and P4.

**Second P1 slice landed on `master` @ `f50b29ab`** (still opt-in behind `Scripting:Runner=OutOfProcess`; InProcess unchanged). Code only: no installer work, nothing deployed. **What is in** (items from the P1-remaining list in the previous comment) 1. **HMAC'd compiled-PE cache** (§7, ★8), at `Scripting:Worker:PeCacheDir`, optional. The host creates a per-node random key at `<dir>/pe-cache.key`; on Unix the key file is 0600 and the directory 0700, and on Windows it relies on the directory ACL (installer). The key reaches the worker only in `Hello`. Entries live at `<dir>/pe/<id>.pe`, and `id = SHA-256(compile fingerprint + kind + source)`. The fingerprint covers the Roslyn version, the Core.Scripting build (analyzer, wrapper, stack guard), the sandbox reference set and the runtime, so a compiler or analyzer change misses every old entry. The HMAC covers magic + id + PE. An entry that has been edited, truncated, written under another key or copied to another id is rejected, deleted and recompiled. Compile errors are never cached. An unusable directory gives a Warning and a runner without a cache. Deviation: pruning is by recency at startup (30 d / 50 000 entries), not by generation, because this runner has no generation object. **Measured (40 scripts, worker restart to every first result):** | Host | No cache | Empty cache | Warm cache | |---|---|---|---| | macOS | 820 ms | 840 ms | **85 ms** | | windev (Windows) | 9.5 s | 9.4 s | **1.0 s** | 2. **Quarantine race fixed.** `WorkerConnection` now runs exit attribution synchronously, before it completes any evaluation or load it held and before `Exited`. The shard passes that callback straight to the runner. A killed key is therefore quarantined before its single-flight slot is released. 3. **Observability + readiness** (§10, ★12): - gauges `otopcua.script.runner.state` (0 ready / 1 degraded / 2 circuit open), `worker.up`, `worker.inflight`, `worker.queue_depth` and `worker.heap_bytes` (tagged `shard`, from heartbeats), and `isolation.suspects`; - histograms `ipc.roundtrip` and `eval.duration` (µs, tagged `kind`, no per-key tag); - counter `compile{cache=hit|miss|rejected|uncached|negative}`, and `exits` gains `reason=idle`; - the `script-runner` health check is on the **ready** tier and is at worst **Degraded** (still 200). It is never Unhealthy. - New crash-loop circuit, `Scripting:Worker:CircuitOpenAfter` (default 2 min): when it opens, one Error is logged and evaluations are answered at once while the shards keep retrying at the backoff cap. 4. **Supervisor gaps:** - **Victims are re-evaluated when their shard comes back.** A dead worker restarts by itself after its backoff. An Unavailable outcome carries `ScriptRunOutcome.Recovery` → `VirtualTagEvalResult.RetryWhen`, and the VirtualTagActor and CalculationDriver re-evaluate once against their current inputs when it completes. - **Idle isolation worker stop**, `Scripting:Worker:IsolationIdleTimeout` (default 10 min). It is an Idle exit: no backoff and not counted as down. - **Orphan sweep at host startup.** Each launched worker gets a marker file recording its pid, start time and image, plus the host's pid and start time. At startup the sweep kills a process only when the marker's host is gone **and** that process's pid, start time and image all match. It never kills a live host's worker or a reused pid. - **Culprit-finding for a crash during script load.** Keys that were still compiling when a worker died become suspects. They then compile alone in the isolation worker, where a further death strikes and quarantines them (`"its compile crashed the isolation worker"`). This is isolation per key, not the §5.3.5 three-strikes bisection, because loads here are per key. 5. **Batch submit (P0 §4.5): not done.** It means a batch API on `IScriptRunner` plus rework of the Calc pump and the alarm engine, and the alarm seam is not on the runner yet. The connection writer already merges concurrent submits into one `EvalBatch`. **Evidence** - `dotnet build ZB.MOM.WW.OtOpcUa.slnx -c Release --no-incremental`: 0 warnings, 0 errors. - `AKKA_TEST_TIMEFACTOR=3`, all on macOS: | Suite | Result | |---|---| | Core.Scripting | 221 passed / 1 skipped | | ScriptedAlarms | 106 | | VirtualTags | 16 | | Calculation | 83 | | Host | 269 | | Runtime | 991 passed / 3 skipped | | **ScriptWorker** | **122/122** | | **OpcUaServer.IntegrationTests** | **132/132** | **ScriptWorker.Tests is also 122/122 on windev (Windows).** - New tests: `SupervisionTests` (real workers), `PeCacheStoreTests`, `PeCacheWorkerTests` (cold/warm measurement and a forged entry that returns 666 and never runs), `ScriptRunnerHealthCheckTests`, and recovery tests for the VirtualTag actor and the Calculation driver. - 14 mutations, each turned its test red and was reverted: - attribution moved after completion; - the old async shard wiring; - no proactive restart; - no idle stop; - no sweep at startup; - no host-alive check; - no start-time check; - loading keys not counted in attribution; - no circuit fast path; - worker ignores the cache; - no HMAC check; - VirtualTag actor does not re-arm; - Calc does not re-arm; - an open circuit reported Unhealthy. **Still open on #780 (keep open)** - A self-contained worker publish for the Windows installs. - Installer work: the low-privilege worker account, the pipe ACL, the firewall outbound rule, and the PeCacheDir ACL on Windows. - Enable on docker-dev and run the live drill (§13). - The scripted-alarm seam, with its batch submit. - The §4.5 batch submit for Calc. - The §8 perf harness (`Category=Perf`). Then P2 (Shadow), P3 (flip, after ScadaBridge#112) and P4.
dohertj2 commented 2026-11-10 03:01:18 -05:00
Author
Owner
Copy Link
Copy Source

Third P1 slice landed on master @ 1ead8f45. It is still opt-in, and InProcess stays the default and is unchanged. Nothing was deployed and no wonder-box was touched. The full record is in design §12.1.

1. Self-contained worker publish (for the Windows installs)

  • A -r win-x64 --self-contained Host publish now publishes the ScriptWorker self-contained for the same RID into script-worker/. That is 236 files, about 97 MB, with its own runtime; it is not trimmed and not single-file.
  • The launcher starts ZB.MOM.WW.OtOpcUa.ScriptWorker.exe directly, but only when the worker's runtimeconfig.json lists includedFrameworks (OutOfProcessScriptRunner.ResolveWorkerEntry). There is no dotnet muxer in that path.
  • A framework-dependent publish (docker-dev) is unchanged: 51 files, started with dotnet exec <dll>.
  • Verified on windev: the Host was published win-x64 self-contained under C:\build, and the whole ScriptWorker.Tests suite was run against the published apphost via OTOPCUA_SCRIPTWORKER_TEST_PATH. Result: 132/132. This includes a gate test asserting that the worker process's MainModule is the .exe.

2. docker-dev live drill (new overlay docker-dev/docker-compose.script-runner.yml: central pair, VT + Calc seams)

  • Normal evaluation. After a fixture write of HR5 = 21, Calc HR5x2 = 42 Good on both nodes and the VT sibling (HR5×3) = 63 Good.
  • Runaway. A while(true) VT was killed at its 4000 ms hard deadline twice and then quarantined, on both nodes. The VT and Calc on the same shard stayed Good.
  • kill -9 of both workers.
    • The in-flight VT published BadResourceUnavailable.
    • It went to isolation, came back Good 30 ms later, and after the next input change read 1022.
    • Isolation probation returned it to its shard.
  • Open circuit (worker dll renamed for about 3 minutes).
    • /health/ready stayed 200: Degraded, then circuit-open Unavailable after 2 min.
    • otopcua_script_runner_state = 2.
    • VT and Calc published BadResourceUnavailable while driver tags stayed Good.
    • After the dll was restored, they recovered to 46 / 69 by themselves.
  • Metrics are visible on :9000/metrics.
  • A note for ScadaBridge#112: on the wire, a Bad status carries a null Value (true of every Bad status on this server), so OPC UA clients do not see the "last value".
  • Rig left as:
    • drill rows deleted and HR5 back to 5;
    • redeployed with revision e221b179…, identical to the pre-drill gate-cleanup deployment;
    • otopcua-host:dev re-tagged to the pre-drill image 24f9d7287ed2;
    • central-1/2 recreated from the base compose: InProcess, no Scripting__*.
    • The site nodes, sql and traefik were not touched. The central containers' local secret stores were reset by the recreate.
    • The drill ran on an image built before the two fixes below.

3. §8 perf harness

  • ScriptRunnerPerfTests, Category=Perf. It is opt-in with OTOPCUA_SCRIPT_PERF=1, so it skips in a default run.
  • Accepted gate (≥ 5k evals/s/node at K=2): PASS. macOS 336k/s, windev 51k/s, every result correct.
  • p99 at 10k/s: 0.7 ms / 1.9 ms (budget ≤ 2 ms).
  • Warm restart of 1,000 cached scripts: 0.2 s / 1.7 s (budget ≤ 2 s).
  • Misses → #789:
    • host CPU at 10k/s: about 240 % (macOS) and 153 % (windev), against a ≤ 25 % budget;
    • worker RSS: 158–190 MB, against ≤ 80 MB;
    • on windev only, idle p50 of 795 µs and kill latency of about 210 ms.

Defects the harness found, fixed here with tests that go red under mutation:

  1. A caller refused by its own finished run. The slot was released by a continuation that was not ordered against the caller's await. 32 of 200 back-to-back runs came back "previous evaluation still running", which the seams publish as Bad. The slot is now released before the caller resumes.
  2. A compile burst starved the heartbeat (windev). 500 cold loads, each on its own thread-pool work item, delayed the heartbeat; the host killed the shard (KilledHeartbeat) and all 500 keys became suspects. Compiles now run on min(8, max(1, CPUs/2)) dedicated threads.

New issues

  • #787: a disabled VirtualTag is still deployed and evaluated (Enabled is ignored by the composer and the artifact).
  • #788: worker start failures log an Error with a stack trace on every retry.
  • #789: §8.2 budget misses.
  • #790: the load timeout counts time spent queued behind other compiles.

Evidence

  • Release build of the solution with --no-incremental: 0 warnings, 0 errors.
  • AKKA_TEST_TIMEFACTOR=3:
    Suite Result
    ScriptWorker 131 passed / 3 skipped (the perf tests and the self-contained gate); 132/132 on windev self-contained
    Core.Scripting 221 passed / 1 skipped
    Host 269
    Runtime 991 passed / 3 skipped
    OpcUaServer.IntegrationTests 132/132

Still open on #780 (I am keeping it open)

  • Installer work: the low-privilege worker account, the pipe ACL, the firewall outbound rule and the PeCacheDir ACL.
  • The scripted-alarm seam, with its batch submit.
  • The §4.5 batch submit for Calc.
  • #789 and #790.

Then P2 (Shadow) and P3 (after ScadaBridge#112).

**Third P1 slice landed on `master` @ `1ead8f45`.** It is still opt-in, and `InProcess` stays the default and is unchanged. Nothing was deployed and no wonder-box was touched. The full record is in design §12.1. **1. Self-contained worker publish (for the Windows installs)** - A `-r win-x64 --self-contained` Host publish now publishes the ScriptWorker self-contained for the same RID into `script-worker/`. That is 236 files, about 97 MB, with its own runtime; it is not trimmed and not single-file. - The launcher starts `ZB.MOM.WW.OtOpcUa.ScriptWorker.exe` directly, but only when the worker's `runtimeconfig.json` lists `includedFrameworks` (`OutOfProcessScriptRunner.ResolveWorkerEntry`). There is no `dotnet` muxer in that path. - A framework-dependent publish (docker-dev) is unchanged: 51 files, started with `dotnet exec <dll>`. - **Verified on windev:** the Host was published win-x64 self-contained under `C:\build`, and the whole `ScriptWorker.Tests` suite was run against the published apphost via `OTOPCUA_SCRIPTWORKER_TEST_PATH`. Result: **132/132**. This includes a gate test asserting that the worker process's `MainModule` is the `.exe`. **2. docker-dev live drill** (new overlay `docker-dev/docker-compose.script-runner.yml`: central pair, VT + Calc seams) - **Normal evaluation.** After a fixture write of HR5 = 21, Calc `HR5x2` = **42 Good** on both nodes and the VT sibling (HR5×3) = **63 Good**. - **Runaway.** A `while(true)` VT was killed at its 4000 ms hard deadline twice and then quarantined, on both nodes. The VT and Calc on the same shard stayed Good. - **`kill -9` of both workers.** - The in-flight VT published **BadResourceUnavailable**. - It went to isolation, came back Good 30 ms later, and after the next input change read 1022. - Isolation probation returned it to its shard. - **Open circuit** (worker dll renamed for about 3 minutes). - `/health/ready` stayed **200**: Degraded, then circuit-open Unavailable after 2 min. - `otopcua_script_runner_state` = 2. - VT and Calc published BadResourceUnavailable while driver tags stayed Good. - After the dll was restored, they recovered to 46 / 69 by themselves. - Metrics are visible on `:9000/metrics`. - A note for ScadaBridge#112: on the wire, a Bad status carries a **null Value** (true of every Bad status on this server), so OPC UA clients do not see the "last value". - **Rig left as:** - drill rows deleted and HR5 back to 5; - redeployed with revision `e221b179…`, identical to the pre-drill `gate-cleanup` deployment; - `otopcua-host:dev` re-tagged to the pre-drill image `24f9d7287ed2`; - central-1/2 recreated from the base compose: InProcess, no `Scripting__*`. - The site nodes, sql and traefik were not touched. The central containers' local secret stores were reset by the recreate. - The drill ran on an image built **before** the two fixes below. **3. §8 perf harness** - `ScriptRunnerPerfTests`, `Category=Perf`. It is opt-in with `OTOPCUA_SCRIPT_PERF=1`, so it skips in a default run. - **Accepted gate (≥ 5k evals/s/node at K=2): PASS.** macOS 336k/s, windev 51k/s, every result correct. - p99 at 10k/s: 0.7 ms / 1.9 ms (budget ≤ 2 ms). - Warm restart of 1,000 cached scripts: 0.2 s / 1.7 s (budget ≤ 2 s). - **Misses → #789:** - host CPU at 10k/s: about 240 % (macOS) and 153 % (windev), against a ≤ 25 % budget; - worker RSS: 158–190 MB, against ≤ 80 MB; - on windev only, idle p50 of 795 µs and kill latency of about 210 ms. **Defects the harness found, fixed here with tests that go red under mutation:** 1. **A caller refused by its own finished run.** The slot was released by a continuation that was not ordered against the caller's await. 32 of 200 back-to-back runs came back "previous evaluation still running", which the seams publish as Bad. The slot is now released before the caller resumes. 2. **A compile burst starved the heartbeat (windev).** 500 cold loads, each on its own thread-pool work item, delayed the heartbeat; the host killed the shard (KilledHeartbeat) and all 500 keys became suspects. Compiles now run on `min(8, max(1, CPUs/2))` dedicated threads. **New issues** - #787: a disabled VirtualTag is still deployed and evaluated (`Enabled` is ignored by the composer and the artifact). - #788: worker start failures log an Error with a stack trace on every retry. - #789: §8.2 budget misses. - #790: the load timeout counts time spent queued behind other compiles. **Evidence** - Release build of the solution with `--no-incremental`: 0 warnings, 0 errors. - `AKKA_TEST_TIMEFACTOR=3`: | Suite | Result | |---|---| | ScriptWorker | 131 passed / 3 skipped (the perf tests and the self-contained gate); 132/132 on windev self-contained | | Core.Scripting | 221 passed / 1 skipped | | Host | 269 | | Runtime | 991 passed / 3 skipped | | **OpcUaServer.IntegrationTests** | **132/132** | **Still open on #780** (I am keeping it open) - Installer work: the low-privilege worker account, the pipe ACL, the firewall outbound rule and the PeCacheDir ACL. - The scripted-alarm seam, with its batch submit. - The §4.5 batch submit for Calc. - #789 and #790. Then P2 (Shadow) and P3 (after ScadaBridge#112).
dohertj2 commented 2026-11-10 06:38:42 -05:00
Author
Owner
Copy Link
Copy Source

Fourth P1 slice landed on master @ 07969285. It is still opt-in, and InProcess remains the default and is unchanged. Nothing was deployed. The record is in design §12.3.

1. Batch submit (design §4.5, P0)

  • IScriptRunner.RunBatch(invocations) returns one task per item, and results stream back as each item finishes.
    • The default implementation is RunAsync per item, started together. InProcessScriptRunner uses it.
    • The out-of-process runner sends every item that is ready to run as one EvalBatch frame per worker. Ready means admitted, not quarantined, not a suspect, and already loaded in a live worker.
    • Any other item (it needs a compile, its worker is down, or it is a suspect bound for isolation) takes the full RunAsync path on its own. A batch never waits on a load or a restart as a whole.
  • Where it is used (out-of-process path only):
    • Calculation pump. It dequeues everything pending, submits one batch and publishes in dequeue order. A tag queued twice ends the batch, so the published sequence is exactly the one-at-a-time pump's.
    • VirtualTag seam. ScriptBatchCoalescer batches evaluations from tags that become ready together, since there is one actor per tag. It has no timer, so an evaluation on an idle seam is submitted at once.
    • Scripted-alarm engine. It submits the whole dirty set of a pass.
  • In process, the Calc pump and the alarm engine still run one evaluation at a time, unchanged.

2. Per-item deadlines and blame stay exact under batching

  • Before this change, work queued in the worker behind busy threads had its clock running from submit. It could time out, take strikes, and even be the key blamed for the kill at T+G.
  • Now the worker either hands an evaluation to an idle thread or defers it (EvalResults.deferred_eval_ids).
    • The host stops a deferred evaluation's clock.
    • It restarts the full T, then G, when the evaluation starts (started_eval_ids, sent on the result frame of the run that freed the thread).
    • The two notices are unordered on the wire. The deadlines are absolute timestamps, so a stale timer tick never fires early.
  • A run queued behind a runaway is therefore neither timed out nor struck. At the kill it is a victim (script worker restarted), never the culprit.

3. Scripted-alarm seam behind Scripting:Runner:Seams

  • An unset Seams list now means VirtualTag + Calculation. Alarms go out of process only when ScriptedAlarm is listed.
  • The runner reaches DriverHostActor through a new IScriptRunnerProvider (new last parameter scriptedAlarmRunner).
  • Engine refactor (runner path only):
    • LoadAsync still compiles each predicate, to PE bytes only (CompileToImage), so a broken predicate still fails the load. No assembly is loaded into the host.
    • Startup recovery and every re-evaluation pass submit their predicates as one batch, then apply the results in serial order under _evalGate.
    • A test shows the Part 9 transition sequence is identical to an in-process engine's.
  • Failure semantics (§9, alarm column):
    • timed out, killed or quarantined: the condition is frozen with a BadInternalError annotation, as today;
    • the runner failed (worker restarted, or no worker ready): frozen with a BadResourceUnavailable annotation, consistent with VT/Calc, and re-evaluated by itself on recovery;
    • the predicate threw: state unchanged and no annotation, as in process;
    • a dead worker never clears an alarm: only a completed bool reaches the state machine.

4. Perf harness (OTOPCUA_SCRIPT_PERF=1, new Batch_throughput_and_alarm_pass_latency)

  • Batch submit vs the same evaluations as concurrent RunAsync calls, out of process, K=2:

    • macOS: 369k vs 233k evals/s at a batch of 100, and 463k vs 256k at 500.
    • windev: 49k vs 41k at a batch of 100, at about half the host CPU (129 % vs 224 %).

  • Alarm pass, measured from push to the last transition:

    windev, alarms per pass InProcess p99 OOP batched p99 OOP serial (pre-batch) p99
    100 39 ms 5 ms 65 ms
    500 53 ms 36 ms 273 ms — misses 250 ms
    1,000 84 ms 53 ms 519 ms — misses 250 ms

    On macOS every row is ≤ 41 ms. Without batching, a large out-of-process alarm engine would miss its budget on the VM, which is the design's "batch submit is a prerequisite" in numbers. No predicate failed to complete in any pass.

  • Rerun of the §8.2 harness after the scheduler change:

    • gate PASS on both hosts: K=2 closed loop at 366k/s (macOS) and 50k/s (windev);
    • kill latency 9–19 ms;
    • #789's misses are unchanged in kind.

Evidence

  • Release build --no-incremental: 0 warnings, 0 errors.

  • Test results (AKKA_TEST_TIMEFACTOR=3, macOS):

    Suite Result
    ScriptWorker 157/157 (+4 skipped: perf and self-contained)
    Core.Scripting 224 (+1 skipped)
    ScriptedAlarms 109
    Calculation 84
    VirtualTags 16
    Host 270
    Runtime 1002 (+3 skipped)
    OpcUaServer.IntegrationTests 133/133

  • ScriptWorker.Tests on windev is 157/157, and the alarm-seam tests and batch perf run were repeated there.

  • The Runtime and integration suites are green, so #791 (VT→VT routing) and #787 (disabled VT → BadOutOfService) still hold.

  • New tests:

    • BatchSubmitTests: one frame per batch; parity with RunAsync, including a duplicate key; a runaway in a batch is blamed alone; a deferred run is timed from when it starts.
    • AlarmSeamTests (real worker): runaway → kill → quarantine with the sibling still evaluating; a worker killed under a predicate → BadResourceUnavailable, the alarm is never cleared, and it is re-evaluated on return.
    • ScriptedAlarmEngineScriptRunnerTests: same transitions as in process, one batch per pass, annotations by whose fault it was.
    • CalculationDriverBatchPumpTests and ScriptBatchCoalescerTests.
    • ScriptRunnerSelectionTests updated for the explicit alarm seam.

  • 15 mutations, every one turned its test red and was reverted:

    • host ignores the deferral notice;
    • worker never defers;
    • start notice does not re-arm the clock;
    • no batch fast path;
    • Calc pump stays serial;
    • Calc pump does not split at a repeated tag;
    • alarm pass runs one at a time;
    • Unavailable annotated as BadInternalError;
    • no recovery re-evaluation (fake runner and real worker);
    • runner failure skips its annotation;
    • runner failure read as false;
    • coalescer does not queue;
    • alarms included in the default seams.

  • A test issue the full suite exposed: a worker's Exited completes before the evaluations it held are released, so a push right after can be refused with "previous evaluation still running". That is not a strike, and in-process behaves the same. The alarm test now re-pushes instead of assuming the first push lands. I filed no product defect for it.

Still open on #780 (keeping it open)

  • Installer work: low-privilege worker account, pipe ACL, firewall outbound rule, PeCacheDir ACL.
  • #789's remaining misses: Windows host CPU at 10k/s, and worker RSS after a compile.
  • A docker-dev drill with ScriptedAlarm listed. The overlay comment shows how to list it.

Then P2 (Shadow), P3 (flip; alarms last, after ScadaBridge#112), and P4.

**Fourth P1 slice landed on `master` @ `07969285`.** It is still opt-in, and `InProcess` remains the default and is unchanged. Nothing was deployed. The record is in design §12.3. **1. Batch submit (design §4.5, P0)** - `IScriptRunner.RunBatch(invocations)` returns one task per item, and results stream back as each item finishes. - The default implementation is `RunAsync` per item, started together. `InProcessScriptRunner` uses it. - The out-of-process runner sends every item that is ready to run as **one `EvalBatch` frame per worker**. Ready means admitted, not quarantined, not a suspect, and already loaded in a live worker. - Any other item (it needs a compile, its worker is down, or it is a suspect bound for isolation) takes the full `RunAsync` path on its own. A batch never waits on a load or a restart as a whole. - Where it is used (out-of-process path only): - **Calculation pump.** It dequeues everything pending, submits one batch and publishes in dequeue order. A tag queued twice ends the batch, so the published sequence is exactly the one-at-a-time pump's. - **VirtualTag seam.** `ScriptBatchCoalescer` batches evaluations from tags that become ready together, since there is one actor per tag. It has no timer, so an evaluation on an idle seam is submitted at once. - **Scripted-alarm engine.** It submits the whole dirty set of a pass. - In process, the Calc pump and the alarm engine still run one evaluation at a time, unchanged. **2. Per-item deadlines and blame stay exact under batching** - Before this change, work queued in the worker behind busy threads had its clock running from submit. It could time out, take strikes, and even be the key blamed for the kill at T+G. - Now the worker either hands an evaluation to an idle thread or **defers** it (`EvalResults.deferred_eval_ids`). - The host stops a deferred evaluation's clock. - It restarts the full T, then G, when the evaluation **starts** (`started_eval_ids`, sent on the result frame of the run that freed the thread). - The two notices are unordered on the wire. The deadlines are absolute timestamps, so a stale timer tick never fires early. - A run queued behind a runaway is therefore neither timed out nor struck. At the kill it is a victim (`script worker restarted`), never the culprit. **3. Scripted-alarm seam behind `Scripting:Runner:Seams`** - An unset `Seams` list now means **VirtualTag + Calculation**. Alarms go out of process **only when `ScriptedAlarm` is listed**. - The runner reaches `DriverHostActor` through a new `IScriptRunnerProvider` (new last parameter `scriptedAlarmRunner`). - Engine refactor (runner path only): - `LoadAsync` still compiles each predicate, to PE bytes only (`CompileToImage`), so a broken predicate still fails the load. No assembly is loaded into the host. - Startup recovery and every re-evaluation pass submit their predicates as one batch, then apply the results in serial order under `_evalGate`. - A test shows the Part 9 transition sequence is identical to an in-process engine's. - Failure semantics (§9, alarm column): - timed out, killed or quarantined: the condition is frozen with a `BadInternalError` annotation, as today; - the runner failed (worker restarted, or no worker ready): frozen with a **`BadResourceUnavailable`** annotation, consistent with VT/Calc, and re-evaluated by itself on recovery; - the predicate threw: state unchanged and no annotation, as in process; - **a dead worker never clears an alarm**: only a completed `bool` reaches the state machine. **4. Perf harness** (`OTOPCUA_SCRIPT_PERF=1`, new `Batch_throughput_and_alarm_pass_latency`) - Batch submit vs the same evaluations as concurrent `RunAsync` calls, out of process, K=2: - macOS: 369k vs 233k evals/s at a batch of 100, and 463k vs 256k at 500. - windev: 49k vs 41k at a batch of 100, at about half the host CPU (129 % vs 224 %). - Alarm pass, measured from push to the last transition: | windev, alarms per pass | InProcess p99 | OOP batched p99 | OOP serial (pre-batch) p99 | |---:|---:|---:|---:| | 100 | 39 ms | 5 ms | 65 ms | | 500 | 53 ms | 36 ms | **273 ms — misses 250 ms** | | 1,000 | 84 ms | 53 ms | **519 ms — misses 250 ms** | On macOS every row is ≤ 41 ms. Without batching, a large out-of-process alarm engine would miss its budget on the VM, which is the design's "batch submit is a prerequisite" in numbers. No predicate failed to complete in any pass. - Rerun of the §8.2 harness after the scheduler change: - gate PASS on both hosts: K=2 closed loop at 366k/s (macOS) and 50k/s (windev); - kill latency 9–19 ms; - #789's misses are unchanged in kind. **Evidence** - Release build `--no-incremental`: 0 warnings, 0 errors. - Test results (`AKKA_TEST_TIMEFACTOR=3`, macOS): | Suite | Result | |---|---| | ScriptWorker | **157/157** (+4 skipped: perf and self-contained) | | Core.Scripting | 224 (+1 skipped) | | ScriptedAlarms | 109 | | Calculation | 84 | | VirtualTags | 16 | | Host | 270 | | Runtime | 1002 (+3 skipped) | | **OpcUaServer.IntegrationTests** | **133/133** | - `ScriptWorker.Tests` on **windev is 157/157**, and the alarm-seam tests and batch perf run were repeated there. - The Runtime and integration suites are green, so #791 (VT→VT routing) and #787 (disabled VT → BadOutOfService) still hold. - New tests: - `BatchSubmitTests`: one frame per batch; parity with `RunAsync`, including a duplicate key; a runaway in a batch is blamed alone; a deferred run is timed from when it starts. - `AlarmSeamTests` (real worker): runaway → kill → quarantine with the sibling still evaluating; a worker killed under a predicate → `BadResourceUnavailable`, the alarm is never cleared, and it is re-evaluated on return. - `ScriptedAlarmEngineScriptRunnerTests`: same transitions as in process, one batch per pass, annotations by whose fault it was. - `CalculationDriverBatchPumpTests` and `ScriptBatchCoalescerTests`. - `ScriptRunnerSelectionTests` updated for the explicit alarm seam. - 15 mutations, every one turned its test red and was reverted: - host ignores the deferral notice; - worker never defers; - start notice does not re-arm the clock; - no batch fast path; - Calc pump stays serial; - Calc pump does not split at a repeated tag; - alarm pass runs one at a time; - Unavailable annotated as `BadInternalError`; - no recovery re-evaluation (fake runner and real worker); - runner failure skips its annotation; - runner failure read as `false`; - coalescer does not queue; - alarms included in the default seams. - A test issue the full suite exposed: a worker's `Exited` completes before the evaluations it held are released, so a push right after can be refused with "previous evaluation still running". That is not a strike, and in-process behaves the same. The alarm test now re-pushes instead of assuming the first push lands. I filed no product defect for it. **Still open on #780** (keeping it open) - Installer work: low-privilege worker account, pipe ACL, firewall outbound rule, PeCacheDir ACL. - #789's remaining misses: Windows host CPU at 10k/s, and worker RSS after a compile. - A docker-dev drill with `ScriptedAlarm` listed. The overlay comment shows how to list it. Then P2 (Shadow), P3 (flip; alarms last, after ScadaBridge#112), and P4.
dohertj2 commented 2026-11-10 07:36:39 -05:00
Author
Owner
Copy Link
Copy Source

P2 Shadow landed on master @ b1eedcea (03686eb5 code, b1eedcea record). It is opt-in, and InProcess remains the default and is unchanged. Nothing was deployed and no wonder box was touched. The full record is in design §12.4.

1. Scripting:Runner=Shadow

  • The selected seams use the same rule as OutOfProcess: unset means VirtualTag + Calculation, and ScriptedAlarm is included only when listed. Those seams get a ShadowScriptRunner and keep their in-process path byte for byte: no coalescer, the one-at-a-time Calc pump, the in-process alarm engine. What they publish is what InProcess publishes.
  • After each compiled evaluation the seam calls Mirror. That copies the inputs, drops the script logger and queues the shadow run on the pool. It never blocks the seam and never throws.
  • What is compared: value (structural for arrays), exact CLR type, status, and failure class: compile, budget (timed out / killed / refused / quarantined), threw (including the message), unsupported_type.
  • A shadow outcome that is the runner's own fault (Unavailable, worker busy) is not compared; it is counted as shadow_unavailable.
  • ★11: a value the protocol cannot carry is a mismatch with reason unsupported_type.
  • Bounded: at most Scripting:Shadow:MaxInFlight (256) shadow runs outstanding, and at most one per script. Anything beyond that is dropped and counted (behind | key_busy).
  • Metrics: otopcua.script.shadow.compared{kind,result}, .mismatch{kind,reason,script}, .dropped{kind,reason}, .inflight.
  • Log: one Warning per script and reason, carrying a diff summary (published Ok Int32 1; shadow Ok Int32 2). It is re-armed at the seam's deploy boundary and capped at Scripting:Shadow:MismatchLogsPerMinute (20).
  • Readiness is unaffected: Status() is null under Shadow, and the script-runner entry is always Healthy, with the shadow pool's state and counters as data. The shadow pool's kill and quarantine lines go to the host log tagged ScriptRunnerRole=Shadow, never to the script log.
  • New overlay: docker-dev/docker-compose.script-runner-shadow.yml. Documented in docs/VirtualTags.md under "Shadow mode".

2. Tests

  • Mutation-proved: 16 mutants, all red. The one survivor got an assertion and is now red.

  • ShadowScriptRunnerTests (fakes):

    • the comparison table;
    • per-script metric plus a single diff-summary line;
    • the rate limit;
    • a shadow runner that hangs, blocks its caller or throws never reaches the caller;
    • MaxInFlight drop accounting;
    • the input copy.

  • ShadowModeTests (real worker):

    • Ordinary scripts match.
    • A deliberately divergent script and a decimal result are counted against exactly those two scripts.
    • A runaway that runs away only in the worker is killed twice and quarantined there, while every published value stays in-process-correct and no published evaluation waits more than 250 ms.
    • Alarm emissions equal an in-process engine's across a kill -9 of the shadow worker.

  • Release build has 0 warnings; AKKA_TEST_TIMEFACTOR=3.

  • macOS results:

    Suite Passed Skipped
    Core.Scripting 255 1
    ScriptedAlarms 109 0
    VirtualTags 16 0
    Calculation 84 0
    ScriptWorker 160 4
    Host 275 0
    Runtime 1002 3
    OpcUaServer.IntegrationTests 133 0

  • windev: ScriptWorker 160 passed / 4 skipped. The first windev run caught a test-only DateTime bug, fixed before the push.

3. docker-dev drill (central pair, image built from the branch)

  • (a) Shadow, all three seams.
    • VTs (×3, raw ushort), Calc (+1000) and the alarms produced zero mismatches on both nodes.
    • w34-vt-div (divergent) counted value mismatches and w34-vt-dec counted unsupported_type mismatches. Nothing else mismatched.
    • kill -9 of both shadow workers on the leader left every published value Good, and /health/ready stayed 200 / Healthy.
    • [A-7]: a Modbus UInt16 arrives boxed as ushort and round-trips unchanged.
    • [A-8]: in process, a VT that returns a decimal already publishes BadNotSupported. Out of process it would be BadInternalError "unsupported type": Bad either way, so the only change is the status code.
  • (b) OutOfProcess with ScriptedAlarm listed.
    • Alarms raise and clear, and the predicates ran in the worker (ipc_roundtrip{kind=alarm}).
    • A while(true) predicate was killed at 4000 ms twice and then quarantined. Its condition reads Quality BadInternalError. Its siblings on the same shard stayed Active / Good.
    • A kill -9 during a 1.5 s predicate froze that condition Active + BadResourceUnavailable, without clearing it. It moved to isolation and re-evaluated to Inactive / Good with no further input change.
  • Rig left as it was before the drill:
    • drill rows deleted and HR5 back to 5;
    • redeployed with revision e221b179…, identical to before;
    • central-1/2 recreated from the base compose in the same project: image otopcua-host:dev 24f9d7287ed2 (never re-tagged), no Scripting__*, config hash equal to the base file's;
    • the drill image removed; site nodes, sql and traefik untouched.
    • The secret stores now live on the central volumes, so they survived the recreates: central-1's is byte-identical, and central-2's differs only by a WAL checkpoint.
    • Both centrals log Mesh PSK secret ot-mesh-psk-* … absent. central-1 does so with its unchanged store too, so that is pre-existing rig state, not caused by this drill.

Filed: lmxopcua#792 — scripts can reach System.AppContext (SetSwitch / SetData / GetData), p3 review-finding.

Still open on #780:

  • the installer work (low-privilege account, pipe ACL, firewall rule, PeCacheDir ACL), deferred by request;
  • the QA shadow week (P2 exit criterion);
  • then the P3 flip;
  • #789's remaining Windows misses.
**P2 Shadow landed on `master` @ `b1eedcea`** (`03686eb5` code, `b1eedcea` record). It is opt-in, and `InProcess` remains the default and is unchanged. Nothing was deployed and no wonder box was touched. The full record is in design §12.4. **1. `Scripting:Runner=Shadow`** - The selected seams use the same rule as OutOfProcess: unset means VirtualTag + Calculation, and `ScriptedAlarm` is included only when listed. Those seams get a `ShadowScriptRunner` and keep their **in-process path byte for byte**: no coalescer, the one-at-a-time Calc pump, the in-process alarm engine. What they publish is what InProcess publishes. - After each compiled evaluation the seam calls `Mirror`. That copies the inputs, drops the script logger and queues the shadow run on the pool. It never blocks the seam and never throws. - **What is compared:** value (structural for arrays), exact CLR type, status, and failure class: `compile`, `budget` (timed out / killed / refused / quarantined), `threw` (including the message), `unsupported_type`. - A shadow outcome that is the runner's own fault (Unavailable, worker busy) is **not compared**; it is counted as `shadow_unavailable`. - ★11: a value the protocol cannot carry is a mismatch with reason `unsupported_type`. - **Bounded:** at most `Scripting:Shadow:MaxInFlight` (256) shadow runs outstanding, and at most one per script. Anything beyond that is dropped and counted (`behind` | `key_busy`). - **Metrics:** `otopcua.script.shadow.compared{kind,result}`, `.mismatch{kind,reason,script}`, `.dropped{kind,reason}`, `.inflight`. - **Log:** one Warning per script and reason, carrying a diff summary (`published Ok Int32 1; shadow Ok Int32 2`). It is re-armed at the seam's deploy boundary and capped at `Scripting:Shadow:MismatchLogsPerMinute` (20). - **Readiness is unaffected:** `Status()` is null under Shadow, and the `script-runner` entry is always Healthy, with the shadow pool's state and counters as data. The shadow pool's kill and quarantine lines go to the host log tagged `ScriptRunnerRole=Shadow`, never to the script log. - New overlay: `docker-dev/docker-compose.script-runner-shadow.yml`. Documented in `docs/VirtualTags.md` under "Shadow mode". **2. Tests** - Mutation-proved: 16 mutants, all red. The one survivor got an assertion and is now red. - `ShadowScriptRunnerTests` (fakes): - the comparison table; - per-script metric plus a single diff-summary line; - the rate limit; - a shadow runner that hangs, blocks its caller or throws never reaches the caller; - MaxInFlight drop accounting; - the input copy. - `ShadowModeTests` (real worker): - Ordinary scripts match. - A deliberately divergent script and a `decimal` result are counted against exactly those two scripts. - A runaway that runs away **only in the worker** is killed twice and quarantined there, while every published value stays in-process-correct and no published evaluation waits more than 250 ms. - Alarm emissions equal an in-process engine's across a `kill -9` of the shadow worker. - Release build has 0 warnings; `AKKA_TEST_TIMEFACTOR=3`. - macOS results: | Suite | Passed | Skipped | |---|---:|---:| | Core.Scripting | 255 | 1 | | ScriptedAlarms | 109 | 0 | | VirtualTags | 16 | 0 | | Calculation | 84 | 0 | | ScriptWorker | 160 | 4 | | Host | 275 | 0 | | Runtime | 1002 | 3 | | OpcUaServer.IntegrationTests | 133 | 0 | - **windev:** ScriptWorker 160 passed / 4 skipped. The first windev run caught a test-only `DateTime` bug, fixed before the push. **3. docker-dev drill** (central pair, image built from the branch) - **(a) Shadow, all three seams.** - VTs (×3, raw `ushort`), Calc (+1000) and the alarms produced **zero mismatches** on both nodes. - `w34-vt-div` (divergent) counted `value` mismatches and `w34-vt-dec` counted `unsupported_type` mismatches. Nothing else mismatched. - `kill -9` of both shadow workers on the leader left every published value Good, and `/health/ready` stayed 200 / Healthy. - **[A-7]:** a Modbus UInt16 arrives boxed as `ushort` and round-trips unchanged. - **[A-8]:** in process, a VT that returns a `decimal` already publishes **BadNotSupported**. Out of process it would be BadInternalError "unsupported type": Bad either way, so the only change is the status code. - **(b) OutOfProcess with `ScriptedAlarm` listed.** - Alarms raise and clear, and the predicates ran in the worker (`ipc_roundtrip{kind=alarm}`). - A `while(true)` predicate was killed at 4000 ms twice and then quarantined. Its condition reads Quality **BadInternalError**. Its siblings on the same shard stayed Active / Good. - A `kill -9` during a 1.5 s predicate froze that condition **Active + BadResourceUnavailable**, without clearing it. It moved to isolation and re-evaluated to Inactive / Good with no further input change. - **Rig left as it was before the drill:** - drill rows deleted and HR5 back to 5; - redeployed with revision `e221b179…`, identical to before; - central-1/2 recreated from the base compose in the same project: image `otopcua-host:dev` `24f9d7287ed2` (never re-tagged), no `Scripting__*`, config hash equal to the base file's; - the drill image removed; site nodes, sql and traefik untouched. - The secret stores now live on the central volumes, so they survived the recreates: central-1's is byte-identical, and central-2's differs only by a WAL checkpoint. - Both centrals log `Mesh PSK secret ot-mesh-psk-* … absent`. central-1 does so with its unchanged store too, so that is pre-existing rig state, not caused by this drill. **Filed:** lmxopcua#792 — scripts can reach `System.AppContext` (`SetSwitch` / `SetData` / `GetData`), p3 review-finding. **Still open on #780:** - the installer work (low-privilege account, pipe ACL, firewall rule, PeCacheDir ACL), deferred by request; - the QA shadow week (P2 exit criterion); - then the P3 flip; - #789's remaining Windows misses.
dohertj2 commented 2026-11-10 08:52:06 -05:00
Author
Owner
Copy Link
Copy Source

Installer slice landed: 2a8e2981 on master (owner decisions §14 #5/#6 plus the PeCacheDir ACL; record in design §12.5).

  • Worker identity. Set with Scripting:Worker:Identity = NT SERVICE\OtOpcUaScriptWorker. Default unset; Windows only; fail-closed validation.
    • Windows only lets the SCM start a process as a virtual account, so the host starts each worker under a restricted token derived from its own. That token has no privileges except SeChangeNotify, and its restricting SIDs are the identity, Everyone, Users and RESTRICTED.
    • The worker is created suspended, put in its Job Object before it runs, and given an ACL'd work dir as its TEMP. DOTNET_EnableDiagnostics=0 is set.
    • Found on windev: the runtime would not load under the restricted token (coreclr ERROR_DLL_INIT_FAILED, from USER32's window-station connection). The fix grants the identity on the host's window station and desktop. The logon SID is deliberately not used, because the host's process DACL gives it PROCESS_VM_READ.
  • Pipe ACL. On Windows the pipe is always created with an explicit, protected DACL that admits only the host account and the identity. Unlike CurrentUserOnly, it gives no ACE to the owner (Administrators on an elevated host).
  • Installer. scripts/install/Install-ScriptWorkerIsolation.ps1 (+ .psm1) is idempotent and supports -WhatIf, -Uninstall (rollback) and -PlanOnly. It sets up:
    • a disabled anchor service for the virtual account;
    • PeCacheDir: protected, no Users, with the key and markers host-only; pe\ gets Modify for the identity;
    • read+execute for the identity on script-worker\;
    • a deny on the install root, so the worker cannot read appsettings or logs;
    • an outbound firewall block on the worker exe;
    • the host-service env, optionally.
  • Tests.
    • The dry-run harness scripts/install/tests/Test-ScriptWorkerIsolation.ps1 has 46 assertions and is mutation-checked.
    • ScriptWorkerIdentityTests and WorkerIsolationTests cover the pipe ACL, the token's access, and a real restricted worker running a script.
    • Suites (Release, 0 warnings, AKKA_TEST_TIMEFACTOR=3): macOS ScriptWorker 165/8 skip, Core.Scripting 296/1, Host 279, OpcUaServer.IntegrationTests 133; windev ScriptWorker 168/5 skip. Nothing persistent was changed on windev.
  • Live install on QA is deploy gate #794, with the exact checklist. Keeping this issue open for the QA shadow week and the P3 flip.
**Installer slice landed: `2a8e2981` on master** (owner decisions §14 #5/#6 plus the PeCacheDir ACL; record in design §12.5). - **Worker identity.** Set with `Scripting:Worker:Identity = NT SERVICE\OtOpcUaScriptWorker`. Default unset; Windows only; fail-closed validation. - Windows only lets the SCM start a process *as* a virtual account, so the host starts each worker under a **restricted token** derived from its own. That token has no privileges except SeChangeNotify, and its restricting SIDs are the identity, Everyone, Users and RESTRICTED. - The worker is created suspended, put in its Job Object before it runs, and given an ACL'd work dir as its TEMP. `DOTNET_EnableDiagnostics=0` is set. - Found on windev: the runtime would not load under the restricted token (coreclr `ERROR_DLL_INIT_FAILED`, from USER32's window-station connection). The fix grants the identity on the host's window station and desktop. The logon SID is deliberately not used, because the host's process DACL gives it `PROCESS_VM_READ`. - **Pipe ACL.** On Windows the pipe is always created with an explicit, protected DACL that admits only the host account and the identity. Unlike `CurrentUserOnly`, it gives no ACE to the owner (Administrators on an elevated host). - **Installer.** `scripts/install/Install-ScriptWorkerIsolation.ps1` (+ `.psm1`) is idempotent and supports `-WhatIf`, `-Uninstall` (rollback) and `-PlanOnly`. It sets up: - a disabled anchor service for the virtual account; - PeCacheDir: protected, no Users, with the key and markers host-only; `pe\` gets Modify for the identity; - read+execute for the identity on `script-worker\`; - a **deny** on the install root, so the worker cannot read appsettings or logs; - an **outbound firewall block** on the worker exe; - the host-service env, optionally. - **Tests.** - The dry-run harness `scripts/install/tests/Test-ScriptWorkerIsolation.ps1` has 46 assertions and is mutation-checked. - `ScriptWorkerIdentityTests` and `WorkerIsolationTests` cover the pipe ACL, the token's access, and a real restricted worker running a script. - Suites (Release, 0 warnings, `AKKA_TEST_TIMEFACTOR=3`): macOS ScriptWorker 165/8 skip, Core.Scripting 296/1, Host 279, OpcUaServer.IntegrationTests 133; **windev ScriptWorker 168/5 skip**. Nothing persistent was changed on windev. - **Live install on QA** is deploy gate **#794**, with the exact checklist. Keeping this issue open for the QA shadow week and the P3 flip.
dohertj2 added the blocked label 2026-11-10 08:59:05 -05:00
dohertj2 commented 2026-11-10 08:59:05 -05:00
Author
Owner
Copy Link
Copy Source

Labelled blocked. All code slices are on master, including the Windows isolation installer in 2a8e2981. What's left is the QA shadow week (deploy gate #794 and #793) and the P3 flip.

Labelled `blocked`. All code slices are on master, including the Windows isolation installer in `2a8e2981`. What's left is the QA shadow week (deploy gate #794 and #793) and the P3 flip.
dohertj2 changed title from Implement the out-of-process script runner (design #750) — blocked on owner decisions to Implement the out-of-process script runner (design #750) — code on master; QA shadow week (#793, #794) + P3 flip remain 2026-11-10 09:41:01 -05:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
auto-managed
queue management
 blocked
Blocked by dependency or external hold
 cross-driver
queue management
 cross-repo
Needs coordinated change in another family repo (wire contract etc.)
 driver/abcip
queue management
 driver/ablegacy
queue management
 driver/focas
queue management
 driver/mqtt
MQTT / Sparkplug B broker-subscribe driver
 driver/opcuaclient
queue management
 driver/s7
queue management
 driver/twincat
queue management
 migrated
Migrated from a markdown backlog (2026-11 standardization)
 phase/1
queue management
 phase/2
queue management
 phase/3
queue management
 phase/4
queue management
 phase/5
queue management
 phase/6
queue management
 priority:p0
Must-have / blocking
 priority:p1
High priority
 priority:p2
Normal priority
 priority:p3
Low priority / nice to have
 queue/blocked
queue management
 queue/done
queue management
 queue/failed
queue management
 queue/in-progress
queue management
 queue/queued
queue management
 type:bug
Defect in shipped behaviour
 type:debt
Technical debt / hygiene
 type:deploy-gate
Rollout/deploy/migration step not yet done on an environment
 type:docs
Documentation work
 type:feature
Feature implementation
 type:followup
Follow-up deferred from a plan, PR or gate
 type:infra
Infrastructure and scaffolding
 type:review-finding
Unfixed finding from an architecture/code review
 type:test
Test work
No Label blocked priority:p3 type:feature
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
dohertj2 (dohertj2)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: dohertj2/lmxopcua#780
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?