feat(scripting): self-contained script worker publish, docker-dev drill, §8 perf harness (lmxopcua#780 P1)
v2-ci / unit-tests (push) Waiting to run

- A self-contained Host publish (-r win-x64 --self-contained, the Windows installs) now publishes the
  ScriptWorker self-contained for the same RID into script-worker/. The launcher starts its apphost
  directly when runtimeconfig.json lists includedFrameworks. A framework-dependent worker (docker-dev)
  keeps running through the dotnet muxer.
- docker-dev/docker-compose.script-runner.yml: OutOfProcess for the central pair's VT + Calc seams.
  Live drill recorded in design §12.1 (runaway kill + quarantine, kill -9 recovery, open circuit
  with /health/ready still 200, metrics).
- ScriptRunnerPerfTests (Category=Perf, opt-in via OTOPCUA_SCRIPT_PERF=1): InProcess vs OutOfProcess
  at K=1/2/4. The accepted 5k evals/s/node gate passes on macOS and windev. The host-CPU, RSS and
  Windows latency budget misses are tracked as #789.
- Fix: the single-flight slot is released before the awaiting caller resumes. Before, a caller that
  started its next run right after the previous one returned could be refused with "previous
  evaluation still running", which the seams publish as Bad.
- Fix: the worker compiles on dedicated threads, not the thread pool. A burst of cold loads starved
  the heartbeat, and the host killed the shard (found on windev). Load-timeout queueing is #790.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
This commit is contained in:
Joseph Doherty
2026-11-10 02:53:53 -05:00
parent f50b29abae
commit 1ead8f458a
13 changed files with 836 additions and 20 deletions
@@ -100,6 +100,55 @@ public sealed class SupervisionTests
attributed.ShouldBeTrue("the runner's verdict is recorded before the callback returns");
}
/// <summary>
/// A burst of cold compiles must not starve the worker's heartbeat. Found on windev by the §8 perf harness: 500
/// loads at once, each compiled synchronously on its own thread-pool work item, queued the heartbeat timer and
/// the pipe writer behind them; the host saw no frame for 3 s, killed the shard (KilledHeartbeat) and made all 500
/// keys suspects. Here a 100 ms heartbeat (kill after 300 ms of silence) and 400 distinct cold scripts at once:
/// every script answers and the shard's worker is the one that started. Mutation check: compile on the thread
/// pool again and the worker is killed for silence.
/// </summary>
[Fact]
public async Task A_burst_of_cold_compiles_does_not_starve_the_heartbeat()
{
var sink = new CapturingSink();
await using var runner = new OutOfProcessScriptRunner(Scripts.Options(), sink.CreateLogger(), null,
heartbeatInterval: TimeSpan.FromMilliseconds(50), loadTimeout: TimeSpan.FromSeconds(120));
(await runner.RunAsync(Scripts.Vt("first", Healthy), Ct)).Value.ShouldBe(7);
var worker = runner.Shards[0].Current.ShouldNotBeNull();
const int count = 1500;
var outcomes = await Task.WhenAll(Enumerable.Range(0, count).Select(i => runner.RunAsync(
Scripts.Vt("burst-" + i, $"return (int)ctx.GetTag(\"x\").Value * {i} + 1;", TimeSpan.FromSeconds(60),
Scripts.Input("x", 2)), Ct)));
var log = string.Join(Environment.NewLine, sink.Rendered.Where(l => l.Contains("ScriptWorker", StringComparison.Ordinal)).Take(5));
for (var i = 0; i < count; i++) outcomes[i].Value.ShouldBe(2 * i + 1, $"{outcomes[i].Status} {outcomes[i].Reason}\n{log}");
runner.Shards[0].Current.ShouldNotBeNull().Pid.ShouldBe(worker.Pid, log);
worker.Exited.IsCompleted.ShouldBeFalse(log);
}
/// <summary>
/// Single flight must not refuse a caller's NEXT run because of the run it already awaited: the slot is freed
/// before the awaiting caller resumes. The §8 perf harness found the slot freed by a continuation unordered
/// against the caller's await, so 32 of 200 back-to-back runs of one key came back "previous evaluation still
/// running" — which the VirtualTag and Calculation seams publish as Bad. Mutation check: drop the inline
/// release after the await and this goes red.
/// </summary>
[Fact]
public async Task A_caller_that_awaited_its_run_is_never_refused_by_it()
{
await using var runner = Scripts.Runner(Scripts.Options());
var refused = new List<string>();
for (var i = 0; i < 1_000; i++)
{
var outcome = await runner.RunAsync(
Scripts.Vt("seq", "return (int)ctx.GetTag(\"x\").Value + 7;", inputs: Scripts.Input("x", i)), Ct);
if (outcome.Status != ScriptRunStatus.Ok || !Equals(outcome.Value, i + 7)) refused.Add($"#{i}: {outcome.Reason}");
}
refused.ShouldBeEmpty();
}
/// <summary>
/// The race itself, end to end: with QuarantineAfterKills = 1, the moment the killed script's single-flight slot is
/// released the quarantine is already in place — so no trigger can re-run it in the restart gap, and the next