feat(scripting): self-contained script worker publish, docker-dev drill, §8 perf harness (lmxopcua#780 P1)
v2-ci / unit-tests (push) Waiting to run
v2-ci / unit-tests (push) Waiting to run
- A self-contained Host publish (-r win-x64 --self-contained, the Windows installs) now publishes the ScriptWorker self-contained for the same RID into script-worker/. The launcher starts its apphost directly when runtimeconfig.json lists includedFrameworks. A framework-dependent worker (docker-dev) keeps running through the dotnet muxer. - docker-dev/docker-compose.script-runner.yml: OutOfProcess for the central pair's VT + Calc seams. Live drill recorded in design §12.1 (runaway kill + quarantine, kill -9 recovery, open circuit with /health/ready still 200, metrics). - ScriptRunnerPerfTests (Category=Perf, opt-in via OTOPCUA_SCRIPT_PERF=1): InProcess vs OutOfProcess at K=1/2/4. The accepted 5k evals/s/node gate passes on macOS and windev. The host-CPU, RSS and Windows latency budget misses are tracked as #789. - Fix: the single-flight slot is released before the awaiting caller resumes. Before, a caller that started its next run right after the previous one returned could be refused with "previous evaluation still running", which the seams publish as Bad. - Fix: the worker compiles on dedicated threads, not the thread pool. A burst of cold loads starved the heartbeat, and the host killed the shard (found on windev). Load-timeout queueing is #790. Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
This commit is contained in:
@@ -100,6 +100,55 @@ public sealed class SupervisionTests
|
||||
attributed.ShouldBeTrue("the runner's verdict is recorded before the callback returns");
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// A burst of cold compiles must not starve the worker's heartbeat. Found on windev by the §8 perf harness: 500
|
||||
/// loads at once, each compiled synchronously on its own thread-pool work item, queued the heartbeat timer and
|
||||
/// the pipe writer behind them; the host saw no frame for 3 s, killed the shard (KilledHeartbeat) and made all 500
|
||||
/// keys suspects. Here a 100 ms heartbeat (kill after 300 ms of silence) and 400 distinct cold scripts at once:
|
||||
/// every script answers and the shard's worker is the one that started. Mutation check: compile on the thread
|
||||
/// pool again and the worker is killed for silence.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public async Task A_burst_of_cold_compiles_does_not_starve_the_heartbeat()
|
||||
{
|
||||
var sink = new CapturingSink();
|
||||
await using var runner = new OutOfProcessScriptRunner(Scripts.Options(), sink.CreateLogger(), null,
|
||||
heartbeatInterval: TimeSpan.FromMilliseconds(50), loadTimeout: TimeSpan.FromSeconds(120));
|
||||
(await runner.RunAsync(Scripts.Vt("first", Healthy), Ct)).Value.ShouldBe(7);
|
||||
var worker = runner.Shards[0].Current.ShouldNotBeNull();
|
||||
|
||||
const int count = 1500;
|
||||
var outcomes = await Task.WhenAll(Enumerable.Range(0, count).Select(i => runner.RunAsync(
|
||||
Scripts.Vt("burst-" + i, $"return (int)ctx.GetTag(\"x\").Value * {i} + 1;", TimeSpan.FromSeconds(60),
|
||||
Scripts.Input("x", 2)), Ct)));
|
||||
|
||||
var log = string.Join(Environment.NewLine, sink.Rendered.Where(l => l.Contains("ScriptWorker", StringComparison.Ordinal)).Take(5));
|
||||
for (var i = 0; i < count; i++) outcomes[i].Value.ShouldBe(2 * i + 1, $"{outcomes[i].Status} {outcomes[i].Reason}\n{log}");
|
||||
runner.Shards[0].Current.ShouldNotBeNull().Pid.ShouldBe(worker.Pid, log);
|
||||
worker.Exited.IsCompleted.ShouldBeFalse(log);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Single flight must not refuse a caller's NEXT run because of the run it already awaited: the slot is freed
|
||||
/// before the awaiting caller resumes. The §8 perf harness found the slot freed by a continuation unordered
|
||||
/// against the caller's await, so 32 of 200 back-to-back runs of one key came back "previous evaluation still
|
||||
/// running" — which the VirtualTag and Calculation seams publish as Bad. Mutation check: drop the inline
|
||||
/// release after the await and this goes red.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public async Task A_caller_that_awaited_its_run_is_never_refused_by_it()
|
||||
{
|
||||
await using var runner = Scripts.Runner(Scripts.Options());
|
||||
var refused = new List<string>();
|
||||
for (var i = 0; i < 1_000; i++)
|
||||
{
|
||||
var outcome = await runner.RunAsync(
|
||||
Scripts.Vt("seq", "return (int)ctx.GetTag(\"x\").Value + 7;", inputs: Scripts.Input("x", i)), Ct);
|
||||
if (outcome.Status != ScriptRunStatus.Ok || !Equals(outcome.Value, i + 7)) refused.Add($"#{i}: {outcome.Reason}");
|
||||
}
|
||||
refused.ShouldBeEmpty();
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// The race itself, end to end: with QuarantineAfterKills = 1, the moment the killed script's single-flight slot is
|
||||
/// released the quarantine is already in place — so no trigger can re-run it in the restart gap, and the next
|
||||
|
||||
Reference in New Issue
Block a user