Joseph Doherty
f50b29abae
v2-ci / unit-tests (push) Failing after 1h12m10s
Second P1 slice of docs/plans/2026-11-09-out-of-process-script-runner-design.md; still opt-in behind Scripting:Runner=OutOfProcess (InProcess unchanged). - Compiled-PE cache (design §7, §14 #8): Scripting:Worker:PeCacheDir holds <dir>/pe/<id>.pe entries authenticated with HMAC-SHA256 under a per-node random key (<dir>/pe-cache.key, created by the host, 0600/0700 on Unix), handed to the worker in Hello. Entry id = SHA-256(compile fingerprint + kind + source); the fingerprint covers the Roslyn version, the Core.Scripting build (analyzer, wrapper, stack guard), the sandbox reference set and the runtime. Edited, truncated, other-key or renamed entries are rejected, deleted and recompiled; the worker never loads unverified bytes. Pruned by recency at startup (30 d / 50 000). ScriptEvaluator splits into CompileToImage + FromImage. Measured on 40 scripts, restart to every first result: macOS 820 ms no cache / 840 ms empty / 85 ms warm; windev (Windows) 9.5 s / 9.4 s / 1.0 s. - Quarantine race: WorkerConnection runs exit attribution synchronously BEFORE completing the evaluations/loads it held (and before Exited), and the shard passes it straight to the runner — a kill strike is in place before the killed key's slot is released. - Supervision: a dead worker restarts by itself after its backoff; Unavailable outcomes carry ScriptRunOutcome.Recovery (-> VirtualTagEvalResult.RetryWhen) and the VT actor / Calculation driver re-evaluate once against current inputs when it completes. Isolation worker stops after Scripting:Worker:IsolationIdleTimeout (10 min). Keys compiling when a worker dies are suspects too (isolation finds a poison compile and quarantines it). Startup orphan sweep by launch marker: kills only a process whose marker's host is gone and whose pid + start time + image match. - Crash-loop circuit (Scripting:Worker:CircuitOpenAfter, 2 min): runner Unavailable, one Error, evaluations answered at once while shards keep retrying at the cap. - Observability (§10): otopcua.script.runner.state, worker.up/inflight/queue_depth/heap_bytes (shard), ipc.roundtrip + eval.duration (kind), compile{cache}, isolation.suspects, exits reason=idle. `script-runner` health check on the ready tier, at worst Degraded (§14 #12: degraded but still ready). Left on #780: self-contained worker publish, installer (account, pipe ACL, firewall), docker-dev live drill, scripted-alarm seam, the §4.5 per-seam batch submit, the §8 perf harness. Tests: SupervisionTests, PeCacheStoreTests/PeCacheWorkerTests (real workers), ScriptRunnerHealthCheckTests, VT + Calc recovery seam tests; 14 mutations each turned their test red. Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
80 lines
3.6 KiB
C#
80 lines
3.6 KiB
C#
using Microsoft.Extensions.DependencyInjection;
|
|
using Microsoft.Extensions.Diagnostics.HealthChecks;
|
|
using Microsoft.Extensions.Options;
|
|
using Shouldly;
|
|
using Xunit;
|
|
using ZB.MOM.WW.Health;
|
|
using ZB.MOM.WW.OtOpcUa.Core.Scripting.OutOfProcess;
|
|
using ZB.MOM.WW.OtOpcUa.Host.Health;
|
|
|
|
namespace ZB.MOM.WW.OtOpcUa.Host.Tests.Health;
|
|
|
|
/// <summary>
|
|
/// lmxopcua#780 design §10, owner decision §14 #12: the script runner is a READY-tier component that is degraded but
|
|
/// still ready when its circuit is open — a sick runner must never fail readiness (on a redundant pair that would move
|
|
/// ServiceLevel to the peer over a script problem while OPC UA still serves driver data).
|
|
/// </summary>
|
|
public sealed class ScriptRunnerHealthCheckTests
|
|
{
|
|
private static ScriptWorkerShardStatus Shard(string label, bool up, double? downSeconds = null)
|
|
=> new(label, label == "iso", up, downSeconds is { } s ? TimeSpan.FromSeconds(s) : null, up ? 1234 : null, up ? 0 : 3, 0, 0, 0);
|
|
|
|
private static ScriptRunnerStatus Status(ScriptRunnerState state, params ScriptWorkerShardStatus[] shards)
|
|
=> new(state, shards, SuspectCount: 0, QuarantinedCount: 0, PeCacheEnabled: true);
|
|
|
|
[Fact]
|
|
public void In_process_is_healthy()
|
|
=> ScriptRunnerHealthDecision.Evaluate(null).Status.ShouldBe(HealthStatus.Healthy);
|
|
|
|
[Fact]
|
|
public void Ready_is_healthy()
|
|
=> ScriptRunnerHealthDecision.Evaluate(Status(ScriptRunnerState.Ready, Shard("0", true), Shard("1", true))).Status
|
|
.ShouldBe(HealthStatus.Healthy);
|
|
|
|
[Fact]
|
|
public void A_shard_down_is_degraded_and_names_it()
|
|
{
|
|
var (status, description, data) = ScriptRunnerHealthDecision.Evaluate(
|
|
Status(ScriptRunnerState.Degraded, Shard("0", true), Shard("1", false, 4)));
|
|
status.ShouldBe(HealthStatus.Degraded);
|
|
description.ShouldContain("ScriptWorker 1 down");
|
|
data["shard.1"].ShouldBe("down 4 s (3 failure(s))");
|
|
data["state"].ShouldBe("Degraded");
|
|
}
|
|
|
|
/// <summary>★ decision 12: the circuit-open runner is Degraded — never Unhealthy, so /health/ready stays 200.</summary>
|
|
[Fact]
|
|
public void An_open_circuit_is_degraded_never_unhealthy()
|
|
{
|
|
var (status, description, _) = ScriptRunnerHealthDecision.Evaluate(
|
|
Status(ScriptRunnerState.Unavailable, Shard("0", false, 200), Shard("1", false, 200)));
|
|
status.ShouldBe(HealthStatus.Degraded);
|
|
description.ShouldContain("circuit open");
|
|
}
|
|
|
|
[Theory]
|
|
[InlineData(false, false)]
|
|
[InlineData(true, true)]
|
|
public async Task It_is_registered_on_the_ready_tier_only_on_driver_nodes(bool hasDriver, bool registered)
|
|
{
|
|
var registrations = new ServiceCollection()
|
|
.AddOtOpcUaHealth(hasAdmin: true, hasDriver: hasDriver)
|
|
.BuildServiceProvider()
|
|
.GetRequiredService<IOptions<HealthCheckServiceOptions>>()
|
|
.Value.Registrations
|
|
.Where(r => r.Name == ScriptRunnerHealthCheck.Name)
|
|
.ToList();
|
|
if (!registered)
|
|
{
|
|
registrations.ShouldBeEmpty();
|
|
return;
|
|
}
|
|
var registration = registrations.ShouldHaveSingleItem();
|
|
registration.Tags.ShouldBe([ZbHealthTags.Ready]);
|
|
// A node without a runner selection (in-process, or a harness) resolves it as Healthy.
|
|
var check = registration.Factory(new ServiceCollection().BuildServiceProvider()).ShouldBeOfType<ScriptRunnerHealthCheck>();
|
|
(await check.CheckHealthAsync(new HealthCheckContext { Registration = registration }, CancellationToken.None))
|
|
.Status.ShouldBe(HealthStatus.Healthy);
|
|
}
|
|
}
|