f50b29abae
v2-ci / unit-tests (push) Failing after 1h12m10s
Second P1 slice of docs/plans/2026-11-09-out-of-process-script-runner-design.md; still opt-in behind Scripting:Runner=OutOfProcess (InProcess unchanged). - Compiled-PE cache (design §7, §14 #8): Scripting:Worker:PeCacheDir holds <dir>/pe/<id>.pe entries authenticated with HMAC-SHA256 under a per-node random key (<dir>/pe-cache.key, created by the host, 0600/0700 on Unix), handed to the worker in Hello. Entry id = SHA-256(compile fingerprint + kind + source); the fingerprint covers the Roslyn version, the Core.Scripting build (analyzer, wrapper, stack guard), the sandbox reference set and the runtime. Edited, truncated, other-key or renamed entries are rejected, deleted and recompiled; the worker never loads unverified bytes. Pruned by recency at startup (30 d / 50 000). ScriptEvaluator splits into CompileToImage + FromImage. Measured on 40 scripts, restart to every first result: macOS 820 ms no cache / 840 ms empty / 85 ms warm; windev (Windows) 9.5 s / 9.4 s / 1.0 s. - Quarantine race: WorkerConnection runs exit attribution synchronously BEFORE completing the evaluations/loads it held (and before Exited), and the shard passes it straight to the runner — a kill strike is in place before the killed key's slot is released. - Supervision: a dead worker restarts by itself after its backoff; Unavailable outcomes carry ScriptRunOutcome.Recovery (-> VirtualTagEvalResult.RetryWhen) and the VT actor / Calculation driver re-evaluate once against current inputs when it completes. Isolation worker stops after Scripting:Worker:IsolationIdleTimeout (10 min). Keys compiling when a worker dies are suspects too (isolation finds a poison compile and quarantines it). Startup orphan sweep by launch marker: kills only a process whose marker's host is gone and whose pid + start time + image match. - Crash-loop circuit (Scripting:Worker:CircuitOpenAfter, 2 min): runner Unavailable, one Error, evaluations answered at once while shards keep retrying at the cap. - Observability (§10): otopcua.script.runner.state, worker.up/inflight/queue_depth/heap_bytes (shard), ipc.roundtrip + eval.duration (kind), compile{cache}, isolation.suspects, exits reason=idle. `script-runner` health check on the ready tier, at worst Degraded (§14 #12: degraded but still ready). Left on #780: self-contained worker publish, installer (account, pipe ACL, firewall), docker-dev live drill, scripted-alarm seam, the §4.5 per-seam batch submit, the §8 perf harness. Tests: SupervisionTests, PeCacheStoreTests/PeCacheWorkerTests (real workers), ScriptRunnerHealthCheckTests, VT + Calc recovery seam tests; 14 mutations each turned their test red. Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
69 lines
3.6 KiB
C#
69 lines
3.6 KiB
C#
using Microsoft.Extensions.Diagnostics.HealthChecks;
|
|
using ZB.MOM.WW.OtOpcUa.Core.Scripting.OutOfProcess;
|
|
using ZB.MOM.WW.OtOpcUa.Host.Configuration;
|
|
|
|
namespace ZB.MOM.WW.OtOpcUa.Host.Health;
|
|
|
|
/// <summary>
|
|
/// Pure decision function for the script-runner probe (lmxopcua#780 design §10, owner decision §14 #12), factored
|
|
/// out of <see cref="ScriptRunnerHealthCheck"/> so the matrix is table-testable without worker processes.
|
|
/// </summary>
|
|
public static class ScriptRunnerHealthDecision
|
|
{
|
|
/// <summary>
|
|
/// Maps the runner's status to a health status. <b>Never Unhealthy:</b> a sick script runner does not stop the
|
|
/// node serving driver data over OPC UA, so it must not fail readiness — on a redundant pair that would shift
|
|
/// ServiceLevel to the peer over a script problem. Degraded is still ready (200) but visible.
|
|
/// </summary>
|
|
/// <param name="status">The out-of-process runner's status, or null when every seam runs in process.</param>
|
|
/// <returns>The status, a description and the per-shard data.</returns>
|
|
public static (HealthStatus Status, string Description, IReadOnlyDictionary<string, object> Data) Evaluate(ScriptRunnerStatus? status)
|
|
{
|
|
if (status is null)
|
|
return (HealthStatus.Healthy, "Scripts run in process (Scripting:Runner=InProcess).", new Dictionary<string, object>());
|
|
|
|
var data = new Dictionary<string, object>(StringComparer.Ordinal)
|
|
{
|
|
["state"] = status.State.ToString(),
|
|
["suspects"] = status.SuspectCount,
|
|
["quarantined"] = status.QuarantinedCount,
|
|
["peCache"] = status.PeCacheEnabled,
|
|
};
|
|
foreach (var shard in status.Shards)
|
|
{
|
|
data[$"shard.{shard.Label}"] = shard.Up
|
|
? $"up pid {shard.Pid}, {shard.InFlight} in flight"
|
|
: shard.DownFor is { } downFor ? $"down {downFor.TotalSeconds:F0} s ({shard.ConsecutiveFailures} failure(s))" : "not started";
|
|
}
|
|
|
|
var down = status.Shards.Where(s => s.DownFor is not null).Select(s => s.Label).ToArray();
|
|
return status.State switch
|
|
{
|
|
ScriptRunnerState.Ready => (HealthStatus.Healthy, "Out-of-process script runner ready.", data),
|
|
ScriptRunnerState.Degraded => (HealthStatus.Degraded,
|
|
$"Out-of-process script runner degraded: ScriptWorker {string.Join(", ", down)} down; scripts placed there report BadResourceUnavailable until it restarts.",
|
|
data),
|
|
_ => (HealthStatus.Degraded,
|
|
"Out-of-process script runner unavailable (circuit open): every ScriptWorker shard is down; every compiled script reports BadResourceUnavailable. OPC UA keeps serving driver data.",
|
|
data),
|
|
};
|
|
}
|
|
}
|
|
|
|
/// <summary>The script-runner readiness component: Healthy, or Degraded-but-ready (see <see cref="ScriptRunnerHealthDecision"/>).</summary>
|
|
/// <param name="selection">The node's runner selection; null on nodes that never register one.</param>
|
|
public sealed class ScriptRunnerHealthCheck(ScriptRunnerSelection? selection) : IHealthCheck
|
|
{
|
|
/// <summary>The check's registration name.</summary>
|
|
public const string Name = "script-runner";
|
|
|
|
/// <inheritdoc />
|
|
public Task<HealthCheckResult> CheckHealthAsync(HealthCheckContext context, CancellationToken cancellationToken = default)
|
|
{
|
|
var (status, description, data) = ScriptRunnerHealthDecision.Evaluate(selection?.Status());
|
|
return Task.FromResult(status == HealthStatus.Healthy
|
|
? HealthCheckResult.Healthy(description, data)
|
|
: HealthCheckResult.Degraded(description, data: data));
|
|
}
|
|
}
|