ShadowModeTests' runaway case asserted "slowest < 250 ms" around evaluations
whose timing included cold Roslyn compile/JIT of new script text, so it failed
on a healthy published path (540-880 ms measured, both SDK baselines).
The test now puts a test-owned barrier runner (GatedShadow) between the real
ShadowScriptRunner and the real worker pool. The runaway's first shadow run is
held before it reaches the pool, then (after it runs away in the worker, the
pool answers T and the worker is KilledDeadline with the runaway as culprit)
its outcome is held before comparison. Published evaluations made while a
barrier is closed must return the in-process values; a Mirror that waited for
its shadow hangs into a 30 s liveness failure (mutation-checked). Output
correctness, pool kill/quarantine, published-side never quarantined, the single
runaway mismatch (sibling is a victim, not compared) and key_busy drops are kept.
The wall-clock number moves to an opt-in Category=Perf case
(Shadow_published_path_latency_while_the_shadow_runs_away, OTOPCUA_SCRIPT_PERF=1)
on warmed evaluators, reporting hardware and load average.
Sibling: ContainmentTests' hard-deadline case timed the runaway's first
(cold-compile) run; it now compiles the runaway's source first with an input
that does not spin.
Central pin 1.1.0 -> 3.2.2; AdminUI/ControlPlane/Runtime/Host.Tests no longer carry VersionOverride.
The three driver IntegrationTests fixtures' Fact/Theory attributes gain the source-info ctor (xUnit3003).
No xUnit1051 fallout remained (already cleaned). Test counts unchanged.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Akka.TestKit.Xunit2 -> Akka.TestKit.Xunit 1.5.62 (xunit.v3 TestKit; needs xunit.v3 3.2.2, set via
VersionOverride on the four projects). Removes the xunit 2.9.3 pin. IAsyncLifetime returns ValueTask,
SqlServerFactAttribute gains the source-info ctor, and xUnit1051 call sites pass
TestContext.Current.CancellationToken. Test counts unchanged.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Owner decisions §14 #5/#6 as built (design §12.5):
- Scripting:Worker:Identity (default unset, Windows only, fail-closed validation): each worker
starts under a restricted token derived from the host's. Every privilege but SeChangeNotify is
removed; the restricting SIDs are NT SERVICE\OtOpcUaScriptWorker's SID + Everyone + Users +
RESTRICTED; the token and default DACLs are host + SYSTEM + identity + RESTRICTED. The worker
is created suspended, put in its Job Object before it runs, inherits only its stdio pipes, gets
an ACL'd work dir as TEMP, and runs with DOTNET_EnableDiagnostics=0. The identity is granted on
the host's window station and desktop, because USER32 init made coreclr.dll fail with
ERROR_DLL_INIT_FAILED on windev. The logon SID is not used because the host's process DACL
gives it PROCESS_VM_READ.
- The worker pipe on Windows always has an explicit protected DACL: host account + identity
only. There is no owner ACE, so an elevated host's Administrators gets no access.
- The worker treats access denied on its parent-pid poll as alive; the Job Object covers that case.
- scripts/install/Install-ScriptWorkerIsolation.ps1 + ScriptWorkerIsolation.psm1 are idempotent
and support -WhatIf, -Uninstall and -PlanOnly. They create the disabled anchor service, the
PeCacheDir ACL (protected, no Users, key and markers host-only, pe\ Modify for the identity),
read+execute on script-worker\, a ReadData deny on the install root, the outbound firewall
block on the worker exe, and optionally the host-service env.
- Tests: the dry-run harness scripts/install/tests/Test-ScriptWorkerIsolation.ps1 (46
assertions, mutation-checked); ScriptWorkerIdentityTests; WorkerIsolationTests, which cover
the pipe ACL, token access and a real restricted worker running a script, all green on windev.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
ForbiddenTypeAnalyzer now denies System.AppContext, System.Console, GCSettings, ProfileOptimization,
EncodingProvider/CodePagesEncodingProvider, TypeDescriptor and CryptoConfig as types, and the ambient
CultureInfo.{DefaultThreadCurrentCulture,DefaultThreadCurrentUICulture,CurrentCulture,CurrentUICulture}
and TimeZoneInfo.ClearCachedData members (member deny-list now covers properties and member-access reads).
Shadow-mode tests diverge via culture-sensitive string.Compare (worker is InvariantGlobalization) instead of
AppContext.BaseDirectory, with a premise test. Mutation-proved: 21 new cases fail with the analyzer reverted.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
- ShadowScriptRunner (Core.Scripting): IScriptShadow.Mirror copies the inputs and queues the shadow run on the
pool; compares value (structural), CLR type, status and failure class (compile / budget / threw+message /
unsupported_type); runner-fault shadow outcomes (Unavailable, worker busy) are not compared. Bounded:
Scripting:Shadow:MaxInFlight (256) and one per script; drops counted (behind | key_busy).
- Metrics otopcua.script.shadow.{compared,mismatch{script},dropped,inflight}; one rate-limited Warning per
script and reason with a diff summary (Scripting:Shadow:MismatchLogsPerMinute, re-armed at the seam's deploy).
- The VirtualTag, Calculation and scripted-alarm seams recognise the shadow and keep their in-process path
byte-for-byte (no coalescer, one-at-a-time pump, in-process engine), mirroring after each compiled run.
- ScriptRunnerSelection builds the shadow over an OutOfProcessScriptRunner whose lines carry
ScriptRunnerRole=Shadow; Status() is null under Shadow so readiness never sees the shadow pool; the
script-runner health entry stays Healthy and reports the shadow counters as data.
- docker-dev/docker-compose.script-runner-shadow.yml; docs/VirtualTags.md "Shadow mode".
- Tests: comparison table, per-script metric + once-per-reason log, rate limit, hung/blocking/throwing shadow
never reaches the caller, MaxInFlight drop accounting, input copy; real-worker divergent + unsupported-type
scripts counted against exactly those scripts, a worker-only runaway killed/quarantined there with every
published value intact, alarm transitions identical to in process across a kill -9 of the shadow worker.
Mutation-proved (16 mutants, 15 red; the one survivor added an assertion).
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
- IScriptRunner.RunBatch: one EvalBatch frame per worker for every evaluation ready to run; per-item, streamed results.
The default (InProcessScriptRunner) is RunAsync per item. Used by the Calculation pump (dequeue all, split at a
repeated tag, publish in dequeue order), the VirtualTag seam (ScriptBatchCoalescer, no timer) and the scripted-alarm
engine (whole dirty set). In process the pump and the engine stay one at a time, unchanged.
- Exact per-item deadlines: the worker hands work to an idle evaluation thread or defers it (EvalResults
deferred_eval_ids); the host stops that run's clock and restarts T then G when it starts (started_eval_ids). A run
queued behind a runaway is neither timed out nor struck and is a victim, not the culprit, at the kill.
- Scripted-alarm seam behind Scripting:Runner:Seams; unset Seams now means VirtualTag + Calculation, so alarms run out of
process only when listed. Load still compiles each predicate (to PE bytes only) so a broken one fails the load; passes
batch and apply in the serial order under the gate. Script faults freeze the condition BadInternalError; runner faults
freeze it BadResourceUnavailable and re-evaluate on recovery; a dead worker never clears an alarm. IScriptRunnerProvider
carries the runner to DriverHostActor.
- Perf harness: batch throughput (RunBatch vs RunAsync x n) and alarm-pass latency (in process / batched / serial);
design §12.3 records macOS + windev. On windev the serial out-of-process alarm pass misses 250 ms from ~500 alarms;
batched p99 <= 53 ms at 1,000.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
- NodeManagerLockTimeBenchmark: wall-clock budget enforced only with OTOPCUA_BENCH=1; default run keeps the state-correctness asserts and logs timings.
- OpcUaClientDriverWire host-status: signal (semaphore) wait that only accepts a non-Running state observed after the server is disposed.
- AbCip PollLatencyComparison: retry once on a fixture connection reset; p95 criterion enforced only with AB_LAT_ENFORCE=1.
- ScriptRunGuardTests: orphan scripts held by a ctx.Released signal instead of a 250 ms wall-clock spin that could finish inside the 60 ms budget under load; stuck-orphan wall-clock rule disabled in the tests that pin the strike counting.
- DependencyMuxReplayTests: split the inside-grace positive control (1 h grace) from the release test (control asserted before unregister), removing the 200 ms grace race.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Proven broken first: VirtualTagToVirtualTagFlowTests (real mux + host + Roslyn
evaluator) failed on master - VT A evaluated, VT B reading A by its key never did.
VT outputs only reached their OPC UA node; the mux carried driver values only.
- VirtualTagHostActor also sends each real result (not re-asserts) and a disabled
tag's BadOutOfService to the mux as VirtualTagOutput, keyed by the VT's UNS
NodeId identifier {EquipmentId}/{Name}. Publish path unchanged (single source).
- DependencyMuxActor pins VT outputs (retained with no subscriber, no grace,
immune to SourceRemoved) and replays them; VirtualTagOutputsRemoved releases
keys that leave the deployed set.
- DraftValidator: blocking VirtualTagDependencyCycle (A->B->A, self-read, loops
through Calculation tags, {{equip}} refs resolved); disabled VTs are not nodes.
EquipReferenceUnresolved names the key form when {{equip}}/<name> is a VT.
- Runtime backstop: the host re-detects VT cycles per apply and does not route
outputs of tags on one.
- AdminUI ScriptTagCatalog lists VTs by key (the bare leaf name never resolved).
- docs/VirtualTags.md documents the routing, key, retention and cycle rules.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Unsubscribe paths now delete through the existing DeleteSubscriptionAsync seam so a test can hold the delete open >grace. Mutation: dropping the post-delete restamp fails both slow-delete tests.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Joseph Doherty