lmxopcua/archreview/02-scripting-alarms.md

Architecture Review 02 — Scripting, Virtual Tags, Scripted Alarms, Alarm Historian

• Date: 2026-07-08
• Commit: 9cad9ed0
• Scope:
  • src/Core/ZB.MOM.WW.OtOpcUa.Core.Scripting + Core.Scripting.Abstractions (Roslyn compile/eval engine)
  • src/Core/ZB.MOM.WW.OtOpcUa.Core.VirtualTags (virtual-tag runtime)
  • src/Core/ZB.MOM.WW.OtOpcUa.Core.ScriptedAlarms (scripted-alarm engine + Part 9 state machine)
  • src/Core/ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian (SQLite store-and-forward sink)
  • Test projects under tests/Core/ for coverage assessment
• Method: full read of the load-bearing files (ScriptEvaluator, ScriptSandbox, ForbiddenTypeAnalyzer, CompiledScriptCache, TimedScriptEvaluator, VirtualTagEngine, ScriptedAlarmEngine, Part9StateMachine, SqliteStoreAndForwardSink) plus their support types; production-consumer tracing into Runtime/Host to distinguish live vs dormant code; TODO/stub grep; test-breadth vs source-surface comparison.

---

Architecture Overview

Script compile/eval pipeline (Core.Scripting)

User scripts are C# statement bodies ending in return …;. ScriptEvaluator<TContext,TResult>.Compile (Core.Scripting/ScriptEvaluator.cs) synthesizes a wrapper class (CompiledScript.Run(globals)), then runs a five-step gate:

1. Parse the wrapper source with #line 1 remapping so diagnostics point at operator source.
2. Wrapper-injection guard (EnforceSingleRunMember, ScriptEvaluator.cs:222) — rejects brace-balanced injections that declare sibling members/types (diagnostics LMX001/LMX002).
3. Roslyn compile against the ScriptSandbox reference set (pinned OtOpcUa assemblies + System.*/netstandard TPA subset) — the allow-list is explicitly not the security boundary (type forwarding makes it porous).
4. ForbiddenTypeAnalyzer — the real gate. Two semantic passes: (1) member/call surface via GetSymbolInfo on creations/invocations/member-access/identifiers; (2) every TypeSyntax via GetTypeInfo, recursing generic args + array element types. Deny-list = namespace prefixes (System.IO, System.Net, System.Diagnostics, System.Reflection, System.Threading.Tasks, System.Runtime.InteropServices, System.Runtime.Loader, Microsoft.Win32) plus type-granular names for dangerous residents of allowed namespaces (Environment, AppDomain, GC, Activator, Thread, ThreadPool, Timer, Unsafe).
5. Emit to an in-memory PE, load into a per-script collectible AssemblyLoadContext, bind a static Func<ScriptGlobals<TContext>,TResult> delegate. Dispose() unloads the ALC.

Around the core evaluator sit three wrappers: CompiledScriptCache (SHA-256-source-keyed, Lazy single-compile, disposes ALCs on Clear() at publish-replace), TimedScriptEvaluator (Task.Run + WaitAsync wall-clock budget, default 250 ms, wraps timeout as ScriptTimeoutException; documents the known orphan-thread leak for CPU-bound scripts), and the script-log pipeline (ScriptRootLogger → rolling scripts-*.log + ScriptLogCompanionSink error mirror to the main log + ScriptLogTopicSink → DPS script-logs topic via IScriptLogPublisher).

Core.Scripting.Abstractions holds the Roslyn-free closure the sandbox pins: ScriptContext (the ctx API — GetTag/SetVirtualTag/Now/Logger/Deadband), ScriptGlobals<T>, PassthroughScript (regex fast-path classifier for the return ctx.GetTag("X").Value; mirror shape), and — notably — the concrete VirtualTagContext and AlarmPredicateContext, which live in this assembly but declare Core.VirtualTags / Core.ScriptedAlarms namespaces (see C2).

DependencyExtractor statically harvests ctx.GetTag("literal") / ctx.SetVirtualTag("literal", …) call sites from the AST, rejecting dynamic paths at publish so the change-trigger subscription graph is knowable up front.

Virtual-tag runtime (Core.VirtualTags) — two implementations, one live

The in-scope VirtualTagEngine is a self-contained multi-tag engine: Load() compiles all scripts through a CompiledScriptCache, validates data types and SetVirtualTag targets, builds a DependencyGraph (iterative Tarjan cycle detection + Kahn topo sort + cached rank), subscribes to upstream driver tags via ITagUpstreamSource, and serializes all evaluations under a single SemaphoreSlim gate. Change events cascade dependents in topological order; TimerTriggerScheduler adds interval-grouped timer triggers with in-flight tick skipping; VirtualTagSource adapts the engine to the driver-agnostic IReadable/ISubscribable surface.

However, production does not run this engine. The live path is VirtualTagHostActor → per-tag VirtualTagActor → IVirtualTagEvaluator bound to Host/Engines/RoslynVirtualTagEvaluator (single-tag adapter; fan-out owned by DependencyMuxActor). Grep confirms VirtualTagEngine, TimerTriggerScheduler, VirtualTagSource, and DependencyGraph have no production instantiation — they are exercised only by their unit tests (see U1). Only VirtualTagContext, IHistoryWriter/NullHistoryWriter, and PassthroughScript cross into the live path.

Scripted-alarm flow (Core.ScriptedAlarms) — live end to end

ScriptedAlarmHostActor (Runtime) owns a long-lived ScriptedAlarmEngine and calls LoadAsync per config apply (generation-tagged to discard stale completions). Inside the engine:

1. LoadAsync (under _evalGate) tears down the prior generation (shelving timer, subscriptions, alarms, scratch, compile-cache ALCs), compiles every predicate via CompiledScriptCache<AlarmPredicateContext,bool>, builds an inverse index tag-path → alarm ids (alarms are DAG leaves; no topo sort), seeds the value cache, restores persisted AlarmConditionState from IAlarmStateStore (EF-backed in prod, in-memory for dev/tests), re-derives ActiveState from the live predicate (startup recovery), then subscribes upstream and starts the 5 s shelving-check timer.
2. Upstream changes (OnUpstreamChange, fed by DependencyMuxTagUpstreamSource) update the value cache and spawn tracked fire-and-forget ReevaluateAsync tasks that serialize on _evalGate, run the predicate through TimedScriptEvaluator against a reused per-alarm AlarmScratch (read-cache dictionary + context, refilled in place — the hot path allocates nothing), and apply the result through Part9StateMachine.ApplyPredicate.
3. Part9StateMachine is a pure-function transition table over the immutable AlarmConditionState record (Enabled/Active/Acked/Confirmed/Shelving + audit Comments as ImmutableList). Operator verbs (Acknowledge/Confirm/OneShotShelve/TimedShelve/Unshelve/Enable/Disable/AddComment) come in via the engine's ApplyAsync (persist-before-update-in-memory; emissions built under the gate, fired after release to avoid re-entrancy deadlock).
4. Emissions (ScriptedAlarmEvent) fan out through OnEvent → ScriptedAlarmSource (the IAlarmSource adapter with equipment-path-prefix subscription filters) into the Part 9 condition nodes / /alerts / historian adapter, carrying the per-alarm HistorizeToAveva opt-out. MessageTemplate resolves {TagPath} tokens at emission with a stricter-than-predicate quality bar (Good only; else {?}).

Alarm-history write path (Core.AlarmHistorian)

IAlarmHistorianSink is the fire-and-forget ingestion contract; production is SqliteStoreAndForwardSink: WAL-mode SQLite Queue table, EnqueueAsync commits a row per event (capacity fast path consults an Interlocked cached counter, falling back to COUNT(*) + oldest-row eviction at the wall, with periodic 10k-enqueue resync). A self-rescheduling one-shot timer drains batches (default 100) through IAlarmHistorianWriter (the HistorianGateway SendEvent gRPC client), applying per-event outcomes: Ack → delete, PermanentFail → dead-letter, RetryPlease → attempt-bump with a max-attempts (10) poison cap; corrupt payloads dead-letter immediately so they can't stall the queue head. Backoff ladder 1→2→5→15→60 s applied to the timer due-time; dead letters retained 30 days with operator RetryDeadLettered(); GetStatus() surfaces depth/dead-letter/last-error/evictions to the Admin UI and health checks.

---

Findings

Severity scale: Critical (production correctness/availability today) / High (real risk under realistic conditions) / Medium (defect or debt worth scheduling) / Low (note).

1. Stability

S1 — HIGH — VirtualTagEngine.Load mutates shared state with no gate against in-flight evaluations

VirtualTagEngine.Load (Core.VirtualTags/VirtualTagEngine.cs:78-173) clears and rebuilds the plain Dictionary _tags (line 24), the DependencyGraph, and disposes the compile cache without acquiring _evalGate, while three concurrent paths read that state from arbitrary threads: EvaluateInternalAsync reads _tags before taking the gate (line 293), CascadeAsync walks _graph (line 278) whose internals are plain dictionaries plus a nullable _cachedRank, and timer ticks call EvaluateOneAsync. A Load during an in-flight cascade is a torn-read on non-thread-safe collections. The sibling ScriptedAlarmEngine solved exactly this (gate-held LoadAsync, ConcurrentDictionary with an explanatory comment at ScriptedAlarmEngine.cs:42-50); the VT engine never received the same treatment. Mitigating factor: the engine is dormant in production (U1) — but it is fully unit-tested and presented as production-ready, so anyone wiring it up inherits the race. Recommendation: either retire the engine (preferred — see U1) or port the alarm engine's discipline: gate-held load, ConcurrentDictionary for _tags, and re-check-after-gate in the eval path.

S2 — HIGH — No coalescing/backpressure on upstream-change fan-out (both engines)

Every upstream change spawns a fire-and-forget task that queues on the single eval gate: ScriptedAlarmEngine.OnUpstreamChange (ScriptedAlarmEngine.cs:442-450) spawns one ReevaluateAsync per change event, and VirtualTagEngine.OnUpstreamChange (VirtualTagEngine.cs:263-272) spawns one CascadeAsync per change (untracked). When a referenced tag changes faster than the serialized evaluations drain (a 100 Hz analog referenced by one alarm is enough), tasks accumulate without bound — memory growth plus an ever-staler processing lag, and each queued task re-evaluates against the current cache so the extra work is pure waste. The alarm engine tracks these tasks for dispose-drain (_inFlight) but does not bound them. Recommendation: replace per-event task spawning with a dirty-set + single pump: mark the alarm/tag dirty, and have one loop drain the dirty set under the gate. This makes cost proportional to distinct dirty alarms, not event rate.

S3 — MEDIUM — Failed reload is fail-stop, not fail-back (both engines)

Both engines tear down the previous generation before compiling the new one. On a compile failure, VirtualTagEngine.Load has already unsubscribed, cleared _tags/_graph, and cleared the compile cache (VirtualTagEngine.cs:84-88) before throwing (line 148) — leaving _loaded == true from the prior load, an empty tag set, and a stale _valueCache that Read/VirtualTagSource keep serving as Good values that will never update. ScriptedAlarmEngine.LoadAsync is the same shape (ScriptedAlarmEngine.cs:196-209, throw at 250-255): after a failed apply, zero alarms are loaded — no predicates evaluate until the next successful publish. The host actor explicitly accepts this ("log it and stay inert", ScriptedAlarmHostActor.cs:188, 245), and publish-time validation upstream makes a bad definition reaching the engine unlikely — but a transient IAlarmStateStore.LoadAsync/SaveAsync failure during the state-restore loop (ScriptedAlarmEngine.cs:273-281) also lands here, and that is not operator-preventable. Recommendation: build the new generation into local structures and swap under the gate only on success (compile is already side-effect-free; the subscriptions and timer are the only external effects). At minimum, distinguish store-failures from compile-failures and retry the former.

S4 — MEDIUM — _alarmsReferencing read without synchronization

ScriptedAlarmEngine._alarmsReferencing is a plain Dictionary (ScriptedAlarmEngine.cs:120-121) mutated inside gate-held LoadAsync (lines 201, 237-242) but read by OnUpstreamChange (line 446) from upstream callback threads with no gate. LoadAsync disposes the old subscriptions first, but IDisposable.Dispose on a subscription does not guarantee an already-executing callback has finished — a callback in flight during reload can race Clear()/Add() on a non-thread-safe dictionary. The class comment (lines 42-50) carefully justifies ConcurrentDictionary for _alarms for exactly this class of reader; _alarmsReferencing was missed. Recommendation: make it a ConcurrentDictionary or swap-in an immutable snapshot (Dictionary rebuilt per load, published via Volatile.Write to a field).

S5 — MEDIUM — Timed-shelve expiry via the predicate path swallows the Unshelved emission

Part9StateMachine.ApplyPredicate silently expires timed shelving (Part9StateMachine.cs:47-49 via MaybeExpireShelving, lines 318-322) and returns EmissionKind.None (or Activated/Cleared for the value transition) — the Unshelved emission only ever comes from ApplyShelvingCheck on the 5 s timer. If a predicate re-evaluation lands between expiry time and the next timer tick, the state record is persisted as unshelved but no Unshelved event is ever published — OPC UA clients and the /alerts page tracking ShelvingState from events never learn the shelve ended (the next Activated/Cleared arrives un-suppressed, which is at least confusing for an operator who believes the alarm is still shelved). Recommendation: have ApplyPredicate return a compound result (or the engine detect Shelving kind change between seed and result) so the expiry emits Unshelved before the value-transition emission. Add the missing state-machine test (Part9StateMachineTests.cs does not cover this path).

S6 — MEDIUM — SqliteStoreAndForwardSink.Dispose races an in-flight drain

Dispose (SqliteStoreAndForwardSink.cs:679-686) sets _disposed, disposes the timer, then immediately disposes _drainGate and the writer. Timer.Dispose() (no WaitHandle overload used) does not wait for a running callback, so an in-flight DrainOnceAsync can still be holding the gate and mid-WriteBatchAsync: the finally { _drainGate.Release(); } (line 467) then throws ObjectDisposedException (caught by the timer callback's catch and logged as a drain fault), and the writer can be disposed under an active call. Not a crash — but shutdown noise and an undefined writer contract. Recommendation: mirror the alarm engine's dispose-drain: _drainGate.Wait() (or an in-flight task handle) before disposing the gate/writer, or use Timer.Dispose(WaitHandle).

S7 — MEDIUM — Accepted sandbox limits: CPU/memory unbounded, timed-out scripts leak a thread

Documented and deliberate: the sandbox cannot bound allocation or CPU (ScriptSandbox.cs:30-35), and a timed-out CPU-bound script leaves its thread-pool thread spinning forever (TimedScriptEvaluator.cs:17-26). One orphan per bad-script evaluation attempt — under change-triggered evaluation a hot looping script orphans a thread per upstream change until the pool starves, since the engine keeps re-evaluating (alarm engine holds prior state on timeout, ScriptedAlarmEngine.cs:535-538, but does not quarantine the predicate). Recommendation: add a per-script circuit breaker (N consecutive timeouts → suspend evaluation + surface to Admin UI) as a cheap interim before any out-of-process runner. The three-layer compile-time sandbox itself (allow-list refs → wrapper-injection guard → semantic deny-list with the full TypeSyntax pass) is notably strong — including the type-forwarding rationale, var-inference coverage via GetTypeInfo, and the Unsafe/Activator/ThreadPool type-granular entries.

S8 — LOW — ScriptedAlarmEngine.Dispose blocks sync-over-async

Task.WhenAll(toAwait).GetAwaiter().GetResult() (ScriptedAlarmEngine.cs:761) — fine under the actor host (no sync context) but a deadlock trap if ever disposed from a context-bound thread. Consider IAsyncDisposable.

S9 — LOW — At-least-once delivery duplicates on crash window

A crash between a successful WriteBatchAsync and the outcome transaction commit (SqliteStoreAndForwardSink.cs:372-441) re-delivers the batch. Correct choice for alarm history (dupes over loss) — worth one line in docs/Historian.md so the gateway side knows to tolerate replays.

S10 — LOW — VirtualTagSource.SubscribeAsync initial-value window can miss one update

Emitting the seed read before registering the observer (VirtualTagSource.cs:63-75) guarantees ordering but means a change landing between the Read and the Subscribe is missed until the next change — the inverse trade-off of the race the comment describes. Register first, then emit the seed, and let the (idempotent, newer-wins) consumer handle a possible out-of-order pair — or document that seed-then-subscribe can serve a stale value on slow-changing tags.

S11 — LOW — Capacity eviction drops the oldest accepted alarm events

Documented behavior with an EvictedCount operator counter (SqliteStoreAndForwardSink.cs:602-636). Note only: for a compliance-oriented alarm trail, dropping newest (or refusing enqueue) is sometimes preferred over silently losing the oldest; the current choice is defensible but should be a deliberate, documented policy in docs/AlarmTracking.md.

2. Performance

P1 — HIGH — ScriptSandbox.Build rebuilds the full BCL reference set on every compile

Every ScriptEvaluator.Compile calls ScriptSandbox.Build(typeof(TContext)) (ScriptEvaluator.cs:74), which calls MetadataReference.CreateFromFile for every System.*/netstandard DLL in the trusted-platform-assemblies list (ScriptSandbox.cs:83-87, 100-130) — on the order of 100+ files, each a fresh PortableExecutableReference whose AssemblyMetadata Roslyn must load and parse per compilation. CompiledScriptCache de-dupes per source, but a publish with 200 distinct scripts pays this ~200 times, and the Admin UI ScriptAnalysis endpoints (which reuse this compiler context per keystroke-ish cadence) pay it continuously. The references are immutable per contextType for the process lifetime. Recommendation: cache SandboxConfig (or at least the MetadataReference list) in a static ConcurrentDictionary<Assembly, …> keyed by contextType.Assembly. This is likely a 10-100× compile-latency win and directly reduces publish time and script-editor diagnostic latency.

P2 — MEDIUM — Task.Run + WaitAsync per evaluation on the hot path

TimedScriptEvaluator.RunAsync (TimedScriptEvaluator.cs:78-81) pushes every predicate/tag evaluation to the pool and registers a timeout. For the alarm engine this happens once per referencing alarm per upstream change. The hop is required for the timeout to work at all (see U2 for what happens without it), but the common case (sub-millisecond script) pays scheduling + WaitAsync machinery every time. Recommendation: acceptable at current scale; if profiling ever shows it, an inline-run-with-watchdog design (run synchronously, watchdog flags overruns for quarantine rather than aborting) trades hard timeout for zero hop.

P3 — MEDIUM — VT engine allocates per evaluation what the alarm engine pools

VirtualTagEngine.EvaluateInternalAsync builds a fresh read-cache Dictionary and a fresh VirtualTagContext per evaluation (VirtualTagEngine.cs:298, 313-317); the alarm engine got the AlarmScratch reuse optimization (ScriptedAlarmEngine.cs:52-63, 798-835) for exactly this pattern. Inconsistent — moot while the VT engine is dormant (U1), but whichever way U1 resolves, the two should match.

P4 — LOW — Single global eval gate per engine

All evaluations serialize on one SemaphoreSlim (VirtualTagEngine.cs:44; ScriptedAlarmEngine.cs:124). Correct and simple; it caps throughput at single-threaded script execution per engine. Fine for operator-authored low-thousands scale (the compile cache doc states the same bound); a per-alarm or sharded gate is the escape hatch if scale demands it. Not worth changing now.

P5 — LOW — SQLite per-call overheads

EnqueueAsync opens a (pooled) connection and re-runs two PRAGMAs per event (SqliteStoreAndForwardSink.cs:244-246); GetStatus runs a synchronous COUNT(*) per call (line 479-484). Both fine at alarm-event rates; the capacity fast path (cached counter, probe only near the wall — lines 271-290) already removed the real hot-path cost, and the drain-order index (IX_Queue_Drain) is right.

P6 — LOW — DependencyGraph is well optimized

Cached topological rank invalidated on mutation, shared empty set for misses, iterative Tarjan/Kahn (DependencyGraph.cs:34-44, 141-149). The per-cascade List/HashSet/Stack allocations in TransitiveDependentsInOrder are minor. No action.

3. Conventions

C1 — MEDIUM — ITagUpstreamSource is defined twice

Core.VirtualTags/ITagUpstreamSource.cs and a second, byte-for-byte-shaped interface at the bottom of ScriptedAlarmEngine.cs:860-872 ("intentionally identical shape so Stream G can compose them behind one driver bridge"). Two distinct .NET types means DependencyMuxTagUpstreamSource and any composing bridge must implement/adapt each separately, and the "identical shape" invariant is enforced only by comment. Recommendation: hoist a single ITagUpstreamSource into Core.Scripting.Abstractions (both projects already reference it) and retire the duplicate.

C2 — MEDIUM — Core.Scripting.Abstractions declares types across three namespaces

ScriptContext/ScriptGlobals/PassthroughScript claim namespace Core.Scripting, VirtualTagContext claims Core.VirtualTags, AlarmPredicateContext claims Core.ScriptedAlarms — all physically in the Abstractions assembly. The reason is sound (the sandbox pins contextType.Assembly, so concrete contexts must live in the Roslyn-free assembly to keep Roslyn out of the script compilation closure — ScriptSandbox.cs:60-70), but it is documented only in a PassthroughScript remark and violates the repo's namespace-follows-assembly norm; find-by-namespace navigation actively misleads here. Recommendation: add a project-level README or AssemblyInfo doc comment stating the rule ("this assembly is the sandbox-pinned closure; types keep their consumer namespaces deliberately"), or bite the bullet and align namespaces with a TypeForwardedTo migration.

C3 — LOW — Trailing type definitions in engine files

ScriptedAlarmEvent and the duplicate ITagUpstreamSource live at the bottom of ScriptedAlarmEngine.cs (843-872); CompilationErrorException and ScriptAssemblyLoadContext at the bottom of ScriptEvaluator.cs (391-432); DependencyCycleException in DependencyGraph.cs. Mostly reasonable co-location, but ScriptedAlarmEvent is a public cross-project contract and deserves its own file.

C4 — LOW — Repeated raw OPC UA status-code literals

0x80340000u /* BadNodeIdUnknown */, 0x80020000u /* BadInternalError */, 0x80320000u /* BadWaitingForInitialData */, 0x80000000u severity mask are hand-rolled with comments in at least five files (VirtualTagEngine.cs:215-218, 307, 328; VirtualTagContext.cs; AlarmPredicateContext.cs; ScriptedAlarmEngine.cs:584-587). A small KnownStatusCodes static in Core.Abstractions would end the copy-paste drift risk (these layers deliberately avoid the OPC Foundation package, so the constants can't come from there).

C5 — LOW — Core.AlarmHistorian mixes contract and implementation

IAlarmHistorianSink/IAlarmHistorianWriter/status types share the project with SqliteStoreAndForwardSink, while scripting got a dedicated Abstractions assembly. Tolerable at three files, but the Runtime adapter and the gateway driver both reference the whole project (dragging the Microsoft.Data.Sqlite dependency) to get the interfaces.

C6 — LOW — Plan-era naming residue

XML docs throughout reference "Phase 7 plan Stream A.4/B/C/E/F/G" (TimedScriptEvaluator.cs:5, 39; ScriptSandbox.cs; IAlarmStateStore.cs:13; many more). The repo already purged the meaningless Phase7 type prefixes (master 40e8a23e); the doc references now point at plans that have been superseded by docs/ScriptedAlarms.md/docs/ScriptEditor.md. Sweep them to current doc anchors during the next doc pass.

C7 — LOW — Doc-drift: ApplyPredicate claims branch handling that doesn't exist

Part9StateMachine.ApplyPredicate's summary promises "branch-stack increment when a new active arrives while prior active is still un-acked" (Part9StateMachine.cs:31-34), but no branch/previous-instance logic exists anywhere in the file — re-activation simply resets Acked/Confirmed. See U5; at minimum fix the comment.

4. Underdeveloped Areas

U1 — HIGH — The entire Core.VirtualTags engine stack is dormant in production

VirtualTagEngine (568 lines), TimerTriggerScheduler (162), VirtualTagSource (106), and DependencyGraph (324) have no production instantiation — repo-wide grep finds new VirtualTagEngine only in tests/Core/ZB.MOM.WW.OtOpcUa.Core.VirtualTags.Tests. The live virtual-tag runtime is Runtime/VirtualTags/VirtualTagHostActor + DependencyMuxActor + Host/Engines/RoslynVirtualTagEvaluator (single-tag adapter, IVirtualTagEvaluator seam), which re-implements read-cache building, cross-tag-write handling (drops them — the actor mux owns fan-out), timeout, and compile caching independently. ~1,160 lines of engine + ~1,260 lines of tests are being maintained, reviewed, and (per S1/S3) carrying latent bugs for a component nothing runs. The RoslynVirtualTagEvaluator doc comment even acknowledges the split ("Cycle detection + cascade ordering live in VirtualTagEngine; this adapter stays single-tag scoped"). Recommendation: decide explicitly. Either (a) retire VirtualTagEngine/TimerTriggerScheduler/VirtualTagSource (keep DependencyGraph only if the publish-time cycle check uses it — verify) and let the actor pipeline be the one implementation, or (b) document the engine as the sanctioned embedded/non-Akka path and fix S1-S3. The current state — two divergent implementations where the tested one is dormant and the live one bypasses the safety wrappers (U2/U3) — is the worst of both.

U2 — CRITICAL (cross-cutting; defect sits in Host, root cause is a dormant in-scope component) — Production script timeout is ineffective: TimedScriptEvaluator is bypassed on the live path

RoslynVirtualTagEvaluator.Evaluate runs evaluator.RunAsync(context, cts.Token).GetAwaiter().GetResult() with a 2 s CancellationTokenSource (Host/Engines/RoslynVirtualTagEvaluator.cs:102-106). But ScriptEvaluator.RunAsync executes the compiled delegate synchronously on the calling thread and only checks the token before invoking it (ScriptEvaluator.cs:178-190 — "TimedScriptEvaluator wraps this in Task.Run so a long-running script still honours its wall-clock budget"). The token can therefore never interrupt a running script: the catch (OperationCanceledException) timeout branch in the Host adapter is dead code, and a CPU-bound or infinite-loop virtual-tag script hangs the owning VirtualTagActor's message loop indefinitely — no timeout, no log, no recovery. The in-scope TimedScriptEvaluator exists precisely to prevent this and is used by the (live) alarm engine and the (dormant) VT engine — but not by the one component that evaluates operator-authored virtual-tag scripts in production. Recommendation: wrap the adapter's evaluator in TimedScriptEvaluator (accepting the documented orphan-thread trade-off), or at minimum Task.Run + WaitAsync inside Evaluate. Add a regression test with a while(true) script. This should be fixed before any other finding in this report.

U3 — HIGH (same file) — Live path also bypasses CompiledScriptCache: unbounded ALC accretion across republishes

RoslynVirtualTagEvaluator caches compiled evaluators in a raw ConcurrentDictionary<string, ScriptEvaluator<…>> keyed by source (RoslynVirtualTagEvaluator.cs:25-26, 69), never evicted until process dispose. Every edited-script republish adds a new entry while the old source's evaluator — and its collectible AssemblyLoadContext — stays rooted forever. This is exactly the "per-publish recompile accretion" leak that CompiledScriptCache.Clear() was built to fix (its own doc, CompiledScriptCache.cs:34-40, and the engine comments at ScriptedAlarmEngine.cs:66-77 describe the production leak from skipping it). Additionally, GetOrAdd with a value factory can compile twice under a race and silently leak the losing ALC (the cache solved this with Lazy + ExecutionAndPublication). Recommendation: switch the adapter to CompiledScriptCache<VirtualTagContext, object?> and clear it on config apply (the host actor already sees apply boundaries).

U4 — MEDIUM — IHistoryWriter is a permanently-Null surface

VirtualTagDefinition.Historize routes to IHistoryWriter.Record, but DI binds only NullHistoryWriter (Runtime/ServiceCollectionExtensions.cs:57-58, 236-237 — "durable AVEVA sink is infra-gated; a deployment binding a real IHistoryWriter overrides"). No real implementation exists in the tree; the operator-visible Historize checkbox on virtual tags is a silent no-op (the separate ContinuousHistorization path taps the mux directly and is itself gated + ref-set-empty per CLAUDE.md Known Limitation 2). Either wire IHistoryWriter to the gateway value-writer or hide/annotate the checkbox until it does something.

U5 — MEDIUM — Part 9 surface gaps are real but undeclared in one place

No condition branches / previous-instances (re-activation while un-acked just resets Acked — and the xmldoc overpromises, C7); Severity is static per definition ("not currently computed by the predicate", ScriptedAlarmDefinition.cs); AlarmKind affects only node typing; MessageTemplate has no brace escaping (documented as "reach for a feature request", MessageTemplate.cs:16-19); Retain is carried on the definition but not consulted anywhere in this scope. Each is a defensible v1 cut; collect them in docs/ScriptedAlarms.md as an explicit conformance statement so OPC UA client integrators know what subset to expect.

U6 — LOW — No TODO/HACK/stub markers

Grep found zero TODO|HACK|FIXME|NotImplemented across all five projects — unfinished edges are instead documented in XML remarks (a healthier pattern, though it makes gaps greppable only by reading).

U7 — LOW — Test coverage: broad and behavior-focused, with specific holes

~6,690 test lines against ~5,760 source lines, all four impl projects have dedicated suites: SqliteStoreAndForwardSinkTests (888 — outcomes, poison cap, corrupt payloads, capacity fast path), ScriptedAlarmEngineTests (1,270), ScriptSandboxTests (614 — the escape catalog), CompiledScriptCacheTests (409 — including ALC-reclaim via WeakReference), VirtualTagEngineTests (800), state machine, extractor, graph, loggers, sources. Holes worth filling: (a) S5's expiry-during-predicate emission swallow — no Part9StateMachineTests case; (b) no concurrency test for load-vs-cascade (would expose S1/S4); (c) no sink Dispose-during-drain test (S6); (d) ForbiddenTypeAnalyzer has no dedicated unit suite (exercised only through end-to-end Compile in sandbox tests — fine functionally, but analyzer-pass regressions produce opaque failures); (e) nothing tests the production RoslynVirtualTagEvaluator timeout (which would have caught U2 — it lives in Host, whose test project should own it).

---

Maturity Ratings

Scale: 1 (prototype) → 5 (production-hardened). Rating reflects the scoped code as it stands, including how it is (or isn't) consumed.

Dimension	Rating	Justification
Stability	3.5	Alarm engine + sink show genuinely careful engineering (gate discipline, emit-outside-gate, dispose drains, persist-before-memory, poison-row caps); dragged down by fail-stop reloads, unbounded change fan-out, the VT engine's un-gated Load, and the shelve-expiry emission swallow.
Performance	3	Right structures where it counts (compile cache, alarm scratch reuse, graph rank cache, SQLite capacity fast path), but every compile rebuilds the full BCL reference set (P1), every evaluation pays a pool hop, and one global gate serializes each engine.
Conventions	4	Strongly consistent with repo norms (Serilog, Null-object defaults, xUnit/Shouldly, exemplary "why"-comments); duplicated ITagUpstreamSource, the three-namespace Abstractions assembly, and literal status codes are the debt.
Underdeveloped areas	2	A complete, tested virtual-tag engine nobody runs, while the live path bypasses the scope's two key defenses (timeout → hung actor risk, ALC cache → leak); Historize is a Null no-op; Part 9 gaps undeclared. The split between "what is built and tested" and "what actually runs" is the defining problem of this subsystem.

---

Priority Recommendations (ordered)

1. U2 — Wrap the production virtual-tag evaluation in TimedScriptEvaluator (or equivalent). A single bad script can currently hang a VirtualTagActor forever with no timeout.
2. U3 — Use CompiledScriptCache (with apply-boundary Clear) in RoslynVirtualTagEvaluator to stop cross-publish ALC accretion.
3. P1 — Statically cache the ScriptSandbox reference set; large publish-time and script-editor latency win for one small change.
4. U1 — Decide the fate of the dormant VirtualTagEngine stack (retire preferred); until then S1/S3 fixes are latent-bug maintenance on dead code.
5. S2 — Add dirty-set coalescing to ScriptedAlarmEngine upstream fan-out before a fast tag meets a popular alarm in production.
6. S5 + C7 — Fix the timed-shelve expiry emission and the state-machine doc drift together, with a state-machine test.
7. S3/S4/S6 — Schedule the reload-swap, _alarmsReferencing synchronization, and sink dispose-drain as one hardening pass.