# Design + Implementation Plan — R2-03: VT script-failure quality + cache-clear hardening - **Source report:** `archreview/02-scripting-alarms.md` (2026-07-12 re-review) - **Review commit:** `f6eaa267` · **Plan verified against tree at:** `f6eaa267` (master, clean) - **Scope:** OVERALL prioritized action item **#3** — "Publish Bad quality on VT script failure/timeout (match the dormant engine's BadInternalError); optionally gate the apply-boundary cache clear on changed scripts" — covering findings **02/S13** (Medium — failure/timeout never degrades node quality), **02/S12** (Medium — apply-boundary `ClearCompiledScripts` races in-flight child evaluations), and **02/P7** (Low — unconditional per-apply cache flush forces full VT recompilation on every deploy). - **Touches:** `Runtime/VirtualTags/VirtualTagActor.cs`, `Runtime/VirtualTags/VirtualTagHostActor.cs`, `Host/Engines/RoslynVirtualTagEvaluator.cs`, `Runtime.Tests/VirtualTags/*`, `Host.IntegrationTests/RoslynVirtualTagEvaluatorTests.cs`. **No Core.Scripting / Commons-contract / OPC UA-sink changes** except one additive field on `VirtualTagActor.EvaluationResult`. ## Verification summary Every anchor in the three findings was opened at the cited file:line against the working tree at `f6eaa267`. **All three findings CONFIRMED. One minor anchor drift** (noted below); everything else is line-exact. - **S13** — Confirmed. `VirtualTagActor.OnDependencyChanged` returns without any publish on `!result.Success` (`Runtime/VirtualTags/VirtualTagActor.cs:123-128`) and on evaluator-threw (`:115-121`) — Warning `ScriptLogEntry` + `fail` metric only. The host bridge hard-codes `OpcUaQuality.Good` into every `AttributeValueUpdate` (`Runtime/VirtualTags/VirtualTagHostActor.cs:179-180`) and hard-codes `0u /* StatusCodes.Good */` into the historize snapshot (`:187-188`). `EvaluationResult` (`VirtualTagActor.cs:23`) carries no quality at all, so the child *cannot* express degradation today. Confirmed downstream that Bad quality *is* honored end to end once published: `OpcUaPublishActor.HandleAttributeUpdate` → `IOpcUaAddressSpaceSink.WriteValue` → `OtOpcUaNodeManager.WriteValue` (`OpcUaServer/OtOpcUaNodeManager.cs:261-281`) sets `variable.StatusCode = StatusFromQuality(quality)` + `ClearChangeMasks`, where `StatusFromQuality` (`:2418-2423`) maps `Good→StatusCodes.Good`, `Uncertain→StatusCodes.Uncertain`, `_→StatusCodes.Bad`. So the *only* missing plumbing is child → bridge. - **S13 anchor drift (minor):** the report cites the dormant engine's failure mapping at `VirtualTagEngine.cs:307,328`. At `f6eaa267`, line **307** is the *cold-start* guard (`BadWaitingForInitialData` when inputs aren't ready yet — `AreInputsReady`); the actual failure→`BadInternalError` (`0x80020000u`) mappings are at **327-329** (uncoercible result), **334** (`ScriptTimeoutException`), and **343** (script threw). The substance of the finding is unaffected — and line 305-311 turns out to be load-bearing for this plan's design (the inputs-ready gate, below). - **S12** — Confirmed. `VirtualTagHostActor.OnApply` calls `ClearCompiledScripts()` first thing (`VirtualTagHostActor.cs:92`), on the host's dispatcher thread, with nothing gating concurrent child evaluations. The race window is inside one `RoslynVirtualTagEvaluator.Evaluate` call: the evaluator is fetched at `RoslynVirtualTagEvaluator.cs:69` (`_cache.GetOrCompile`), the host's `Clear()` value-scoped-removes **and disposes** it (`CompiledScriptCache.cs:113-116` → `ScriptEvaluator.Dispose` → `_alc.Unload()`), and the child's subsequent run (`RoslynVirtualTagEvaluator.cs:108-109`, via `TimedScriptEvaluator.RunAsync`'s `Task.Run`) trips the disposed guard (`ScriptEvaluator.cs:180`) → `ObjectDisposedException` surfaces through `GetAwaiter().GetResult()` into the generic `catch (Exception)` (`RoslynVirtualTagEvaluator.cs:116-120`) → `Failure("script threw: Cannot access a disposed object…")` → dropped update (and, per S13, no quality degradation either). The contrast with `ScriptedAlarmEngine` holds: its per-generation `_compileCache.Clear()` (`ScriptedAlarmEngine.cs:209`) runs **under `_evalGate` with upstream subscriptions already torn down**, so no alarm evaluation can hold a disposed evaluator. The secondary wrinkle also confirmed: the clear runs *before* the removed/changed children are stopped (`VirtualTagHostActor.cs:98-118`) and `Context.Stop` is asynchronous, so a doomed child can recompile its stale expression into the fresh cache for one generation (bounded — not a leak). Note a mid-*execution* clear is safe: `AssemblyLoadContext.Unload()` is cooperative (the executing delegate keeps the ALC alive); only the guard at `ScriptEvaluator.cs:180` — i.e. dispose landing between fetch and run-start — produces the dropped update. The `Task.Run` hop inside `TimedScriptEvaluator` widens exactly that window. - **P7** — Confirmed. The clear at `VirtualTagHostActor.cs:92` is unconditional per `ApplyVirtualTags`, including a byte-identical redeploy whose plans all diff equal (the no-churn path proven by `ApplyVirtualTags_does_not_respawn_child_when_plan_unchanged`). The host already keeps last generation's plans in `_planByVtag` (`:50`, rebuilt at `:160-164`) — the comparison input for a conditional clear is already sitting in actor state. `EquipmentVirtualTagPlan` has element-wise value equality including `Expression` (`OpcUaServer/AddressSpaceComposer.cs:179-188`). The existing wiring guard `VirtualTagHostActorTests.ApplyVirtualTags_clears_the_evaluator_compile_cache_each_generation` asserts the clear fires on an **identical** second apply — that test encodes the P7 behavior and must be rewritten, not just kept green (called out in the task breakdown). - **Existing-test impact identified:** `VirtualTagActorTests.Evaluator_failure_publishes_ScriptLogEntry_warning` ends with `parent.ExpectNoMsg(...)` — under the S13 fix a failure now *does* send the parent a Bad-quality `EvaluationResult`, so that assertion inverts (task R2-03-T3 updates it deliberately). - **Design constraint verified:** the value fan-out's quality vocabulary is the coarse 3-value `OpcUaQuality { Good, Uncertain, Bad }` (`Commons/OpcUa/IOpcUaAddressSpaceSink.cs:99`) — there is no sub-code channel to the node, and `AttributeValueUpdate` (`Runtime/OpcUa/OpcUaPublishActor.cs:42`) already carries it. `NullVirtualTagEvaluator` returns `NoChange` (Success=true), so the dev/Mac path can never emit Bad. The passthrough fast-path returns `Ok(null)` for a missing dep (never Failure), so it is unaffected by all three fixes. --- ## Priority ordering Per the source report's Priority Recommendations (#1 = S13, #2 = S12) and OVERALL action item #3: 1. **S13** (Medium, report's top item) — failure/timeout → node quality. The stale-Good repro test lands first and must fail on current code. 2. **S12** (Medium) — close the clear-vs-in-flight-evaluate race (retry-once in the adapter). 3. **P7** (Low) — gate the apply-boundary clear on an actually-changed script set. All three are one PR (`fix/archreview-r2-03-vt-failure-quality`): S13 and S12 touch disjoint files; P7 touches the same `OnApply` region S13's bridge work sits next to. --- ## 1. S13 — MEDIUM — Script failure/timeout never degrades the OPC UA node's quality: stale value stays Good forever **Restatement:** When a VT script evaluation fails (compile error, runtime throw, sandbox violation, or the U2-era timeout), `VirtualTagActor` logs + meters and returns without publishing anything, and the host bridge only ever publishes `OpcUaQuality.Good`. A previously-working script that starts failing leaves its node serving the last Good value with its old timestamp indefinitely — OPC UA clients, the continuous-historization recorder, and ScadaBridge cannot distinguish "healthy but unchanged" from "script broken for a week". The dormant `VirtualTagEngine` maps this correctly to `BadInternalError`; the live actor path never implemented it. **Verification:** Confirmed (see summary). Bad quality is fully plumbed from `AttributeValueUpdate` through the sink to the SDK node — the gap is exactly `VirtualTagActor` (no quality concept) and the `VirtualTagHostActor` bridge (hard-coded Good at `:179-180`, hard-coded `0u` historize status at `:187-188`). **Root cause:** `EvaluationResult` was designed as a success-only message ("emit to the parent whenever the value actually changes"); failures were treated purely as an observability event (script-log + metric). The quality dimension was left to the materialiser's initial `BadWaitingForInitialData` and never revisited when the U2 remediation made timeouts a first-class, expected failure mode. **Proposed design:** *(a) StatusCode policy — coarse `OpcUaQuality.Bad` for both failure and timeout.* The node will read `StatusCodes.Bad` (0x80000000) via the existing `StatusFromQuality` mapping. This "matches the dormant engine's BadInternalError" at the level that matters — the severity bit that flips client/consumer handling — and the failure *kind* (timeout vs throw vs compile) is already delivered verbatim to operators as the script-log `Reason` string on the `script-logs` topic. Alternatives rejected: - *Widen `OpcUaQuality` with `BadInternalError`/`BadTimeout` members* — additive on the enum, but it ripples a Commons contract consumed by every sink implementation and driver publish path, for a sub-code no consumer acts on. Contained follow-up if a real client requirement appears. - *Carry a raw `uint` status through `AttributeValueUpdate`/`IOpcUaAddressSpaceSink.WriteValue`* — the precise-parity option, but it changes an interface with four implementations and re-runs the F10b/DeferredAddressSpaceSink forwarding trap for zero operator value. Rejected. - Exception: the **historize snapshot** takes raw `uint` status already (`DataValueSnapshot`), so there the bridge records exact dormant-engine parity: `0x80020000u /* BadInternalError */` on Bad results (the dormant engine historizes its Bad snapshots unconditionally at `VirtualTagEngine.cs:346-348`; we mirror that). *(b) Publish cadence — once per transition, not per failure.* Every publish is a sink write + `ClearChangeMasks` → client notification on the pinned OPC UA dispatcher. A failing script on a hot dependency would otherwise emit one identical Bad write per dependency change (each also paying the P2 pool hop + 2 s worst-case actor block per S7). The actor tracks `_lastPublishedBad`; it publishes Bad on the Good→Bad (or initial→Bad) transition only, and on recovery it **force-publishes the next success even if the value equals the pre-failure value** — otherwise the value-dedup at `VirtualTagActor.cs:138` would leave the node Bad forever after a same-value recovery. (Script-log Warning entries stay per-failure, unchanged — that's the existing operator diagnostic and its volume is pre-existing.) *(c) Cold-start suppression — an inputs-ready gate, mirroring the dormant engine.* The live path has no `AreInputsReady` equivalent: a multi-dep script fails on the first dependency arrival (the missing dep's `GetTag(...).Value` is null → cast throws) until all deps have landed. Publishing Bad on those would make every fresh/respawned multi-dep VT flash `BadWaitingForInitialData → Bad → Good` on every deploy — operator-visible churn for healthy scripts. Worse, gating on "had a prior success in this child" is **insufficient for S13's headline scenario**: an *edited-now-broken* script respawns its child (plan change ⇒ stop+respawn at `VirtualTagHostActor.cs:109-118`), so the fresh child has no prior success and would never degrade — reintroducing stale-Good exactly where it matters most. The correct gate is the dormant engine's: **suppress Bad publishes until every `dependencyRefs` entry has been received at least once** (`_dependencyRefs.All(_dependencies.ContainsKey)`); after that, any failure degrades. A never-arriving dependency keeps the node at the materialiser's `BadWaitingForInitialData` (already Bad — acceptable, and per the report "acceptable" for never-succeeded scripts). *(d) Message/bridge shape — additive `Quality` on `EvaluationResult`.* Add `OpcUaQuality Quality = OpcUaQuality.Good` as a trailing optional parameter on the `EvaluationResult` positional record: every existing construction and destructuring site compiles unchanged, and the message is a local `Context.Parent.Tell` (no wire serialization concern). The bridge replaces the hard-coded `OpcUaQuality.Good` with `result.Quality`, and maps the historize status (`Good→0u`, else `0x80020000u`). **The dependency mux is untouched** — quality flows child→parent→publish actor; the mux only fans values *into* children (`DependencyValueChanged` is unchanged), so no mux registration, routing, or message contract moves. On failure the Bad `EvaluationResult` carries the **last-known value** (`_lastValue` if `_hasLastValue`, else `null`) with the triggering `msg.TimestampUtc` — clients see the stale value explicitly flagged Bad (the report's "publish the existing value with Bad quality"), rather than the dormant engine's null-value choice; either is defensible, this one preserves more information for trend displays. **Implementation steps:** 1. `Runtime/VirtualTags/VirtualTagActor.cs`: - `using ZB.MOM.WW.OtOpcUa.Commons.OpcUa;` - `public sealed record EvaluationResult(string VirtualTagId, object? Value, DateTime TimestampUtc, CorrelationId Correlation, OpcUaQuality Quality = OpcUaQuality.Good);` - Add fields `private bool _lastPublishedBad;` and a helper `private bool AllDependenciesArrived() => _dependencyRefs.All(_dependencies.ContainsKey);` (refs list is small; called once per failure — no caching needed). - Extract the failure tail shared by the evaluator-threw catch (`:115-121`) and the `!Success` branch (`:123-128`) into `private void OnEvaluationFailed(string reason, string level, DateTime timestampUtc)`: keep the existing metric + `PublishLog`, then `if (_lastPublishedBad || !AllDependenciesArrived()) return;` else `_lastPublishedBad = true; Context.Parent.Tell(new EvaluationResult(_virtualTagId, _hasLastValue ? _lastValue : null, timestampUtc, CorrelationId.NewId(), OpcUaQuality.Bad));` - Success path: change the dedup guard at `:138` to `if (!_lastPublishedBad && _hasLastValue && Equals(_lastValue, result.Value))` and set `_lastPublishedBad = false;` alongside `_hasLastValue = true;` before the Good publish. The `NoChange` short-circuit (`:132-136`) stays *above* the dedup and does **not** touch `_lastPublishedBad` (NoChange = "no fresh value", not a recovery). - Update the class summary: failures now degrade quality once per transition. 2. `Runtime/VirtualTags/VirtualTagHostActor.cs` (`OnResult`, `:170-190`): - `_publishActor.Tell(new OpcUaPublishActor.AttributeValueUpdate(nodeId, result.Value, result.Quality, result.TimestampUtc));` - Historize: `var status = result.Quality == OpcUaQuality.Good ? 0u : 0x80020000u /* BadInternalError — dormant-engine parity */;` and pass `status` into the `DataValueSnapshot`. - Update the `:179-180` comment (no longer hard-coded Good). 3. Update `VirtualTagActorTests.Evaluator_failure_publishes_ScriptLogEntry_warning`: the trailing `parent.ExpectNoMsg` becomes `parent.ExpectMsg()` with `Quality == Bad` (the test constructs the actor with empty `dependencyRefs`, so the inputs-ready gate is vacuously satisfied — `All` over an empty list is true). **Tests:** (all deterministic; probes only) - **Repro first (must FAIL at `f6eaa267`):** `Runtime.Tests` `VirtualTagHostActorTests.` `Failing_script_after_success_degrades_node_quality_to_Bad` — host + real child + a scripted fail-after-success `IVirtualTagEvaluator` stub; drive the child (captured via the mux probe's `RegisterInterest` sender) with two `DependencyValueChanged`s; assert the publish probe receives update 1 `(Good, 42)` then update 2 with `Quality == OpcUaQuality.Bad` carrying the last value. Uses only current types (`AttributeValueUpdate` already has `Quality`), so it compiles today and times out on the second `ExpectMsg` — the stale-Good repro. - `VirtualTagActorTests`: failure-after-success emits Bad `EvaluationResult` with last value; second consecutive failure emits **nothing** (once-per-transition); recovery to the *same* value as before the failure **does** publish (dedup bypass); evaluator-*threw* path (not just `Failure`) also degrades; multi-dep cold-start failure with a missing dep publishes **no** Bad (inputs-ready gate); after all deps arrive, failure publishes Bad. - `VirtualTagHostActorTests`: a Bad-quality `EvaluationResult` bridges to an `AttributeValueUpdate` with `Quality == Bad`; a historized plan's Bad result records status `0x80020000u` (BadInternalError). - Wiring note (theme #1, "unit-green ≠ wired"): the repro test above IS the wiring assertion — it runs the real host + real child + parent-Tell bridge, not a hand-delivered message. The remaining live wire (publish actor → sink → SDK node) is pre-existing, already exercised by `OpcUaPublishActor` tests, and covered by the live-`/run` gate below. **Effort:** S. **Risk:** Low-Medium. The behavioral deltas are (1) parents now receive Bad results — only `VirtualTagHostActor` parents children in production, and it handles the new field natively; (2) a node that used to freeze Good now goes Bad on persistent failure — the intended, strictly-safer change; (3) one existing test's expectation inverts (deliberate, task-tracked). Blast radius = VT value path only; alarms/drivers untouched. --- ## 2. S12 — MEDIUM — Apply-boundary `ClearCompiledScripts` races in-flight child evaluations **Restatement:** The host's per-apply `Clear()` disposes cached evaluators while unchanged children keep evaluating on other dispatcher threads. A child that fetched its evaluator (`GetOrCompile`, `RoslynVirtualTagEvaluator.cs:69`) before the clear hits the disposed guard on run (`ScriptEvaluator.cs:180`) → `ObjectDisposedException` → generic catch → `Failure` → silently dropped update (self-heals only on the tag's *next* dependency change — a long staleness window for slow tags). Secondary: a stopping child can recompile its stale script into the fresh cache for one generation. **Verification:** Confirmed (see summary). The cache itself is blameless — its value-scoped `TryRemove` protects concurrent `GetOrCompile` re-adds; the gap is clear-vs-*use* of an already-handed-out evaluator. The `ScriptedAlarmEngine` mirror is inexact exactly as the report says (gate-held clear after subscription teardown, `ScriptedAlarmEngine.cs:199-209`). **Root cause:** `IScriptCacheOwner.ClearCompiledScripts` was wired at the apply boundary (U3) without a coordination primitive between the host actor's clear and child actors' fetch→run sequence — the `Task.Run` hop inside `TimedScriptEvaluator` guarantees a real window between fetch and the disposed check. **Proposed design — catch-`ObjectDisposedException`, re-fetch, retry once (in the adapter).** The recompile is idempotent (same source → fresh evaluator in the *current* cache generation), the retry is entirely local to `RoslynVirtualTagEvaluator.Evaluate`, and it converts the dropped update into a correct evaluation against the post-clear generation. If a *second* clear lands mid-retry (two applies inside one evaluation — not a realistic cadence), the second `ObjectDisposedException` falls through to the existing `Failure` path — same as today's worst case, now with an S13 Bad-quality publish instead of silence. Alternatives rejected: - *`ReaderWriterLockSlim` (children read-lock fetch→run; clear write-locks)* — blocks the **host actor** for up to a full in-flight timeout budget (2 s) during every apply, and adds lock traffic to the hot path, to prevent a race the retry heals for free. Rejected. - *Per-generation cache instances with deferred dispose of the old generation* — disposing the old generation safely still requires knowing when in-flight evaluations finish, i.e. ref-counting — the same coordination problem with more machinery. Rejected. - *Move the clear after the stop/respawn reconciliation* — doesn't close the race for **unchanged** children (the common case), and the stale-recompile wrinkle survives either way because `Context.Stop` is asynchronous. Rejected as a fix; the wrinkle stays **accepted + documented** (bounded to one generation, released at the next apply-with-changes; P7's gate also means no-op redeploys no longer create the window at all). **Implementation steps** (`Host/Engines/RoslynVirtualTagEvaluator.cs`, the compiled path `:66-121`): 1. Restructure fetch+run into a bounded loop (`for (var attempt = 0; ; attempt++)`): - fetch: `evaluator = _cache.GetOrCompile(expression);` — keep the existing `CompilationErrorException` / `ScriptSandboxViolationException` / generic catches on the fetch (unchanged behavior). A disposed **cache** here means the whole evaluator is being shut down — `GetOrCompile` throws `ObjectDisposedException`, which the new handler must NOT retry: check `_disposed` and return `VirtualTagEvalResult.Failure("evaluator disposed")`. - run: existing `TimedScriptEvaluator` + `GetAwaiter().GetResult()` block, adding **before** the generic catch: ```csharp catch (ObjectDisposedException) when (!_disposed && attempt == 0) { // S12: the apply-boundary ClearCompiledScripts disposed the evaluator between our // GetOrCompile and the run's disposed-guard. Re-fetch — GetOrCompile recompiles the same // source into the CURRENT cache generation — and retry once. A second disposal mid-retry // (two applies inside one evaluation) falls through to the generic failure path. continue; } ``` - `catch (ObjectDisposedException)` (second hit, or `_disposed`) → `Failure("evaluator disposed during evaluation")`. 2. Update the class doc: the apply-boundary clear is retry-safe for in-flight evaluations; the one-generation stale-recompile wrinkle for stopping children is the documented residual. **Tests** (`Host.IntegrationTests/RoslynVirtualTagEvaluatorTests.cs`): - `Evaluate_racing_ClearCompiledScripts_never_fails_with_disposed` — one evaluator; task A loops N (≈300) compiled (non-passthrough) evaluations of the same expression; task B loops `ClearCompiledScripts()` with `Thread.Yield()` between calls until A finishes; whole test `WaitAsync`-bounded (30 s). Assert **every** result `Success == true` (and specifically no `Reason` containing "disposed"). *Determinism note:* the pre-fix failure is probabilistic per iteration (the window is fetch → `Task.Run`-hopped disposed-guard, widened by pool queuing), but at N=300 racing clears it reproduces reliably in practice; post-fix it is deterministic-green by construction (the retry is exhaustive for a single concurrent clear). This is the report's prescribed "concurrent clear-during-evaluate test". - `Evaluate_after_ClearCompiledScripts_recompiles_and_succeeds` — deterministic sequential guard: evaluate, clear, evaluate same source again → Success both times, cache Count back to 1 (uses the existing `CompiledCacheCount` reflection helper). - Existing `Evaluate_after_dispose_returns_Failure` still green (the `_disposed` short-circuit at `:51` is untouched; the new ODE handler's `!_disposed` filter keeps shutdown semantics). **Effort:** S (the report's "10-line fix" is accurate; the loop restructure is the only care point). **Risk:** Low. Happy path unchanged (attempt 0, no exception); passthrough fast-path never enters the loop. Blast radius = one method in the Host adapter. --- ## 3. P7 — LOW — Unconditional per-apply cache flush forces full VT recompilation on every deploy **Restatement:** `OnApply` clears the compile cache on **every** `ApplyVirtualTags` (`VirtualTagHostActor.cs:92`) — including a byte-identical redeploy that stops/spawns nothing. Every VT script then recompiles on its next evaluation, each paying the still-open P1 full-BCL-reference-set rebuild (5-20 ms+ per script). Amplifies P1; also each unnecessary clear is an unnecessary S12 race window. **Verification:** Confirmed (see summary). The host already holds last generation's plans in `_planByVtag`; `EquipmentVirtualTagPlan.Expression` participates in value equality. The existing wiring guard asserts the *current* (unconditional) behavior on an identical redeploy and must be rewritten. **Root cause:** The U3 fix chose the simplest correct lifecycle ("clear per generation", mirroring the alarm engine) without consulting the plan diff the same method computes a few lines later. **Proposed design — clear only when the deployed *expression set* changed.** At the top of `OnApply`, compare the desired expression set against the previous generation's (`_planByVtag.Values.Select(p => p.Expression)`), and clear only on inequality: ```csharp var newExpressions = msg.Plans.Select(p => p.Expression).ToHashSet(StringComparer.Ordinal); var prevExpressions = _planByVtag.Values.Select(p => p.Expression).ToHashSet(StringComparer.Ordinal); if (!newExpressions.SetEquals(prevExpressions)) { (_evaluator as IScriptCacheOwner)?.ClearCompiledScripts(); } ``` Decisions inside this shape: - **Expression set, not full-plan set:** the cache is keyed by source hash only, so a rename / FolderPath move / Historize toggle (which respawns the child) does not need a recompile — comparing full plans would clear on those spuriously. Two vtags sharing one expression are handled naturally (set semantics: the expression survives while any plan still uses it). - **Any-change clears everything** (not per-source eviction): editing 1 of 200 scripts still recompiles 200. Accepted per the report ("cheap to leave as-is at current scale; fix alongside P1") — the headline P7 cost is the **no-op redeploy**, which this eliminates entirely. Rejected alternative: a `CompiledScriptCache.RetainOnly(IEnumerable sources)` API — precise, but new Core.Scripting surface + tests for a Low; note it as the follow-up shape if P1's memoization doesn't land first. - **First apply after actor (re)start:** `_planByVtag` is empty → sets differ → clear fires (harmless: the singleton evaluator's cache may hold a prior incarnation's entries; clearing converges state). - **Interaction with S12 (design question d):** independent and composing — the gate reduces clear *frequency* (no-op redeploys stop racing entirely), the retry remains the correctness backstop for genuine changes. The gate must NOT be treated as the S12 fix: a real script edit still clears while unchanged children evaluate. - **Interaction with S13:** none (quality path doesn't touch the cache). **Implementation steps:** 1. Replace `VirtualTagHostActor.cs:88-92` (comment + unconditional call) with the guarded block above + a comment explaining the expression-set gate (and that `_planByVtag` is "the previous generation" at this point — the method rebuilds it at the end). 2. Rewrite the wiring guard `ApplyVirtualTags_clears_the_evaluator_compile_cache_each_generation` → `ApplyVirtualTags_clears_the_compile_cache_only_when_the_expression_set_changes`: apply plan A (ClearCount 1) → re-apply identical plan A (ClearCount **still 1**) → apply plan A′ with an edited `Expression` (ClearCount 2) → re-apply with a vtag removed but its expression still used by another vtag (ClearCount still 2). Keeps the load-bearing "the host actually calls it" wiring assertion while pinning the new gate. **Tests:** the rewritten wiring guard above (in `Runtime.Tests/VirtualTags/VirtualTagHostActorTests.cs`, using the existing `CacheOwningStubEvaluator` spy). New-behavior red state: the *identical-redeploy* step fails on current code (ClearCount is 2, expected 1). **Effort:** S. **Risk:** Low. The only hazard is under-clearing (a stale ALC surviving a deploy whose expression set is unchanged) — by construction those entries are exactly the still-deployed sources, so nothing stale survives. Blast radius = one method + one test. --- ## Live-`/run` verification (after all tasks green) Deterministic units + wiring guards above are necessary but not sufficient — **unit-green ≠ wired** (the F10b lesson). Close with the docker-dev rig (login disabled; do it yourself, no user sign-in): 1. Rebuild **both** `central-1` and `central-2` images (`:9200` round-robins them) and bring `docker-dev/docker-compose.yml` up. 2. On `/uns` (AdminUI `http://localhost:9200`), author a VirtualTag on a rig equipment with a script that succeeds against live deps (e.g. `return (double)ctx.GetTag("").Value * 2;`), deploy (`POST :9200/api/deployments`, `X-Api-Key`), and confirm via Client.CLI: `dotnet run --project src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI -- read -u opc.tcp://localhost:4840 -n ""` → value present, StatusCode Good. 3. Edit the script to a guaranteed runtime throw (e.g. `int z = 0; return 1 / z;`), redeploy, wait one dependency change → Client.CLI `read` again → **StatusCode Bad** (0x80000000), value = last-known. This exercises the exact S13 headline path (edited-broken script ⇒ respawned child ⇒ inputs-ready gate ⇒ Bad publish) end to end through the SDK node. 4. Fix the script back, redeploy → `read` returns Good with a fresh value (recovery + dedup-bypass live). 5. P7 spot-check: redeploy the identical config; central logs show no recompile burst (and the S12 stress guard covers the race in CI). --- ## Task breakdown Branch: `fix/archreview-r2-03-vt-failure-quality` off `master@f6eaa267`. Tasks are ≤5 min each; TDD-ordered (repro/red test precedes each implementation). Test commands run from the repo root. --- ### R2-03-T1 — S13 stale-Good repro test (RED — must fail on current code) **Classification:** test-first repro (deterministic unit, host-level wiring shape) **Estimated implement time:** 5 min **Parallelizable with:** T7 (different test file); everything else depends on this landing red first. **Files:** `tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/VirtualTags/VirtualTagHostActorTests.cs` 1. Add a `FailAfterFirstSuccessEvaluator : IVirtualTagEvaluator` private stub (call 1 → `Ok(42.0)`, thereafter → `Failure("boom")`; interlocked counter). 2. Add test `Failing_script_after_success_degrades_node_quality_to_Bad`: host with publish + mux probes and the stub; apply one plan (deps `["a"]`); capture the child via `mux.ExpectMsg()` + `mux.LastSender`; tell the child `DependencyValueChanged("a", 1, ts1)` → publish probe expects `AttributeValueUpdate` with `Quality == OpcUaQuality.Good`, `Value == 42.0`; tell `DependencyValueChanged("a", 2, ts2)` → expect a second `AttributeValueUpdate` with `Quality == OpcUaQuality.Bad` and `Value == 42.0` (last-known) and `TimestampUtc == ts2`. 3. `dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests --filter "FullyQualifiedName~VirtualTagHostActorTests.Failing_script_after_success_degrades_node_quality_to_Bad"` → **expected: FAIL** (timeout on the second ExpectMsg — the stale-Good bug, verbatim). Commit: `test(runtime): red repro — VT script failure after success leaves node stale-Good (02/S13)` --- ### R2-03-T2 — EvaluationResult.Quality + bridge pass-through + historize parity **Classification:** implementation (additive message field + bridge) **Estimated implement time:** 5 min **Parallelizable with:** T7/T8 (disjoint files). Blocked by T1 (repro must be red first). **Files:** `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/VirtualTags/VirtualTagActor.cs` (record only), `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/VirtualTags/VirtualTagHostActor.cs`, `tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/VirtualTags/VirtualTagHostActorTests.cs` 1. RED: add bridge tests `Bad_quality_EvaluationResult_is_bridged_as_Bad` (tell the host a synthetic `EvaluationResult(..., Quality: OpcUaQuality.Bad)` → publish probe's `AttributeValueUpdate.Quality` is Bad) and `Historized_Bad_result_is_recorded_with_BadInternalError_status` (PlanH historize:true; Bad result → `CapturingHistoryWriter` snapshot `StatusCode == 0x80020000u`). These require the record change to compile — so within this task: add `OpcUaQuality Quality = OpcUaQuality.Good` to `EvaluationResult` (+ `using ...Commons.OpcUa;`) FIRST, then run the two tests: `dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests --filter "FullyQualifiedName~VirtualTagHostActorTests.Bad_quality_EvaluationResult_is_bridged_as_Bad|FullyQualifiedName~VirtualTagHostActorTests.Historized_Bad_result_is_recorded_with_BadInternalError_status"` → **expected: FAIL** (bridge still hard-codes Good / `0u`). 2. GREEN: `OnResult` — pass `result.Quality` into `AttributeValueUpdate`; historize status `result.Quality == OpcUaQuality.Good ? 0u : 0x80020000u /* BadInternalError — dormant-engine parity */`; fix the stale comments at `:179-188`. Re-run the filter → **PASS**. Also run the full host-actor class (`--filter "FullyQualifiedName~VirtualTagHostActorTests"`) — all pre-existing tests (Good-path bridging, historize Good `0u`) must stay green, proving the default keeps byte-parity. Commit: `feat(runtime): EvaluationResult carries OpcUaQuality; VT bridge publishes it + historizes BadInternalError on Bad (02/S13)` --- ### R2-03-T3 — VirtualTagActor failure→Bad transition (turns T1 green) **Classification:** implementation (actor behavior) **Estimated implement time:** 5 min **Parallelizable with:** T7/T8. Blocked by T2. **Files:** `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/VirtualTags/VirtualTagActor.cs`, `tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/VirtualTags/VirtualTagActorTests.cs` 1. Implement per design §1(b)/(c)/(d): `_lastPublishedBad` field, `AllDependenciesArrived()`, `OnEvaluationFailed(...)` shared by the throw-catch and `!Success` branch (metric + script-log preserved per-failure; Bad publish once per transition, gated on inputs-ready, carrying `_hasLastValue ? _lastValue : null` + `msg.TimestampUtc`); success path resets `_lastPublishedBad` and bypasses the value-dedup when recovering; class-summary xmldoc updated. 2. Update `Evaluator_failure_publishes_ScriptLogEntry_warning`: replace the trailing `parent.ExpectNoMsg(...)` with `parent.ExpectMsg().Quality.ShouldBe(OpcUaQuality.Bad)` — the deliberate behavioral inversion (empty `dependencyRefs` ⇒ inputs-ready gate vacuously true). 3. `dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests --filter "FullyQualifiedName~VirtualTagHostActorTests.Failing_script_after_success_degrades_node_quality_to_Bad"` → **expected: PASS** (T1 repro green). `dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests --filter "FullyQualifiedName~VirtualTagActorTests"` → **expected: PASS** (incl. the updated failure test). Commit: `fix(runtime): VT script failure/timeout degrades node quality to Bad once per transition (02/S13)` --- ### R2-03-T4 — S13 transition-semantics unit guards **Classification:** tests (deterministic units pinning the state machine) **Estimated implement time:** 5 min **Parallelizable with:** T5, T7/T8. Blocked by T3. **Files:** `tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/VirtualTags/VirtualTagActorTests.cs` Add (scriptable stub evaluator with a queue of results): 1. `Second_consecutive_failure_does_not_publish_a_second_Bad` — fail, fail → exactly one Bad `EvaluationResult` at the parent (`ExpectMsg` then `ExpectNoMsg`). 2. `Recovery_to_the_same_value_republishes_Good_after_Bad` — Ok(42), fail, Ok(42) → three parent messages; the third is `Quality == Good`, `Value == 42` (dedup bypass — this is the guard against the node staying Bad forever after a same-value recovery). 3. `Evaluator_throw_also_degrades_quality` — evaluator that throws (not `Failure`) → Bad `EvaluationResult` published (the `:115-121` path). 4. `dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests --filter "FullyQualifiedName~VirtualTagActorTests"` → **expected: PASS** (all should pass immediately if T3 is correct; any red here is a T3 bug, fix in place). Commit: `test(runtime): pin VT Bad-quality transition semantics — once-per-transition, same-value recovery, throw path (02/S13)` --- ### R2-03-T5 — S13 cold-start suppression guard (inputs-ready gate) **Classification:** tests (deterministic unit pinning the gate) **Estimated implement time:** 5 min **Parallelizable with:** T4, T7/T8. Blocked by T3. **Files:** `tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/VirtualTags/VirtualTagActorTests.cs` 1. `Cold_start_failure_with_missing_dependency_publishes_no_Bad` — actor with `dependencyRefs: ["a","b"]`, always-failing evaluator; send `DependencyValueChanged("a", ...)` only → `parent.ExpectNoMsg` (node stays at the materialiser's `BadWaitingForInitialData`; no false Bad flash during deploy/respawn warm-up). 2. `Failure_after_all_dependencies_arrived_publishes_Bad` — same actor; send `"a"` then `"b"` → the second change (all refs now seen) publishes exactly one Bad `EvaluationResult`. 3. `dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests --filter "FullyQualifiedName~VirtualTagActorTests.Cold_start_failure_with_missing_dependency_publishes_no_Bad|FullyQualifiedName~VirtualTagActorTests.Failure_after_all_dependencies_arrived_publishes_Bad"` → **expected: PASS**. Commit: `test(runtime): pin VT inputs-ready gate — no Bad flash before all dependencies arrive (02/S13)` --- ### R2-03-T6 — S12 sequential clear-then-recompile guard (RED-able) **Classification:** test-first (deterministic unit) **Estimated implement time:** 3 min **Parallelizable with:** T1-T5 (different project/file). **Files:** `tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/RoslynVirtualTagEvaluatorTests.cs` 1. Add `Evaluate_after_ClearCompiledScripts_recompiles_and_succeeds`: evaluate a compiled expression, `ClearCompiledScripts()`, evaluate the same source again → both `Success`, `CompiledCacheCount == 1` after the second call. (Green on current code — this pins the sequential contract the T8 retry loop must not regress; the restructure moves the fetch inside a loop.) 2. `dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests --filter "FullyQualifiedName~RoslynVirtualTagEvaluatorTests.Evaluate_after_ClearCompiledScripts_recompiles_and_succeeds"` → **expected: PASS** (pre-fix baseline pin). Commit: `test(host): pin sequential clear-then-recompile contract of RoslynVirtualTagEvaluator (02/S12 baseline)` --- ### R2-03-T7 — S12 concurrent clear-vs-evaluate stress guard (RED — fails on current code) **Classification:** test-first repro (concurrency stress; probabilistic-per-iteration, reliable at volume) **Estimated implement time:** 5 min **Parallelizable with:** T1-T6. **Files:** `tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/RoslynVirtualTagEvaluatorTests.cs` 1. Add `async Task Evaluate_racing_ClearCompiledScripts_never_fails_with_disposed`: one evaluator (generous 5 s timeout so the budget never fires); `Task.Run` loop A doing 300 sequential `Evaluate("vt-race", "return (int)ctx.GetTag(\"a\").Value + 1;", …)` calls collecting results; concurrent loop B calling `ClearCompiledScripts()` + `Thread.Yield()` until A completes; whole test awaited with `WaitAsync(TimeSpan.FromSeconds(30), TestContext.Current.CancellationToken)` (regression fails, never hangs — the U2 test's pattern). Assert every result `Success` shouldBeTrue with `result.Reason` as the message (surfaces "script threw: Cannot access a disposed object" on red). 2. `dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests --filter "FullyQualifiedName~RoslynVirtualTagEvaluatorTests.Evaluate_racing_ClearCompiledScripts_never_fails_with_disposed"` → **expected: FAIL** on current code (disposed-reason failures observed within 300 racing iterations; the window is GetOrCompile-return → Task.Run-hopped disposed guard). If a run happens to pass, bump iterations — do not weaken the assertion. Commit: `test(host): red repro — apply-boundary cache clear races in-flight VT evaluation into ObjectDisposedException (02/S12)` --- ### R2-03-T8 — S12 retry-once on ObjectDisposedException (turns T7 green) **Classification:** implementation (adapter hardening) **Estimated implement time:** 5 min **Parallelizable with:** T1-T5. Blocked by T6 + T7. **Files:** `src/Server/ZB.MOM.WW.OtOpcUa.Host/Engines/RoslynVirtualTagEvaluator.cs` 1. Restructure the compiled path per design §2: `for (var attempt = 0; ; attempt++)` around fetch (`GetOrCompile` — existing catch chain preserved) + run (`TimedScriptEvaluator` block), adding `catch (ObjectDisposedException) when (!_disposed && attempt == 0) { continue; }` before the generic catch, and `catch (ObjectDisposedException)` → `Failure("evaluator disposed during evaluation")` for the shutdown / double-clear case. Update the class xmldoc (retry-safe vs apply-boundary clear; one-generation stale-recompile residual documented). 2. `dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests --filter "FullyQualifiedName~RoslynVirtualTagEvaluatorTests"` → **expected: PASS** (T7 stress green; T6 sequential pin green; all pre-existing U2/U3/passthrough tests green — proves the loop restructure didn't disturb timeout, cache, or fast-path semantics). Commit: `fix(host): retry VT evaluation once on ObjectDisposedException from the apply-boundary cache clear (02/S12)` --- ### R2-03-T9 — P7 conditional-clear wiring guard (RED — fails on current code) **Classification:** test-first (rewrites an existing wiring guard — deliberate behavior change) **Estimated implement time:** 5 min **Parallelizable with:** T7/T8. Blocked by T3 (same test file churn as T1-T5; serialize after S13 lands). **Files:** `tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/VirtualTags/VirtualTagHostActorTests.cs` 1. Rewrite `ApplyVirtualTags_clears_the_evaluator_compile_cache_each_generation` → `ApplyVirtualTags_clears_the_compile_cache_only_when_the_expression_set_changes` (keep the `CacheOwningStubEvaluator` spy; keep the "the host actually calls it" wiring-assertion role — update the xmldoc to say the clear is now gated, citing 02/P7): apply plan A (`ClearCount == 1`) → re-apply value-identical plan A (**`ClearCount == 1`**) → apply plan A with an edited `Expression` (`ClearCount == 2`) → apply two vtags sharing one expression, then remove one of them (expression set unchanged ⇒ count unchanged). 2. `dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests --filter "FullyQualifiedName~VirtualTagHostActorTests.ApplyVirtualTags_clears_the_compile_cache_only_when_the_expression_set_changes"` → **expected: FAIL** on current code (identical-redeploy step observes `ClearCount == 2`). Commit: `test(runtime): red — apply-boundary compile-cache clear must be gated on a changed expression set (02/P7)` --- ### R2-03-T10 — P7 expression-set gate (turns T9 green) **Classification:** implementation (host-actor apply path) **Estimated implement time:** 3 min **Parallelizable with:** — (same method as T2's file; do after T9). Blocked by T9. **Files:** `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/VirtualTags/VirtualTagHostActor.cs` 1. Replace `OnApply`'s unconditional clear (`:88-92`) with the guarded block from design §3 (`HashSet.SetEquals` over `msg.Plans` vs `_planByVtag.Values` expressions; ordinal comparer), with a comment covering: `_planByVtag` is the previous generation at this point; rename/Historize-only changes don't clear (cache is source-keyed); first-apply-after-restart clears by construction; the S12 retry — not this gate — is the in-flight-race fix. 2. `dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests --filter "FullyQualifiedName~VirtualTagHostActorTests"` → **expected: PASS** (T9 guard green + every other host-actor test green). Commit: `perf(runtime): gate the VT apply-boundary compile-cache clear on a changed expression set (02/P7)` --- ### R2-03-T11 — Full-suite sweep + STATUS/report bookkeeping **Classification:** verification + docs **Estimated implement time:** 5 min **Parallelizable with:** —. Blocked by T1-T10. **Files:** `archreview/plans/STATUS.md`, `archreview/plans/R2-03-vt-failure-quality-plan.md.tasks.json` 1. `dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests` → **expected: PASS** (whole project). 2. `dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests --filter "FullyQualifiedName~RoslynVirtualTagEvaluatorTests"` → **expected: PASS**. 3. `dotnet build ZB.MOM.WW.OtOpcUa.slnx` → clean (analyzer OTOPCUA0001 + xmldoc warnings unchanged). 4. Record S13/S12/P7 as fixed-on-branch in `STATUS.md` (Re-review 2026-07-12 section) + tick the tasks.json. Commit: `docs(archreview): record R2-03 (02/S13+S12+P7) fixed on branch; full VT suites green` --- ### R2-03-T12 — Live-`/run` gate on docker-dev (fail→Bad→recover, end to end) **Classification:** live verification (docker-dev rig; no user sign-in needed — login disabled) **Estimated implement time:** 15-20 min (rig rebuild dominates) **Parallelizable with:** —. Blocked by T11. **Files:** none (verification only; findings, if any, loop back as new tasks) Follow the "Live-`/run` verification" section above: rebuild **both** central images; author a working VT → Client.CLI `read` Good; edit it to a runtime throw → redeploy → after one dependency change, Client.CLI `read` shows **StatusCode Bad** with the last-known value (this is the edited-broken-script respawn path — the S13 headline — through the real mux/child/bridge/publish-actor/SDK-node chain); fix → redeploy → `read` Good with a fresh value. Spot-check an identical redeploy produces no recompile burst in central logs (P7). Do NOT merge on unit-green alone — this is the wiring bar. Commit: none (verification). If the rig exposes a gap, open follow-up tasks before merge. --- ## Task dependency summary ``` T1 (S13 red repro) ──► T2 (record+bridge) ──► T3 (actor fix, T1 green) ──► T4, T5 (guards) └────────► T9 (P7 red) ──► T10 (P7 green) T6 (S12 pin) ─┐ T7 (S12 red) ─┴──► T8 (S12 retry, T7 green) T4, T5, T8, T10 ──► T11 (sweep + docs) ──► T12 (live /run gate) ``` **Overall effort:** Small (≈2-3 h implementation + the rig session), matching the OVERALL table's estimate for action item #3. **Overall risk:** Low-Medium, concentrated in S13's deliberate behavior change (nodes that used to freeze Good now go Bad — the point of the fix) and the one inverted legacy test expectation, both explicitly task-tracked.