lmxopcua/archreview/plans/R2-03-vt-failure-quality-plan.md

Design + Implementation Plan — R2-03: VT script-failure quality + cache-clear hardening

• Source report: archreview/02-scripting-alarms.md (2026-07-12 re-review)
• Review commit: f6eaa267 · Plan verified against tree at: f6eaa267 (master, clean)
• Scope: OVERALL prioritized action item #3 — "Publish Bad quality on VT script failure/timeout (match the dormant engine's BadInternalError); optionally gate the apply-boundary cache clear on changed scripts" — covering findings 02/S13 (Medium — failure/timeout never degrades node quality), 02/S12 (Medium — apply-boundary ClearCompiledScripts races in-flight child evaluations), and 02/P7 (Low — unconditional per-apply cache flush forces full VT recompilation on every deploy).
• Touches: Runtime/VirtualTags/VirtualTagActor.cs, Runtime/VirtualTags/VirtualTagHostActor.cs, Host/Engines/RoslynVirtualTagEvaluator.cs, Runtime.Tests/VirtualTags/*, Host.IntegrationTests/RoslynVirtualTagEvaluatorTests.cs. No Core.Scripting / Commons-contract / OPC UA-sink changes except one additive field on VirtualTagActor.EvaluationResult.

Verification summary

Every anchor in the three findings was opened at the cited file:line against the working tree at f6eaa267. All three findings CONFIRMED. One minor anchor drift (noted below); everything else is line-exact.

• S13 — Confirmed. VirtualTagActor.OnDependencyChanged returns without any publish on !result.Success (Runtime/VirtualTags/VirtualTagActor.cs:123-128) and on evaluator-threw (:115-121) — Warning ScriptLogEntry + fail metric only. The host bridge hard-codes OpcUaQuality.Good into every AttributeValueUpdate (Runtime/VirtualTags/VirtualTagHostActor.cs:179-180) and hard-codes 0u /* StatusCodes.Good */ into the historize snapshot (:187-188). EvaluationResult (VirtualTagActor.cs:23) carries no quality at all, so the child cannot express degradation today. Confirmed downstream that Bad quality is honored end to end once published: OpcUaPublishActor.HandleAttributeUpdate → IOpcUaAddressSpaceSink.WriteValue → OtOpcUaNodeManager.WriteValue (OpcUaServer/OtOpcUaNodeManager.cs:261-281) sets variable.StatusCode = StatusFromQuality(quality) + ClearChangeMasks, where StatusFromQuality (:2418-2423) maps Good→StatusCodes.Good, Uncertain→StatusCodes.Uncertain, _→StatusCodes.Bad. So the only missing plumbing is child → bridge.
• S13 anchor drift (minor): the report cites the dormant engine's failure mapping at VirtualTagEngine.cs:307,328. At f6eaa267, line 307 is the cold-start guard (BadWaitingForInitialData when inputs aren't ready yet — AreInputsReady); the actual failure→BadInternalError (0x80020000u) mappings are at 327-329 (uncoercible result), 334 (ScriptTimeoutException), and 343 (script threw). The substance of the finding is unaffected — and line 305-311 turns out to be load-bearing for this plan's design (the inputs-ready gate, below).
• S12 — Confirmed. VirtualTagHostActor.OnApply calls ClearCompiledScripts() first thing (VirtualTagHostActor.cs:92), on the host's dispatcher thread, with nothing gating concurrent child evaluations. The race window is inside one RoslynVirtualTagEvaluator.Evaluate call: the evaluator is fetched at RoslynVirtualTagEvaluator.cs:69 (_cache.GetOrCompile), the host's Clear() value-scoped-removes and disposes it (CompiledScriptCache.cs:113-116 → ScriptEvaluator.Dispose → _alc.Unload()), and the child's subsequent run (RoslynVirtualTagEvaluator.cs:108-109, via TimedScriptEvaluator.RunAsync's Task.Run) trips the disposed guard (ScriptEvaluator.cs:180) → ObjectDisposedException surfaces through GetAwaiter().GetResult() into the generic catch (Exception) (RoslynVirtualTagEvaluator.cs:116-120) → Failure("script threw: Cannot access a disposed object…") → dropped update (and, per S13, no quality degradation either). The contrast with ScriptedAlarmEngine holds: its per-generation _compileCache.Clear() (ScriptedAlarmEngine.cs:209) runs under _evalGate with upstream subscriptions already torn down, so no alarm evaluation can hold a disposed evaluator. The secondary wrinkle also confirmed: the clear runs before the removed/changed children are stopped (VirtualTagHostActor.cs:98-118) and Context.Stop is asynchronous, so a doomed child can recompile its stale expression into the fresh cache for one generation (bounded — not a leak). Note a mid-execution clear is safe: AssemblyLoadContext.Unload() is cooperative (the executing delegate keeps the ALC alive); only the guard at ScriptEvaluator.cs:180 — i.e. dispose landing between fetch and run-start — produces the dropped update. The Task.Run hop inside TimedScriptEvaluator widens exactly that window.
• P7 — Confirmed. The clear at VirtualTagHostActor.cs:92 is unconditional per ApplyVirtualTags, including a byte-identical redeploy whose plans all diff equal (the no-churn path proven by ApplyVirtualTags_does_not_respawn_child_when_plan_unchanged). The host already keeps last generation's plans in _planByVtag (:50, rebuilt at :160-164) — the comparison input for a conditional clear is already sitting in actor state. EquipmentVirtualTagPlan has element-wise value equality including Expression (OpcUaServer/AddressSpaceComposer.cs:179-188). The existing wiring guard VirtualTagHostActorTests.ApplyVirtualTags_clears_the_evaluator_compile_cache_each_generation asserts the clear fires on an identical second apply — that test encodes the P7 behavior and must be rewritten, not just kept green (called out in the task breakdown).
• Existing-test impact identified: VirtualTagActorTests.Evaluator_failure_publishes_ScriptLogEntry_warning ends with parent.ExpectNoMsg(...) — under the S13 fix a failure now does send the parent a Bad-quality EvaluationResult, so that assertion inverts (task R2-03-T3 updates it deliberately).
• Design constraint verified: the value fan-out's quality vocabulary is the coarse 3-value OpcUaQuality { Good, Uncertain, Bad } (Commons/OpcUa/IOpcUaAddressSpaceSink.cs:99) — there is no sub-code channel to the node, and AttributeValueUpdate (Runtime/OpcUa/OpcUaPublishActor.cs:42) already carries it. NullVirtualTagEvaluator returns NoChange (Success=true), so the dev/Mac path can never emit Bad. The passthrough fast-path returns Ok(null) for a missing dep (never Failure), so it is unaffected by all three fixes.

---

Priority ordering

Per the source report's Priority Recommendations (#1 = S13, #2 = S12) and OVERALL action item #3:

1. S13 (Medium, report's top item) — failure/timeout → node quality. The stale-Good repro test lands first and must fail on current code.
2. S12 (Medium) — close the clear-vs-in-flight-evaluate race (retry-once in the adapter).
3. P7 (Low) — gate the apply-boundary clear on an actually-changed script set.

All three are one PR (fix/archreview-r2-03-vt-failure-quality): S13 and S12 touch disjoint files; P7 touches the same OnApply region S13's bridge work sits next to.

---

1. S13 — MEDIUM — Script failure/timeout never degrades the OPC UA node's quality: stale value stays Good forever

Restatement: When a VT script evaluation fails (compile error, runtime throw, sandbox violation, or the U2-era timeout), VirtualTagActor logs + meters and returns without publishing anything, and the host bridge only ever publishes OpcUaQuality.Good. A previously-working script that starts failing leaves its node serving the last Good value with its old timestamp indefinitely — OPC UA clients, the continuous-historization recorder, and ScadaBridge cannot distinguish "healthy but unchanged" from "script broken for a week". The dormant VirtualTagEngine maps this correctly to BadInternalError; the live actor path never implemented it.

Verification: Confirmed (see summary). Bad quality is fully plumbed from AttributeValueUpdate through the sink to the SDK node — the gap is exactly VirtualTagActor (no quality concept) and the VirtualTagHostActor bridge (hard-coded Good at :179-180, hard-coded 0u historize status at :187-188).

Root cause: EvaluationResult was designed as a success-only message ("emit to the parent whenever the value actually changes"); failures were treated purely as an observability event (script-log + metric). The quality dimension was left to the materialiser's initial BadWaitingForInitialData and never revisited when the U2 remediation made timeouts a first-class, expected failure mode.

Proposed design:

(a) StatusCode policy — coarse OpcUaQuality.Bad for both failure and timeout. The node will read StatusCodes.Bad (0x80000000) via the existing StatusFromQuality mapping. This "matches the dormant engine's BadInternalError" at the level that matters — the severity bit that flips client/consumer handling — and the failure kind (timeout vs throw vs compile) is already delivered verbatim to operators as the script-log Reason string on the script-logs topic. Alternatives rejected:

• Widen OpcUaQuality with BadInternalError/BadTimeout members — additive on the enum, but it ripples a Commons contract consumed by every sink implementation and driver publish path, for a sub-code no consumer acts on. Contained follow-up if a real client requirement appears.
• Carry a raw uint status through AttributeValueUpdate/IOpcUaAddressSpaceSink.WriteValue — the precise-parity option, but it changes an interface with four implementations and re-runs the F10b/DeferredAddressSpaceSink forwarding trap for zero operator value. Rejected.
• Exception: the historize snapshot takes raw uint status already (DataValueSnapshot), so there the bridge records exact dormant-engine parity: 0x80020000u /* BadInternalError */ on Bad results (the dormant engine historizes its Bad snapshots unconditionally at VirtualTagEngine.cs:346-348; we mirror that).

(b) Publish cadence — once per transition, not per failure. Every publish is a sink write + ClearChangeMasks → client notification on the pinned OPC UA dispatcher. A failing script on a hot dependency would otherwise emit one identical Bad write per dependency change (each also paying the P2 pool hop + 2 s worst-case actor block per S7). The actor tracks _lastPublishedBad; it publishes Bad on the Good→Bad (or initial→Bad) transition only, and on recovery it force-publishes the next success even if the value equals the pre-failure value — otherwise the value-dedup at VirtualTagActor.cs:138 would leave the node Bad forever after a same-value recovery. (Script-log Warning entries stay per-failure, unchanged — that's the existing operator diagnostic and its volume is pre-existing.)

(c) Cold-start suppression — an inputs-ready gate, mirroring the dormant engine. The live path has no AreInputsReady equivalent: a multi-dep script fails on the first dependency arrival (the missing dep's GetTag(...).Value is null → cast throws) until all deps have landed. Publishing Bad on those would make every fresh/respawned multi-dep VT flash BadWaitingForInitialData → Bad → Good on every deploy — operator-visible churn for healthy scripts. Worse, gating on "had a prior success in this child" is insufficient for S13's headline scenario: an edited-now-broken script respawns its child (plan change ⇒ stop+respawn at VirtualTagHostActor.cs:109-118), so the fresh child has no prior success and would never degrade — reintroducing stale-Good exactly where it matters most. The correct gate is the dormant engine's: suppress Bad publishes until every dependencyRefs entry has been received at least once (_dependencyRefs.All(_dependencies.ContainsKey)); after that, any failure degrades. A never-arriving dependency keeps the node at the materialiser's BadWaitingForInitialData (already Bad — acceptable, and per the report "acceptable" for never-succeeded scripts).

(d) Message/bridge shape — additive Quality on EvaluationResult. Add OpcUaQuality Quality = OpcUaQuality.Good as a trailing optional parameter on the EvaluationResult positional record: every existing construction and destructuring site compiles unchanged, and the message is a local Context.Parent.Tell (no wire serialization concern). The bridge replaces the hard-coded OpcUaQuality.Good with result.Quality, and maps the historize status (Good→0u, else 0x80020000u). The dependency mux is untouched — quality flows child→parent→publish actor; the mux only fans values into children (DependencyValueChanged is unchanged), so no mux registration, routing, or message contract moves.

On failure the Bad EvaluationResult carries the last-known value (_lastValue if _hasLastValue, else null) with the triggering msg.TimestampUtc — clients see the stale value explicitly flagged Bad (the report's "publish the existing value with Bad quality"), rather than the dormant engine's null-value choice; either is defensible, this one preserves more information for trend displays.

Implementation steps:

1. Runtime/VirtualTags/VirtualTagActor.cs:
   • using ZB.MOM.WW.OtOpcUa.Commons.OpcUa;
   • public sealed record EvaluationResult(string VirtualTagId, object? Value, DateTime TimestampUtc, CorrelationId Correlation, OpcUaQuality Quality = OpcUaQuality.Good);
   • Add fields private bool _lastPublishedBad; and a helper private bool AllDependenciesArrived() => _dependencyRefs.All(_dependencies.ContainsKey); (refs list is small; called once per failure — no caching needed).
   • Extract the failure tail shared by the evaluator-threw catch (:115-121) and the !Success branch (:123-128) into private void OnEvaluationFailed(string reason, string level, DateTime timestampUtc): keep the existing metric + PublishLog, then if (_lastPublishedBad || !AllDependenciesArrived()) return; else _lastPublishedBad = true; Context.Parent.Tell(new EvaluationResult(_virtualTagId, _hasLastValue ? _lastValue : null, timestampUtc, CorrelationId.NewId(), OpcUaQuality.Bad));
   • Success path: change the dedup guard at :138 to if (!_lastPublishedBad && _hasLastValue && Equals(_lastValue, result.Value)) and set _lastPublishedBad = false; alongside _hasLastValue = true; before the Good publish. The NoChange short-circuit (:132-136) stays above the dedup and does not touch _lastPublishedBad (NoChange = "no fresh value", not a recovery).
   • Update the class summary: failures now degrade quality once per transition.
2. Runtime/VirtualTags/VirtualTagHostActor.cs (OnResult, :170-190):
   • _publishActor.Tell(new OpcUaPublishActor.AttributeValueUpdate(nodeId, result.Value, result.Quality, result.TimestampUtc));
   • Historize: var status = result.Quality == OpcUaQuality.Good ? 0u : 0x80020000u /* BadInternalError — dormant-engine parity */; and pass status into the DataValueSnapshot.
   • Update the :179-180 comment (no longer hard-coded Good).
3. Update VirtualTagActorTests.Evaluator_failure_publishes_ScriptLogEntry_warning: the trailing parent.ExpectNoMsg becomes parent.ExpectMsg<EvaluationResult>() with Quality == Bad (the test constructs the actor with empty dependencyRefs, so the inputs-ready gate is vacuously satisfied — All over an empty list is true).

Tests: (all deterministic; probes only)

• Repro first (must FAIL at f6eaa267): Runtime.Tests VirtualTagHostActorTests. Failing_script_after_success_degrades_node_quality_to_Bad — host + real child + a scripted fail-after-success IVirtualTagEvaluator stub; drive the child (captured via the mux probe's RegisterInterest sender) with two DependencyValueChangeds; assert the publish probe receives update 1 (Good, 42) then update 2 with Quality == OpcUaQuality.Bad carrying the last value. Uses only current types (AttributeValueUpdate already has Quality), so it compiles today and times out on the second ExpectMsg — the stale-Good repro.
• VirtualTagActorTests: failure-after-success emits Bad EvaluationResult with last value; second consecutive failure emits nothing (once-per-transition); recovery to the same value as before the failure does publish (dedup bypass); evaluator-threw path (not just Failure) also degrades; multi-dep cold-start failure with a missing dep publishes no Bad (inputs-ready gate); after all deps arrive, failure publishes Bad.
• VirtualTagHostActorTests: a Bad-quality EvaluationResult bridges to an AttributeValueUpdate with Quality == Bad; a historized plan's Bad result records status 0x80020000u (BadInternalError).
• Wiring note (theme #1, "unit-green ≠ wired"): the repro test above IS the wiring assertion — it runs the real host + real child + parent-Tell bridge, not a hand-delivered message. The remaining live wire (publish actor → sink → SDK node) is pre-existing, already exercised by OpcUaPublishActor tests, and covered by the live-/run gate below.

Effort: S. Risk: Low-Medium. The behavioral deltas are (1) parents now receive Bad results — only VirtualTagHostActor parents children in production, and it handles the new field natively; (2) a node that used to freeze Good now goes Bad on persistent failure — the intended, strictly-safer change; (3) one existing test's expectation inverts (deliberate, task-tracked). Blast radius = VT value path only; alarms/drivers untouched.

---

2. S12 — MEDIUM — Apply-boundary ClearCompiledScripts races in-flight child evaluations

Restatement: The host's per-apply Clear() disposes cached evaluators while unchanged children keep evaluating on other dispatcher threads. A child that fetched its evaluator (GetOrCompile, RoslynVirtualTagEvaluator.cs:69) before the clear hits the disposed guard on run (ScriptEvaluator.cs:180) → ObjectDisposedException → generic catch → Failure → silently dropped update (self-heals only on the tag's next dependency change — a long staleness window for slow tags). Secondary: a stopping child can recompile its stale script into the fresh cache for one generation.

Verification: Confirmed (see summary). The cache itself is blameless — its value-scoped TryRemove protects concurrent GetOrCompile re-adds; the gap is clear-vs-use of an already-handed-out evaluator. The ScriptedAlarmEngine mirror is inexact exactly as the report says (gate-held clear after subscription teardown, ScriptedAlarmEngine.cs:199-209).

Root cause: IScriptCacheOwner.ClearCompiledScripts was wired at the apply boundary (U3) without a coordination primitive between the host actor's clear and child actors' fetch→run sequence — the Task.Run hop inside TimedScriptEvaluator guarantees a real window between fetch and the disposed check.

Proposed design — catch-ObjectDisposedException, re-fetch, retry once (in the adapter). The recompile is idempotent (same source → fresh evaluator in the current cache generation), the retry is entirely local to RoslynVirtualTagEvaluator.Evaluate, and it converts the dropped update into a correct evaluation against the post-clear generation. If a second clear lands mid-retry (two applies inside one evaluation — not a realistic cadence), the second ObjectDisposedException falls through to the existing Failure path — same as today's worst case, now with an S13 Bad-quality publish instead of silence. Alternatives rejected:

• ReaderWriterLockSlim (children read-lock fetch→run; clear write-locks) — blocks the host actor for up to a full in-flight timeout budget (2 s) during every apply, and adds lock traffic to the hot path, to prevent a race the retry heals for free. Rejected.
• Per-generation cache instances with deferred dispose of the old generation — disposing the old generation safely still requires knowing when in-flight evaluations finish, i.e. ref-counting — the same coordination problem with more machinery. Rejected.
• Move the clear after the stop/respawn reconciliation — doesn't close the race for unchanged children (the common case), and the stale-recompile wrinkle survives either way because Context.Stop is asynchronous. Rejected as a fix; the wrinkle stays accepted + documented (bounded to one generation, released at the next apply-with-changes; P7's gate also means no-op redeploys no longer create the window at all).

Implementation steps (Host/Engines/RoslynVirtualTagEvaluator.cs, the compiled path :66-121):

1. Restructure fetch+run into a bounded loop (for (var attempt = 0; ; attempt++)):
   • fetch: evaluator = _cache.GetOrCompile(expression); — keep the existing CompilationErrorException / ScriptSandboxViolationException / generic catches on the fetch (unchanged behavior). A disposed cache here means the whole evaluator is being shut down — GetOrCompile throws ObjectDisposedException, which the new handler must NOT retry: check _disposed and return VirtualTagEvalResult.Failure("evaluator disposed").
   • run: existing TimedScriptEvaluator + GetAwaiter().GetResult() block, adding before the generic catch:

     catch (ObjectDisposedException) when (!_disposed && attempt == 0)
     {
         // S12: the apply-boundary ClearCompiledScripts disposed the evaluator between our
         // GetOrCompile and the run's disposed-guard. Re-fetch — GetOrCompile recompiles the same
         // source into the CURRENT cache generation — and retry once. A second disposal mid-retry
         // (two applies inside one evaluation) falls through to the generic failure path.
         continue;
     }
     

   • catch (ObjectDisposedException) (second hit, or _disposed) → Failure("evaluator disposed during evaluation").
2. Update the class doc: the apply-boundary clear is retry-safe for in-flight evaluations; the one-generation stale-recompile wrinkle for stopping children is the documented residual.

Tests (Host.IntegrationTests/RoslynVirtualTagEvaluatorTests.cs):

• Evaluate_racing_ClearCompiledScripts_never_fails_with_disposed — one evaluator; task A loops N (≈300) compiled (non-passthrough) evaluations of the same expression; task B loops ClearCompiledScripts() with Thread.Yield() between calls until A finishes; whole test WaitAsync-bounded (30 s). Assert every result Success == true (and specifically no Reason containing "disposed"). Determinism note: the pre-fix failure is probabilistic per iteration (the window is fetch → Task.Run-hopped disposed-guard, widened by pool queuing), but at N=300 racing clears it reproduces reliably in practice; post-fix it is deterministic-green by construction (the retry is exhaustive for a single concurrent clear). This is the report's prescribed "concurrent clear-during-evaluate test".
• Evaluate_after_ClearCompiledScripts_recompiles_and_succeeds — deterministic sequential guard: evaluate, clear, evaluate same source again → Success both times, cache Count back to 1 (uses the existing CompiledCacheCount reflection helper).
• Existing Evaluate_after_dispose_returns_Failure still green (the _disposed short-circuit at :51 is untouched; the new ODE handler's !_disposed filter keeps shutdown semantics).

Effort: S (the report's "10-line fix" is accurate; the loop restructure is the only care point). Risk: Low. Happy path unchanged (attempt 0, no exception); passthrough fast-path never enters the loop. Blast radius = one method in the Host adapter.

---

3. P7 — LOW — Unconditional per-apply cache flush forces full VT recompilation on every deploy

Restatement: OnApply clears the compile cache on every ApplyVirtualTags (VirtualTagHostActor.cs:92) — including a byte-identical redeploy that stops/spawns nothing. Every VT script then recompiles on its next evaluation, each paying the still-open P1 full-BCL-reference-set rebuild (5-20 ms+ per script). Amplifies P1; also each unnecessary clear is an unnecessary S12 race window.

Verification: Confirmed (see summary). The host already holds last generation's plans in _planByVtag; EquipmentVirtualTagPlan.Expression participates in value equality. The existing wiring guard asserts the current (unconditional) behavior on an identical redeploy and must be rewritten.

Root cause: The U3 fix chose the simplest correct lifecycle ("clear per generation", mirroring the alarm engine) without consulting the plan diff the same method computes a few lines later.

Proposed design — clear only when the deployed expression set changed. At the top of OnApply, compare the desired expression set against the previous generation's (_planByVtag.Values.Select(p => p.Expression)), and clear only on inequality:

var newExpressions = msg.Plans.Select(p => p.Expression).ToHashSet(StringComparer.Ordinal);
var prevExpressions = _planByVtag.Values.Select(p => p.Expression).ToHashSet(StringComparer.Ordinal);
if (!newExpressions.SetEquals(prevExpressions))
{
    (_evaluator as IScriptCacheOwner)?.ClearCompiledScripts();
}

Decisions inside this shape:

• Expression set, not full-plan set: the cache is keyed by source hash only, so a rename / FolderPath move / Historize toggle (which respawns the child) does not need a recompile — comparing full plans would clear on those spuriously. Two vtags sharing one expression are handled naturally (set semantics: the expression survives while any plan still uses it).
• Any-change clears everything (not per-source eviction): editing 1 of 200 scripts still recompiles 200. Accepted per the report ("cheap to leave as-is at current scale; fix alongside P1") — the headline P7 cost is the no-op redeploy, which this eliminates entirely. Rejected alternative: a CompiledScriptCache.RetainOnly(IEnumerable<string> sources) API — precise, but new Core.Scripting surface + tests for a Low; note it as the follow-up shape if P1's memoization doesn't land first.
• First apply after actor (re)start: _planByVtag is empty → sets differ → clear fires (harmless: the singleton evaluator's cache may hold a prior incarnation's entries; clearing converges state).
• Interaction with S12 (design question d): independent and composing — the gate reduces clear frequency (no-op redeploys stop racing entirely), the retry remains the correctness backstop for genuine changes. The gate must NOT be treated as the S12 fix: a real script edit still clears while unchanged children evaluate.
• Interaction with S13: none (quality path doesn't touch the cache).

Implementation steps:

1. Replace VirtualTagHostActor.cs:88-92 (comment + unconditional call) with the guarded block above + a comment explaining the expression-set gate (and that _planByVtag is "the previous generation" at this point — the method rebuilds it at the end).
2. Rewrite the wiring guard ApplyVirtualTags_clears_the_evaluator_compile_cache_each_generation → ApplyVirtualTags_clears_the_compile_cache_only_when_the_expression_set_changes: apply plan A (ClearCount 1) → re-apply identical plan A (ClearCount still 1) → apply plan A′ with an edited Expression (ClearCount 2) → re-apply with a vtag removed but its expression still used by another vtag (ClearCount still 2). Keeps the load-bearing "the host actually calls it" wiring assertion while pinning the new gate.

Tests: the rewritten wiring guard above (in Runtime.Tests/VirtualTags/VirtualTagHostActorTests.cs, using the existing CacheOwningStubEvaluator spy). New-behavior red state: the identical-redeploy step fails on current code (ClearCount is 2, expected 1).

Effort: S. Risk: Low. The only hazard is under-clearing (a stale ALC surviving a deploy whose expression set is unchanged) — by construction those entries are exactly the still-deployed sources, so nothing stale survives. Blast radius = one method + one test.

---

Live-/run verification (after all tasks green)

Deterministic units + wiring guards above are necessary but not sufficient — unit-green ≠ wired (the F10b lesson). Close with the docker-dev rig (login disabled; do it yourself, no user sign-in):

1. Rebuild both central-1 and central-2 images (:9200 round-robins them) and bring docker-dev/docker-compose.yml up.
2. On /uns (AdminUI http://localhost:9200), author a VirtualTag on a rig equipment with a script that succeeds against live deps (e.g. return (double)ctx.GetTag("<dep>").Value * 2;), deploy (POST :9200/api/deployments, X-Api-Key), and confirm via Client.CLI: dotnet run --project src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI -- read -u opc.tcp://localhost:4840 -n "<folder-scoped NodeId>" → value present, StatusCode Good.
3. Edit the script to a guaranteed runtime throw (e.g. int z = 0; return 1 / z;), redeploy, wait one dependency change → Client.CLI read again → StatusCode Bad (0x80000000), value = last-known. This exercises the exact S13 headline path (edited-broken script ⇒ respawned child ⇒ inputs-ready gate ⇒ Bad publish) end to end through the SDK node.
4. Fix the script back, redeploy → read returns Good with a fresh value (recovery + dedup-bypass live).
5. P7 spot-check: redeploy the identical config; central logs show no recompile burst (and the S12 stress guard covers the race in CI).

---

Task breakdown

Branch: fix/archreview-r2-03-vt-failure-quality off master@f6eaa267. Tasks are ≤5 min each; TDD-ordered (repro/red test precedes each implementation). Test commands run from the repo root.

---

R2-03-T1 — S13 stale-Good repro test (RED — must fail on current code)

Classification: test-first repro (deterministic unit, host-level wiring shape) Estimated implement time: 5 min Parallelizable with: T7 (different test file); everything else depends on this landing red first. Files: tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/VirtualTags/VirtualTagHostActorTests.cs

1. Add a FailAfterFirstSuccessEvaluator : IVirtualTagEvaluator private stub (call 1 → Ok(42.0), thereafter → Failure("boom"); interlocked counter).
2. Add test Failing_script_after_success_degrades_node_quality_to_Bad: host with publish + mux probes and the stub; apply one plan (deps ["a"]); capture the child via mux.ExpectMsg<DependencyMuxActor.RegisterInterest>() + mux.LastSender; tell the child DependencyValueChanged("a", 1, ts1) → publish probe expects AttributeValueUpdate with Quality == OpcUaQuality.Good, Value == 42.0; tell DependencyValueChanged("a", 2, ts2) → expect a second AttributeValueUpdate with Quality == OpcUaQuality.Bad and Value == 42.0 (last-known) and TimestampUtc == ts2.
3. dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests --filter "FullyQualifiedName~VirtualTagHostActorTests.Failing_script_after_success_degrades_node_quality_to_Bad" → expected: FAIL (timeout on the second ExpectMsg — the stale-Good bug, verbatim).

Commit: test(runtime): red repro — VT script failure after success leaves node stale-Good (02/S13)

---

R2-03-T2 — EvaluationResult.Quality + bridge pass-through + historize parity

Classification: implementation (additive message field + bridge) Estimated implement time: 5 min Parallelizable with: T7/T8 (disjoint files). Blocked by T1 (repro must be red first). Files: src/Server/ZB.MOM.WW.OtOpcUa.Runtime/VirtualTags/VirtualTagActor.cs (record only), src/Server/ZB.MOM.WW.OtOpcUa.Runtime/VirtualTags/VirtualTagHostActor.cs, tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/VirtualTags/VirtualTagHostActorTests.cs

1. RED: add bridge tests Bad_quality_EvaluationResult_is_bridged_as_Bad (tell the host a synthetic EvaluationResult(..., Quality: OpcUaQuality.Bad) → publish probe's AttributeValueUpdate.Quality is Bad) and Historized_Bad_result_is_recorded_with_BadInternalError_status (PlanH historize:true; Bad result → CapturingHistoryWriter snapshot StatusCode == 0x80020000u). These require the record change to compile — so within this task: add OpcUaQuality Quality = OpcUaQuality.Good to EvaluationResult (+ using ...Commons.OpcUa;) FIRST, then run the two tests: dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests --filter "FullyQualifiedName~VirtualTagHostActorTests.Bad_quality_EvaluationResult_is_bridged_as_Bad|FullyQualifiedName~VirtualTagHostActorTests.Historized_Bad_result_is_recorded_with_BadInternalError_status" → expected: FAIL (bridge still hard-codes Good / 0u).
2. GREEN: OnResult — pass result.Quality into AttributeValueUpdate; historize status result.Quality == OpcUaQuality.Good ? 0u : 0x80020000u /* BadInternalError — dormant-engine parity */; fix the stale comments at :179-188. Re-run the filter → PASS. Also run the full host-actor class (--filter "FullyQualifiedName~VirtualTagHostActorTests") — all pre-existing tests (Good-path bridging, historize Good 0u) must stay green, proving the default keeps byte-parity.

Commit: feat(runtime): EvaluationResult carries OpcUaQuality; VT bridge publishes it + historizes BadInternalError on Bad (02/S13)

---

R2-03-T3 — VirtualTagActor failure→Bad transition (turns T1 green)

Classification: implementation (actor behavior) Estimated implement time: 5 min Parallelizable with: T7/T8. Blocked by T2. Files: src/Server/ZB.MOM.WW.OtOpcUa.Runtime/VirtualTags/VirtualTagActor.cs, tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/VirtualTags/VirtualTagActorTests.cs

1. Implement per design §1(b)/(c)/(d): _lastPublishedBad field, AllDependenciesArrived(), OnEvaluationFailed(...) shared by the throw-catch and !Success branch (metric + script-log preserved per-failure; Bad publish once per transition, gated on inputs-ready, carrying _hasLastValue ? _lastValue : null + msg.TimestampUtc); success path resets _lastPublishedBad and bypasses the value-dedup when recovering; class-summary xmldoc updated.
2. Update Evaluator_failure_publishes_ScriptLogEntry_warning: replace the trailing parent.ExpectNoMsg(...) with parent.ExpectMsg<VirtualTagActor.EvaluationResult>().Quality.ShouldBe(OpcUaQuality.Bad) — the deliberate behavioral inversion (empty dependencyRefs ⇒ inputs-ready gate vacuously true).
3. dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests --filter "FullyQualifiedName~VirtualTagHostActorTests.Failing_script_after_success_degrades_node_quality_to_Bad" → expected: PASS (T1 repro green). dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests --filter "FullyQualifiedName~VirtualTagActorTests" → expected: PASS (incl. the updated failure test).

Commit: fix(runtime): VT script failure/timeout degrades node quality to Bad once per transition (02/S13)

---

R2-03-T4 — S13 transition-semantics unit guards

Classification: tests (deterministic units pinning the state machine) Estimated implement time: 5 min Parallelizable with: T5, T7/T8. Blocked by T3. Files: tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/VirtualTags/VirtualTagActorTests.cs

Add (scriptable stub evaluator with a queue of results):

1. Second_consecutive_failure_does_not_publish_a_second_Bad — fail, fail → exactly one Bad EvaluationResult at the parent (ExpectMsg then ExpectNoMsg).
2. Recovery_to_the_same_value_republishes_Good_after_Bad — Ok(42), fail, Ok(42) → three parent messages; the third is Quality == Good, Value == 42 (dedup bypass — this is the guard against the node staying Bad forever after a same-value recovery).
3. Evaluator_throw_also_degrades_quality — evaluator that throws (not Failure) → Bad EvaluationResult published (the :115-121 path).
4. dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests --filter "FullyQualifiedName~VirtualTagActorTests" → expected: PASS (all should pass immediately if T3 is correct; any red here is a T3 bug, fix in place).

Commit: test(runtime): pin VT Bad-quality transition semantics — once-per-transition, same-value recovery, throw path (02/S13)

---

R2-03-T5 — S13 cold-start suppression guard (inputs-ready gate)

Classification: tests (deterministic unit pinning the gate) Estimated implement time: 5 min Parallelizable with: T4, T7/T8. Blocked by T3. Files: tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/VirtualTags/VirtualTagActorTests.cs

1. Cold_start_failure_with_missing_dependency_publishes_no_Bad — actor with dependencyRefs: ["a","b"], always-failing evaluator; send DependencyValueChanged("a", ...) only → parent.ExpectNoMsg (node stays at the materialiser's BadWaitingForInitialData; no false Bad flash during deploy/respawn warm-up).
2. Failure_after_all_dependencies_arrived_publishes_Bad — same actor; send "a" then "b" → the second change (all refs now seen) publishes exactly one Bad EvaluationResult.
3. dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests --filter "FullyQualifiedName~VirtualTagActorTests.Cold_start_failure_with_missing_dependency_publishes_no_Bad|FullyQualifiedName~VirtualTagActorTests.Failure_after_all_dependencies_arrived_publishes_Bad" → expected: PASS.

Commit: test(runtime): pin VT inputs-ready gate — no Bad flash before all dependencies arrive (02/S13)

---

R2-03-T6 — S12 sequential clear-then-recompile guard (RED-able)

Classification: test-first (deterministic unit) Estimated implement time: 3 min Parallelizable with: T1-T5 (different project/file). Files: tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/RoslynVirtualTagEvaluatorTests.cs

1. Add Evaluate_after_ClearCompiledScripts_recompiles_and_succeeds: evaluate a compiled expression, ClearCompiledScripts(), evaluate the same source again → both Success, CompiledCacheCount == 1 after the second call. (Green on current code — this pins the sequential contract the T8 retry loop must not regress; the restructure moves the fetch inside a loop.)
2. dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests --filter "FullyQualifiedName~RoslynVirtualTagEvaluatorTests.Evaluate_after_ClearCompiledScripts_recompiles_and_succeeds" → expected: PASS (pre-fix baseline pin).

Commit: test(host): pin sequential clear-then-recompile contract of RoslynVirtualTagEvaluator (02/S12 baseline)

---

R2-03-T7 — S12 concurrent clear-vs-evaluate stress guard (RED — fails on current code)

Classification: test-first repro (concurrency stress; probabilistic-per-iteration, reliable at volume) Estimated implement time: 5 min Parallelizable with: T1-T6. Files: tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/RoslynVirtualTagEvaluatorTests.cs

1. Add async Task Evaluate_racing_ClearCompiledScripts_never_fails_with_disposed: one evaluator (generous 5 s timeout so the budget never fires); Task.Run loop A doing 300 sequential Evaluate("vt-race", "return (int)ctx.GetTag(\"a\").Value + 1;", …) calls collecting results; concurrent loop B calling ClearCompiledScripts() + Thread.Yield() until A completes; whole test awaited with WaitAsync(TimeSpan.FromSeconds(30), TestContext.Current.CancellationToken) (regression fails, never hangs — the U2 test's pattern). Assert every result Success shouldBeTrue with result.Reason as the message (surfaces "script threw: Cannot access a disposed object" on red).
2. dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests --filter "FullyQualifiedName~RoslynVirtualTagEvaluatorTests.Evaluate_racing_ClearCompiledScripts_never_fails_with_disposed" → expected: FAIL on current code (disposed-reason failures observed within 300 racing iterations; the window is GetOrCompile-return → Task.Run-hopped disposed guard). If a run happens to pass, bump iterations — do not weaken the assertion.

Commit: test(host): red repro — apply-boundary cache clear races in-flight VT evaluation into ObjectDisposedException (02/S12)

---

R2-03-T8 — S12 retry-once on ObjectDisposedException (turns T7 green)

Classification: implementation (adapter hardening) Estimated implement time: 5 min Parallelizable with: T1-T5. Blocked by T6 + T7. Files: src/Server/ZB.MOM.WW.OtOpcUa.Host/Engines/RoslynVirtualTagEvaluator.cs

1. Restructure the compiled path per design §2: for (var attempt = 0; ; attempt++) around fetch (GetOrCompile — existing catch chain preserved) + run (TimedScriptEvaluator block), adding catch (ObjectDisposedException) when (!_disposed && attempt == 0) { continue; } before the generic catch, and catch (ObjectDisposedException) → Failure("evaluator disposed during evaluation") for the shutdown / double-clear case. Update the class xmldoc (retry-safe vs apply-boundary clear; one-generation stale-recompile residual documented).
2. dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests --filter "FullyQualifiedName~RoslynVirtualTagEvaluatorTests" → expected: PASS (T7 stress green; T6 sequential pin green; all pre-existing U2/U3/passthrough tests green — proves the loop restructure didn't disturb timeout, cache, or fast-path semantics).

Commit: fix(host): retry VT evaluation once on ObjectDisposedException from the apply-boundary cache clear (02/S12)

---

R2-03-T9 — P7 conditional-clear wiring guard (RED — fails on current code)

Classification: test-first (rewrites an existing wiring guard — deliberate behavior change) Estimated implement time: 5 min Parallelizable with: T7/T8. Blocked by T3 (same test file churn as T1-T5; serialize after S13 lands). Files: tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/VirtualTags/VirtualTagHostActorTests.cs

1. Rewrite ApplyVirtualTags_clears_the_evaluator_compile_cache_each_generation → ApplyVirtualTags_clears_the_compile_cache_only_when_the_expression_set_changes (keep the CacheOwningStubEvaluator spy; keep the "the host actually calls it" wiring-assertion role — update the xmldoc to say the clear is now gated, citing 02/P7): apply plan A (ClearCount == 1) → re-apply value-identical plan A (ClearCount == 1) → apply plan A with an edited Expression (ClearCount == 2) → apply two vtags sharing one expression, then remove one of them (expression set unchanged ⇒ count unchanged).
2. dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests --filter "FullyQualifiedName~VirtualTagHostActorTests.ApplyVirtualTags_clears_the_compile_cache_only_when_the_expression_set_changes" → expected: FAIL on current code (identical-redeploy step observes ClearCount == 2).

Commit: test(runtime): red — apply-boundary compile-cache clear must be gated on a changed expression set (02/P7)

---

R2-03-T10 — P7 expression-set gate (turns T9 green)

Classification: implementation (host-actor apply path) Estimated implement time: 3 min Parallelizable with: — (same method as T2's file; do after T9). Blocked by T9. Files: src/Server/ZB.MOM.WW.OtOpcUa.Runtime/VirtualTags/VirtualTagHostActor.cs

1. Replace OnApply's unconditional clear (:88-92) with the guarded block from design §3 (HashSet.SetEquals over msg.Plans vs _planByVtag.Values expressions; ordinal comparer), with a comment covering: _planByVtag is the previous generation at this point; rename/Historize-only changes don't clear (cache is source-keyed); first-apply-after-restart clears by construction; the S12 retry — not this gate — is the in-flight-race fix.
2. dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests --filter "FullyQualifiedName~VirtualTagHostActorTests" → expected: PASS (T9 guard green + every other host-actor test green).

Commit: perf(runtime): gate the VT apply-boundary compile-cache clear on a changed expression set (02/P7)

---

R2-03-T11 — Full-suite sweep + STATUS/report bookkeeping

Classification: verification + docs Estimated implement time: 5 min Parallelizable with: —. Blocked by T1-T10. Files: archreview/plans/STATUS.md, archreview/plans/R2-03-vt-failure-quality-plan.md.tasks.json

1. dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests → expected: PASS (whole project).
2. dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests --filter "FullyQualifiedName~RoslynVirtualTagEvaluatorTests" → expected: PASS.
3. dotnet build ZB.MOM.WW.OtOpcUa.slnx → clean (analyzer OTOPCUA0001 + xmldoc warnings unchanged).
4. Record S13/S12/P7 as fixed-on-branch in STATUS.md (Re-review 2026-07-12 section) + tick the tasks.json.

Commit: docs(archreview): record R2-03 (02/S13+S12+P7) fixed on branch; full VT suites green

---

R2-03-T12 — Live-/run gate on docker-dev (fail→Bad→recover, end to end)

Classification: live verification (docker-dev rig; no user sign-in needed — login disabled) Estimated implement time: 15-20 min (rig rebuild dominates) Parallelizable with: —. Blocked by T11. Files: none (verification only; findings, if any, loop back as new tasks)

Follow the "Live-/run verification" section above: rebuild both central images; author a working VT → Client.CLI read Good; edit it to a runtime throw → redeploy → after one dependency change, Client.CLI read shows StatusCode Bad with the last-known value (this is the edited-broken-script respawn path — the S13 headline — through the real mux/child/bridge/publish-actor/SDK-node chain); fix → redeploy → read Good with a fresh value. Spot-check an identical redeploy produces no recompile burst in central logs (P7). Do NOT merge on unit-green alone — this is the wiring bar.

Commit: none (verification). If the rig exposes a gap, open follow-up tasks before merge.

---

Task dependency summary

T1 (S13 red repro) ──► T2 (record+bridge) ──► T3 (actor fix, T1 green) ──► T4, T5 (guards)
                                                                └────────► T9 (P7 red) ──► T10 (P7 green)
T6 (S12 pin) ─┐
T7 (S12 red) ─┴──► T8 (S12 retry, T7 green)
T4, T5, T8, T10 ──► T11 (sweep + docs) ──► T12 (live /run gate)

Overall effort: Small (≈2-3 h implementation + the rig session), matching the OVERALL table's estimate for action item #3. Overall risk: Low-Medium, concentrated in S13's deliberate behavior change (nodes that used to freeze Good now go Bad — the point of the fix) and the one inverted legacy test expectation, both explicitly task-tracked.

Execution deviations (R2-03)

Executed on branch r2/03-vt-failure-quality off master 1676c8f4 (not f6eaa267 — the plan's verified tree; 1676c8f4 is the current master and includes the round-2 plan/docs merge, so the code anchors are unchanged). Deviations, all reversible:

• T7 stress-test config retuned for a reliable RED on fast hardware. The plan's prescribed shape (single loop × 300 sequential evaluations racing a back-to-back Thread.Yield clear loop) did not reproduce the ObjectDisposedException on this dev machine (arm64 macOS, many idle cores → the fetch→Task.Run-hop→disposed-guard window is sub-microsecond and the Task.Run body dispatches instantly). Escalating to a back-to-back clear loop instead compile-swamped the run into a 60 s WaitAsync timeout (every evaluation recompiles under constant clearing) — and a back-to-back clear also risks double-disposal within one evaluation, which a single retry cannot heal (would make the fix's GREEN flaky). Final config: 12 concurrent evaluation loops × 400 iterations (thread-pool pressure widens the fetch→run window) racing a clear loop that clears then Thread.SpinWait(4000) (~tens of µs — frequent but not back-to-back, so at most one clear lands inside a single evaluation and the warm cache keeps compile cost bounded). This reproduces reliably (133 disposed-reason failures observed on the unfixed tree) and is deterministically green post-fix (verified stable across 3 repeat runs). The assertion was not weakened — it still requires every evaluation to succeed with no "disposed" reason.
• Test-run scope narrowed per controller memory constraint. A mid-execution controller directive flagged a ~16 GB/run memory leak in the *.IntegrationTests suites and forbade whole-project/whole- solution dotnet test. Verified locally instead: dotnet build ZB.MOM.WW.OtOpcUa.slnx clean; the whole Runtime.Tests unit project (372/372); and RoslynVirtualTagEvaluatorTests filtered (20/20). T11 (full cross-suite sweep) and T12 (docker-dev live-/run gate) are marked deferred-live in .tasks.json for the controller's serialized heavy pass — the build + STATUS/tasks bookkeeping portion of T11 was completed and committed.