ScadaBridge/archreview/plans/PLAN-R2-03-site-runtime-dcl.md

PLAN-R2-03 — Site Runtime & DCL Round-2 Hardening Implementation Plan

For Claude: REQUIRED SUB-SKILL: Use superpowers-extended-cc:executing-plans to implement this plan task-by-task.

Goal: Fix the round-2 findings in archreview/03-site-runtime-dcl.md (dated 2026-07-12) — the one Medium residual on the MxGateway reconnect path (a stale event loop's non-OCE fault can still flap the fresh connection through the generic catch) plus five Low polish items: missing failure mapping on the expression-eval PipeTo, two PLAN-03 option knobs absent from the eager options validator, the deploy-path double-compile (adopt the earmarked compile cache), the reintroduced "off-thread" spec-drift sentence, and the undocumented Expression-trigger/script-scheduler coupling. Round-1 deferrals P5 and P6 stay deferred (coverage-table rows only; Tasks 5/6's compile cache shrinks P6's warm-process cost as a side effect).

Architecture: All changes are site-side, inside ZB.MOM.WW.ScadaBridge.SiteRuntime and ZB.MOM.WW.ScadaBridge.DataConnectionLayer, plus their design doc. The established discipline is preserved: adapter lifecycle faults are classified on the loop's own cancellation token (never the mutable fields a newer generation may have replaced), every off-thread result — including a faulted task — returns to the actor via PipeTo and is applied on the actor thread, and options fail fast at boot with key-naming messages per the PLAN-08 §1.5 convention. The N4 compile cache mirrors TemplateEngine's ScriptCompileVerdictCache (SHA-256-keyed, bounded, clear-on-overflow) but is site-local and stores the full ScriptCompilationResult — sound because Roslyn Script<T> objects are immutable and thread-safe and the result's error text is name-free — so the synchronous deploy gate (deliberately on the singleton mailbox, see N5) and the InstanceActor.PreStart recompile share one Roslyn compile per script body per process lifetime. Docs travel with code.

Tech Stack: C#/.NET, Akka.NET (ReceiveActor, TestKit.Xunit2), Roslyn scripting, gRPC (Grpc.Core exception shapes via the ZB.MOM.WW.MxGateway.Client package), xunit + NSubstitute.

Build: dotnet build ZB.MOM.WW.ScadaBridge.slnx. Test per-project: dotnet test tests/<project> --filter <name>.

---

Task 1: MxGatewayDataConnection — a stale event loop's fault on a cancelled token must not signal Disconnected

Classification: high-risk (adapter reconnect lifecycle, concurrency with the dying loop) Estimated implement time: ~5 min Parallelizable with: 2, 3, 4, 5 Files:

• Modify: src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Adapters/MxGatewayDataConnection.cs (ConnectAsync ~lines 106–108, RunEventLoopAsync ~lines 111–130)
• Test: tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests/Adapters/MxGatewayDataConnectionReconnectTests.cs

The S1 fix cancels the previous loop's CTS and disposes the previous client (:77–89), then resets _disconnectFired (:91) — but Cancel() is not synchronous with the loop's exit. The old loop's await foreach over the gRPC stream surfaces its failure milliseconds later, and a cancelled/disposed Grpc.Net stream commonly faults with RpcException(StatusCode.Cancelled) (the default without ThrowOperationCanceledOnCancellation) or ObjectDisposedException from the concurrent DisposeAsync — neither is an OperationCanceledException, so both fall into the generic catch (:125–129) → RaiseDisconnected() against the reset guard → the exact S1 scenario-2 flap, potentially self-sustaining.

1. Add failing tests to the existing reconnect test class (extend its per-client substitute fixture so each client's RunEventLoopAsync returns a controllable TaskCompletionSource task instead of Task.Delay(Timeout.Infinite)):

[Fact]
public async Task StaleEventLoopRpcFault_AfterReconnect_DoesNotSignalDisconnected()
{
    var clients = new List<IMxGatewayClient>();
    var loops = new List<TaskCompletionSource>();
    var loopTokens = new List<CancellationToken>();
    var factory = Substitute.For<IMxGatewayClientFactory>();
    factory.Create().Returns(_ =>
    {
        var c = Substitute.For<IMxGatewayClient>();
        var tcs = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously);
        loops.Add(tcs);
        c.RunEventLoopAsync(Arg.Any<Action<MxValueUpdate>>(), Arg.Do<CancellationToken>(t => loopTokens.Add(t)))
            .Returns(tcs.Task);
        clients.Add(c);
        return c;
    });
    var adapter = new MxGatewayDataConnection(factory, NullLogger<MxGatewayDataConnection>.Instance);
    var disconnects = 0;
    adapter.Disconnected += () => Interlocked.Increment(ref disconnects);
    var details = new Dictionary<string, string> { ["Endpoint"] = "http://gw:5000", ["ApiKey"] = "k" };

    await adapter.ConnectAsync(details);
    await WaitUntilAsync(() => loopTokens.Count == 1);
    await adapter.ConnectAsync(details);          // reconnect: cancels loop #0, resets _disconnectFired
    await WaitUntilAsync(() => loopTokens.Count == 2);

    // The OLD loop observes its cancellation as a gRPC fault, not an OCE —
    // the Grpc.Net default without ThrowOperationCanceledOnCancellation.
    loops[0].SetException(new RpcException(new Status(StatusCode.Cancelled, "call cancelled")));
    await Task.Delay(200);
    Assert.Equal(0, disconnects);                 // stale fault absorbed — the fresh connection must not flap

    // The CURRENT loop's genuine fault must still signal, exactly once.
    loops[1].SetException(new RpcException(new Status(StatusCode.Unavailable, "gateway gone")));
    await WaitUntilAsync(() => disconnects == 1);
}

[Fact]
public async Task StaleEventLoopObjectDisposedFault_AfterReconnect_DoesNotSignalDisconnected()
{
    // Identical drive, but the old loop faults with the concurrent-DisposeAsync shape.
    // … same fixture …
    loops[0].SetException(new ObjectDisposedException("MxGatewayClient"));
    await Task.Delay(200);
    Assert.Equal(0, disconnects);
}

(RpcException/Status/StatusCode come from Grpc.Core, transitively via the ZB.MOM.WW.MxGateway.Client package the test project already references — RealMxGatewayClient.cs:227 catches RpcException in the main project. If the test assembly cannot resolve them, add a Grpc.Net.Client PackageReference; the version is already pinned in Directory.Packages.props.) 2. Run: dotnet test tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests --filter "FullyQualifiedName~StaleEventLoopRpcFault_AfterReconnect|FullyQualifiedName~StaleEventLoopObjectDisposedFault_AfterReconnect" → expect FAIL (the generic catch raises Disconnected against the reset guard: disconnects == 1 after the stale fault). 3. Implement:

• Change RunEventLoopAsync to take the client AND token as parameters — private async Task RunEventLoopAsync(IMxGatewayClient client, CancellationToken ct) — and use client.RunEventLoopAsync(…) instead of the _client! field read (the old loop must never bind a newer generation's client).
• Add a filtered catch before the generic one:

catch (Exception ex) when (ct.IsCancellationRequested)
{
    // Teardown fault of a superseded loop (N1): Cancel() is not synchronous with
    // the loop's exit — a cancelled/disposed gRPC stream commonly surfaces as
    // RpcException(StatusCode.Cancelled) (Grpc.Net default without
    // ThrowOperationCanceledOnCancellation) or ObjectDisposedException from the
    // concurrent DisposeAsync, milliseconds AFTER ConnectAsync has already reset
    // _disconnectFired. Any fault on a cancelled token is normal shutdown;
    // raising Disconnected here would flap the NEW healthy session.
    _logger.LogDebug(ex,
        "MxGateway event loop faulted after cancellation — superseded loop teardown; not signalling disconnect");
}

• In ConnectAsync, capture the token and client into locals before the lambda (a rapid double-reconnect must not hand loop #1 loop #2's token/client — the current Task.Run(() => RunEventLoopAsync(_eventLoopCts.Token)) reads the field lazily):

_eventLoopCts = new CancellationTokenSource();
var loopToken = _eventLoopCts.Token;   // capture NOW: the field may be replaced by a
var loopClient = _client;              // subsequent reconnect before Task.Run executes (N1)
_ = Task.Run(() => RunEventLoopAsync(loopClient, loopToken));

4. Run the two new tests → PASS; then the full adapter suite: dotnet test tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests --filter "FullyQualifiedName~MxGatewayDataConnection" → all green (the existing Reconnect_DisposesPreviousClient_AndCancelsPreviousEventLoop test pins the S1 behavior and must survive the signature change).
5. Commit: git add src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Adapters/MxGatewayDataConnection.cs tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests/Adapters/MxGatewayDataConnectionReconnectTests.cs && git commit -m "fix(dcl): stale MxGateway event-loop fault on a cancelled token no longer flaps the fresh connection (plan R2-03 T1)"

Task 2: ScriptActor — failure mapping on the expression-eval PipeTo (a faulted scheduler task must not kill the trigger)

Classification: high-risk (actor concurrency, trigger liveness) Estimated implement time: ~5 min Parallelizable with: 1, 3, 4, 5, 6 Files:

• Modify: src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/ScriptActor.cs (Receive block ~line 161, StartExpressionEvaluation PipeTo ~lines 311–313, internal messages ~line 623)
• Test: tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/ScriptActorTests.cs

StartExpressionEvaluation pipes with success: only (:312–313). The inner body catches everything, so the only fault source is the outer Task.Factory.StartNew itself (e.g. ObjectDisposedException from a disposed ScriptExecutionScheduler during shutdown/test teardown) — but if it ever fires, the actor gets an unhandled Status.Failure and _evalInFlight stays true forever (:284–285): every subsequent change parks in _evalPending and the expression trigger is dead for the actor's lifetime with no log.

1. Failing test. Add a public static gate hook to the test file (the CompileRawTriggerExpression helper compiles without the trust validator, so the expression may reference it — same trick as the P1 thread-name tests):

public static class EvalGate
{
    public static readonly SemaphoreSlim Gate = new(0);
    private static int _entries;
    public static int Entries => Volatile.Read(ref _entries);
    public static bool Block() { Interlocked.Increment(ref _entries); Gate.Wait(); return false; }
}

[Fact]
public void ExpressionEvalTaskFault_ClearsInFlight_AndDrainsPendingEvaluation()
{
    var expr = CompileRawTriggerExpression(
        "ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.Actors.EvalGate.Block()");
    var (actor, _) = CreateTriggeredActor(
        "ExprFault", "Expression", "{\"expression\":\"true\",\"mode\":\"OnTrue\"}", null, expr);
    try
    {
        actor.Tell(Change("A", "1"));                              // eval starts on the scheduler and BLOCKS → _evalInFlight = true
        AwaitAssert(() => Assert.Equal(1, EvalGate.Entries), TimeSpan.FromSeconds(10));
        actor.Tell(Change("A", "2"));                              // coalesces → _evalPending = true

        // Simulate the faulted scheduler task the PipeTo failure mapping now surfaces
        // (e.g. ObjectDisposedException from a disposed ScriptExecutionScheduler).
        actor.Tell(new ScriptActor.ExpressionEvalFailed(new ObjectDisposedException("scheduler")));

        // In-flight cleared + pending drained ⇒ a SECOND evaluation starts and blocks too.
        AwaitAssert(() => Assert.Equal(2, EvalGate.Entries), TimeSpan.FromSeconds(10));
    }
    finally
    {
        EvalGate.Gate.Release(EvalGate.Entries); // ALWAYS free the shared scheduler threads
    }
}

(Note the expression must reference the gate's fully-qualified name; the raw compile references typeof(object).Assembly only, so also add the test assembly to CompileRawTriggerExpression's .WithReferences(...) for this test — or add an overload taking extra references. The finally release is mandatory: the gate blocks threads of the process-wide ScriptExecutionScheduler.Shared.) 2. Run: dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~ExpressionEvalTaskFault_ClearsInFlight" → expect FAIL — the RED step is the compile failure (ScriptActor.ExpressionEvalFailed does not exist); behaviorally, without the fix the second evaluation never starts either. 3. Implement:

• New message next to ExpressionEvalResult (:623) — internal (not private) so the TestKit can Tell it, same visibility idiom as IntervalTick/ScriptExecutionCompleted:

/// <summary>
/// Piped back to self when the off-dispatcher evaluation TASK itself faults
/// (e.g. ObjectDisposedException from a disposed ScriptExecutionScheduler
/// during shutdown) — the inner body's catch never sees that. Without this
/// mapping the actor receives an unhandled Status.Failure and _evalInFlight
/// stays true forever, permanently parking the expression trigger (N2).
/// </summary>
internal sealed record ExpressionEvalFailed(Exception Cause);

• Ctor Receive block (after the ExpressionEvalResult registration at :161): Receive<ExpressionEvalFailed>(HandleExpressionEvalFailed);
• PipeTo (:312–313): …​.PipeTo(self, success: r => new ExpressionEvalResult(r), failure: ex => new ExpressionEvalFailed(ex));
• Handler — log via the existing LogExpressionError (:397, already increments the health counter), then reuse the result path with false (consistent with the inner catch's treated-as-false semantics — clears in-flight, applies the false edge, drains pending):

private void HandleExpressionEvalFailed(ExpressionEvalFailed msg)
{
    LogExpressionError(msg.Cause);
    HandleExpressionEvalResult(new ExpressionEvalResult(false));
}

4. Run the new test → PASS; then dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~ScriptActorTests" → all green (the P1 expression/WhileTrue suites are the regression canary).
5. Commit: git add src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/ScriptActor.cs tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/ScriptActorTests.cs && git commit -m "fix(siteruntime): faulted expression-eval task clears _evalInFlight instead of killing the ScriptActor trigger (plan R2-03 T2)"

Task 3: AlarmActor — same failure mapping on the expression-eval PipeTo

Classification: high-risk (alarm state machine liveness) Estimated implement time: ~4 min Parallelizable with: 1, 2, 4, 5, 6 Files:

• Modify: src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/AlarmActor.cs (Receive block ~line 188, StartExpressionEvaluation PipeTo ~lines 552–554, internal messages ~line 797)
• Test: tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/AlarmActorTests.cs

Structurally identical to Task 2 — AlarmActor.cs:552–554 pipes success-only, :523–524 sets _evalInFlight, so a faulted task permanently kills the Expression alarm.

1. Failing test (reuse the ExpressionAlarm_EvaluatesOnSchedulerThread_AndActivates harness in AlarmActorTests.cs:1031 — same ResolvedAlarm/instanceProbe construction):

[Fact]
public void ExpressionEvalTaskFault_DoesNotKillTheAlarmTrigger()
{
    var expr = CompileRawTriggerExpression("true");
    var alarmConfig = new ResolvedAlarm
    {
        CanonicalName = "ExprFaultAlarm",
        TriggerType = "Expression",
        TriggerConfiguration = "{\"expression\":\"true\"}",
        PriorityLevel = 1
    };
    var instanceProbe = CreateTestProbe();
    var alarm = ActorOf(Props.Create(() => new AlarmActor(
        "ExprFaultAlarm", "Pump1", instanceProbe.Ref, alarmConfig,
        null, _sharedLibrary, _options,
        NullLogger<AlarmActor>.Instance, expr)));

    // Simulate the faulted scheduler task: must be handled (logged, in-flight
    // cleared, false applied) — NOT an unhandled Status.Failure.
    alarm.Tell(new AlarmActor.ExpressionEvalFailed(new ObjectDisposedException("scheduler")));

    // The trigger must still be alive: the next change evaluates and activates.
    alarm.Tell(new AttributeValueChanged("Pump1", "A", "A", 1, "Good", DateTimeOffset.UtcNow));
    var change = instanceProbe.ExpectMsg<AlarmStateChanged>(TimeSpan.FromSeconds(10));
    Assert.Equal(AlarmState.Active, change.State);
}

2. Run: dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~ExpressionEvalTaskFault_DoesNotKillTheAlarmTrigger" → expect FAIL (compile error — AlarmActor.ExpressionEvalFailed does not exist).
3. Implement, mirroring Task 2:
   • internal sealed record ExpressionEvalFailed(Exception Cause); next to the existing ExpressionEvalResult (:797), with the same N2 XML doc.
   • Receive<ExpressionEvalFailed>(HandleExpressionEvalFailed); after :188.
   • PipeTo (:553–554) gains failure: ex => new ExpressionEvalFailed(ex).
   • Handler — same shape as the inner catch's error path, then delegate to the false result (the class doc already promises "a throwing … expression is treated as false so that the state machine still runs — an Active alarm correctly clears"):

private void HandleExpressionEvalFailed(ExpressionEvalFailed msg)
{
    _healthCollector?.IncrementAlarmError();
    _logger.LogError(msg.Cause,
        "Alarm {Alarm} trigger-expression evaluation task faulted on {Instance}; treated as false",
        _alarmName, _instanceName);
    HandleExpressionEvalResult(new ExpressionEvalResult(false));
}

4. Run the new test → PASS; then dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~AlarmActorTests" → all green.
5. Commit: git add src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/AlarmActor.cs tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/AlarmActorTests.cs && git commit -m "fix(siteruntime): faulted expression-eval task clears _evalInFlight instead of killing the AlarmActor trigger (plan R2-03 T3)"

Task 4: Eagerly validate TagSubscribeRetryIntervalMs and StuckScriptGraceMs

Classification: small (additive validator rules per the PLAN-08 §1.5 convention) Estimated implement time: ~3 min Parallelizable with: 1, 2, 3, 5, 6 Files:

• Modify: src/ZB.MOM.WW.ScadaBridge.SiteRuntime/SiteRuntimeOptionsValidator.cs (append after the ConfigFetchRetryCount rule, lines 56–58)
• Test: tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/SiteRuntimeOptionsValidatorTests.cs

The validator covers ten options but not the two PLAN-03 added: TagSubscribeRetryIntervalMs (SiteRuntimeOptions.cs:76 — a 0 hot-loops the subscribe retry, a negative value throws inside InstanceActor.SendTagSubscribe's ScheduleTellOnceCancelable) and StuckScriptGraceMs (SiteRuntimeOptions.cs:86 — a negative value throws ArgumentOutOfRangeException inside the watchdog's Task.Delay, ScriptExecutionActor.cs:144).

1. Failing tests (mirror the existing Validate(...) helper and assertion shape in the test file):

[Fact]
public void ZeroTagSubscribeRetryIntervalMs_IsRejected()
{
    var result = Validate(new SiteRuntimeOptions { TagSubscribeRetryIntervalMs = 0 });

    Assert.True(result.Failed);
    Assert.Contains("TagSubscribeRetryIntervalMs", result.FailureMessage);
}

[Fact]
public void NegativeStuckScriptGraceMs_IsRejected()
{
    var result = Validate(new SiteRuntimeOptions { StuckScriptGraceMs = -1 });

    Assert.True(result.Failed);
    Assert.Contains("StuckScriptGraceMs", result.FailureMessage);
}

2. Run: dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~SiteRuntimeOptionsValidatorTests" → the two new tests FAIL (validator succeeds on both).
3. Implement — two RequireThat lines matching the file's key-naming message style:

builder.RequireThat(options.TagSubscribeRetryIntervalMs > 0,
    $"ScadaBridge:SiteRuntime:TagSubscribeRetryIntervalMs must be greater than 0 " +
    $"(was {options.TagSubscribeRetryIntervalMs}); a zero interval hot-loops the tag-subscribe " +
    "retry and a negative one throws inside the Instance Actor's retry scheduling.");

builder.RequireThat(options.StuckScriptGraceMs >= 0,
    $"ScadaBridge:SiteRuntime:StuckScriptGraceMs must be >= 0 " +
    $"(was {options.StuckScriptGraceMs}); a negative grace throws inside the stuck-script watchdog's delay.");

4. Run → PASS, including DefaultOptions_AreValid (defaults 5000/30000 satisfy both rules).
5. Commit: git add src/ZB.MOM.WW.ScadaBridge.SiteRuntime/SiteRuntimeOptionsValidator.cs tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/SiteRuntimeOptionsValidatorTests.cs && git commit -m "fix(siteruntime): eagerly validate TagSubscribeRetryIntervalMs + StuckScriptGraceMs at boot (plan R2-03 T4)"

Task 5: SiteScriptCompileCache — bounded process-wide compiled-script cache (N4 groundwork)

Classification: standard (pure new class, no consumers yet) Estimated implement time: ~5 min Parallelizable with: 1, 2, 3, 4 Files:

• Create: src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Scripts/SiteScriptCompileCache.cs
• Test: create tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Scripts/SiteScriptCompileCacheTests.cs

This is the site-side adoption of the compile-cache abstraction the master tracker earmarked when PLAN-03's Task 3/4 shipped ("later refactor, not a blocker") — TemplateEngine's ScriptCompileVerdictCache pattern, but storing the full ScriptCompilationResult so both the deploy gate and the PreStart recompile reuse the compiled Script<object?> itself. Soundness: the result is a pure function of (code, globals type, compile-time-static trust policy); the stored error text is name-free (Diagnostic.GetMessage() / "Compilation exception: …" — callers prefix their own context); Roslyn Script<T> is immutable and thread-safe (RunAsync creates per-run state), so one instance may be shared across actors.

1. Failing tests (the type does not exist — compile failure is the RED step). The cache is process-wide static state: put this test class and Task 6's ScriptCompilationServiceTests additions in one xunit collection ([Collection("SiteScriptCompileCache")]) and call Clear() at the top of each test, so parallel classes cannot skew Hits/Count:

[Collection("SiteScriptCompileCache")]
public class SiteScriptCompileCacheTests
{
    private static ScriptCompilationResult Ok() =>
        ScriptCompilationResult.Succeeded(CSharpScript.Create<object?>("1", ScriptOptions.Default, typeof(ScriptGlobals)));

    [Fact]
    public void GetOrAdd_SameCodeAndGlobals_ComputesOnce()
    {
        SiteScriptCompileCache.Clear();
        var factoryCalls = 0;
        ScriptCompilationResult Factory() { factoryCalls++; return Ok(); }

        var r1 = SiteScriptCompileCache.GetOrAdd("return 1;", typeof(ScriptGlobals), Factory);
        var r2 = SiteScriptCompileCache.GetOrAdd("return 1;", typeof(ScriptGlobals), Factory);

        Assert.Equal(1, factoryCalls);
        Assert.Same(r1, r2);
        Assert.Equal(1, SiteScriptCompileCache.Hits);
    }

    [Fact]
    public void GetOrAdd_SameCode_DifferentGlobals_AreSeparateEntries()
    {
        SiteScriptCompileCache.Clear();
        SiteScriptCompileCache.GetOrAdd("1 > 0", typeof(ScriptGlobals), Ok);
        SiteScriptCompileCache.GetOrAdd("1 > 0", typeof(TriggerExpressionGlobals), Ok);

        Assert.Equal(2, SiteScriptCompileCache.Count);
        Assert.Equal(0, SiteScriptCompileCache.Hits); // no cross-globals hit
    }

    [Fact]
    public void Overflow_ClearsWholesale()
    {
        SiteScriptCompileCache.Clear();
        for (var i = 0; i <= SiteScriptCompileCache.MaxEntries; i++)
            SiteScriptCompileCache.GetOrAdd($"return {i};", typeof(ScriptGlobals), Ok);

        Assert.True(SiteScriptCompileCache.Count <= SiteScriptCompileCache.MaxEntries);
    }
}

2. Run: dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~SiteScriptCompileCacheTests" → FAIL (type does not exist).
3. Implement, mirroring ScriptCompileVerdictCache (src/ZB.MOM.WW.ScadaBridge.TemplateEngine/Validation/ScriptCompileVerdictCache.cs): internal static class SiteScriptCompileCache (InternalsVisibleTo already covers the test assembly) with
   • internal const int MaxEntries = 1024; — deliberately smaller than the verdict cache's 4096: entries pin compiled assemblies, not just verdict strings; on overflow the cache is cleared wholesale (results are recomputable, coarse reset avoids eviction bookkeeping).
   • Key = globalsType.FullName + ":" + Convert.ToHexString(SHA256.HashData(Encoding.UTF8.GetBytes(code))) — the globals discriminator keeps ScriptGlobals and TriggerExpressionGlobals compiles of identical source apart.
   • public static ScriptCompilationResult GetOrAdd(string code, Type globalsType, Func<ScriptCompilationResult> factory) over a ConcurrentDictionary<string, ScriptCompilationResult>, incrementing Hits on a hit; Hits/Count/Clear() exposed for tests and diagnostics — same surface as the TemplateEngine cache.
   • XML doc stating the three soundness invariants above (pure function of code+globals+static policy; name-free errors; Script<T> immutability) and that failures are cached too — a bad script body fails identically for every caller.
4. Run → PASS.
5. Commit: git add src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Scripts/SiteScriptCompileCache.cs tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Scripts/SiteScriptCompileCacheTests.cs && git commit -m "feat(siteruntime): bounded process-wide compiled-script cache, keyed on code hash + globals type (plan R2-03 T5)"

Task 6: Wire the cache into ScriptCompilationService — deploy gate + PreStart share one compile per script per revision

Classification: standard (single choke point; all call sites inherit) Estimated implement time: ~4 min Parallelizable with: 1, 2, 3, 4 (needs 5 merged) Files:

• Modify: src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Scripts/ScriptCompilationService.cs (CompileCore, ~lines 98–139)
• Modify: src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/DeploymentManagerActor.cs (gate rationale comment ~lines 510–521 — comment only)
• Test: tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Scripts/ScriptCompilationServiceTests.cs

CompileCore is the single funnel for every site-side compile — the DeployCompileValidator gate (DeploymentManagerActor.cs:522), the InstanceActor.CreateChildActors recompile (InstanceActor.cs:1485, 1497–1498), and SharedScriptLibrary all flow through it — so caching here collapses the 2×N-compiles-per-bulk-redeploy cost (N4) and shrinks P6's warm-process staggered-startup recompiles for free, with zero call-site changes.

1. Failing test (add to ScriptCompilationServiceTests, tagged [Collection("SiteScriptCompileCache")] per Task 5's isolation note):

[Fact]
public void Compile_SameCodeTwice_SharesOneRoslynCompile()
{
    SiteScriptCompileCache.Clear();
    var r1 = _service.Compile("deploy-gate-copy", "return 1 + 1;");
    var r2 = _service.Compile("prestart-copy", "return 1 + 1;");

    Assert.True(r1.IsSuccess);
    Assert.Same(r1.CompiledScript, r2.CompiledScript);   // one compile, shared Script<T> (N4)
    Assert.Equal(1, SiteScriptCompileCache.Hits);
}

[Fact]
public void Compile_ScriptAndTriggerExpression_DoNotCrossContaminate()
{
    SiteScriptCompileCache.Clear();
    var script = _service.Compile("s", "1 > 0");
    var trigger = _service.CompileTriggerExpression("t", "1 > 0");

    Assert.NotSame(script.CompiledScript, trigger.CompiledScript); // different globals surfaces
}

2. Run: dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~Compile_SameCodeTwice_SharesOneRoslynCompile|FullyQualifiedName~Compile_ScriptAndTriggerExpression_DoNotCrossContaminate" → FAIL (Assert.Same fails: two independent compiles today).
3. Implement:
   • Rename the current CompileCore body to private ScriptCompilationResult CompileUncached(string name, string code, Type globalsType) (byte-identical, including the trust-model check — the verdict is deterministic on code, so it is cacheable with the compile).
   • New CompileCore:

private ScriptCompilationResult CompileCore(string name, string code, Type globalsType)
{
    var result = SiteScriptCompileCache.GetOrAdd(code, globalsType,
        () => CompileUncached(name, code, globalsType));
    if (!result.IsSuccess)
        _logger.LogDebug(
            "Script {Script}: returning cached compile failure ({ErrorCount} error(s))",
            name, result.Errors.Count); // miss-path logs name the FIRST caller; keep this caller visible
    return result;
}

• Update the class XML doc: results (success AND failure) are memoised process-wide in SiteScriptCompileCache; error text is name-free by construction, so cached failures render correctly for every caller.
• Append one sentence to the HandleDeploy gate comment (DeploymentManagerActor.cs:519–521): "Redeploys and multi-instance deploys of unchanged scripts hit the process-wide compile cache (SiteScriptCompileCache), so the synchronous gate recompiles only genuinely new code and the Instance Actor's PreStart reuses the gate's compile (N4)."

4. Run the two new tests → PASS; then the funnel's consumer suites: dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~ScriptCompilationServiceTests|FullyQualifiedName~DeployCompileValidatorTests|FullyQualifiedName~SharedScriptLibraryTests|FullyQualifiedName~TrustModelSemanticTests" → all green (these pin that caching changed no verdicts).
5. Commit: git add src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Scripts/ScriptCompilationService.cs src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/DeploymentManagerActor.cs tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Scripts/ScriptCompilationServiceTests.cs && git commit -m "perf(siteruntime): deploy gate + InstanceActor PreStart share one Roslyn compile via the compile cache (plan R2-03 T6)"

Task 7: Design-doc sync — synchronous compile gate (N5), Expression/scheduler coupling (N6), P6/N4 note refresh

Classification: trivial (docs + one stale XML doc; repo rule: docs travel with code) Estimated implement time: ~4 min Parallelizable with: none (write after the behavior tasks land so the docs describe reality) Files:

• Modify: docs/requirements/Component-SiteRuntime.md (lines ~489, ~234–246, ~99)
• Modify: src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Deployment/DeployCompileValidator.cs (class XML doc, ~lines 16–18)

1. N5 — Component-SiteRuntime.md:489: the sentence claims the gate compiles "off-thread"; the shipped gate is deliberately synchronous (DeploymentManagerActor.cs:510–522). Replace the off-thread clause so the bullet reads: "…the Deployment Manager compiles all of the instance's scripts, alarm on-trigger scripts, and trigger expressions synchronously, as a pure prefix step of the deploy handler on the singleton's actor thread — deliberately not off-thread, because piping the verdict back to self would reorder concurrent deploys against delete/disable and break the redeploy-supersede mailbox-FIFO ordering (see the in-code rationale in DeploymentManagerActor.HandleDeploy); unchanged script bodies hit the process-wide compile cache, so the gate recompiles only genuinely new code. Any compile failure rejects the deployment with the collected compile errors…" (keep the existing honesty tail verbatim).
2. N6 — Component-SiteRuntime.md Alarm Evaluation section (~line 236, after the trigger-type list): add one bullet: "Expression trigger evaluation runs on the bounded script-execution thread pool shared with script bodies (and Expression script triggers): scheduler saturation — e.g. stuck scripts occupying all threads — delays Expression alarm transitions site-wide until a thread frees. This is a deliberate trade-off (evaluation no longer blocks dispatcher threads; per-actor coalescing bounds queue growth) and the scheduler gauges + stuck-script watchdog make the saturated state visible."
3. P6/N4 note refresh — Component-SiteRuntime.md:99 (the existing P6 follow-up note): append "As of the round-2 hardening, a process-wide compile cache dedupes identical script bodies within a node's process lifetime (the deploy gate's compile is reused by Instance Actor start), shrinking the recompile cost; the first compile after process start still runs inside Instance Actor start, so the deferral stands."
4. N5 (code-comment drift) — DeployCompileValidator.cs:16–18: the XML doc still says "it is safe to run off the singleton mailbox on a Task.Run thread — the deploy path never blocks on Roslyn", which describes the plan's original shape, not the shipped synchronous gate. Reword: "Because it holds no actor state and only calls the (stateless) ScriptCompilationService, it is pure and thread-safe; the Deployment Manager nevertheless invokes it synchronously on the singleton's actor thread as a pure prefix step, preserving mailbox-FIFO deploy ordering (see HandleDeploy). Compiles of unchanged script bodies are deduped by the process-wide compile cache."
5. Review with git diff docs/requirements/Component-SiteRuntime.md src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Deployment/DeployCompileValidator.cs — verify no other "off-thread" claim about the gate remains (grep -n "off-thread" docs/requirements/Component-SiteRuntime.md should return only the line-99 P6 note, which refers to the startup compiles, not the gate).
6. Commit: git add docs/requirements/Component-SiteRuntime.md src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Deployment/DeployCompileValidator.cs && git commit -m "docs(siteruntime): compile gate is synchronous by design; Expression alarms share the bounded script pool (plan R2-03 T7)"

---

Dependencies on other plans

• R2-05 (Templates/Deployment/Transport round 2) owns the Transport options-validator knob; R2-08 owns the AuditLog/Host validator gaps — Task 4 here covers ONLY the two SiteRuntime knobs (TagSubscribeRetryIntervalMs, StuckScriptGraceMs), per the cross-plan ruling.
• TemplateEngine's ScriptCompileVerdictCache stays untouched: it caches verdicts for central validation; Task 5/6's SiteScriptCompileCache caches compiled scripts for site execution. The two are intentionally separate (different result shapes, different assemblies); no shared abstraction is introduced.
• The site-side live-alarm-stream code (SiteStreamManager.SubscribeSiteAlarms) was reviewed clean in round 2 — nothing from the aggregated-live-alarm-stream plan lands here.

Execution order

P0 (do first): Task 1 (N1 — the only Medium; closes the last open S1 mechanism).

P1 (parallel, file-disjoint): Tasks 2, 3 (N2 — liveness of every Expression trigger/alarm), Task 4 (N3), Task 5 (N4 groundwork).

P2: Task 6 (after 5 — consumes the cache; comment-only touch on DeploymentManagerActor.cs).

Last: Task 7 (docs describe the landed behavior).

Same-file sequencing summary: ScriptCompilationService.cs: 6 only. DeploymentManagerActor.cs: 6 only (comment). DeployCompileValidator.cs: 7 only (XML doc). No file is touched by two tasks.

After the final task: dotnet build ZB.MOM.WW.ScadaBridge.slnx && dotnet test ZB.MOM.WW.ScadaBridge.slnx, and for cluster-runtime verification rebuild the docker image (bash docker/deploy.sh).

---

Findings Coverage

Finding	Severity	Task(s) / Disposition
N1 — MxGateway: stale event loop's non-OCE fault (RpcException(Cancelled)/ObjectDisposedException) can flap the fresh connection through the generic catch; lazy _eventLoopCts field read in the Task.Run lambda	Medium	1 (ct-guarded catch + token/client captured into locals; churn scenario regression-tested with both fault shapes)
N2 — expression-eval PipeTo has a success mapping only; a faulted scheduler task permanently kills the trigger with _evalInFlight stuck	Low	2 (ScriptActor), 3 (AlarmActor)
N3 — TagSubscribeRetryIntervalMs / StuckScriptGraceMs missing from the eager options validator	Low	4 (SiteRuntime knobs only — Transport → R2-05, AuditLog/Host → R2-08 per the cross-plan ruling)
N4 — every deploy compiles each script twice (synchronous gate + PreStart recompile), no compile cache	Low	5, 6 — full adoption ships here (one compile per script body per process; gate, PreStart, and SharedScriptLibrary all funnel through the cached CompileCore), so no deferred-register remainder. The gate itself stays synchronous by design (N5).
N5 — spec drift (reintroduced): Component-SiteRuntime.md:489 says the gate runs "off-thread"; the shipped gate is deliberately synchronous	Low	7 (spec sentence + the matching stale DeployCompileValidator XML doc)
N6 — Expression triggers/alarms share the bounded script scheduler; saturation silently stalls all Expression evaluation site-wide	Low	7 (documented as the accepted trade-off the report judged it to be — gauges/watchdog already make it visible; no behavior change)
P5 — per-attribute script read is an Ask round-trip (round-1 deferral)	Low	Deferred (accepted, unchanged) — spec-consistent, no profiling evidence; batch GetAttributes remains the profiling-gated follow-up. No task.
P6 — Roslyn compiles inside InstanceActor.PreStart during staggered startup (round-1 deferral)	Low	Deferred (accepted, further mitigated) — Tasks 5/6's cache dedupes identical bodies within a warm process (the report's own observation); the first compile after process start still runs in PreStart, and moving it out stays out of scope. Doc note refreshed in 7.
UA1 removal-reconcile / cross-node list-aggregation sub-scope (round-1 residue)	—	Deferred behind the documented central-persistence-of-cert-trust follow-up (Component-SiteRuntime.md:125). No task.
S1–S10, P1–P4, C2–C6, UA1–UA7 (round-1 findings)	—	Verified fixed by the round-2 report's disposition table (26 fixed, evidence-read) — no action.