ScadaBridge/archreview/plans/PLAN-R2-03-site-runtime-dcl.md

# PLAN-R2-03 — Site Runtime & DCL Round-2 Hardening Implementation Plan

> **For Claude:** REQUIRED SUB-SKILL: Use superpowers-extended-cc:executing-plans to implement this plan task-by-task.

**Goal:** Fix the round-2 findings in `archreview/03-site-runtime-dcl.md` (dated 2026-07-12) — the one Medium residual on the MxGateway reconnect path (a stale event loop's non-OCE fault can still flap the fresh connection through the generic catch) plus five Low polish items: missing failure mapping on the expression-eval `PipeTo`, two PLAN-03 option knobs absent from the eager options validator, the deploy-path double-compile (adopt the earmarked compile cache), the reintroduced "off-thread" spec-drift sentence, and the undocumented Expression-trigger/script-scheduler coupling. Round-1 deferrals P5 and P6 stay deferred (coverage-table rows only; Tasks 5/6's compile cache shrinks P6's warm-process cost as a side effect).

**Architecture:** All changes are site-side, inside `ZB.MOM.WW.ScadaBridge.SiteRuntime` and `ZB.MOM.WW.ScadaBridge.DataConnectionLayer`, plus their design doc. The established discipline is preserved: adapter lifecycle faults are classified on the *loop's own* cancellation token (never the mutable fields a newer generation may have replaced), every off-thread result — including a *faulted* task — returns to the actor via `PipeTo` and is applied on the actor thread, and options fail fast at boot with key-naming messages per the PLAN-08 §1.5 convention. The N4 compile cache mirrors TemplateEngine's `ScriptCompileVerdictCache` (SHA-256-keyed, bounded, clear-on-overflow) but is site-local and stores the full `ScriptCompilationResult` — sound because Roslyn `Script<T>` objects are immutable and thread-safe and the result's error text is name-free — so the synchronous deploy gate (deliberately on the singleton mailbox, see N5) and the `InstanceActor.PreStart` recompile share one Roslyn compile per script body per process lifetime. Docs travel with code.

**Tech Stack:** C#/.NET, Akka.NET (ReceiveActor, TestKit.Xunit2), Roslyn scripting, gRPC (Grpc.Core exception shapes via the `ZB.MOM.WW.MxGateway.Client` package), xunit + NSubstitute.

Build: `dotnet build ZB.MOM.WW.ScadaBridge.slnx`. Test per-project: `dotnet test tests/<project> --filter <name>`.

---

### Task 1: MxGatewayDataConnection — a stale event loop's fault on a cancelled token must not signal Disconnected

**Classification:** high-risk (adapter reconnect lifecycle, concurrency with the dying loop)
**Estimated implement time:** ~5 min
**Parallelizable with:** 2, 3, 4, 5
**Files:**
- Modify: `src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Adapters/MxGatewayDataConnection.cs` (`ConnectAsync` ~lines 106–108, `RunEventLoopAsync` ~lines 111–130)
- Test: `tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests/Adapters/MxGatewayDataConnectionReconnectTests.cs`

The S1 fix cancels the previous loop's CTS and disposes the previous client (`:77–89`), then resets `_disconnectFired` (`:91`) — but `Cancel()` is not synchronous with the loop's exit. The old loop's `await foreach` over the gRPC stream surfaces its failure milliseconds later, and a cancelled/disposed Grpc.Net stream commonly faults with `RpcException(StatusCode.Cancelled)` (the default without `ThrowOperationCanceledOnCancellation`) or `ObjectDisposedException` from the concurrent `DisposeAsync` — neither is an `OperationCanceledException`, so both fall into the generic `catch` (`:125–129`) → `RaiseDisconnected()` against the *reset* guard → the exact S1 scenario-2 flap, potentially self-sustaining.

1. Add failing tests to the existing reconnect test class (extend its per-client substitute fixture so each client's `RunEventLoopAsync` returns a controllable `TaskCompletionSource` task instead of `Task.Delay(Timeout.Infinite)`):

```csharp
[Fact]
public async Task StaleEventLoopRpcFault_AfterReconnect_DoesNotSignalDisconnected()
{
    var clients = new List<IMxGatewayClient>();
    var loops = new List<TaskCompletionSource>();
    var loopTokens = new List<CancellationToken>();
    var factory = Substitute.For<IMxGatewayClientFactory>();
    factory.Create().Returns(_ =>
    {
        var c = Substitute.For<IMxGatewayClient>();
        var tcs = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously);
        loops.Add(tcs);
        c.RunEventLoopAsync(Arg.Any<Action<MxValueUpdate>>(), Arg.Do<CancellationToken>(t => loopTokens.Add(t)))
            .Returns(tcs.Task);
        clients.Add(c);
        return c;
    });
    var adapter = new MxGatewayDataConnection(factory, NullLogger<MxGatewayDataConnection>.Instance);
    var disconnects = 0;
    adapter.Disconnected += () => Interlocked.Increment(ref disconnects);
    var details = new Dictionary<string, string> { ["Endpoint"] = "http://gw:5000", ["ApiKey"] = "k" };

    await adapter.ConnectAsync(details);
    await WaitUntilAsync(() => loopTokens.Count == 1);
    await adapter.ConnectAsync(details);          // reconnect: cancels loop #0, resets _disconnectFired
    await WaitUntilAsync(() => loopTokens.Count == 2);

    // The OLD loop observes its cancellation as a gRPC fault, not an OCE —
    // the Grpc.Net default without ThrowOperationCanceledOnCancellation.
    loops[0].SetException(new RpcException(new Status(StatusCode.Cancelled, "call cancelled")));
    await Task.Delay(200);
    Assert.Equal(0, disconnects);                 // stale fault absorbed — the fresh connection must not flap

    // The CURRENT loop's genuine fault must still signal, exactly once.
    loops[1].SetException(new RpcException(new Status(StatusCode.Unavailable, "gateway gone")));
    await WaitUntilAsync(() => disconnects == 1);
}

[Fact]
public async Task StaleEventLoopObjectDisposedFault_AfterReconnect_DoesNotSignalDisconnected()
{
    // Identical drive, but the old loop faults with the concurrent-DisposeAsync shape.
    // … same fixture …
    loops[0].SetException(new ObjectDisposedException("MxGatewayClient"));
    await Task.Delay(200);
    Assert.Equal(0, disconnects);
}
```

   (`RpcException`/`Status`/`StatusCode` come from `Grpc.Core`, transitively via the `ZB.MOM.WW.MxGateway.Client` package the test project already references — `RealMxGatewayClient.cs:227` catches `RpcException` in the main project. If the test assembly cannot resolve them, add a `Grpc.Net.Client` PackageReference; the version is already pinned in `Directory.Packages.props`.)
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests --filter "FullyQualifiedName~StaleEventLoopRpcFault_AfterReconnect|FullyQualifiedName~StaleEventLoopObjectDisposedFault_AfterReconnect"` → expect **FAIL** (the generic catch raises `Disconnected` against the reset guard: `disconnects == 1` after the stale fault).
3. Implement:
   - Change `RunEventLoopAsync` to take the client AND token as parameters — `private async Task RunEventLoopAsync(IMxGatewayClient client, CancellationToken ct)` — and use `client.RunEventLoopAsync(…)` instead of the `_client!` field read (the old loop must never bind a newer generation's client).
   - Add a filtered catch **before** the generic one:

```csharp
catch (Exception ex) when (ct.IsCancellationRequested)
{
    // Teardown fault of a superseded loop (N1): Cancel() is not synchronous with
    // the loop's exit — a cancelled/disposed gRPC stream commonly surfaces as
    // RpcException(StatusCode.Cancelled) (Grpc.Net default without
    // ThrowOperationCanceledOnCancellation) or ObjectDisposedException from the
    // concurrent DisposeAsync, milliseconds AFTER ConnectAsync has already reset
    // _disconnectFired. Any fault on a cancelled token is normal shutdown;
    // raising Disconnected here would flap the NEW healthy session.
    _logger.LogDebug(ex,
        "MxGateway event loop faulted after cancellation — superseded loop teardown; not signalling disconnect");
}
```

   - In `ConnectAsync`, capture the token and client into locals before the lambda (a rapid double-reconnect must not hand loop #1 loop #2's token/client — the current `Task.Run(() => RunEventLoopAsync(_eventLoopCts.Token))` reads the *field* lazily):

```csharp
_eventLoopCts = new CancellationTokenSource();
var loopToken = _eventLoopCts.Token;   // capture NOW: the field may be replaced by a
var loopClient = _client;              // subsequent reconnect before Task.Run executes (N1)
_ = Task.Run(() => RunEventLoopAsync(loopClient, loopToken));
```

4. Run the two new tests → **PASS**; then the full adapter suite: `dotnet test tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests --filter "FullyQualifiedName~MxGatewayDataConnection"` → all green (the existing `Reconnect_DisposesPreviousClient_AndCancelsPreviousEventLoop` test pins the S1 behavior and must survive the signature change).
5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Adapters/MxGatewayDataConnection.cs tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests/Adapters/MxGatewayDataConnectionReconnectTests.cs && git commit -m "fix(dcl): stale MxGateway event-loop fault on a cancelled token no longer flaps the fresh connection (plan R2-03 T1)"`

### Task 2: ScriptActor — failure mapping on the expression-eval `PipeTo` (a faulted scheduler task must not kill the trigger)

**Classification:** high-risk (actor concurrency, trigger liveness)
**Estimated implement time:** ~5 min
**Parallelizable with:** 1, 3, 4, 5, 6
**Files:**
- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/ScriptActor.cs` (Receive block ~line 161, `StartExpressionEvaluation` PipeTo ~lines 311–313, internal messages ~line 623)
- Test: `tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/ScriptActorTests.cs`

`StartExpressionEvaluation` pipes with `success:` only (`:312–313`). The inner body catches everything, so the only fault source is the outer `Task.Factory.StartNew` itself (e.g. `ObjectDisposedException` from a disposed `ScriptExecutionScheduler` during shutdown/test teardown) — but if it ever fires, the actor gets an unhandled `Status.Failure` and `_evalInFlight` stays `true` forever (`:284–285`): every subsequent change parks in `_evalPending` and the expression trigger is dead for the actor's lifetime with no log.

1. Failing test. Add a public static gate hook to the test file (the `CompileRawTriggerExpression` helper compiles *without* the trust validator, so the expression may reference it — same trick as the P1 thread-name tests):

```csharp
public static class EvalGate
{
    public static readonly SemaphoreSlim Gate = new(0);
    private static int _entries;
    public static int Entries => Volatile.Read(ref _entries);
    public static bool Block() { Interlocked.Increment(ref _entries); Gate.Wait(); return false; }
}

[Fact]
public void ExpressionEvalTaskFault_ClearsInFlight_AndDrainsPendingEvaluation()
{
    var expr = CompileRawTriggerExpression(
        "ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.Actors.EvalGate.Block()");
    var (actor, _) = CreateTriggeredActor(
        "ExprFault", "Expression", "{\"expression\":\"true\",\"mode\":\"OnTrue\"}", null, expr);
    try
    {
        actor.Tell(Change("A", "1"));                              // eval starts on the scheduler and BLOCKS → _evalInFlight = true
        AwaitAssert(() => Assert.Equal(1, EvalGate.Entries), TimeSpan.FromSeconds(10));
        actor.Tell(Change("A", "2"));                              // coalesces → _evalPending = true

        // Simulate the faulted scheduler task the PipeTo failure mapping now surfaces
        // (e.g. ObjectDisposedException from a disposed ScriptExecutionScheduler).
        actor.Tell(new ScriptActor.ExpressionEvalFailed(new ObjectDisposedException("scheduler")));

        // In-flight cleared + pending drained ⇒ a SECOND evaluation starts and blocks too.
        AwaitAssert(() => Assert.Equal(2, EvalGate.Entries), TimeSpan.FromSeconds(10));
    }
    finally
    {
        EvalGate.Gate.Release(EvalGate.Entries); // ALWAYS free the shared scheduler threads
    }
}
```

   (Note the expression must reference the gate's fully-qualified name; the raw compile references `typeof(object).Assembly` only, so also add the test assembly to `CompileRawTriggerExpression`'s `.WithReferences(...)` for this test — or add an overload taking extra references. The `finally` release is mandatory: the gate blocks threads of the *process-wide* `ScriptExecutionScheduler.Shared`.)
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~ExpressionEvalTaskFault_ClearsInFlight"` → expect **FAIL** — the RED step is the compile failure (`ScriptActor.ExpressionEvalFailed` does not exist); behaviorally, without the fix the second evaluation never starts either.
3. Implement:
   - New message next to `ExpressionEvalResult` (`:623`) — `internal` (not `private`) so the TestKit can Tell it, same visibility idiom as `IntervalTick`/`ScriptExecutionCompleted`:

```csharp
/// <summary>
/// Piped back to self when the off-dispatcher evaluation TASK itself faults
/// (e.g. ObjectDisposedException from a disposed ScriptExecutionScheduler
/// during shutdown) — the inner body's catch never sees that. Without this
/// mapping the actor receives an unhandled Status.Failure and _evalInFlight
/// stays true forever, permanently parking the expression trigger (N2).
/// </summary>
internal sealed record ExpressionEvalFailed(Exception Cause);
```

   - Ctor Receive block (after the `ExpressionEvalResult` registration at `:161`): `Receive<ExpressionEvalFailed>(HandleExpressionEvalFailed);`
   - PipeTo (`:312–313`): `…​.PipeTo(self, success: r => new ExpressionEvalResult(r), failure: ex => new ExpressionEvalFailed(ex));`
   - Handler — log via the existing `LogExpressionError` (`:397`, already increments the health counter), then reuse the result path with `false` (consistent with the inner catch's treated-as-false semantics — clears in-flight, applies the false edge, drains pending):

```csharp
private void HandleExpressionEvalFailed(ExpressionEvalFailed msg)
{
    LogExpressionError(msg.Cause);
    HandleExpressionEvalResult(new ExpressionEvalResult(false));
}
```

4. Run the new test → **PASS**; then `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~ScriptActorTests"` → all green (the P1 expression/WhileTrue suites are the regression canary).
5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/ScriptActor.cs tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/ScriptActorTests.cs && git commit -m "fix(siteruntime): faulted expression-eval task clears _evalInFlight instead of killing the ScriptActor trigger (plan R2-03 T2)"`

### Task 3: AlarmActor — same failure mapping on the expression-eval `PipeTo`

**Classification:** high-risk (alarm state machine liveness)
**Estimated implement time:** ~4 min
**Parallelizable with:** 1, 2, 4, 5, 6
**Files:**
- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/AlarmActor.cs` (Receive block ~line 188, `StartExpressionEvaluation` PipeTo ~lines 552–554, internal messages ~line 797)
- Test: `tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/AlarmActorTests.cs`

Structurally identical to Task 2 — `AlarmActor.cs:552–554` pipes success-only, `:523–524` sets `_evalInFlight`, so a faulted task permanently kills the Expression alarm.

1. Failing test (reuse the `ExpressionAlarm_EvaluatesOnSchedulerThread_AndActivates` harness in `AlarmActorTests.cs:1031` — same `ResolvedAlarm`/`instanceProbe` construction):

```csharp
[Fact]
public void ExpressionEvalTaskFault_DoesNotKillTheAlarmTrigger()
{
    var expr = CompileRawTriggerExpression("true");
    var alarmConfig = new ResolvedAlarm
    {
        CanonicalName = "ExprFaultAlarm",
        TriggerType = "Expression",
        TriggerConfiguration = "{\"expression\":\"true\"}",
        PriorityLevel = 1
    };
    var instanceProbe = CreateTestProbe();
    var alarm = ActorOf(Props.Create(() => new AlarmActor(
        "ExprFaultAlarm", "Pump1", instanceProbe.Ref, alarmConfig,
        null, _sharedLibrary, _options,
        NullLogger<AlarmActor>.Instance, expr)));

    // Simulate the faulted scheduler task: must be handled (logged, in-flight
    // cleared, false applied) — NOT an unhandled Status.Failure.
    alarm.Tell(new AlarmActor.ExpressionEvalFailed(new ObjectDisposedException("scheduler")));

    // The trigger must still be alive: the next change evaluates and activates.
    alarm.Tell(new AttributeValueChanged("Pump1", "A", "A", 1, "Good", DateTimeOffset.UtcNow));
    var change = instanceProbe.ExpectMsg<AlarmStateChanged>(TimeSpan.FromSeconds(10));
    Assert.Equal(AlarmState.Active, change.State);
}
```

2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~ExpressionEvalTaskFault_DoesNotKillTheAlarmTrigger"` → expect **FAIL** (compile error — `AlarmActor.ExpressionEvalFailed` does not exist).
3. Implement, mirroring Task 2:
   - `internal sealed record ExpressionEvalFailed(Exception Cause);` next to the existing `ExpressionEvalResult` (`:797`), with the same N2 XML doc.
   - `Receive<ExpressionEvalFailed>(HandleExpressionEvalFailed);` after `:188`.
   - PipeTo (`:553–554`) gains `failure: ex => new ExpressionEvalFailed(ex)`.
   - Handler — same shape as the inner catch's error path, then delegate to the false result (the class doc already promises "a throwing … expression is treated as false so that the state machine still runs — an Active alarm correctly clears"):

```csharp
private void HandleExpressionEvalFailed(ExpressionEvalFailed msg)
{
    _healthCollector?.IncrementAlarmError();
    _logger.LogError(msg.Cause,
        "Alarm {Alarm} trigger-expression evaluation task faulted on {Instance}; treated as false",
        _alarmName, _instanceName);
    HandleExpressionEvalResult(new ExpressionEvalResult(false));
}
```

4. Run the new test → **PASS**; then `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~AlarmActorTests"` → all green.
5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/AlarmActor.cs tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/AlarmActorTests.cs && git commit -m "fix(siteruntime): faulted expression-eval task clears _evalInFlight instead of killing the AlarmActor trigger (plan R2-03 T3)"`

### Task 4: Eagerly validate `TagSubscribeRetryIntervalMs` and `StuckScriptGraceMs`

**Classification:** small (additive validator rules per the PLAN-08 §1.5 convention)
**Estimated implement time:** ~3 min
**Parallelizable with:** 1, 2, 3, 5, 6
**Files:**
- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/SiteRuntimeOptionsValidator.cs` (append after the `ConfigFetchRetryCount` rule, lines 56–58)
- Test: `tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/SiteRuntimeOptionsValidatorTests.cs`

The validator covers ten options but not the two PLAN-03 added: `TagSubscribeRetryIntervalMs` (`SiteRuntimeOptions.cs:76` — a 0 hot-loops the subscribe retry, a negative value throws inside `InstanceActor.SendTagSubscribe`'s `ScheduleTellOnceCancelable`) and `StuckScriptGraceMs` (`SiteRuntimeOptions.cs:86` — a negative value throws `ArgumentOutOfRangeException` inside the watchdog's `Task.Delay`, `ScriptExecutionActor.cs:144`).

1. Failing tests (mirror the existing `Validate(...)` helper and assertion shape in the test file):

```csharp
[Fact]
public void ZeroTagSubscribeRetryIntervalMs_IsRejected()
{
    var result = Validate(new SiteRuntimeOptions { TagSubscribeRetryIntervalMs = 0 });

    Assert.True(result.Failed);
    Assert.Contains("TagSubscribeRetryIntervalMs", result.FailureMessage);
}

[Fact]
public void NegativeStuckScriptGraceMs_IsRejected()
{
    var result = Validate(new SiteRuntimeOptions { StuckScriptGraceMs = -1 });

    Assert.True(result.Failed);
    Assert.Contains("StuckScriptGraceMs", result.FailureMessage);
}
```

2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~SiteRuntimeOptionsValidatorTests"` → the two new tests **FAIL** (validator succeeds on both).
3. Implement — two `RequireThat` lines matching the file's key-naming message style:

```csharp
builder.RequireThat(options.TagSubscribeRetryIntervalMs > 0,
    $"ScadaBridge:SiteRuntime:TagSubscribeRetryIntervalMs must be greater than 0 " +
    $"(was {options.TagSubscribeRetryIntervalMs}); a zero interval hot-loops the tag-subscribe " +
    "retry and a negative one throws inside the Instance Actor's retry scheduling.");

builder.RequireThat(options.StuckScriptGraceMs >= 0,
    $"ScadaBridge:SiteRuntime:StuckScriptGraceMs must be >= 0 " +
    $"(was {options.StuckScriptGraceMs}); a negative grace throws inside the stuck-script watchdog's delay.");
```

4. Run → **PASS**, including `DefaultOptions_AreValid` (defaults 5000/30000 satisfy both rules).
5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.SiteRuntime/SiteRuntimeOptionsValidator.cs tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/SiteRuntimeOptionsValidatorTests.cs && git commit -m "fix(siteruntime): eagerly validate TagSubscribeRetryIntervalMs + StuckScriptGraceMs at boot (plan R2-03 T4)"`

### Task 5: SiteScriptCompileCache — bounded process-wide compiled-script cache (N4 groundwork)

**Classification:** standard (pure new class, no consumers yet)
**Estimated implement time:** ~5 min
**Parallelizable with:** 1, 2, 3, 4
**Files:**
- Create: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Scripts/SiteScriptCompileCache.cs`
- Test: create `tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Scripts/SiteScriptCompileCacheTests.cs`

This is the site-side adoption of the compile-cache abstraction the master tracker earmarked when PLAN-03's Task 3/4 shipped ("later refactor, not a blocker") — TemplateEngine's `ScriptCompileVerdictCache` pattern, but storing the full `ScriptCompilationResult` so both the deploy gate and the `PreStart` recompile reuse the compiled `Script<object?>` itself. Soundness: the result is a pure function of (code, globals type, compile-time-static trust policy); the stored error text is name-free (`Diagnostic.GetMessage()` / `"Compilation exception: …"` — callers prefix their own context); Roslyn `Script<T>` is immutable and thread-safe (`RunAsync` creates per-run state), so one instance may be shared across actors.

1. Failing tests (the type does not exist — compile failure is the RED step). The cache is process-wide static state: put this test class and Task 6's `ScriptCompilationServiceTests` additions in one xunit collection (`[Collection("SiteScriptCompileCache")]`) and call `Clear()` at the top of each test, so parallel classes cannot skew `Hits`/`Count`:

```csharp
[Collection("SiteScriptCompileCache")]
public class SiteScriptCompileCacheTests
{
    private static ScriptCompilationResult Ok() =>
        ScriptCompilationResult.Succeeded(CSharpScript.Create<object?>("1", ScriptOptions.Default, typeof(ScriptGlobals)));

    [Fact]
    public void GetOrAdd_SameCodeAndGlobals_ComputesOnce()
    {
        SiteScriptCompileCache.Clear();
        var factoryCalls = 0;
        ScriptCompilationResult Factory() { factoryCalls++; return Ok(); }

        var r1 = SiteScriptCompileCache.GetOrAdd("return 1;", typeof(ScriptGlobals), Factory);
        var r2 = SiteScriptCompileCache.GetOrAdd("return 1;", typeof(ScriptGlobals), Factory);

        Assert.Equal(1, factoryCalls);
        Assert.Same(r1, r2);
        Assert.Equal(1, SiteScriptCompileCache.Hits);
    }

    [Fact]
    public void GetOrAdd_SameCode_DifferentGlobals_AreSeparateEntries()
    {
        SiteScriptCompileCache.Clear();
        SiteScriptCompileCache.GetOrAdd("1 > 0", typeof(ScriptGlobals), Ok);
        SiteScriptCompileCache.GetOrAdd("1 > 0", typeof(TriggerExpressionGlobals), Ok);

        Assert.Equal(2, SiteScriptCompileCache.Count);
        Assert.Equal(0, SiteScriptCompileCache.Hits); // no cross-globals hit
    }

    [Fact]
    public void Overflow_ClearsWholesale()
    {
        SiteScriptCompileCache.Clear();
        for (var i = 0; i <= SiteScriptCompileCache.MaxEntries; i++)
            SiteScriptCompileCache.GetOrAdd($"return {i};", typeof(ScriptGlobals), Ok);

        Assert.True(SiteScriptCompileCache.Count <= SiteScriptCompileCache.MaxEntries);
    }
}
```

2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~SiteScriptCompileCacheTests"` → **FAIL** (type does not exist).
3. Implement, mirroring `ScriptCompileVerdictCache` (`src/ZB.MOM.WW.ScadaBridge.TemplateEngine/Validation/ScriptCompileVerdictCache.cs`): `internal static class SiteScriptCompileCache` (InternalsVisibleTo already covers the test assembly) with
   - `internal const int MaxEntries = 1024;` — deliberately smaller than the verdict cache's 4096: entries pin compiled assemblies, not just verdict strings; on overflow the cache is cleared wholesale (results are recomputable, coarse reset avoids eviction bookkeeping).
   - Key = `globalsType.FullName + ":" + Convert.ToHexString(SHA256.HashData(Encoding.UTF8.GetBytes(code)))` — the globals discriminator keeps `ScriptGlobals` and `TriggerExpressionGlobals` compiles of identical source apart.
   - `public static ScriptCompilationResult GetOrAdd(string code, Type globalsType, Func<ScriptCompilationResult> factory)` over a `ConcurrentDictionary<string, ScriptCompilationResult>`, incrementing `Hits` on a hit; `Hits`/`Count`/`Clear()` exposed for tests and diagnostics — same surface as the TemplateEngine cache.
   - XML doc stating the three soundness invariants above (pure function of code+globals+static policy; name-free errors; `Script<T>` immutability) and that **failures are cached too** — a bad script body fails identically for every caller.
4. Run → **PASS**.
5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Scripts/SiteScriptCompileCache.cs tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Scripts/SiteScriptCompileCacheTests.cs && git commit -m "feat(siteruntime): bounded process-wide compiled-script cache, keyed on code hash + globals type (plan R2-03 T5)"`

### Task 6: Wire the cache into ScriptCompilationService — deploy gate + PreStart share one compile per script per revision

**Classification:** standard (single choke point; all call sites inherit)
**Estimated implement time:** ~4 min
**Parallelizable with:** 1, 2, 3, 4 (needs 5 merged)
**Files:**
- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Scripts/ScriptCompilationService.cs` (`CompileCore`, ~lines 98–139)
- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/DeploymentManagerActor.cs` (gate rationale comment ~lines 510–521 — comment only)
- Test: `tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Scripts/ScriptCompilationServiceTests.cs`

`CompileCore` is the single funnel for every site-side compile — the `DeployCompileValidator` gate (`DeploymentManagerActor.cs:522`), the `InstanceActor.CreateChildActors` recompile (`InstanceActor.cs:1485, 1497–1498`), and `SharedScriptLibrary` all flow through it — so caching here collapses the 2×N-compiles-per-bulk-redeploy cost (N4) and shrinks P6's warm-process staggered-startup recompiles for free, with zero call-site changes.

1. Failing test (add to `ScriptCompilationServiceTests`, tagged `[Collection("SiteScriptCompileCache")]` per Task 5's isolation note):

```csharp
[Fact]
public void Compile_SameCodeTwice_SharesOneRoslynCompile()
{
    SiteScriptCompileCache.Clear();
    var r1 = _service.Compile("deploy-gate-copy", "return 1 + 1;");
    var r2 = _service.Compile("prestart-copy", "return 1 + 1;");

    Assert.True(r1.IsSuccess);
    Assert.Same(r1.CompiledScript, r2.CompiledScript);   // one compile, shared Script<T> (N4)
    Assert.Equal(1, SiteScriptCompileCache.Hits);
}

[Fact]
public void Compile_ScriptAndTriggerExpression_DoNotCrossContaminate()
{
    SiteScriptCompileCache.Clear();
    var script = _service.Compile("s", "1 > 0");
    var trigger = _service.CompileTriggerExpression("t", "1 > 0");

    Assert.NotSame(script.CompiledScript, trigger.CompiledScript); // different globals surfaces
}
```

2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~Compile_SameCodeTwice_SharesOneRoslynCompile|FullyQualifiedName~Compile_ScriptAndTriggerExpression_DoNotCrossContaminate"` → **FAIL** (`Assert.Same` fails: two independent compiles today).
3. Implement:
   - Rename the current `CompileCore` body to `private ScriptCompilationResult CompileUncached(string name, string code, Type globalsType)` (byte-identical, including the trust-model check — the verdict is deterministic on code, so it is cacheable with the compile).
   - New `CompileCore`:

```csharp
private ScriptCompilationResult CompileCore(string name, string code, Type globalsType)
{
    var result = SiteScriptCompileCache.GetOrAdd(code, globalsType,
        () => CompileUncached(name, code, globalsType));
    if (!result.IsSuccess)
        _logger.LogDebug(
            "Script {Script}: returning cached compile failure ({ErrorCount} error(s))",
            name, result.Errors.Count); // miss-path logs name the FIRST caller; keep this caller visible
    return result;
}
```

   - Update the class XML doc: results (success AND failure) are memoised process-wide in `SiteScriptCompileCache`; error text is name-free by construction, so cached failures render correctly for every caller.
   - Append one sentence to the `HandleDeploy` gate comment (`DeploymentManagerActor.cs:519–521`): "Redeploys and multi-instance deploys of unchanged scripts hit the process-wide compile cache (`SiteScriptCompileCache`), so the synchronous gate recompiles only genuinely new code and the Instance Actor's PreStart reuses the gate's compile (N4)."
4. Run the two new tests → **PASS**; then the funnel's consumer suites: `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~ScriptCompilationServiceTests|FullyQualifiedName~DeployCompileValidatorTests|FullyQualifiedName~SharedScriptLibraryTests|FullyQualifiedName~TrustModelSemanticTests"` → all green (these pin that caching changed no verdicts).
5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Scripts/ScriptCompilationService.cs src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/DeploymentManagerActor.cs tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Scripts/ScriptCompilationServiceTests.cs && git commit -m "perf(siteruntime): deploy gate + InstanceActor PreStart share one Roslyn compile via the compile cache (plan R2-03 T6)"`

### Task 7: Design-doc sync — synchronous compile gate (N5), Expression/scheduler coupling (N6), P6/N4 note refresh

**Classification:** trivial (docs + one stale XML doc; repo rule: docs travel with code)
**Estimated implement time:** ~4 min
**Parallelizable with:** none (write after the behavior tasks land so the docs describe reality)
**Files:**
- Modify: `docs/requirements/Component-SiteRuntime.md` (lines ~489, ~234–246, ~99)
- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Deployment/DeployCompileValidator.cs` (class XML doc, ~lines 16–18)

1. **N5** — `Component-SiteRuntime.md:489`: the sentence claims the gate compiles "**off-thread**"; the shipped gate is deliberately synchronous (`DeploymentManagerActor.cs:510–522`). Replace the off-thread clause so the bullet reads: "…the Deployment Manager compiles **all** of the instance's scripts, alarm on-trigger scripts, and trigger expressions **synchronously, as a pure prefix step of the deploy handler on the singleton's actor thread** — deliberately not off-thread, because piping the verdict back to self would reorder concurrent deploys against delete/disable and break the redeploy-supersede mailbox-FIFO ordering (see the in-code rationale in `DeploymentManagerActor.HandleDeploy`); unchanged script bodies hit the process-wide compile cache, so the gate recompiles only genuinely new code. Any compile failure rejects the deployment with the collected compile errors…" (keep the existing honesty tail verbatim).
2. **N6** — `Component-SiteRuntime.md` Alarm Evaluation section (~line 236, after the trigger-type list): add one bullet: "**Expression** trigger evaluation runs on the bounded script-execution thread pool shared with script bodies (and Expression *script* triggers): scheduler saturation — e.g. stuck scripts occupying all threads — delays Expression alarm transitions site-wide until a thread frees. This is a deliberate trade-off (evaluation no longer blocks dispatcher threads; per-actor coalescing bounds queue growth) and the scheduler gauges + stuck-script watchdog make the saturated state visible."
3. **P6/N4 note refresh** — `Component-SiteRuntime.md:99` (the existing P6 follow-up note): append "As of the round-2 hardening, a process-wide compile cache dedupes identical script bodies within a node's process lifetime (the deploy gate's compile is reused by Instance Actor start), shrinking the recompile cost; the *first* compile after process start still runs inside Instance Actor start, so the deferral stands."
4. **N5 (code-comment drift)** — `DeployCompileValidator.cs:16–18`: the XML doc still says "it is safe to run off the singleton mailbox on a `Task.Run` thread — the deploy path never blocks on Roslyn", which describes the plan's original shape, not the shipped synchronous gate. Reword: "Because it holds no actor state and only calls the (stateless) `ScriptCompilationService`, it is pure and thread-safe; the Deployment Manager nevertheless invokes it *synchronously on the singleton's actor thread* as a pure prefix step, preserving mailbox-FIFO deploy ordering (see `HandleDeploy`). Compiles of unchanged script bodies are deduped by the process-wide compile cache."
5. Review with `git diff docs/requirements/Component-SiteRuntime.md src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Deployment/DeployCompileValidator.cs` — verify no other "off-thread" claim about the gate remains (`grep -n "off-thread" docs/requirements/Component-SiteRuntime.md` should return only the line-99 P6 note, which refers to the *startup* compiles, not the gate).
6. Commit: `git add docs/requirements/Component-SiteRuntime.md src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Deployment/DeployCompileValidator.cs && git commit -m "docs(siteruntime): compile gate is synchronous by design; Expression alarms share the bounded script pool (plan R2-03 T7)"`

---

## Dependencies on other plans

- **R2-05 (Templates/Deployment/Transport round 2)** owns the Transport options-validator knob; **R2-08** owns the AuditLog/Host validator gaps — Task 4 here covers ONLY the two SiteRuntime knobs (`TagSubscribeRetryIntervalMs`, `StuckScriptGraceMs`), per the cross-plan ruling.
- **TemplateEngine's `ScriptCompileVerdictCache`** stays untouched: it caches *verdicts* for central validation; Task 5/6's `SiteScriptCompileCache` caches *compiled scripts* for site execution. The two are intentionally separate (different result shapes, different assemblies); no shared abstraction is introduced.
- The site-side live-alarm-stream code (`SiteStreamManager.SubscribeSiteAlarms`) was reviewed clean in round 2 — nothing from the aggregated-live-alarm-stream plan lands here.

## Execution order

**P0 (do first):** Task 1 (N1 — the only Medium; closes the last open S1 mechanism).

**P1 (parallel, file-disjoint):** Tasks 2, 3 (N2 — liveness of every Expression trigger/alarm), Task 4 (N3), Task 5 (N4 groundwork).

**P2:** Task 6 (after 5 — consumes the cache; comment-only touch on `DeploymentManagerActor.cs`).

**Last:** Task 7 (docs describe the landed behavior).

Same-file sequencing summary: `ScriptCompilationService.cs`: 6 only. `DeploymentManagerActor.cs`: 6 only (comment). `DeployCompileValidator.cs`: 7 only (XML doc). No file is touched by two tasks.

After the final task: `dotnet build ZB.MOM.WW.ScadaBridge.slnx && dotnet test ZB.MOM.WW.ScadaBridge.slnx`, and for cluster-runtime verification rebuild the docker image (`bash docker/deploy.sh`).

---

## Findings Coverage

| Finding | Severity | Task(s) / Disposition |
|---|---|---|
| N1 — MxGateway: stale event loop's non-OCE fault (`RpcException(Cancelled)`/`ObjectDisposedException`) can flap the fresh connection through the generic catch; lazy `_eventLoopCts` field read in the `Task.Run` lambda | Medium | 1 (ct-guarded catch + token/client captured into locals; churn scenario regression-tested with both fault shapes) |
| N2 — expression-eval `PipeTo` has a success mapping only; a faulted scheduler task permanently kills the trigger with `_evalInFlight` stuck | Low | 2 (ScriptActor), 3 (AlarmActor) |
| N3 — `TagSubscribeRetryIntervalMs` / `StuckScriptGraceMs` missing from the eager options validator | Low | 4 (SiteRuntime knobs only — Transport → R2-05, AuditLog/Host → R2-08 per the cross-plan ruling) |
| N4 — every deploy compiles each script twice (synchronous gate + `PreStart` recompile), no compile cache | Low | 5, 6 — **full adoption ships here** (one compile per script body per process; gate, PreStart, and SharedScriptLibrary all funnel through the cached `CompileCore`), so no deferred-register remainder. The gate itself stays synchronous by design (N5). |
| N5 — spec drift (reintroduced): `Component-SiteRuntime.md:489` says the gate runs "off-thread"; the shipped gate is deliberately synchronous | Low | 7 (spec sentence + the matching stale `DeployCompileValidator` XML doc) |
| N6 — Expression triggers/alarms share the bounded script scheduler; saturation silently stalls all Expression evaluation site-wide | Low | 7 (documented as the accepted trade-off the report judged it to be — gauges/watchdog already make it visible; no behavior change) |
| P5 — per-attribute script read is an Ask round-trip (round-1 deferral) | Low | **Deferred (accepted, unchanged)** — spec-consistent, no profiling evidence; batch `GetAttributes` remains the profiling-gated follow-up. No task. |
| P6 — Roslyn compiles inside `InstanceActor.PreStart` during staggered startup (round-1 deferral) | Low | **Deferred (accepted, further mitigated)** — Tasks 5/6's cache dedupes identical bodies within a warm process (the report's own observation); the first compile after process start still runs in `PreStart`, and moving it out stays out of scope. Doc note refreshed in 7. |
| UA1 removal-reconcile / cross-node list-aggregation sub-scope (round-1 residue) | — | **Deferred** behind the documented central-persistence-of-cert-trust follow-up (`Component-SiteRuntime.md:125`). No task. |
| S1–S10, P1–P4, C2–C6, UA1–UA7 (round-1 findings) | — | **Verified fixed** by the round-2 report's disposition table (26 fixed, evidence-read) — no action. |