Files

T

Joseph Doherty ef69fab6f6 docs(archreview): register PLAN-03 Task 15 readiness-handshake deviation as a follow-up (owner: Task 26 doc capture)

2026-07-09 00:36:20 -04:00

98 KiB

Raw Blame History

Site Runtime & DCL Hardening Implementation Plan

For Claude: REQUIRED SUB-SKILL: Use superpowers-extended-cc:executing-plans to implement this plan task-by-task.

Goal: Fix every finding in archreview/03-site-runtime-dcl.md — adapter reconnect lifecycle leaks, script-containment observability, deployment honesty (site-side compile failure must reject the deployment per spec), subscription/startup retry holes, dead-child bookkeeping, hot-path fan-out cost, and site SQLite hardening — with design docs updated alongside the code.

Architecture: All changes are site-side, inside ZB.MOM.WW.ScadaBridge.SiteRuntime and ZB.MOM.WW.ScadaBridge.DataConnectionLayer (plus additive message/report contracts in Commons/HealthMonitoring). The actor model is preserved: every fix keeps state mutation on the actor thread (background work returns via PipeTo), retries use the existing fixed-interval timer idiom, and new health metrics ride the existing ISiteHealthCollector → SiteHealthReport path with additive-only contract changes. The deploy compile gate runs off-thread before actor creation so the singleton mailbox never blocks on Roslyn.

Tech Stack: C#/.NET, Akka.NET (ReceiveActor/UntypedActor, TestKit.Xunit2), Roslyn scripting, Microsoft.Data.Sqlite, xunit + NSubstitute.

Build: dotnet build ZB.MOM.WW.ScadaBridge.slnx. Test per-project: dotnet test tests/<project> --filter <name>.

Findings Coverage

Finding	Severity	Task(s) / Disposition
S1 — reconnect abandons previous client, stale `ConnectionLost` flaps healthy connection	High	1 (OPC UA), 2 (MxGateway)
S2 — cooperative-only timeout; no stuck-thread observability	High	5, 6, 7, 8. Sub-suggestion "inject ADO.NET command timeouts into Database.Connection" deferred: scripts own their ADO.NET commands (`SqlCommand.CommandTimeout` defaults to 30 s already); the watchdog + gauges make the residual case visible.
S3 — site compile failure doesn't reject deployment (spec drift)	High	3, 4
S4 — no retry on failed tag subscription; NativeAlarm lost-response gap	Medium	11 (InstanceActor), 12 (NativeAlarmActor)
S5 — startup SQLite load failure aborts silently, no retry	Medium	13
S6 — dead child ref in `_instanceActors` + Success reported for dead instance	Medium	14, 15
S7 — background tasks read mutable `_adapter` field off-thread	Medium	16
S8 — site SQLite: no WAL/busy-timeout	Low	19
S9 — fire-and-forget adapter dispose in `PostStop` unobserved	Low	25
S10 — alarm condition filter last-subscriber-wins, silent overwrite	Low	25
P1 — trigger expressions block dispatcher threads per attribute change	High	9 (ScriptActor), 10 (AlarmActor)
P2 — attribute-change fan-out broadcast to every child	Medium	18
P3 — DCL tag-value fan-out scans all instances per update	Medium	17
P4 — per-transition SQLite upserts on native-alarm mirror	Low	20
P5 — per-attribute script read is an Ask round-trip	Low	Deferred — correct and spec-consistent; no profiling evidence yet; batch `GetAttributes` is a follow-up when profiling flags it.
P6 — Roslyn compiles inside `InstanceActor.PreStart` during staggered startup	Low	Deferred (partially mitigated) — Task 3/4 moves deploy-path compiles off the singleton mailbox; moving startup-path compiles out of `PreStart` requires an async child-creation phase (messages arriving before children exist) and is out of scope; risk is failover time-to-recover only. Logged as follow-up in Task 26 doc note.
C1 — positive (Akka discipline)	—	No action.
C2 — stale "unbounded channel" comment in SiteEventLogger	Low	25
C3 — spec self-contradiction on reconnect interval scope	Low	26
C4 — quality-only change not delivered to children vs. spec wording	Low	26
C5 — shared scripts run Root scope, undocumented	Low	26
C6 — `EvaluateCondition` fallback uses current-culture `ToString`	Low	25
UA1 — cert-trust convergence after node downtime	Underdev.	22, 23 (+ doc in 26). List-aggregation across both nodes and removal-reconcile deferred to the documented central-persistence follow-up (`Component-SiteRuntime.md:125`).
UA2 — `ConfigFetchRetryCount` dead option	Underdev.	24
UA3 — no aggregated live alarm stream	Underdev.	Deferred — documented M7 follow-up, central-side; out of this plan's site-runtime scope.
UA4 — native-alarm rehydration loses display metadata	Underdev.	21
UA5 — no scheduler stuck-thread/queue-depth observability	Underdev.	5, 6, 7, 8 (same fix as S2)
UA6 — tag-subscription failure path has no retry/site event	Underdev.	11, 12 (same fix as S4)
UA7 — test gaps (reconnect-stale-handler, compile-fail deploy status, subscribe race)	Underdev.	Covered by the TDD tests in Tasks 1, 2, 4, 11

Task 1: OpcUaDataConnection — dispose + detach the previous client on reconnect

Classification: high-risk (adapter lifecycle, concurrency with SDK callback threads) Estimated implement time: ~5 min Parallelizable with: 2, 3, 5, 9, 10, 12, 13, 19, 22, 24 Files:

Modify: src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Adapters/OpcUaDataConnection.cs (ConnectAsync, ~lines 97–108)
Test: tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests/OpcUaDataConnectionTests.cs

Add failing tests (reuse the existing Substitute.For<IOpcUaClient>() / IOpcUaClientFactory fixture; note the fixture returns one _mockClient — for these tests configure the factory to return a fresh substitute per Create() call):

[Fact]
public async Task Reconnect_DisposesAndDetachesPreviousClient()
{
    var clients = new List<IOpcUaClient>();
    _mockFactory.Create().Returns(_ =>
    {
        var c = Substitute.For<IOpcUaClient>();
        clients.Add(c);
        return c;
    });
    var details = new Dictionary<string, string> { ["EndpointUrl"] = "opc.tcp://x:4840" };

    await _adapter.ConnectAsync(details);   // client[0]
    await _adapter.ConnectAsync(details);   // reconnect -> client[1]

    Assert.Equal(2, clients.Count);
    await clients[0].Received(1).DisposeAsync();
    await clients[1].DidNotReceive().DisposeAsync();
}

[Fact]
public async Task StaleClientConnectionLost_DoesNotRaiseDisconnected_AfterReconnect()
{
    var clients = new List<IOpcUaClient>();
    _mockFactory.Create().Returns(_ =>
    {
        var c = Substitute.For<IOpcUaClient>();
        clients.Add(c);
        return c;
    });
    var details = new Dictionary<string, string> { ["EndpointUrl"] = "opc.tcp://x:4840" };
    var disconnects = 0;
    _adapter.Disconnected += () => Interlocked.Increment(ref disconnects);

    await _adapter.ConnectAsync(details);
    await _adapter.ConnectAsync(details);   // reconnect resets _disconnectFired

    // The OLD client's keep-alive dies later — must be a no-op now.
    clients[0].ConnectionLost += Raise.Event<Action>();
    Assert.Equal(0, disconnects);

    // The CURRENT client's loss must still signal.
    clients[1].ConnectionLost += Raise.Event<Action>();
    Assert.Equal(1, disconnects);
}

Run: dotnet test tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests --filter "FullyQualifiedName~Reconnect_DisposesAndDetachesPreviousClient|FullyQualifiedName~StaleClientConnectionLost_DoesNotRaiseDisconnected" → expect FAIL (old client never disposed; stale handler still wired).
Implement in ConnectAsync, immediately before _client = _clientFactory.Create();:

// Reconnect reuses this adapter instance: detach + dispose the previous
// client so (a) its session/keep-alive/socket do not leak per flap and
// (b) its late ConnectionLost cannot flap the NEW healthy session after
// _disconnectFired is reset below. Dispose is best-effort fire-and-forget
// (CloseAsync against a dead server can block for the operation timeout);
// faults are logged, never thrown into the connect path.
if (_client is { } previousClient)
{
    previousClient.ConnectionLost -= OnClientConnectionLost;
    _ = previousClient.DisposeAsync().AsTask().ContinueWith(
        t => _logger.LogWarning(t.Exception?.GetBaseException(),
            "Disposing previous OPC UA client failed for {Endpoint}", _endpointUrl),
        TaskContinuationOptions.OnlyOnFaulted);
}

Also stop the previous heartbeat monitor if active (call the existing StopHeartbeatMonitor() before creating the new client) so the old StaleTagMonitor cannot fire against the new session. 4. Run the two tests → expect PASS. Run the full adapter suite: dotnet test tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests --filter "FullyQualifiedName~OpcUaDataConnectionTests" → all green. 5. Commit: git add src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Adapters/OpcUaDataConnection.cs tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests/OpcUaDataConnectionTests.cs && git commit -m "fix(dcl): dispose+detach previous OPC UA client on reconnect (S1) — stops session leak and stale ConnectionLost flapping"

Task 2: MxGatewayDataConnection — dispose previous client + cancel previous event loop on reconnect

Classification: high-risk (adapter lifecycle) Estimated implement time: ~4 min Parallelizable with: 1, 3, 5, 9, 10, 12, 13, 19, 22, 24 Files:

Modify: src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Adapters/MxGatewayDataConnection.cs (ConnectAsync, ~lines 67–88)
Test: create tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests/Adapters/MxGatewayDataConnectionReconnectTests.cs

Failing test (mirror the OPC UA fixture shape with Substitute.For<IMxGatewayClient>() / IMxGatewayClientFactory; capture the CancellationToken handed to RunEventLoopAsync):

[Fact]
public async Task Reconnect_DisposesPreviousClient_AndCancelsPreviousEventLoop()
{
    var clients = new List<IMxGatewayClient>();
    var loopTokens = new List<CancellationToken>();
    var factory = Substitute.For<IMxGatewayClientFactory>();
    factory.Create().Returns(_ =>
    {
        var c = Substitute.For<IMxGatewayClient>();
        c.RunEventLoopAsync(Arg.Any<Action<MxGatewayValueUpdate>>(), Arg.Do<CancellationToken>(t => loopTokens.Add(t)))
            .Returns(Task.Delay(Timeout.Infinite));
        clients.Add(c);
        return c;
    });
    var adapter = new MxGatewayDataConnection(factory, NullLogger<MxGatewayDataConnection>.Instance);
    var details = new Dictionary<string, string> { ["Endpoint"] = "localhost:5000", ["ApiKey"] = "k" };

    await adapter.ConnectAsync(details);
    await adapter.ConnectAsync(details); // reconnect

    await clients[0].Received(1).DisposeAsync();
    Assert.True(loopTokens[0].IsCancellationRequested);   // old loop cancelled
    Assert.False(loopTokens[1].IsCancellationRequested);  // new loop live
}

(Adjust the RunEventLoopAsync signature/MxGatewayValueUpdate names to the real IMxGatewayClient members — check src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Adapters/IMxGatewayClient.cs.) 2. Run: dotnet test tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests --filter "FullyQualifiedName~MxGatewayDataConnectionReconnectTests" → FAIL. 3. Implement at the top of ConnectAsync (before _client = _clientFactory.Create();):

// Reconnect reuses this adapter: cancel the previous event loop and dispose
// the previous client so a flapping gateway cannot accumulate orphaned
// streams/sockets, and the old loop's fault path cannot signal Disconnected
// against the new session after the guard resets below.
if (_eventLoopCts is { } previousLoopCts)
{
    previousLoopCts.Cancel();
    previousLoopCts.Dispose();
    _eventLoopCts = null;
}
if (_client is { } previousClient)
{
    _ = previousClient.DisposeAsync().AsTask().ContinueWith(
        t => _logger.LogWarning(t.Exception?.GetBaseException(),
            "Disposing previous MxGateway client failed"),
        TaskContinuationOptions.OnlyOnFaulted);
}

Note ordering: keep Interlocked.Exchange(ref _disconnectFired, 0) AFTER the cancel/dispose, so a fault raised by the dying loop during teardown is absorbed by the old guard cycle. Also clear _subs/_tagToSub is NOT needed (re-subscribe repopulates via the actor), but verify RunEventLoopAsync's catch respects the cancelled token → OperationCanceledException path, which it already does. 4. Run → PASS; then full project: dotnet test tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests → green. 5. Commit: git add src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Adapters/MxGatewayDataConnection.cs tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests/Adapters/MxGatewayDataConnectionReconnectTests.cs && git commit -m "fix(dcl): cancel old event loop + dispose old MxGateway client on reconnect (S1)"

Task 3: Pure deploy-time compile validator for a flattened config

Classification: standard Estimated implement time: ~5 min Parallelizable with: 1, 2, 5, 9, 10, 12, 19, 22, 24 Files:

Create: src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Deployment/DeployCompileValidator.cs
Test: create tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Deployment/DeployCompileValidatorTests.cs

Failing test:

public class DeployCompileValidatorTests
{
    private static string ConfigWith(string scriptCode) => JsonSerializer.Serialize(new FlattenedConfiguration
    {
        Scripts = new List<ResolvedScript>
        {
            new() { CanonicalName = "S1", Code = scriptCode, TriggerType = "Call" }
        },
        Attributes = new List<ResolvedAttribute>(),
        Alarms = new List<ResolvedAlarm>(),
        NativeAlarmSources = new List<ResolvedNativeAlarmSource>()
    });

    [Fact]
    public void BrokenScript_ReturnsErrors()
    {
        var validator = new DeployCompileValidator(new ScriptCompilationService(/* mirror ctor args used in ScriptCompilationServiceTests */));
        var errors = validator.Validate(ConfigWith("this is not C# ~~~"));
        Assert.NotEmpty(errors);
        Assert.Contains(errors, e => e.Contains("S1"));
    }

    [Fact]
    public void ValidScript_ReturnsNoErrors()
    {
        var validator = new DeployCompileValidator(new ScriptCompilationService(/* as above */));
        Assert.Empty(validator.Validate(ConfigWith("return 1 + 1;")));
    }

    [Fact]
    public void BrokenTriggerExpression_ReturnsErrors()
    {
        var cfg = /* config with one Expression-triggered script whose TriggerConfiguration
                     JSON carries expression: "1 +" (mirror shape used in ScriptActorTests) */;
        var validator = new DeployCompileValidator(new ScriptCompilationService(/* as above */));
        Assert.NotEmpty(validator.Validate(cfg));
    }
}

(Mirror FlattenedConfiguration/ResolvedScript construction and the ScriptCompilationService ctor from the existing ScriptCompilationServiceTests / InstanceActorTests fixtures.) 2. Run: dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~DeployCompileValidatorTests" → FAIL (type does not exist). 3. Implement DeployCompileValidator: a small class taking the same compilation-service type InstanceActor uses; method IReadOnlyList<string> Validate(string configJson):

Deserialize FlattenedConfiguration; malformed JSON → single error "Configuration JSON failed to deserialize: …" (this also fronts Task 15's poison-config case).
For each config.Scripts: _compilationService.Compile(script.CanonicalName, script.Code); on failure add $"Script '{script.CanonicalName}': {string.Join("; ", result.Errors)}".
For each alarm with OnTriggerScriptCanonicalName: resolve the script def (missing def → error) and compile it (mirroring InstanceActor.CreateChildActors lines 1404–1424).
For each Expression-trigger script/alarm: extract via TriggerExpressionGlobals.ExtractExpression and CompileTriggerExpression; failures are errors (mirroring InstanceActor.CompileTriggerExpression).
Pure and thread-safe (no actor state) — callable from Task.Run.

Run → PASS.
Commit: git add src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Deployment/DeployCompileValidator.cs tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Deployment/DeployCompileValidatorTests.cs && git commit -m "feat(site-runtime): pure deploy-time compile validator for flattened configs (S3 groundwork)"

Task 4: Reject the deployment on site-side compile failure (per spec)

Classification: high-risk (deployment pipeline, singleton actor) Estimated implement time: ~5 min Parallelizable with: 1, 2, 12, 19, 22, 24 Files:

Modify: src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/DeploymentManagerActor.cs (HandleDeploy entry + new piped message)
Modify: src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/InstanceActor.cs (fix the contradictory XML doc at line 1339)
Test: tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/DeploymentManagerActorTests.cs

Failing test (reuse the existing DeploymentManagerActor TestKit harness/fixture in that file):

[Fact]
public void Deploy_WithNonCompilingScript_ReportsFailed_AndCreatesNoInstanceActor()
{
    var badConfig = /* FlattenedConfiguration JSON with one script whose Code = "not C# ~~~" —
                       reuse the config-builder helper the existing deploy tests use */;
    var dm = CreateDeploymentManager(); // existing harness factory

    dm.Tell(new DeployInstanceCommand(/* deploymentId */ Guid.NewGuid().ToString(),
        "bad-instance", badConfig, /* revisionHash */ "r1" /*, …existing args*/));

    var resp = ExpectMsg<DeploymentStatusResponse>(TimeSpan.FromSeconds(10));
    Assert.Equal(DeploymentStatus.Failed, resp.Status);
    Assert.Contains("compile", resp.ErrorMessage, StringComparison.OrdinalIgnoreCase);

    // No partial state applied: instance actor must not exist, config must not persist.
    Sys.ActorSelection(dm.Path / "bad-instance").Tell(new Identify(1));
    Assert.Null(ExpectMsg<ActorIdentity>().Subject);
    _storage.DidNotReceive().StoreDeployedConfigAsync("bad-instance",
        Arg.Any<string>(), Arg.Any<string>(), Arg.Any<string>(), Arg.Any<bool>());
}

Run: dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~Deploy_WithNonCompilingScript" → FAIL (today it reports Success).
Implement:
- Add internal message private sealed record DeployValidationResult(DeployInstanceCommand Cmd, IActorRef ReplyTo, IReadOnlyList<string> Errors); and Receive<DeployValidationResult>(HandleDeployValidationResult);.
- Rename the current HandleDeploy(DeployInstanceCommand) body to ProceedWithDeploy(DeployInstanceCommand command, IActorRef sender) (this is the code that decides redeploy-buffer vs ApplyDeployment — keep it byte-identical).
- New HandleDeploy: capture Sender; run Task.Run(() => _deployCompileValidator.Validate(command.FlattenedConfigurationJson)) → ContinueWith into DeployValidationResult → PipeTo(Self). Validation is pure, so the singleton mailbox never blocks on Roslyn, and results are applied on the actor thread in arrival order (the existing redeploy-supersede handling still governs ordering).
- HandleDeployValidationResult: if Errors.Count > 0 → LogDeploymentEvent("Error", …, "compile validation failed"), reply DeploymentStatusResponse(DeploymentId, Instance, DeploymentStatus.Failed, "Script compilation failed on site: " + string.Join(" | ", errors), UtcNow) and return (no actor creation, no persistence — spec: "No partial state is applied"). Else call ProceedWithDeploy(msg.Cmd, msg.ReplyTo).
- Construct _deployCompileValidator = new DeployCompileValidator(_compilationService) in the ctor.
- Fix InstanceActor.cs:1339 XML doc: replace "Compilation errors reject entire instance deployment (logged but actor still starts)" with "Compile failures are rejected at deploy time by the Deployment Manager's validation gate; a failure reaching this point (startup load of a legacy config) is logged and the script skipped."
Run the new test → PASS; then the deploy suites: dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~DeploymentManager" → green (existing deploy/redeploy/disable-supersede tests must still pass — they exercise ProceedWithDeploy through the new gate with compiling configs).
Commit: git add src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/DeploymentManagerActor.cs src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/InstanceActor.cs tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/DeploymentManagerActorTests.cs && git commit -m "fix(site-runtime): site-side compile failure now rejects the deployment per spec (S3) — off-thread validation gate before actor creation"

Task 5: ScriptExecutionScheduler gauges — queue depth, busy threads, oldest-busy age

Classification: small (concurrency-adjacent but additive instrumentation) Estimated implement time: ~4 min Parallelizable with: 1, 2, 3, 9, 10, 11, 12, 13, 19, 22, 24 Files:

Modify: src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Scripts/ScriptExecutionScheduler.cs
Test: tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Scripts/ScriptExecutionSchedulerTests.cs

Failing test:

[Fact]
public async Task Gauges_ReflectBusyThreadAndQueueDepth()
{
    using var scheduler = new ScriptExecutionScheduler(1);
    using var gate = new ManualResetEventSlim(false);

    var blocking = Task.Factory.StartNew(() => gate.Wait(),
        CancellationToken.None, TaskCreationOptions.None, scheduler);
    var queued = Task.Factory.StartNew(() => { },
        CancellationToken.None, TaskCreationOptions.None, scheduler);

    await WaitUntilAsync(() => scheduler.BusyThreadCount == 1);
    Assert.Equal(1, scheduler.QueueDepth);          // second task waiting
    Assert.True(scheduler.OldestBusyAge > TimeSpan.Zero);

    gate.Set();
    await Task.WhenAll(blocking, queued);
    await WaitUntilAsync(() => scheduler.BusyThreadCount == 0);
    Assert.Null(scheduler.OldestBusyAge);
}
// WaitUntilAsync: small poll helper (100×50ms) — add locally if the test file has none.

Run: dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~Gauges_ReflectBusyThreadAndQueueDepth" → FAIL (members don't exist).
Implement:
- private readonly long[] _busySinceTicks; sized threadCount, initialized 0. Give each worker its index (change new Thread(WorkerLoop) to new Thread(() => WorkerLoop(index)) with a captured local).
- In WorkerLoop(int index): around TryExecuteTask(task) do Volatile.Write(ref _busySinceTicks[index], Environment.TickCount64); before and Volatile.Write(ref _busySinceTicks[index], 0); in a finally.
- Public gauges: public int QueueDepth => _queue.Count; public int BusyThreadCount (count of non-zero slots), public TimeSpan? OldestBusyAge (max of now - since over non-zero slots, else null).
Run → PASS. Run full scheduler suite: dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~ScriptExecutionScheduler".
Commit: git add src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Scripts/ScriptExecutionScheduler.cs tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Scripts/ScriptExecutionSchedulerTests.cs && git commit -m "feat(site-runtime): script scheduler gauges — queue depth, busy threads, oldest-busy age (S2/UA5)"

Task 6: Surface scheduler stats through ISiteHealthCollector + SiteHealthReport (additive)

Classification: small Estimated implement time: ~4 min Parallelizable with: 1, 2, 3, 4, 9, 10, 11, 12, 13, 19, 22, 24 Files:

Modify: src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/ISiteHealthCollector.cs (new default-no-op method)
Modify: src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/SiteHealthCollector.cs (store + include in CollectReport)
Modify: src/ZB.MOM.WW.ScadaBridge.Commons/Messages/Health/SiteHealthReport.cs (additive optional properties)
Test: tests/ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests/ (add to the existing SiteHealthCollector test file)

Failing test (in the existing collector test class):

[Fact]
public void CollectReport_CarriesScriptSchedulerStats()
{
    var collector = CreateCollector(); // existing fixture helper
    collector.SetScriptSchedulerStats(queueDepth: 7, busyThreads: 3, oldestBusyAgeSeconds: 42.5);
    var report = collector.CollectReport("site-1");
    Assert.Equal(7, report.ScriptQueueDepth);
    Assert.Equal(3, report.ScriptBusyThreads);
    Assert.Equal(42.5, report.ScriptOldestBusyAgeSeconds);
}

Run: dotnet test tests/ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests --filter "FullyQualifiedName~CollectReport_CarriesScriptSchedulerStats" → FAIL.
Implement:
- Interface: void SetScriptSchedulerStats(int queueDepth, int busyThreads, double? oldestBusyAgeSeconds) { } (default no-op, matching the SetSiteEventLogWriteFailures pattern so test fakes keep compiling).
- Collector: store via Interlocked/volatile fields; include in CollectReport.
- SiteHealthReport: additive optional init properties with safe defaults (public int ScriptQueueDepth { get; init; }, public int ScriptBusyThreads { get; init; }, public double? ScriptOldestBusyAgeSeconds { get; init; }) — do NOT change the positional parameter list (message-contract additive-only rule).
Run → PASS; run full projects: dotnet test tests/ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests and dotnet test tests/ZB.MOM.WW.ScadaBridge.Commons.Tests.
Commit: git add src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/ISiteHealthCollector.cs src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/SiteHealthCollector.cs src/ZB.MOM.WW.ScadaBridge.Commons/Messages/Health/SiteHealthReport.cs tests/ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests && git commit -m "feat(health): additive script-scheduler stats on ISiteHealthCollector + SiteHealthReport (S2/UA5)"

Task 7: Periodic scheduler-stats reporter (hosted service)

Classification: small Estimated implement time: ~4 min Parallelizable with: 1, 2, 4, 9, 10, 11, 12, 13, 19, 22, 24 (needs 5 + 6 merged) Files:

Create: src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Scripts/ScriptSchedulerStatsReporter.cs
Modify: src/ZB.MOM.WW.ScadaBridge.SiteRuntime/ServiceCollectionExtensions.cs (register hosted service on site hosts)
Test: create tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Scripts/ScriptSchedulerStatsReporterTests.cs

Failing test:

[Fact]
public async Task Reporter_PushesStatsToCollector()
{
    var collector = Substitute.For<ISiteHealthCollector>();
    var options = new SiteRuntimeOptions { ScriptExecutionThreadCount = 1 };
    using var reporter = new ScriptSchedulerStatsReporter(
        collector, options, NullLogger<ScriptSchedulerStatsReporter>.Instance,
        pollInterval: TimeSpan.FromMilliseconds(50));

    await reporter.StartAsync(CancellationToken.None);
    await Task.Delay(300);
    await reporter.StopAsync(CancellationToken.None);

    collector.ReceivedWithAnyArgs().SetScriptSchedulerStats(default, default, default);
}

Run: dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~ScriptSchedulerStatsReporterTests" → FAIL.
Implement a BackgroundService mirroring SiteEventLogFailureCountReporter (src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/SiteEventLogFailureCountReporter.cs): every 10 s (test-overridable ctor param) read ScriptExecutionScheduler.Shared(options) gauges and call collector.SetScriptSchedulerStats(...). Register it in the SiteRuntime ServiceCollectionExtensions next to the existing site-role hosted services. Never throw from the loop (catch + log, matching the sibling reporter).
Run → PASS. Build the solution once (dotnet build ZB.MOM.WW.ScadaBridge.slnx) to confirm Host wiring compiles.
Commit: git add src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Scripts/ScriptSchedulerStatsReporter.cs src/ZB.MOM.WW.ScadaBridge.SiteRuntime/ServiceCollectionExtensions.cs tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Scripts/ScriptSchedulerStatsReporterTests.cs && git commit -m "feat(site-runtime): periodic scheduler-stats reporter feeding site health (S2/UA5)"

Task 8: Stuck-script watchdog — name the script whose thread never came back

Classification: high-risk (script execution path) Estimated implement time: ~5 min Parallelizable with: 1, 2, 4, 9, 10, 12, 13, 19, 22, 24 Files:

Modify: src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/ScriptExecutionActor.cs (~lines 122–260)
Modify: src/ZB.MOM.WW.ScadaBridge.SiteRuntime/SiteRuntimeOptions.cs (new StuckScriptGraceMs, default 30000)
Test: tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/ExecutionActorTests.cs

Failing test (follow the existing ExecutionActorTests pattern for compiling a Script<object?> directly with CSharpScript.Create<object>(code, options, typeof(ScriptGlobals)) — test-compiled, so the trust validator is not in the path):

[Fact]
public async Task TimedOutScript_WithBlockedThread_EmitsStuckThreadSiteEvent()
{
    var gate = new SemaphoreSlim(0);
    StuckTestHooks.Gate = gate; // small public static test hook class in the test assembly
    var script = CompileRaw("ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.Actors.StuckTestHooks.Gate.Wait(); return null;");
    var siteEvents = new RecordingSiteEventLogger(); // reuse/extend the existing fake in TestSupport
    var options = new SiteRuntimeOptions { ScriptExecutionTimeoutSeconds = 1, StuckScriptGraceMs = 200 };

    SpawnExecutionActor(script, options, siteEvents); // existing harness helper

    await WaitUntilAsync(() => siteEvents.Events.Any(e =>
        e.Category == "script" && e.Message.Contains("still executing", StringComparison.OrdinalIgnoreCase)),
        timeout: TimeSpan.FromSeconds(10));

    gate.Release(); // free the scheduler thread so the test run stays clean
}

Run: dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~TimedOutScript_WithBlockedThread_EmitsStuckThreadSiteEvent" → FAIL.
Implement inside the execution lambda in ScriptExecutionActor:
- var completed = 0; before the try; Interlocked.Exchange(ref completed, 1); in the existing finally (add one if missing — the scope-dispose finally exists).
- Immediately after using var cts = new CancellationTokenSource(timeout);:

// Cooperative-only timeout containment (S2): the CTS firing does NOT free a
// thread blocked in synchronous I/O. When the timeout elapses, wait a grace
// period on the thread pool and, if the body still hasn't returned, name the
// script loudly — this is the only signal an operator gets that one of the
// bounded script-execution threads is gone.
var graceMs = options.StuckScriptGraceMs;
cts.Token.Register(() => _ = Task.Run(async () =>
{
    await Task.Delay(graceMs);
    if (Volatile.Read(ref completed) == 0)
    {
        var msg = $"Script '{scriptName}' on instance '{instanceName}' exceeded its " +
                  $"{timeout.TotalSeconds:F0}s timeout and is STILL EXECUTING — its dedicated " +
                  "script-execution thread is blocked (cooperative cancellation not observed).";
        logger.LogError(msg);
        _ = siteEventLogger?.LogEventAsync("script", "Error", instanceName,
            $"ScriptActor:{scriptName}", msg);
    }
}));

Note: Register callback fires on cancellation only; normal completion disposes the CTS with no callback. Task.Run/Task.Delay run on the thread pool, not the (possibly saturated) script scheduler — deliberate.

Run → PASS; run dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~ExecutionActorTests".
Commit: git add src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/ScriptExecutionActor.cs src/ZB.MOM.WW.ScadaBridge.SiteRuntime/SiteRuntimeOptions.cs tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/ExecutionActorTests.cs && git commit -m "feat(site-runtime): stuck-script watchdog names the script holding a scheduler thread past timeout (S2)"

Task 9: ScriptActor — trigger-expression evaluation off the dispatcher (coalesced, actor-confined edge state)

Classification: high-risk (actor concurrency, trigger semantics) Estimated implement time: ~5 min Parallelizable with: 1, 2, 3, 5, 10, 12, 13, 19, 22, 24 Files:

Modify: src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/ScriptActor.cs (~lines 255–307)
Test: tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/ScriptActorTests.cs

Failing test (expression that only returns true on a scheduler thread — proves evaluation left the dispatcher; reuse the ScriptActorTests harness that builds an Expression-triggered ScriptActor with a compiled trigger expression and a script body observable from the test):

[Fact]
public void ExpressionTrigger_EvaluatesOnScriptSchedulerThread_AndStillFires()
{
    // Expression is TRUE only when evaluated on a script-execution thread.
    var expr = CompileTriggerExpression(
        "System.Threading.Thread.CurrentThread.Name != null && " +
        "System.Threading.Thread.CurrentThread.Name.StartsWith(\"script-execution-\")");
    var fired = CreateExpressionScriptActorWithObservableBody(expr); // harness helper returning a signal

    _scriptActor.Tell(new AttributeValueChanged("inst", "A", "A", 1, "Good", DateTimeOffset.UtcNow));

    AwaitAssert(() => Assert.True(fired.IsSet), TimeSpan.FromSeconds(10));
}

[Fact]
public void ExpressionTrigger_OnTrue_StillFiresOncePerFalseTrueEdge()
{
    // Regression: async evaluation must preserve edge semantics.
    // expr: (int)Attributes["A"] > 5, OnTrue mode.
    SendChange("A", 1); SendChange("A", 10); SendChange("A", 11); // true stays true
    AwaitAssert(() => Assert.Equal(1, ExecutionCount()), TimeSpan.FromSeconds(10));
    SendChange("A", 1); SendChange("A", 9);
    AwaitAssert(() => Assert.Equal(2, ExecutionCount()), TimeSpan.FromSeconds(10));
}

Run: dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~ExpressionTrigger_EvaluatesOnScriptSchedulerThread" → FAIL (evaluation currently happens synchronously on the dispatcher thread, whose name is not script-execution-*).
Implement:
- New internal message: private sealed record ExpressionEvalResult(bool Result); + Receive<ExpressionEvalResult>(HandleExpressionEvalResult); in the ctor.
- Fields: private bool _evalInFlight; private bool _evalPending;
- Replace the body of EvaluateExpressionTrigger() with StartExpressionEvaluation():

private void StartExpressionEvaluation()
{
    if (_compiledTriggerExpression == null || _triggerConfig is not ExpressionTriggerConfig) return;
    if (_evalInFlight) { _evalPending = true; return; }   // coalesce bursts: one in flight, one pending
    _evalInFlight = true;

    var snapshot = new Dictionary<string, object?>(_attributeSnapshot); // point-in-time copy, actor thread
    var expression = _compiledTriggerExpression;
    var self = Self;
    Task.Factory.StartNew(async () =>
    {
        try
        {
            using var cts = new CancellationTokenSource(TimeSpan.FromSeconds(2));
            var state = await expression.RunAsync(new TriggerExpressionGlobals(snapshot), cancellationToken: cts.Token);
            return state.ReturnValue is bool b && b;
        }
        catch (Exception ex) { LogExpressionError(ex); return false; }
    }, CancellationToken.None, TaskCreationOptions.DenyChildAttach,
        ScriptExecutionScheduler.Shared(_options)).Unwrap().PipeTo(self,
        success: r => new ExpressionEvalResult(r));
}

private void HandleExpressionEvalResult(ExpressionEvalResult msg)
{
    _evalInFlight = false;
    if (_triggerConfig is ExpressionTriggerConfig exprConfig)
    {
        if (exprConfig.Mode == TriggerMode.WhileTrue) HandleWhileTrueTransition(msg.Result, _lastExpressionResult);
        else if (msg.Result && !_lastExpressionResult) TrySpawnExecution(null);
        _lastExpressionResult = msg.Result;
    }
    if (_evalPending) { _evalPending = false; StartExpressionEvaluation(); }
}

Edge state (_lastExpressionResult, timers) is touched only on the actor thread — the concurrency contract holds. LogExpressionError is called from the background task: it only touches _healthCollector/_logger/_serviceProvider (thread-safe singletons) — verify, and if in doubt route the error through the result message instead.

Run both new tests + dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~ScriptActorTests" → PASS (existing expression tests must survive; they may need AwaitAssert in place of synchronous asserts since evaluation is now async).
Commit: git add src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/ScriptActor.cs tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/ScriptActorTests.cs && git commit -m "perf(site-runtime): ScriptActor trigger expressions evaluate on the script scheduler, coalesced, edge state actor-confined (P1)"

Task 10: AlarmActor — same async trigger-expression evaluation

Classification: high-risk (actor concurrency, alarm state machine) Estimated implement time: ~5 min Parallelizable with: 1, 2, 3, 5, 9, 12, 13, 19, 22, 24 Files:

Modify: src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/AlarmActor.cs (~lines 210–240, 491–519)
Test: tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/AlarmActorTests.cs

Failing test (mirror Task 9's thread-name expression trick with an Expression-triggered alarm; assert the alarm transitions to Active via the existing AlarmStateChanged parent-probe pattern in AlarmActorTests):

[Fact]
public void ExpressionAlarm_EvaluatesOnSchedulerThread_AndActivates()
{
    var expr = CompileTriggerExpression(
        "System.Threading.Thread.CurrentThread.Name != null && " +
        "System.Threading.Thread.CurrentThread.Name.StartsWith(\"script-execution-\")");
    var alarm = CreateExpressionAlarmActor(expr, parentProbe); // existing harness

    alarm.Tell(new AttributeValueChanged("inst", "A", "A", 1, "Good", DateTimeOffset.UtcNow));

    var change = parentProbe.ExpectMsg<AlarmStateChanged>(TimeSpan.FromSeconds(10));
    Assert.Equal(AlarmState.Active, change.State);
}

Run: dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~ExpressionAlarm_EvaluatesOnSchedulerThread" → FAIL.
Implement, structurally identical to Task 9:
- Extract the existing Normal↔Active transition block in HandleAttributeValueChanged (the isTriggered && _currentState == AlarmState.Normal / clear branches) into private void ApplyTriggeredState(bool isTriggered).
- In HandleAttributeValueChanged, the AlarmTriggerType.Expression arm no longer computes EvaluateExpression() inline — it calls StartExpressionEvaluation() (in-flight + pending coalescing, snapshot copy, run on ScriptExecutionScheduler.Shared(_options), PipeTo self as ExpressionEvalResult). Receive<ExpressionEvalResult> → ApplyTriggeredState(msg.Result) + drain pending.
- Non-expression trigger types keep their synchronous path untouched (they are cheap numeric comparisons).
- Delete the now-unused synchronous EvaluateExpression() (or keep as the body invoked inside the task).
Run new test + dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~AlarmActorTests" → PASS (existing expression-alarm tests may need AwaitAssert).
Commit: git add src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/AlarmActor.cs tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/AlarmActorTests.cs && git commit -m "perf(site-runtime): AlarmActor trigger expressions evaluate on the script scheduler (P1)"

Task 11: InstanceActor — retry failed/lost tag subscriptions per connection

Classification: high-risk (actor timers, DCL handshake) Estimated implement time: ~5 min Parallelizable with: 1, 2, 4, 9, 10, 12, 13, 19, 22, 24 Files:

Modify: src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/InstanceActor.cs (line 207 no-op handler; SubscribeToDcl ~962–1010)
Modify: src/ZB.MOM.WW.ScadaBridge.SiteRuntime/SiteRuntimeOptions.cs (new TagSubscribeRetryIntervalMs, default 5000)
Test: tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/InstanceActorTests.cs

Failing tests (TestKit; the InstanceActor harness passes a TestProbe as dclManager):

[Fact]
public void FailedSubscribeResponse_IsRetried()
{
    var options = new SiteRuntimeOptions { TagSubscribeRetryIntervalMs = 200 };
    var (instance, dclProbe) = CreateInstanceWithDataSourcedAttribute(options); // existing harness

    var first = dclProbe.ExpectMsg<SubscribeTagsRequest>();
    instance.Tell(new SubscribeTagsResponse(first.CorrelationId, first.InstanceUniqueName,
        false, "Unknown connection: conn-1", DateTimeOffset.UtcNow), dclProbe.Ref);

    var retry = dclProbe.ExpectMsg<SubscribeTagsRequest>(TimeSpan.FromSeconds(5)); // S4 core
    Assert.Equal(first.ConnectionName, retry.ConnectionName);
    Assert.Equal(first.TagPaths, retry.TagPaths);
}

[Fact]
public void LostSubscribeResponse_IsRetried_AndSuccessStopsRetrying()
{
    var options = new SiteRuntimeOptions { TagSubscribeRetryIntervalMs = 200 };
    var (instance, dclProbe) = CreateInstanceWithDataSourcedAttribute(options);

    var first = dclProbe.ExpectMsg<SubscribeTagsRequest>();
    // No reply at all (dead-lettered response) -> must re-send.
    var retry = dclProbe.ExpectMsg<SubscribeTagsRequest>(TimeSpan.FromSeconds(5));

    instance.Tell(new SubscribeTagsResponse(retry.CorrelationId, retry.InstanceUniqueName,
        true, null, DateTimeOffset.UtcNow), dclProbe.Ref);
    dclProbe.ExpectNoMsg(TimeSpan.FromMilliseconds(600)); // retry timer cancelled
}

Run: dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~FailedSubscribeResponse_IsRetried|FullyQualifiedName~LostSubscribeResponse_IsRetried" → FAIL (response is a no-op; no re-send).
Implement:
- Refactor SubscribeToDcl to build the per-connection groups once into a field Dictionary<string, List<string>> _tagsByConnection, then call SendTagSubscribe(connectionName) per connection.
- SendTagSubscribe(string conn): mint a correlation id, record _pendingTagSubscribes[correlationId] = conn, Tell the DCL manager, and (re)arm a per-connection retry: _tagSubscribeRetryTimers[conn]?.Cancel(); _tagSubscribeRetryTimers[conn] = Context.System.Scheduler.ScheduleTellOnceCancelable(TimeSpan.FromMilliseconds(_options.TagSubscribeRetryIntervalMs), Self, new RetryTagSubscribe(conn), Self); — arming on send covers both the failure reply AND the lost reply with one mechanism (mirrors NativeAlarmActor's timer but closes its lost-response gap too).
- Replace Receive<SubscribeTagsResponse>(_ => { }) with a real handler: resolve conn from _pendingTagSubscribes (remove the entry); on Success → cancel the connection's retry timer, LogDebug; on failure → LogWarning + fire-and-forget an instance_lifecycle/connection site event ("tag subscribe failed … will retry") and leave the armed timer to re-send. Unknown correlation id (stale retry's superseded reply) → ignore.
- Receive<RetryTagSubscribe>(msg => SendTagSubscribe(msg.ConnectionName)); (internal record). PostStop: cancel all retry timers.
- Note the DCL's connection-level-failure reply (Success=false, "connection unavailable — will re-subscribe on reconnect") now ALSO retriggers a subscribe — harmless: the DCL treats already-subscribed tags as AlreadySubscribed and its _subscriptionsByInstance update is idempotent (verified HandleSubscribe/HandleSubscribeCompleted).
Run → PASS; run dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~InstanceActor" → green.
Commit: git add src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/InstanceActor.cs src/ZB.MOM.WW.ScadaBridge.SiteRuntime/SiteRuntimeOptions.cs tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/InstanceActorTests.cs && git commit -m "fix(site-runtime): InstanceActor retries failed/lost tag subscriptions per connection (S4/UA6) — closes the unknown-connection ordering race"

Classification: small Estimated implement time: ~3 min Parallelizable with: 1, 2, 3, 4, 5, 9, 10, 11, 13, 19, 22, 24 Files:

Modify: src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/NativeAlarmActor.cs (SendSubscribe ~123–132, HandleSubscribeResponse ~269–286)
Test: tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/NativeAlarmActorTests.cs

Failing test:

[Fact]
public void LostSubscribeResponse_ResendsSubscribe()
{
    var options = new SiteRuntimeOptions { NativeAlarmRetryIntervalMs = 200 };
    var (actor, dclProbe) = CreateNativeAlarmActor(options); // existing harness

    dclProbe.ExpectMsg<SubscribeAlarmsRequest>();
    // Swallow the response entirely — today the actor never retries.
    dclProbe.ExpectMsg<SubscribeAlarmsRequest>(TimeSpan.FromSeconds(5));
}

Run: dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~LostSubscribeResponse_ResendsSubscribe" → FAIL.
Implement: in SendSubscribe(), after the Tell, arm the response-timeout retry using the existing _retryTimer field: _retryTimer?.Cancel(); _retryTimer = Context.System.Scheduler.ScheduleTellOnceCancelable(TimeSpan.FromMilliseconds(_options.NativeAlarmRetryIntervalMs * 2), Self, RetrySubscribe.Instance, Self);. In HandleSubscribeResponse success branch, add _retryTimer?.Cancel(); before returning. The failure branch already re-arms (keep it — it re-arms at 1× interval, which now simply overrides the 2× response-timeout arm).
Run → PASS; run dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~NativeAlarmActorTests".
Commit: git add src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/NativeAlarmActor.cs tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/NativeAlarmActorTests.cs && git commit -m "fix(site-runtime): NativeAlarmActor retries a lost subscribe response, not only a failed one (S4)"

Task 13: DeploymentManager — retry the startup config load

Classification: standard Estimated implement time: ~4 min Parallelizable with: 1, 2, 3, 5, 9, 10, 11, 12, 19, 22, 24 Files:

Modify: src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/DeploymentManagerActor.cs (PreStart ~240–253, HandleStartupConfigsLoaded ~291–297)
Test: tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/DeploymentManagerActorTests.cs

Failing test:

[Fact]
public void StartupLoadFailure_IsRetried_UntilSuccess()
{
    var calls = 0;
    _storage.GetAllDeployedConfigsAsync().Returns(_ =>
        ++calls == 1
            ? Task.FromException<List<DeployedInstance>>(new IOException("db locked"))
            : Task.FromResult(new List<DeployedInstance>
                { NewDeployedInstance("inst-1", ValidConfigJson()) })); // harness helpers

    var dm = CreateDeploymentManager(); // retry interval overridable to ~200ms for the test

    AwaitAssert(() =>
    {
        Assert.True(calls >= 2);
        Sys.ActorSelection(dm.Path / "inst-1").Tell(new Identify(1));
        Assert.NotNull(ExpectMsg<ActorIdentity>().Subject);
    }, TimeSpan.FromSeconds(10));
}

(If _storage is the concrete SiteStorageService in the harness rather than a substitute, follow the harness's existing seam — several DM tests already fake persistence failures for DeployPersistenceResult; reuse that approach.) 2. Run: dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~StartupLoadFailure_IsRetried" → FAIL (one attempt, then dead). 3. Implement:

Extract the PreStart load block into private void LoadDeployedConfigs() and call it from PreStart.
In HandleStartupConfigsLoaded, replace the error return; with: log the error, then Timers.StartSingleTimer("startup-load-retry", RetryStartupLoad.Instance, StartupLoadRetryInterval); (private static readonly TimeSpan StartupLoadRetryInterval = TimeSpan.FromSeconds(5) — internal-settable or ctor-overridable for the test, matching how other DM test knobs are exposed). Fixed-interval, unlimited — consistent with the system's fixed-interval retry philosophy; each failure logs.
Receive<RetryStartupLoad>(_ => LoadDeployedConfigs()); (internal singleton message). Guard: if configs already loaded (_deployedInstanceNames.Count > 0 after a successful load — add a _startupLoadCompleted bool set in the success path), ignore stale retries.

Run → PASS; run the DM suite.
Commit: git add src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/DeploymentManagerActor.cs tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/DeploymentManagerActorTests.cs && git commit -m "fix(site-runtime): retry the startup deployed-config load on SQLite failure (S5) — no more silent zero-instance node"

Task 14: Watch all Instance Actors; clean the map on unexpected termination

Classification: high-risk (supervision/termination flow interacts with redeploy buffering) Estimated implement time: ~5 min Parallelizable with: 1, 2, 5, 9, 10, 12, 19, 22, 24 Files:

Modify: src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/DeploymentManagerActor.cs (CreateInstanceActor ~1605–1632, HandleTerminated)
Test: tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/DeploymentManagerActorTests.cs

Failing test:

[Fact]
public void InstanceActorInitFailure_RemovesDeadRefFromMap()
{
    var poisonConfig = "{ not json";  // InstanceActor ctor JsonSerializer.Deserialize throws
    var dm = CreateDeploymentManager();
    var healthProbe = _healthCollector; // fake collector already captured by the harness

    dm.Tell(new DeployInstanceCommand(Guid.NewGuid().ToString(), "poison", poisonConfig, "r1" /*…*/));

    // After the child Stops (decider: ActorInitializationException -> Stop), the map
    // must not hold a dead ref: instance counts must return to 0 deployed.
    AwaitAssert(() => Assert.Equal(0, healthProbe.LastDeployedCount), TimeSpan.FromSeconds(10));
}

Note: with Task 4 in place, "{ not json" is rejected by the compile validator (deserialize error) before an actor is created — so drive this test through a config that deserializes and validates but makes the ctor throw (e.g. a config whose Attributes list contains an entry that trips DecodeAttributeValue's non-guarded path, or temporarily bypass via the startup path: seed storage with the poison config and let startup create the actor). Prefer the startup path (storage returns the poison row): it does not pass through the deploy validation gate and is exactly the "dead ref" scenario S6 describes. 2. Run: dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~InstanceActorInitFailure_RemovesDeadRefFromMap" → FAIL (map keeps the dead ref; counts stay 1). 3. Implement:

CreateInstanceActor: after Context.ActorOf, add Context.Watch(actorRef);.
HandleTerminated(Terminated t): keep the existing redeploy-buffer logic FIRST (it matches on its own bookkeeping). Add a fallback branch at the end: if _instanceActors.TryGetValue(t.ActorRef.Path.Name, out var mapped) && mapped.Equals(t.ActorRef) → remove from _instanceActors, UpdateInstanceCounts(), LogError + fire-and-forget a deployment site event ("Instance {name} terminated unexpectedly (init failure or crash) — removed from routing"). Guard: expected stops (disable/delete/redeploy) remove or replace the map entry before the child stops, so mapped.Equals(t.ActorRef) is false for them and the fallback is inert — verify against HandleDisable/HandleDelete/the redeploy shadow map and adjust ordering only if a path removes after stop.
Redeploy path already Context.Watches; double-watch is idempotent in Akka.NET (single Terminated), but confirm the redeploy handler consumes its Terminated before the fallback branch runs (order the if/else so the buffered-redeploy match wins).

Run → PASS; run dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~DeploymentManager" (the redeploy race suites are the regression canary here — all must stay green).
Commit: git add src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/DeploymentManagerActor.cs tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/DeploymentManagerActorTests.cs && git commit -m "fix(site-runtime): watch every Instance Actor; unexpected termination evicts the dead ref from routing (S6)"

Task 15: Fail the in-flight deployment when its Instance Actor dies during init

Classification: high-risk (deployment status protocol) Estimated implement time: ~5 min Parallelizable with: 1, 2, 5, 9, 10, 12, 19, 22, 24 Files:

Modify: src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/DeploymentManagerActor.cs (ApplyDeployment ~538–587, HandleDeployPersistenceResult ~596–636, HandleTerminated)
Test: tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/DeploymentManagerActorTests.cs

Failing test:

[Fact]
public void Deploy_WhoseActorDiesDuringInit_ReportsFailed_NotSuccess()
{
    // Config that passes compile validation but whose InstanceActor ctor throws —
    // reuse/extend the poison-config helper from Task 14 (ctor-throwing but valid JSON),
    // or a config helper stub if none exists (e.g. attribute row that trips ctor indexing).
    var dm = CreateDeploymentManager();
    dm.Tell(new DeployInstanceCommand(Guid.NewGuid().ToString(), "dies-on-init", CtorThrowingConfig(), "r1" /*…*/));

    var resp = ExpectMsg<DeploymentStatusResponse>(TimeSpan.FromSeconds(10));
    Assert.Equal(DeploymentStatus.Failed, resp.Status);
}

Run: dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~Deploy_WhoseActorDiesDuringInit" → FAIL (Success is reported once SQLite persists).
Implement:
- Field: private readonly Dictionary<string, DeployPersistenceResult?> _inFlightDeployments = new(); keyed by instance name — store a small record InFlightDeploy(string DeploymentId, IActorRef ReplyTo) instead (clearer): private readonly Dictionary<string, InFlightDeploy> _inFlightDeployments = new();
- ApplyDeployment: before spawning the persistence task, _inFlightDeployments[instanceName] = new InFlightDeploy(command.DeploymentId, sender);
- Task 14's unexpected-Terminated fallback branch: after evicting the map entry, if _inFlightDeployments.Remove(name, out var inflight) → inflight.ReplyTo.Tell(new DeploymentStatusResponse(inflight.DeploymentId, name, DeploymentStatus.Failed, "Instance actor failed during initialization — see site event log", DateTimeOffset.UtcNow)); and remove the optimistically-persisted row (Task.Run(() => _storage.RemoveDeployedConfigAsync(name)) fire-and-forget with fault log) + _deployedInstanceNames.Remove(name) + UpdateInstanceCounts().
- HandleDeployPersistenceResult: only reply if _inFlightDeployments.Remove(result.InstanceName) returns true (both success and failure branches) — if the Terminated path already failed the deployment, swallow the late persistence Success (log Debug). Both orders (Terminated-then-persistence, persistence-then-Terminated) now produce exactly one reply.
Run → PASS; re-run the full DM suite plus Task 4's test (the validation-gate reply path must not double-reply): dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~DeploymentManager".
Commit: git add src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/DeploymentManagerActor.cs tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/DeploymentManagerActorTests.cs && git commit -m "fix(site-runtime): deployment whose Instance Actor dies during init reports Failed, exactly-once reply (S6)"

Task 16: DataConnectionActor — capture `_adapter` into locals before every background task

Classification: high-risk (actor-state confinement) Estimated implement time: ~4 min Parallelizable with: 3, 5, 9, 10, 11, 12, 13, 19, 21, 22, 24 Files:

Modify: src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Actors/DataConnectionActor.cs (HandleSubscribe ~707–758, HandleWriteBatch ~1155–1202, HandleWrite, ReSubscribeAll ~1594–1605; audit every Task.Run/local-async-function for _adapter reads)
Test: tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests/DataConnectionActorTests.cs

Failing test (deterministic via a gated adapter — the in-flight subscribe must finish against the adapter that STARTED it, even after a failover swaps _adapter):

[Fact]
public async Task InFlightSubscribe_ContinuesOnOriginalAdapter_AfterFailover()
{
    var subscribeGate = new TaskCompletionSource<string>();
    var primaryAdapter = Substitute.For<IDataConnection>();
    primaryAdapter.ConnectAsync(default!).ReturnsForAnyArgs(Task.CompletedTask);
    primaryAdapter.SubscribeAsync(default!, default!).ReturnsForAnyArgs(
        _ => subscribeGate.Task,                    // tag 1 blocks mid-flight
        _ => Task.FromResult("sub-2"));             // tag 2 resumes after failover
    var backupAdapter = Substitute.For<IDataConnection>();
    var factory = Substitute.For<IDataConnectionFactory>();
    factory.Create(default!, default!).ReturnsForAnyArgs(backupAdapter);

    var actor = CreateConnectedActor(primaryAdapter, factory, backupConfig: SomeBackupConfig()); // harness
    actor.Tell(new SubscribeTagsRequest(NewCorrelation(), "inst", "conn", new[] { "t1", "t2" }, Now()));

    TriggerFailoverToBackup(actor); // drive AdapterDisconnected + failed reconnects past retry count

    subscribeGate.SetResult("sub-1"); // release the in-flight background loop
    AwaitAssert(async () =>
    {
        await primaryAdapter.ReceivedWithAnyArgs(2).SubscribeAsync(default!, default!);
        await backupAdapter.DidNotReceiveWithAnyArgs().SubscribeAsync(default!, default!);
    });
}

(Build on the existing DataConnectionActorTests failover harness — it already drives CountFailureAndMaybeFailover via consecutive ConnectResult failures.) 2. Run: dotnet test tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests --filter "FullyQualifiedName~InFlightSubscribe_ContinuesOnOriginalAdapter" → FAIL (later loop iterations read the swapped _adapter field and hit the backup with the old generation). 3. Implement — one pattern, four sites:

HandleSubscribe: var adapter = _adapter; on the actor thread before Task.Run; use adapter.SubscribeAsync(...) and SeedTagsAsync(adapter, tagsToSeed) inside the task. Update the confinement comment at ~689 to note the adapter is captured too.
HandleWriteBatch: var adapter = _adapter; before RunAsync(); use adapter.WriteBatchAsync / adapter.WriteAsync (the second read currently happens after the first await, off-thread).
HandleWrite: same audit/fix (the single-write path has the same local-async-function shape).
ReSubscribeAll: the per-tag SubscribeAsync loop runs on the actor thread (safe), but capture var subscribeAdapter = _adapter; and use it in the loop anyway for symmetry with reseedAdapter — zero cost, removes the last field read reachable from this method's continuations.

Run → PASS; full DCL actor suite: dotnet test tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests --filter "FullyQualifiedName~DataConnectionActor".
Commit: git add src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Actors/DataConnectionActor.cs tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests/DataConnectionActorTests.cs && git commit -m "fix(dcl): capture adapter into locals before background tasks — no off-thread reads of the mutable _adapter field (S7)"

Task 17: DCL — tag→instances reverse index for the value fan-out hot path

Classification: high-risk (hottest DCL loop, index-consistency invariants) Estimated implement time: ~5 min Parallelizable with: 5, 9, 10, 11, 12, 13, 19, 21, 22, 24 Files:

Modify: src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Actors/DataConnectionActor.cs (HandleTagValueReceived ~1693–1738; every _subscriptionsByInstance mutation site)
Test: tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests/DataConnectionActorTests.cs

Failing test (behavioral — index correctness across unsubscribe, which a naive index gets wrong):

[Fact]
public void TagFanOut_AfterOneInstanceUnsubscribes_OnlyRemainingSubscriberReceives()
{
    var actor = CreateConnectedActorWithInstantSubscribes(); // adapter substitute returns sub ids immediately
    var probeA = CreateTestProbe(); var probeB = CreateTestProbe();

    actor.Tell(SubscribeRequest("inst-A", new[] { "shared-tag" }), probeA.Ref);
    actor.Tell(SubscribeRequest("inst-B", new[] { "shared-tag" }), probeB.Ref);
    probeA.ExpectMsg<SubscribeTagsResponse>(); probeB.ExpectMsg<SubscribeTagsResponse>();
    probeA.FishForMessage(m => m is TagValueUpdate, hint: "seed"); // drain seeds as the harness does

    actor.Tell(UnsubscribeRequest("inst-A", new[] { "shared-tag" }), probeA.Ref);
    DeliverTagValue(actor, "shared-tag", 42); // helper: Tell TagValueReceived with current generation

    probeB.ExpectMsg<TagValueUpdate>(u => u.TagPath == "shared-tag");
    probeA.ExpectNoMsg(TimeSpan.FromMilliseconds(300));
}

This passes today via the O(N) scan — write it FIRST and confirm it passes (pin the behavior), then convert HandleTagValueReceived to the index and confirm it STILL passes. Add one more assertion-style test that also passes pre-change (two instances, disjoint tags, each receives only its own) — these are the regression net for the index bookkeeping. 2. Run: dotnet test tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests --filter "FullyQualifiedName~TagFanOut_" → expect PASS pre-change (documented pin), then proceed. 3. Implement:

Field: private readonly Dictionary<string, HashSet<string>> _instancesByTag = new(); with XML doc mirroring _tagSubscriberCount's rationale.
Add/remove alongside every _subscriptionsByInstance[inst] tag-set mutation. Grep _subscriptionsByInstance and cover: HandleSubscribeCompleted (tags added), HandleUnsubscribe (tags removed + instance entry removal), the orphan-release guards (~836–985, 1795–1858 region), and any instance-removal path. Write two private helpers IndexTag(string tag, string inst) / UnindexTag(string tag, string inst) (removing the tag key when its set empties) and use them at every site — no inline mutation.
HandleTagValueReceived: replace the _subscriptionsByInstance scan with if (_instancesByTag.TryGetValue(msg.TagPath, out var insts)) foreach (var inst in insts) if (_subscribers.TryGetValue(inst, out var s)) s.Tell(...).
ReSubscribeAll preserves _subscriptionsByInstance — so it must NOT clear _instancesByTag (it is derived from the same preserved state). Note this in a comment.

Run the fan-out tests + full DCL suite → PASS.
Commit: git add src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Actors/DataConnectionActor.cs tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests/DataConnectionActorTests.cs && git commit -m "perf(dcl): tag->instances reverse index replaces per-update all-instances scan (P3)"

Task 18: InstanceActor — per-attribute child-subscription routing

Classification: high-risk (hot-path routing semantics for scripts AND alarms) Estimated implement time: ~5 min Parallelizable with: 1, 2, 5, 10, 12, 13, 17, 19, 22, 24 Files:

Modify: src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/InstanceActor.cs (CreateChildActors ~1348–1488, PublishAndNotifyChildren ~1179–1209)
Test: tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/InstanceActorChildAttributeRaceTests.cs (or a new InstanceActorChildRoutingTests.cs)

Failing test:

[Fact]
public void AttributeChange_IsRoutedOnlyToInterestedChildren()
{
    // Instance with: script S_A (ValueChange on "A"), script S_B (ValueChange on "B"),
    // script S_expr (Expression trigger). Script bodies record executions via the
    // static-recording pattern the existing tests use.
    var instance = CreateInstance(ScriptsAB_and_Expression());

    SetStatic(instance, "A", 1); // publishes AttributeValueChanged for A

    AwaitAssert(() =>
    {
        Assert.Equal(1, Executions("S_A"));
        Assert.Equal(0, Executions("S_B"));       // FAILS pre-change only via mailbox counting…
    });
}

S_B never executes today either (its own trigger check discards) — so assert on message delivery, not execution: give the routing seam a test hook. Concretely: make the routing maps internal and unit-test map construction directly (InstanceActor exposes internal IReadOnlyDictionary<string, List<IActorRef>> ChildrenByMonitoredAttribute / internal IReadOnlyList<IActorRef> AllChangeChildren next to the existing ScriptActorCount diagnostics), plus a behavioral test that the Expression script still fires on ANY attribute change and the ValueChange script still fires on its own attribute:

[Fact]
public void RoutingMaps_AreBuiltFromTriggerConfigs()
{
    var actor = CreateInstanceActorRef(ScriptsAB_and_Expression());
    var underlying = GetUnderlyingActor(actor); // TestActorRef pattern used elsewhere in this file
    Assert.Single(underlying.ChildrenByMonitoredAttribute["A"]);
    Assert.Single(underlying.ChildrenByMonitoredAttribute["B"]);
    Assert.Single(underlying.AllChangeChildren); // the Expression script
}

Run: dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~RoutingMaps_AreBuiltFromTriggerConfigs" → FAIL.
Implement:
- Fields: private readonly Dictionary<string, List<IActorRef>> _childrenByMonitoredAttribute = new(); private readonly List<IActorRef> _allChangeChildren = new();
- In CreateChildActors, after creating each script/alarm actor, classify:
  - Script trigger valuechange/conditional → parse attributeName from TriggerConfiguration JSON (reuse/extract the parse ScriptActor.ParseTriggerConfig does — lift a tiny static helper TriggerRouting.TryGetMonitoredAttribute(triggerType, triggerConfigJson, out string attr) into Scripts/ so ScriptActor and InstanceActor share one parser). Add to _childrenByMonitoredAttribute[attr].
  - Script trigger expression → _allChangeChildren (its snapshot must stay current).
  - Script trigger interval/call/unparseable → _allChangeChildren if unparseable (fail-open, log warning), else no routing (interval/call scripts never react to attribute changes; their _attributeSnapshot is only read by expression evaluation, which they don't have — verify this before excluding, it is true today at ScriptActor where the snapshot is read only in EvaluateExpressionTrigger).
  - Alarms: same classification via the alarm's trigger type — Expression → all bucket; HiLo/ValueMatch/RangeViolation/RateOfChange → monitored attribute (parse from the alarm config the same way AlarmActor.IsMonitoredAttribute derives it); unparseable → all bucket.
- PublishAndNotifyChildren: replace the two full loops with foreach (var c in _allChangeChildren) c.Tell(changed); + if (_childrenByMonitoredAttribute.TryGetValue(changed.AttributeName, out var interested)) foreach (var c in interested) c.Tell(changed);. Children keep their own trigger gates (defense in depth — routing is an optimization, not the correctness gate).
- Note: changed.AttributeName on the DCL path is the canonical name (HandleTagValueUpdate maps tag→attributes before publishing) — key the map by canonical name; confirm against HandleTagValueUpdate.
Run new tests + the whole InstanceActor/ScriptActor/AlarmActor suites: dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~InstanceActor|FullyQualifiedName~ScriptActor|FullyQualifiedName~AlarmActor" → PASS.
Commit: git add src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/InstanceActor.cs src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/ScriptActor.cs src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Scripts tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests && git commit -m "perf(site-runtime): per-attribute child routing in InstanceActor — attribute changes reach only interested children (P2)"

Task 19: SiteStorageService — WAL journal mode + busy timeout

Classification: small Estimated implement time: ~3 min Parallelizable with: 1, 2, 3, 4, 5, 9, 10, 11, 12, 13, 22, 24 Files:

Modify: src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Persistence/SiteStorageService.cs (ctor ~21–25, InitializeAsync ~41–132)
Test: tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Persistence/ (add to the existing SiteStorageService test file)

Failing test:

[Fact]
public async Task Initialize_EnablesWalJournalMode()
{
    var service = CreateTempFileStorageService(); // existing temp-SQLite fixture pattern
    await service.InitializeAsync();

    await using var conn = service.CreateConnection();
    await conn.OpenAsync();
    await using var cmd = conn.CreateCommand();
    cmd.CommandText = "PRAGMA journal_mode;";
    Assert.Equal("wal", ((string)(await cmd.ExecuteScalarAsync())!).ToLowerInvariant());
}

Note: WAL requires a file-backed DB — if the existing fixture uses :memory:, use the temp-file variant (several persistence tests already do). 2. Run: dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~Initialize_EnablesWalJournalMode" → FAIL (delete). 3. Implement:

In InitializeAsync, before creating tables: execute PRAGMA journal_mode=WAL; (persistent, database-level — one-time). Log the resulting mode; :memory:/unsupported filesystems fall back gracefully (don't throw if the pragma returns non-wal, just warn).
In the ctor, normalize the connection string through SqliteConnectionStringBuilder and ensure DefaultTimeout is at least 5 s (Microsoft.Data.Sqlite uses the command timeout as its busy handler — an explicit floor documents intent even though the library default is 30): only raise it if the caller set something lower. Add an XML remark on CreateConnection() documenting WAL + busy-handling so the repositories layered on it inherit the story.

Run → PASS; run the persistence folder suite: dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~SiteStorageService".
Commit: git add src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Persistence/SiteStorageService.cs tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Persistence && git commit -m "fix(site-runtime): site SQLite runs WAL with an explicit busy-timeout floor (S8) — concurrent writers stop serializing on the rollback journal"

Task 20: NativeAlarmActor — coalesce per-transition SQLite upserts

Classification: standard Estimated implement time: ~5 min Parallelizable with: 1, 2, 4, 5, 9, 10, 11, 13, 17, 22, 24 Files:

Modify: src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/NativeAlarmActor.cs (PersistUpsert ~420–428, timer plumbing)
Modify: src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Persistence/SiteStorageService.cs (new UpsertNativeAlarmsAsync batch — one connection + transaction)
Test: tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/NativeAlarmActorTests.cs

Failing test (against the temp-SQLite storage the existing NativeAlarmActor tests use):

[Fact]
public async Task AlarmStorm_CoalescesPersistence_LatestPerSourceRefWins()
{
    var (actor, storage) = CreateNativeAlarmActorWithTempStorage(flushIntervalMs: 100);

    for (var i = 0; i < 50; i++)
        DeliverLiveTransition(actor, sourceRef: $"ref-{i % 5}", severity: i); // 50 transitions, 5 refs

    await Task.Delay(500); // > flush interval
    var rows = await storage.GetNativeAlarmsAsync("inst", "src");
    Assert.Equal(5, rows.Count);
    // Latest transition per ref persisted (severity of the LAST write for ref-4 is 49):
    Assert.Contains(rows, r => r.ConditionJson.Contains("49"));
}

Run: dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~AlarmStorm_CoalescesPersistence" → FAIL only on the mechanism (row content passes either way) — so also assert the batch API is used: add UpsertNativeAlarmsAsync first (it won't exist → compile-fail = the failing signal), or count connections via a storage wrapper. Simplest honest failing signal: write the test calling storage.UpsertNativeAlarmsAsync(...) shape via the actor and assert rows — compile failure is the RED step.
Implement:
- SiteStorageService.UpsertNativeAlarmsAsync(string instance, string source, IReadOnlyList<(string SourceRef, string ConditionJson, DateTimeOffset At)> rows) — one connection, one transaction, one prepared command re-bound per row.
- NativeAlarmActor: replace direct PersistUpsert(t) calls (in ApplyLiveTransition and ApplySnapshotSwap) with _dirtyUpserts[t.SourceReference] = t; + arm a 100 ms single-shot flush timer if not armed (ScheduleTellOnceCancelable, internal FlushPersistence message). The flush handler snapshots + clears _dirtyUpserts, calls the batch API fire-and-forget with the existing OnlyOnFaulted logging. PersistDelete stays immediate (rare; also remove the ref from _dirtyUpserts so a delete isn't resurrected by a pending upsert). PostStop: best-effort final flush (fire-and-forget).
- Rehydration semantics unchanged: worst case a crash loses ≤100 ms of mirror writes; the (re)subscribe snapshot swap reconciles — note this in the class doc.
Run → PASS; full NativeAlarmActor suite green.
Commit: git add src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/NativeAlarmActor.cs src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Persistence/SiteStorageService.cs tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/NativeAlarmActorTests.cs && git commit -m "perf(site-runtime): coalesce native-alarm mirror persistence into batched flushes (P4)"

Task 21: Persist native-alarm display metadata for rehydration

Classification: standard (SQLite schema migration — additive column) Estimated implement time: ~5 min Parallelizable with: 1, 2, 4, 5, 9, 10, 11, 13, 22, 24 Files:

Modify: src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Persistence/SiteStorageService.cs (native_alarm_state schema ~117–124, MigrateSchemaAsync, upsert/read methods)
Modify: src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/NativeAlarmActor.cs (PersistUpsert/flush ~420–428, HandleRehydration ~134–174)
Test: tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/NativeAlarmActorTests.cs

Failing test:

[Fact]
public async Task Rehydration_RestoresDisplayMetadata()
{
    var (actor1, storage) = CreateNativeAlarmActorWithTempStorage();
    DeliverLiveTransition(actor1, sourceRef: "ref-1",
        alarmTypeName: "HighLevelAlarm", category: "Process", message: "Tank overflow");
    await FlushAndStop(actor1);

    var (actor2, parentProbe) = RecreateActorOnSameStorage(storage); // rehydrates in PreStart
    var emitted = parentProbe.FishForMessage<AlarmStateChanged>(m => m.SourceReference == "ref-1");
    Assert.Equal("HighLevelAlarm", emitted.AlarmTypeName);   // empty string today
    Assert.Equal("Process", emitted.Category);
    Assert.Equal("Tank overflow", emitted.Message);
}

Run: dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~Rehydration_RestoresDisplayMetadata" → FAIL (metadata comes back string.Empty).
Implement:
- Schema: add metadata_json TEXT to the CREATE TABLE native_alarm_state DDL AND await TryAddColumnAsync(connection, "native_alarm_state", "metadata_json", "TEXT"); in MigrateSchemaAsync (existing-DB upgrade).
- Persist: serialize a small record NativeAlarmMetadata(string AlarmTypeName, string Category, string Message, string CurrentValue, string LimitValue) from the transition into metadata_json in the upsert (single + Task 20 batch). NativeAlarmRow gains string? MetadataJson.
- Rehydrate: in HandleRehydration, deserialize MetadataJson (null/malformed → empty strings, current behavior) and build the placeholder NativeAlarmTransition with the restored fields instead of string.Empty.
Run → PASS; full NativeAlarmActor + persistence suites.
Commit: git add src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Persistence/SiteStorageService.cs src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/NativeAlarmActor.cs tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/NativeAlarmActorTests.cs && git commit -m "feat(site-runtime): persist native-alarm display metadata so rehydrated conditions render fully (UA4)"

Task 22: CertStoreActor — export trusted certs (thumbprint + DER) for reconciliation

Classification: standard (additive Commons message contract) Estimated implement time: ~4 min Parallelizable with: 1, 2, 3, 4, 5, 9, 10, 11, 12, 13, 16, 17, 19, 24 Files:

Modify: src/ZB.MOM.WW.ScadaBridge.Commons/Messages/Management/ (the file holding WriteCertToLocalStore/ListLocalCerts — add ExportLocalTrustedCerts + LocalCertExport)
Modify: src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/CertStoreActor.cs (new handler)
Test: tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/CertStoreActorTests.cs

Failing test (reuse the temp-dir PKI fixture in CertStoreActorTests):

[Fact]
public void ExportLocalTrustedCerts_ReturnsThumbprintAndDerBytes()
{
    var (actor, testCertDer, thumbprint) = CreateCertStoreWithOneTrustedCert(); // existing fixture helpers
    actor.Tell(new ExportLocalTrustedCerts());
    var export = ExpectMsg<LocalCertExport>();
    var entry = Assert.Single(export.Certs);
    Assert.Equal(thumbprint, entry.Thumbprint);
    Assert.Equal(testCertDer, entry.DerBytes);
}

Run: dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~ExportLocalTrustedCerts_ReturnsThumbprintAndDerBytes" → FAIL (types don't exist).
Implement:
- Commons (additive, next to the existing cert messages): public sealed record ExportLocalTrustedCerts(); public sealed record ExportedCert(string Thumbprint, byte[] DerBytes); public sealed record LocalCertExport(bool Success, string? Error, IReadOnlyList<ExportedCert> Certs);
- CertStoreActor: Receive<ExportLocalTrustedCerts> — enumerate *.der in _trustedStoreDir, load each via X509CertificateLoader/new X509Certificate2(bytes) (match how HandleList reads them today) to compute the thumbprint, reply LocalCertExport(true, null, list); any IO error → LocalCertExport(false, ex.Message, []).
Run → PASS.
Commit: git add src/ZB.MOM.WW.ScadaBridge.Commons src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/CertStoreActor.cs tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/CertStoreActorTests.cs && git commit -m "feat(site-runtime): CertStoreActor exports trusted certs (thumbprint+DER) for node reconciliation (UA1 groundwork)"

Task 23: Cert-trust reconcile-on-join — singleton pushes its trusted set to a (re)joining site node

Classification: high-risk (cluster membership events) Estimated implement time: ~5 min Parallelizable with: 1, 2, 5, 9, 10, 12, 17, 19, 24 Files:

Modify: src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/DeploymentManagerActor.cs (PreStart/PostStop cluster subscription; new handlers near the cert section ~900–1007)
Test: tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/DeploymentManagerActorTests.cs

Failing test — test the reconcile flow via the internal message (constructing a real ClusterEvent.MemberUp in a unit test is impractical; the MemberUp handler itself is a two-liner producing this message). Use the same "forwarding actor named cert-store → TestProbe" trick the existing cert broadcast tests use:

[Fact]
public void SiteNodeJoined_PushesLocalTrustedCertsToJoinedNode()
{
    var certStoreProbe = InstallCertStoreForwarder(); // /user/cert-store -> probe (existing pattern)
    var dm = CreateDeploymentManager();

    dm.Tell(new DeploymentManagerActor.SiteNodeJoined(Cluster.Get(Sys).SelfAddress));

    certStoreProbe.ExpectMsg<ExportLocalTrustedCerts>(); // 1) read own store
    dm.Tell(new LocalCertExport(true, null,
        new[] { new ExportedCert("ABC123", new byte[] { 1, 2, 3 }) }), certStoreProbe.Ref);
    // 2) pushed to the joined node's cert-store (self address in-test -> same selection):
    var write = certStoreProbe.ExpectMsg<WriteCertToLocalStore>(TimeSpan.FromSeconds(5));
    Assert.Equal("ABC123", write.Thumbprint);
}

(Match WriteCertToLocalStore's real field names/args from Commons when writing the assertion.) 2. Run: dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~SiteNodeJoined_PushesLocalTrustedCerts" → FAIL. 3. Implement:

PreStart: Cluster.Get(Context.System).Subscribe(Self, ClusterEvent.SubscriptionInitialStateMode.InitialStateAsEvents, typeof(ClusterEvent.MemberUp)); PostStop: Cluster.Get(Context.System).Unsubscribe(Self);
Receive<ClusterEvent.MemberUp>: if the member has the site role and is not the singleton's own node → Self.Tell(new SiteNodeJoined(member.Address)) (public/internal record for testability).
Receive<SiteNodeJoined>: capture the address; ask the LOCAL cert store (Context.ActorSelection($"/user/{CertStoreActor.WellKnownName}")) with ExportLocalTrustedCerts (short CertBroadcastTimeout Ask, ContinueWith → pipe a wrapper carrying the target address). On the piped LocalCertExport (wrap it: CertReconcileExport(Address Target, LocalCertExport Export)): for each cert, Tell the target node's cert store (new RootActorPath(target) / "user" / CertStoreActor.WellKnownName) a WriteCertToLocalStore(...) — fire-and-forget, additive union (documented: removals do NOT reconcile until central persistence lands; the write is idempotent — CertStoreActor.HandleWrite overwrites by thumbprint).
Log one info line per reconcile ("Cert-trust reconcile: pushed {N} trusted cert(s) to joining node {Address}"); export failure → warning, no retry (the next MemberUp or manual trust re-broadcast heals).
Simplification for the test flow: since the test drives LocalCertExport directly at the actor, have Receive<CertReconcileExport> be the handler and also accept a bare LocalCertExport only if you keep a single pending target field — prefer the wrapper message; adjust the test to send the wrapper if needed.

Run → PASS; full DM suite green.
Commit: git add src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/DeploymentManagerActor.cs tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/DeploymentManagerActorTests.cs && git commit -m "feat(site-runtime): cert-trust reconcile-on-join — singleton pushes trusted certs to (re)joining site nodes (UA1)"

Task 24: Wire `ConfigFetchRetryCount` into the standby replication fetch

Classification: small Estimated implement time: ~4 min Parallelizable with: 1, 2, 3, 4, 5, 9, 10, 12, 13, 16, 17, 19, 22 Files:

Modify: src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/SiteReplicationActor.cs (HandleApplyConfigDeploy ~153–217; ctor gains SiteRuntimeOptions? options = null)
Modify: src/ZB.MOM.WW.ScadaBridge.SiteRuntime/SiteRuntimeOptions.cs (fix the "Reserved — not yet wired" XML doc, lines 63–67)
Modify: the SiteReplicationActor Props construction site (Host/ServiceCollectionExtensions — grep new SiteReplicationActor( / Props.Create(() => new SiteReplicationActor) to pass options
Test: tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/SiteReplicationActorTests.cs

Failing test:

[Fact]
public void ReplicatedFetch_RetriesUpToConfigFetchRetryCount()
{
    var attempts = 0;
    var fetcher = Substitute.For<IDeploymentConfigFetcher>();
    fetcher.FetchAsync(default!, default!, default!, default).ReturnsForAnyArgs(_ =>
        ++attempts < 3
            ? Task.FromException<string>(new HttpRequestException("central hiccup"))
            : Task.FromResult("{\"config\":true}"));
    var actor = CreateReplicationActor(fetcher,
        new SiteRuntimeOptions { ConfigFetchRetryCount = 3 }); // existing harness + new arg

    actor.Tell(new ApplyConfigDeploy("inst", "dep-1", "r1", true, "http://central", "tok"));

    AwaitAssert(() =>
    {
        Assert.Equal(3, attempts);
        _storage.Received(1).StoreDeployedConfigIfNewerAsync("inst", "{\"config\":true}", "dep-1", "r1", true);
    }, TimeSpan.FromSeconds(10));
}

Run: dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~ReplicatedFetch_RetriesUpToConfigFetchRetryCount" → FAIL (single attempt).
Implement: replace the single FetchAsync(...).ContinueWith(...) with a local async function launched fire-and-forget (same best-effort model):

var maxAttempts = Math.Max(1, _options?.ConfigFetchRetryCount ?? 1);
_ = FetchWithRetryAsync(); // errors fully handled inside; observes every fault
async Task FetchWithRetryAsync()
{
    for (var attempt = 1; attempt <= maxAttempts; attempt++)
    {
        try
        {
            var json = await _configFetcher.FetchAsync(msg.CentralFetchBaseUrl, msg.DeploymentId, msg.FetchToken, CancellationToken.None);
            await _storage.StoreDeployedConfigIfNewerAsync(msg.InstanceName, json, msg.DeploymentId, msg.RevisionHash, msg.IsEnabled);
            return;
        }
        catch (DeploymentConfigFetchException fex) when (fex.IsSuperseded)
        { /* keep the existing "superseded" info log */ return; }
        catch (Exception ex)
        {
            // keep existing per-failure log wording; add attempt counters
            if (attempt < maxAttempts) await Task.Delay(TimeSpan.FromSeconds(2));
            else _logger.LogError(ex, "Replicated config fetch failed after {Attempts} attempt(s) for {Instance} (deployment {DeploymentId}) — reconciliation will backstop", maxAttempts, msg.InstanceName, msg.DeploymentId);
        }
    }
}

Update the SiteRuntimeOptions.ConfigFetchRetryCount XML doc: remove "Reserved … not yet wired"; state "Bounded attempt count (including the first) for the standby's replicated-config fetch; 2 s fixed delay between attempts; superseded fetches never retry." 4. Run → PASS; run the replication suite. 5. Commit: git add src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/SiteReplicationActor.cs src/ZB.MOM.WW.ScadaBridge.SiteRuntime/SiteRuntimeOptions.cs tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/SiteReplicationActorTests.cs && git commit -m "feat(site-runtime): wire ConfigFetchRetryCount into the standby replicated-config fetch (UA2)" (include the Props-site file in the git add if touched)

Task 25: Low-severity code cleanups — S9, S10, C2, C6

Classification: small (batched Low findings; concrete edits below) Estimated implement time: ~5 min Parallelizable with: 13, 19, 21, 22, 24 Files:

Modify: src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Actors/DataConnectionActor.cs (PostStop ~216–221; HandleSubscribeAlarms ~1763)
Modify: src/ZB.MOM.WW.ScadaBridge.SiteEventLogging/SiteEventLogger.cs (comment ~215–218)
Modify: src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/ScriptActor.cs (EvaluateCondition fallback ~474–477)
Test: tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/ScriptActorTests.cs, tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests/DataConnectionActorAlarmTests.cs

C6 failing test (culture-independence of the non-numeric fallback):

[Fact]
public void ConditionalTrigger_NonNumericFallback_IsCultureInvariant()
{
    var original = CultureInfo.CurrentCulture;
    try
    {
        CultureInfo.CurrentCulture = new CultureInfo("de-DE"); // 1.5 -> "1,5"
        // Conditional trigger: operator "==", Threshold = 1.5; value "1,5" fails
        // invariant numeric parse -> string fallback. Old code compares against
        // Threshold.ToString() == "1,5" under de-DE and FIRES; invariant must NOT.
        var fired = DriveConditionalTrigger(threshold: 1.5, op: "==", value: "1,5"); // harness helper
        Assert.False(fired);
    }
    finally { CultureInfo.CurrentCulture = original; }
}

S10 failing test (EventFilter on the warning):

[Fact]
public void SecondAlarmSubscriber_WithDifferentFilter_LogsOverwriteWarning()
{
    var actor = CreateConnectedAlarmCapableActor(); // existing alarm-test harness
    actor.Tell(AlarmSubscribe("src-1", filter: "HighAlarm"), probeA.Ref);
    EventFilter.Warning(contains: "condition filter").ExpectOne(() =>
        actor.Tell(AlarmSubscribe("src-1", filter: "LowAlarm"), probeB.Ref));
}

Run both → FAIL (dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~ConditionalTrigger_NonNumericFallback" and dotnet test tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests --filter "FullyQualifiedName~SecondAlarmSubscriber_WithDifferentFilter").
Implement all four edits:
- C6 (ScriptActor.cs:476): return string.Equals(value.ToString(), config.Threshold.ToString(CultureInfo.InvariantCulture), StringComparison.Ordinal); — matches the invariant-culture handling two lines up.
- S10 (DataConnectionActor.cs, before line 1763's overwrite): if (_alarmSourceFilter.TryGetValue(request.SourceReference, out var existingFilter) && !string.Equals(existingFilter, request.ConditionFilter, StringComparison.Ordinal)) _log.Warning("[{0}] Alarm condition filter for source {1} overwritten: '{2}' -> '{3}' (last subscriber wins; co-subscribers must agree)", _connectionName, request.SourceReference, existingFilter, request.ConditionFilter);
- S9 (DataConnectionActor.PostStop): replace _ = _adapter.DisposeAsync().AsTask(); with _adapter.DisposeAsync().AsTask().ContinueWith(t => _log.Warning("[{0}] Adapter dispose faulted on stop: {1}", _connectionName, t.Exception?.GetBaseException().Message), TaskContinuationOptions.OnlyOnFaulted);
- C2 (SiteEventLogger.cs:215-218): rewrite the comment: "The channel is bounded with DropOldest, so TryWrite only returns false once the channel is completed (shutdown) — full-buffer pressure evicts the oldest pending event instead of failing the write."
Run both tests → PASS; run both touched projects' suites.
Commit: git add src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Actors/DataConnectionActor.cs src/ZB.MOM.WW.ScadaBridge.SiteEventLogging/SiteEventLogger.cs src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/ScriptActor.cs tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/ScriptActorTests.cs tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests/DataConnectionActorAlarmTests.cs && git commit -m "fix(site-runtime,dcl): low-severity cleanups — observed adapter dispose (S9), alarm-filter overwrite warning (S10), bounded-channel comment (C2), invariant-culture condition fallback (C6)"

Task 26: Design-doc updates — C3, C4, C5 + behavior notes for this plan's changes

Classification: trivial (docs only; repo rule: docs travel with code) Estimated implement time: ~5 min Parallelizable with: none (write after the behavior tasks land so the docs describe reality) Files:

Modify: docs/requirements/Component-DataConnectionLayer.md
Modify: docs/requirements/Component-SiteRuntime.md

Component-DataConnectionLayer.md:
- C3 (line ~303): change "The retry interval is defined per data connection" to "The retry interval is a shared setting for all data connections (ReconnectInterval in the Shared Settings table)" — matching the table at lines 170–177 and DataConnectionOptions.
- C4 (line ~302, item 1): qualify — "Instance Actors see the staleness immediately and publish it to the site stream/debug view; script and alarm actors are deliberately NOT re-notified on a quality-only change (no value changed — re-evaluating triggers would cause spurious firings). Scripts observe quality on their next attribute read; a computed alarm holds its last-known state until fresh values arrive."
- S1 note (connection lifecycle section): add "On every (re)connect the adapter detaches and disposes the previous protocol client (session, keep-alive timers, event loop) before creating a new one — a stale client can neither leak nor signal a disconnect against the new session."
- S10 note: add one line documenting the last-subscriber-wins alarm filter + the logged warning on mismatch.
Component-SiteRuntime.md:
- C5 (Shared Script Library section): "Shared scripts always execute with Root scope — Attributes[...] inside a shared script addresses root attributes even when invoked from a composed-module script. Callers needing module-scoped access must pass values as parameters."
- S2/UA5 (Script Trust Model / execution-timeout paragraph ~409–412): qualify "Exceeding the timeout cancels the script" → "…cancels the script cooperatively: a script blocked in synchronous I/O or a CPU loop does not observe cancellation and continues to occupy its dedicated script-execution thread. A watchdog logs the script by name when its thread has not returned within a grace period after timeout, and the scheduler's queue depth / busy-thread-age gauges are reported through Health Monitoring."
- S3 (Script Compilation Errors section ~474–477): add "Enforced at the site by a pre-compile validation gate in the Deployment Manager: all scripts, alarm on-trigger scripts, and trigger expressions are compiled off-thread before the Instance Actor is created or the config persisted; any failure rejects the deployment with the compile errors."
- S4/UA6 (Instance Actor / DCL interaction): add "A failed or unanswered tag-subscription handshake is retried on a fixed interval per connection (default 5 s), mirroring the native-alarm subscribe retry."
- S5: note the fixed-interval retry of the startup deployed-config load.
- S6: note that all Instance Actors are watched and an unexpected termination evicts the ref and fails any in-flight deployment. Deviation from the plan (2026-07-09, see master-tracker "Plan deviations during Wave 1 PLAN-03 execution"): deploy-success is NOT reported on persistence alone — the store can commit before the actor's async PreStart throws, so the design uses a two-signal join: the Instance Actor sends a new InstanceActorInitialized readiness message at the end of a successful PreStart, and the Deployment Manager reports Success only once BOTH persistence has committed AND that signal has arrived. An actor that dies during init never signals readiness → the Terminated fallback replies Failed and rolls the optimistic state back (the persisted-row rollback is deferred until the store commits so it cannot race the write). Document the readiness message + the join.
- UA1 (cert trust section ~125): add "On a site node (re)joining the cluster, the Deployment Manager singleton pushes its trusted-cert set to the joining node's CertStore (additive union). Removal reconciliation and central persistence of trust decisions remain the documented follow-up."
- UA4 (native alarm mirror section): "Persisted mirror rows include display metadata (type/category/message), so rehydrated conditions render fully before the first source snapshot."
- P2 note (attribute change fan-out ~191): "The Instance Actor routes attribute changes per monitored attribute (Expression-trigger children receive all changes); child actors keep their own trigger gates as defense in depth."
- P6 follow-up note (deployment/startup section): "Follow-up: per-instance Roslyn compiles still run inside Instance Actor start during staggered startup; moving them off-thread is deferred (affects failover time-to-recover only)."
Review with git diff docs/ — verify no stale cross-references (e.g., the C3 sentence is the only "per data connection" retry claim).
Commit: git add docs/requirements/Component-DataConnectionLayer.md docs/requirements/Component-SiteRuntime.md && git commit -m "docs(requirements): sync SiteRuntime + DCL specs with hardening changes (C3/C4/C5 + S1-S6, P2, UA1/UA4 behavior notes)"

Dependencies on other plans

Plan 05 (Templates/Deployment/Transport) owns script-trust-policy and Roslyn compile caching in central validation paths. Task 3/4 here adds a site-side compile gate that duplicates compile work per deploy — if plan 05 introduces a shared compile-cache/collectible-ALC abstraction, the DeployCompileValidator should adopt it later; no ordering dependency (this plan's gate is self-contained).
Plan 02 (Communication/S&F) owns the S&F store SQLite (StoreAndForwardStorage) — Task 19 deliberately touches only SiteStorageService. If plan 02 also enables WAL on the S&F store, the two changes are independent files.
Plan 06 (Edge integrations) owns inbound script execution (DI-scope use-after-dispose). Task 8's watchdog pattern (completed flag + grace check) is reusable there; no ordering dependency.
Plan 01 (Cluster/Host/Failover) owns SBR enablement. Task 23 (reconcile-on-join) becomes more valuable once real failover works, but functions independently.
Health-report additive fields (Task 6) touch Commons/Messages/Health/SiteHealthReport.cs — coordinate with any other plan touching that record (additive-only on both sides makes merges trivial).

Execution order

P0 (do first, in order): Task 1, Task 2 (S1 — the single highest-value fix: stops the leak and the healthy-connection flapping), then Task 3 → Task 4 (S3 — deployment honesty, matches OVERALL P1 recommendation #8).

P1: Tasks 5 → 6 → 7 (scheduler observability chain), Task 8 (watchdog), Tasks 9, 10 (P1 dispatcher relief), Task 11 → 12 (subscription retries), Task 13 (startup retry), Task 14 → 15 (dead-child bookkeeping — 15 depends on 14's Terminated fallback).

P2: Task 16 → 17 (both edit DataConnectionActor.cs, sequential), Task 18 (after 11 — both edit InstanceActor.cs), Task 19 → 20 → 21 (19 and 21 both edit SiteStorageService.cs; 20 and 21 both edit NativeAlarmActor.cs — run 19, 20, 21 in that order; 12 must precede 20 for NativeAlarmActor.cs), Task 22 → 23 (23 needs 22's messages and follows 15 in DeploymentManagerActor.cs), Task 24.

Last: Task 25 (after 9 and 17 — shared files), then Task 26 (docs describe the landed behavior).

Same-file sequencing summary: DeploymentManagerActor.cs: 4 → 13 → 14 → 15 → 23. InstanceActor.cs: 4 (doc comment) → 11 → 18. NativeAlarmActor.cs: 12 → 20 → 21. SiteStorageService.cs: 19 → 20 → 21. DataConnectionActor.cs: 16 → 17 → 25. ScriptActor.cs: 9 → 18 (helper extraction) → 25. SiteRuntimeOptions.cs: 8 → 11 → 24.

After the final task: dotnet build ZB.MOM.WW.ScadaBridge.slnx && dotnet test ZB.MOM.WW.ScadaBridge.slnx, and for cluster-runtime verification rebuild the docker image (bash docker/deploy.sh).

98 KiB Raw Blame History Unescape Escape