# Site Runtime & DCL Hardening Implementation Plan > **For Claude:** REQUIRED SUB-SKILL: Use superpowers-extended-cc:executing-plans to implement this plan task-by-task. **Goal:** Fix every finding in `archreview/03-site-runtime-dcl.md` — adapter reconnect lifecycle leaks, script-containment observability, deployment honesty (site-side compile failure must reject the deployment per spec), subscription/startup retry holes, dead-child bookkeeping, hot-path fan-out cost, and site SQLite hardening — with design docs updated alongside the code. **Architecture:** All changes are site-side, inside `ZB.MOM.WW.ScadaBridge.SiteRuntime` and `ZB.MOM.WW.ScadaBridge.DataConnectionLayer` (plus additive message/report contracts in Commons/HealthMonitoring). The actor model is preserved: every fix keeps state mutation on the actor thread (background work returns via `PipeTo`), retries use the existing fixed-interval timer idiom, and new health metrics ride the existing `ISiteHealthCollector` → `SiteHealthReport` path with additive-only contract changes. The deploy compile gate runs off-thread before actor creation so the singleton mailbox never blocks on Roslyn. **Tech Stack:** C#/.NET, Akka.NET (ReceiveActor/UntypedActor, TestKit.Xunit2), Roslyn scripting, Microsoft.Data.Sqlite, xunit + NSubstitute. Build: `dotnet build ZB.MOM.WW.ScadaBridge.slnx`. Test per-project: `dotnet test tests/ --filter `. ## Findings Coverage | Finding | Severity | Task(s) / Disposition | |---|---|---| | S1 — reconnect abandons previous client, stale `ConnectionLost` flaps healthy connection | High | 1 (OPC UA), 2 (MxGateway) | | S2 — cooperative-only timeout; no stuck-thread observability | High | 5, 6, 7, 8. Sub-suggestion "inject ADO.NET command timeouts into Database.Connection" **deferred**: scripts own their ADO.NET commands (`SqlCommand.CommandTimeout` defaults to 30 s already); the watchdog + gauges make the residual case visible. | | S3 — site compile failure doesn't reject deployment (spec drift) | High | 3, 4 | | S4 — no retry on failed tag subscription; NativeAlarm lost-response gap | Medium | 11 (InstanceActor), 12 (NativeAlarmActor) | | S5 — startup SQLite load failure aborts silently, no retry | Medium | 13 | | S6 — dead child ref in `_instanceActors` + Success reported for dead instance | Medium | 14, 15 | | S7 — background tasks read mutable `_adapter` field off-thread | Medium | 16 | | S8 — site SQLite: no WAL/busy-timeout | Low | 19 | | S9 — fire-and-forget adapter dispose in `PostStop` unobserved | Low | 25 | | S10 — alarm condition filter last-subscriber-wins, silent overwrite | Low | 25 | | P1 — trigger expressions block dispatcher threads per attribute change | High | 9 (ScriptActor), 10 (AlarmActor) | | P2 — attribute-change fan-out broadcast to every child | Medium | 18 | | P3 — DCL tag-value fan-out scans all instances per update | Medium | 17 | | P4 — per-transition SQLite upserts on native-alarm mirror | Low | 20 | | P5 — per-attribute script read is an Ask round-trip | Low | **Deferred** — correct and spec-consistent; no profiling evidence yet; batch `GetAttributes` is a follow-up when profiling flags it. | | P6 — Roslyn compiles inside `InstanceActor.PreStart` during staggered startup | Low | **Deferred** (partially mitigated) — Task 3/4 moves *deploy-path* compiles off the singleton mailbox; moving *startup-path* compiles out of `PreStart` requires an async child-creation phase (messages arriving before children exist) and is out of scope; risk is failover time-to-recover only. Logged as follow-up in Task 26 doc note. | | C1 — positive (Akka discipline) | — | No action. | | C2 — stale "unbounded channel" comment in SiteEventLogger | Low | 25 | | C3 — spec self-contradiction on reconnect interval scope | Low | 26 | | C4 — quality-only change not delivered to children vs. spec wording | Low | 26 | | C5 — shared scripts run Root scope, undocumented | Low | 26 | | C6 — `EvaluateCondition` fallback uses current-culture `ToString` | Low | 25 | | UA1 — cert-trust convergence after node downtime | Underdev. | 22, 23 (+ doc in 26). List-aggregation across both nodes and removal-reconcile **deferred** to the documented central-persistence follow-up (`Component-SiteRuntime.md:125`). | | UA2 — `ConfigFetchRetryCount` dead option | Underdev. | 24 | | UA3 — no aggregated live alarm stream | Underdev. | **Deferred** — documented M7 follow-up, central-side; out of this plan's site-runtime scope. | | UA4 — native-alarm rehydration loses display metadata | Underdev. | 21 | | UA5 — no scheduler stuck-thread/queue-depth observability | Underdev. | 5, 6, 7, 8 (same fix as S2) | | UA6 — tag-subscription failure path has no retry/site event | Underdev. | 11, 12 (same fix as S4) | | UA7 — test gaps (reconnect-stale-handler, compile-fail deploy status, subscribe race) | Underdev. | Covered by the TDD tests in Tasks 1, 2, 4, 11 | --- ### Task 1: OpcUaDataConnection — dispose + detach the previous client on reconnect **Classification:** high-risk (adapter lifecycle, concurrency with SDK callback threads) **Estimated implement time:** ~5 min **Parallelizable with:** 2, 3, 5, 9, 10, 12, 13, 19, 22, 24 **Files:** - Modify: `src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Adapters/OpcUaDataConnection.cs` (ConnectAsync, ~lines 97–108) - Test: `tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests/OpcUaDataConnectionTests.cs` 1. Add failing tests (reuse the existing `Substitute.For()` / `IOpcUaClientFactory` fixture; note the fixture returns one `_mockClient` — for these tests configure the factory to return a **fresh substitute per `Create()` call**): ```csharp [Fact] public async Task Reconnect_DisposesAndDetachesPreviousClient() { var clients = new List(); _mockFactory.Create().Returns(_ => { var c = Substitute.For(); clients.Add(c); return c; }); var details = new Dictionary { ["EndpointUrl"] = "opc.tcp://x:4840" }; await _adapter.ConnectAsync(details); // client[0] await _adapter.ConnectAsync(details); // reconnect -> client[1] Assert.Equal(2, clients.Count); await clients[0].Received(1).DisposeAsync(); await clients[1].DidNotReceive().DisposeAsync(); } [Fact] public async Task StaleClientConnectionLost_DoesNotRaiseDisconnected_AfterReconnect() { var clients = new List(); _mockFactory.Create().Returns(_ => { var c = Substitute.For(); clients.Add(c); return c; }); var details = new Dictionary { ["EndpointUrl"] = "opc.tcp://x:4840" }; var disconnects = 0; _adapter.Disconnected += () => Interlocked.Increment(ref disconnects); await _adapter.ConnectAsync(details); await _adapter.ConnectAsync(details); // reconnect resets _disconnectFired // The OLD client's keep-alive dies later — must be a no-op now. clients[0].ConnectionLost += Raise.Event(); Assert.Equal(0, disconnects); // The CURRENT client's loss must still signal. clients[1].ConnectionLost += Raise.Event(); Assert.Equal(1, disconnects); } ``` 2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests --filter "FullyQualifiedName~Reconnect_DisposesAndDetachesPreviousClient|FullyQualifiedName~StaleClientConnectionLost_DoesNotRaiseDisconnected"` → expect **FAIL** (old client never disposed; stale handler still wired). 3. Implement in `ConnectAsync`, immediately before `_client = _clientFactory.Create();`: ```csharp // Reconnect reuses this adapter instance: detach + dispose the previous // client so (a) its session/keep-alive/socket do not leak per flap and // (b) its late ConnectionLost cannot flap the NEW healthy session after // _disconnectFired is reset below. Dispose is best-effort fire-and-forget // (CloseAsync against a dead server can block for the operation timeout); // faults are logged, never thrown into the connect path. if (_client is { } previousClient) { previousClient.ConnectionLost -= OnClientConnectionLost; _ = previousClient.DisposeAsync().AsTask().ContinueWith( t => _logger.LogWarning(t.Exception?.GetBaseException(), "Disposing previous OPC UA client failed for {Endpoint}", _endpointUrl), TaskContinuationOptions.OnlyOnFaulted); } ``` Also stop the previous heartbeat monitor if active (call the existing `StopHeartbeatMonitor()` before creating the new client) so the old `StaleTagMonitor` cannot fire against the new session. 4. Run the two tests → expect **PASS**. Run the full adapter suite: `dotnet test tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests --filter "FullyQualifiedName~OpcUaDataConnectionTests"` → all green. 5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Adapters/OpcUaDataConnection.cs tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests/OpcUaDataConnectionTests.cs && git commit -m "fix(dcl): dispose+detach previous OPC UA client on reconnect (S1) — stops session leak and stale ConnectionLost flapping"` ### Task 2: MxGatewayDataConnection — dispose previous client + cancel previous event loop on reconnect **Classification:** high-risk (adapter lifecycle) **Estimated implement time:** ~4 min **Parallelizable with:** 1, 3, 5, 9, 10, 12, 13, 19, 22, 24 **Files:** - Modify: `src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Adapters/MxGatewayDataConnection.cs` (ConnectAsync, ~lines 67–88) - Test: create `tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests/Adapters/MxGatewayDataConnectionReconnectTests.cs` 1. Failing test (mirror the OPC UA fixture shape with `Substitute.For()` / `IMxGatewayClientFactory`; capture the `CancellationToken` handed to `RunEventLoopAsync`): ```csharp [Fact] public async Task Reconnect_DisposesPreviousClient_AndCancelsPreviousEventLoop() { var clients = new List(); var loopTokens = new List(); var factory = Substitute.For(); factory.Create().Returns(_ => { var c = Substitute.For(); c.RunEventLoopAsync(Arg.Any>(), Arg.Do(t => loopTokens.Add(t))) .Returns(Task.Delay(Timeout.Infinite)); clients.Add(c); return c; }); var adapter = new MxGatewayDataConnection(factory, NullLogger.Instance); var details = new Dictionary { ["Endpoint"] = "localhost:5000", ["ApiKey"] = "k" }; await adapter.ConnectAsync(details); await adapter.ConnectAsync(details); // reconnect await clients[0].Received(1).DisposeAsync(); Assert.True(loopTokens[0].IsCancellationRequested); // old loop cancelled Assert.False(loopTokens[1].IsCancellationRequested); // new loop live } ``` (Adjust the `RunEventLoopAsync` signature/`MxGatewayValueUpdate` names to the real `IMxGatewayClient` members — check `src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Adapters/IMxGatewayClient.cs`.) 2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests --filter "FullyQualifiedName~MxGatewayDataConnectionReconnectTests"` → **FAIL**. 3. Implement at the top of `ConnectAsync` (before `_client = _clientFactory.Create();`): ```csharp // Reconnect reuses this adapter: cancel the previous event loop and dispose // the previous client so a flapping gateway cannot accumulate orphaned // streams/sockets, and the old loop's fault path cannot signal Disconnected // against the new session after the guard resets below. if (_eventLoopCts is { } previousLoopCts) { previousLoopCts.Cancel(); previousLoopCts.Dispose(); _eventLoopCts = null; } if (_client is { } previousClient) { _ = previousClient.DisposeAsync().AsTask().ContinueWith( t => _logger.LogWarning(t.Exception?.GetBaseException(), "Disposing previous MxGateway client failed"), TaskContinuationOptions.OnlyOnFaulted); } ``` Note ordering: keep `Interlocked.Exchange(ref _disconnectFired, 0)` AFTER the cancel/dispose, so a fault raised by the dying loop during teardown is absorbed by the *old* guard cycle. Also clear `_subs`/`_tagToSub` is NOT needed (re-subscribe repopulates via the actor), but verify `RunEventLoopAsync`'s catch respects the cancelled token → `OperationCanceledException` path, which it already does. 4. Run → **PASS**; then full project: `dotnet test tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests` → green. 5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Adapters/MxGatewayDataConnection.cs tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests/Adapters/MxGatewayDataConnectionReconnectTests.cs && git commit -m "fix(dcl): cancel old event loop + dispose old MxGateway client on reconnect (S1)"` ### Task 3: Pure deploy-time compile validator for a flattened config **Classification:** standard **Estimated implement time:** ~5 min **Parallelizable with:** 1, 2, 5, 9, 10, 12, 19, 22, 24 **Files:** - Create: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Deployment/DeployCompileValidator.cs` - Test: create `tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Deployment/DeployCompileValidatorTests.cs` 1. Failing test: ```csharp public class DeployCompileValidatorTests { private static string ConfigWith(string scriptCode) => JsonSerializer.Serialize(new FlattenedConfiguration { Scripts = new List { new() { CanonicalName = "S1", Code = scriptCode, TriggerType = "Call" } }, Attributes = new List(), Alarms = new List(), NativeAlarmSources = new List() }); [Fact] public void BrokenScript_ReturnsErrors() { var validator = new DeployCompileValidator(new ScriptCompilationService(/* mirror ctor args used in ScriptCompilationServiceTests */)); var errors = validator.Validate(ConfigWith("this is not C# ~~~")); Assert.NotEmpty(errors); Assert.Contains(errors, e => e.Contains("S1")); } [Fact] public void ValidScript_ReturnsNoErrors() { var validator = new DeployCompileValidator(new ScriptCompilationService(/* as above */)); Assert.Empty(validator.Validate(ConfigWith("return 1 + 1;"))); } [Fact] public void BrokenTriggerExpression_ReturnsErrors() { var cfg = /* config with one Expression-triggered script whose TriggerConfiguration JSON carries expression: "1 +" (mirror shape used in ScriptActorTests) */; var validator = new DeployCompileValidator(new ScriptCompilationService(/* as above */)); Assert.NotEmpty(validator.Validate(cfg)); } } ``` (Mirror `FlattenedConfiguration`/`ResolvedScript` construction and the `ScriptCompilationService` ctor from the existing `ScriptCompilationServiceTests` / `InstanceActorTests` fixtures.) 2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~DeployCompileValidatorTests"` → **FAIL** (type does not exist). 3. Implement `DeployCompileValidator`: a small class taking the same compilation-service type `InstanceActor` uses; method `IReadOnlyList Validate(string configJson)`: - Deserialize `FlattenedConfiguration`; malformed JSON → single error `"Configuration JSON failed to deserialize: …"` (this also fronts Task 15's poison-config case). - For each `config.Scripts`: `_compilationService.Compile(script.CanonicalName, script.Code)`; on failure add `$"Script '{script.CanonicalName}': {string.Join("; ", result.Errors)}"`. - For each alarm with `OnTriggerScriptCanonicalName`: resolve the script def (missing def → error) and compile it (mirroring `InstanceActor.CreateChildActors` lines 1404–1424). - For each Expression-trigger script/alarm: extract via `TriggerExpressionGlobals.ExtractExpression` and `CompileTriggerExpression`; failures are errors (mirroring `InstanceActor.CompileTriggerExpression`). - Pure and thread-safe (no actor state) — callable from `Task.Run`. 4. Run → **PASS**. 5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Deployment/DeployCompileValidator.cs tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Deployment/DeployCompileValidatorTests.cs && git commit -m "feat(site-runtime): pure deploy-time compile validator for flattened configs (S3 groundwork)"` ### Task 4: Reject the deployment on site-side compile failure (per spec) **Classification:** high-risk (deployment pipeline, singleton actor) **Estimated implement time:** ~5 min **Parallelizable with:** 1, 2, 12, 19, 22, 24 **Files:** - Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/DeploymentManagerActor.cs` (HandleDeploy entry + new piped message) - Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/InstanceActor.cs` (fix the contradictory XML doc at line 1339) - Test: `tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/DeploymentManagerActorTests.cs` 1. Failing test (reuse the existing DeploymentManagerActor TestKit harness/fixture in that file): ```csharp [Fact] public void Deploy_WithNonCompilingScript_ReportsFailed_AndCreatesNoInstanceActor() { var badConfig = /* FlattenedConfiguration JSON with one script whose Code = "not C# ~~~" — reuse the config-builder helper the existing deploy tests use */; var dm = CreateDeploymentManager(); // existing harness factory dm.Tell(new DeployInstanceCommand(/* deploymentId */ Guid.NewGuid().ToString(), "bad-instance", badConfig, /* revisionHash */ "r1" /*, …existing args*/)); var resp = ExpectMsg(TimeSpan.FromSeconds(10)); Assert.Equal(DeploymentStatus.Failed, resp.Status); Assert.Contains("compile", resp.ErrorMessage, StringComparison.OrdinalIgnoreCase); // No partial state applied: instance actor must not exist, config must not persist. Sys.ActorSelection(dm.Path / "bad-instance").Tell(new Identify(1)); Assert.Null(ExpectMsg().Subject); _storage.DidNotReceive().StoreDeployedConfigAsync("bad-instance", Arg.Any(), Arg.Any(), Arg.Any(), Arg.Any()); } ``` 2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~Deploy_WithNonCompilingScript"` → **FAIL** (today it reports Success). 3. Implement: - Add internal message `private sealed record DeployValidationResult(DeployInstanceCommand Cmd, IActorRef ReplyTo, IReadOnlyList Errors);` and `Receive(HandleDeployValidationResult);`. - Rename the current `HandleDeploy(DeployInstanceCommand)` body to `ProceedWithDeploy(DeployInstanceCommand command, IActorRef sender)` (this is the code that decides redeploy-buffer vs `ApplyDeployment` — keep it byte-identical). - New `HandleDeploy`: capture `Sender`; run `Task.Run(() => _deployCompileValidator.Validate(command.FlattenedConfigurationJson))` → `ContinueWith` into `DeployValidationResult` → `PipeTo(Self)`. Validation is pure, so the singleton mailbox never blocks on Roslyn, and results are applied on the actor thread in arrival order (the existing redeploy-supersede handling still governs ordering). - `HandleDeployValidationResult`: if `Errors.Count > 0` → `LogDeploymentEvent("Error", …, "compile validation failed")`, reply `DeploymentStatusResponse(DeploymentId, Instance, DeploymentStatus.Failed, "Script compilation failed on site: " + string.Join(" | ", errors), UtcNow)` and **return** (no actor creation, no persistence — spec: "No partial state is applied"). Else call `ProceedWithDeploy(msg.Cmd, msg.ReplyTo)`. - Construct `_deployCompileValidator = new DeployCompileValidator(_compilationService)` in the ctor. - Fix `InstanceActor.cs:1339` XML doc: replace "Compilation errors reject entire instance deployment (logged but actor still starts)" with "Compile failures are rejected at deploy time by the Deployment Manager's validation gate; a failure reaching this point (startup load of a legacy config) is logged and the script skipped." 4. Run the new test → **PASS**; then the deploy suites: `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~DeploymentManager"` → green (existing deploy/redeploy/disable-supersede tests must still pass — they exercise `ProceedWithDeploy` through the new gate with compiling configs). 5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/DeploymentManagerActor.cs src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/InstanceActor.cs tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/DeploymentManagerActorTests.cs && git commit -m "fix(site-runtime): site-side compile failure now rejects the deployment per spec (S3) — off-thread validation gate before actor creation"` ### Task 5: ScriptExecutionScheduler gauges — queue depth, busy threads, oldest-busy age **Classification:** small (concurrency-adjacent but additive instrumentation) **Estimated implement time:** ~4 min **Parallelizable with:** 1, 2, 3, 9, 10, 11, 12, 13, 19, 22, 24 **Files:** - Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Scripts/ScriptExecutionScheduler.cs` - Test: `tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Scripts/ScriptExecutionSchedulerTests.cs` 1. Failing test: ```csharp [Fact] public async Task Gauges_ReflectBusyThreadAndQueueDepth() { using var scheduler = new ScriptExecutionScheduler(1); using var gate = new ManualResetEventSlim(false); var blocking = Task.Factory.StartNew(() => gate.Wait(), CancellationToken.None, TaskCreationOptions.None, scheduler); var queued = Task.Factory.StartNew(() => { }, CancellationToken.None, TaskCreationOptions.None, scheduler); await WaitUntilAsync(() => scheduler.BusyThreadCount == 1); Assert.Equal(1, scheduler.QueueDepth); // second task waiting Assert.True(scheduler.OldestBusyAge > TimeSpan.Zero); gate.Set(); await Task.WhenAll(blocking, queued); await WaitUntilAsync(() => scheduler.BusyThreadCount == 0); Assert.Null(scheduler.OldestBusyAge); } // WaitUntilAsync: small poll helper (100×50ms) — add locally if the test file has none. ``` 2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~Gauges_ReflectBusyThreadAndQueueDepth"` → **FAIL** (members don't exist). 3. Implement: - `private readonly long[] _busySinceTicks;` sized `threadCount`, initialized 0. Give each worker its index (change `new Thread(WorkerLoop)` to `new Thread(() => WorkerLoop(index))` with a captured local). - In `WorkerLoop(int index)`: around `TryExecuteTask(task)` do `Volatile.Write(ref _busySinceTicks[index], Environment.TickCount64);` before and `Volatile.Write(ref _busySinceTicks[index], 0);` in a `finally`. - Public gauges: `public int QueueDepth => _queue.Count;` `public int BusyThreadCount` (count of non-zero slots), `public TimeSpan? OldestBusyAge` (max of `now - since` over non-zero slots, else null). 4. Run → **PASS**. Run full scheduler suite: `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~ScriptExecutionScheduler"`. 5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Scripts/ScriptExecutionScheduler.cs tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Scripts/ScriptExecutionSchedulerTests.cs && git commit -m "feat(site-runtime): script scheduler gauges — queue depth, busy threads, oldest-busy age (S2/UA5)"` ### Task 6: Surface scheduler stats through ISiteHealthCollector + SiteHealthReport (additive) **Classification:** small **Estimated implement time:** ~4 min **Parallelizable with:** 1, 2, 3, 4, 9, 10, 11, 12, 13, 19, 22, 24 **Files:** - Modify: `src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/ISiteHealthCollector.cs` (new default-no-op method) - Modify: `src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/SiteHealthCollector.cs` (store + include in `CollectReport`) - Modify: `src/ZB.MOM.WW.ScadaBridge.Commons/Messages/Health/SiteHealthReport.cs` (additive optional properties) - Test: `tests/ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests/` (add to the existing SiteHealthCollector test file) 1. Failing test (in the existing collector test class): ```csharp [Fact] public void CollectReport_CarriesScriptSchedulerStats() { var collector = CreateCollector(); // existing fixture helper collector.SetScriptSchedulerStats(queueDepth: 7, busyThreads: 3, oldestBusyAgeSeconds: 42.5); var report = collector.CollectReport("site-1"); Assert.Equal(7, report.ScriptQueueDepth); Assert.Equal(3, report.ScriptBusyThreads); Assert.Equal(42.5, report.ScriptOldestBusyAgeSeconds); } ``` 2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests --filter "FullyQualifiedName~CollectReport_CarriesScriptSchedulerStats"` → **FAIL**. 3. Implement: - Interface: `void SetScriptSchedulerStats(int queueDepth, int busyThreads, double? oldestBusyAgeSeconds) { }` (default no-op, matching the `SetSiteEventLogWriteFailures` pattern so test fakes keep compiling). - Collector: store via `Interlocked`/volatile fields; include in `CollectReport`. - `SiteHealthReport`: **additive** optional init properties with safe defaults (`public int ScriptQueueDepth { get; init; }`, `public int ScriptBusyThreads { get; init; }`, `public double? ScriptOldestBusyAgeSeconds { get; init; }`) — do NOT change the positional parameter list (message-contract additive-only rule). 4. Run → **PASS**; run full projects: `dotnet test tests/ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests` and `dotnet test tests/ZB.MOM.WW.ScadaBridge.Commons.Tests`. 5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/ISiteHealthCollector.cs src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/SiteHealthCollector.cs src/ZB.MOM.WW.ScadaBridge.Commons/Messages/Health/SiteHealthReport.cs tests/ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests && git commit -m "feat(health): additive script-scheduler stats on ISiteHealthCollector + SiteHealthReport (S2/UA5)"` ### Task 7: Periodic scheduler-stats reporter (hosted service) **Classification:** small **Estimated implement time:** ~4 min **Parallelizable with:** 1, 2, 4, 9, 10, 11, 12, 13, 19, 22, 24 (needs 5 + 6 merged) **Files:** - Create: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Scripts/ScriptSchedulerStatsReporter.cs` - Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/ServiceCollectionExtensions.cs` (register hosted service on site hosts) - Test: create `tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Scripts/ScriptSchedulerStatsReporterTests.cs` 1. Failing test: ```csharp [Fact] public async Task Reporter_PushesStatsToCollector() { var collector = Substitute.For(); var options = new SiteRuntimeOptions { ScriptExecutionThreadCount = 1 }; using var reporter = new ScriptSchedulerStatsReporter( collector, options, NullLogger.Instance, pollInterval: TimeSpan.FromMilliseconds(50)); await reporter.StartAsync(CancellationToken.None); await Task.Delay(300); await reporter.StopAsync(CancellationToken.None); collector.ReceivedWithAnyArgs().SetScriptSchedulerStats(default, default, default); } ``` 2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~ScriptSchedulerStatsReporterTests"` → **FAIL**. 3. Implement a `BackgroundService` mirroring `SiteEventLogFailureCountReporter` (`src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/SiteEventLogFailureCountReporter.cs`): every 10 s (test-overridable ctor param) read `ScriptExecutionScheduler.Shared(options)` gauges and call `collector.SetScriptSchedulerStats(...)`. Register it in the SiteRuntime `ServiceCollectionExtensions` next to the existing site-role hosted services. Never throw from the loop (catch + log, matching the sibling reporter). 4. Run → **PASS**. Build the solution once (`dotnet build ZB.MOM.WW.ScadaBridge.slnx`) to confirm Host wiring compiles. 5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Scripts/ScriptSchedulerStatsReporter.cs src/ZB.MOM.WW.ScadaBridge.SiteRuntime/ServiceCollectionExtensions.cs tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Scripts/ScriptSchedulerStatsReporterTests.cs && git commit -m "feat(site-runtime): periodic scheduler-stats reporter feeding site health (S2/UA5)"` ### Task 8: Stuck-script watchdog — name the script whose thread never came back **Classification:** high-risk (script execution path) **Estimated implement time:** ~5 min **Parallelizable with:** 1, 2, 4, 9, 10, 12, 13, 19, 22, 24 **Files:** - Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/ScriptExecutionActor.cs` (~lines 122–260) - Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/SiteRuntimeOptions.cs` (new `StuckScriptGraceMs`, default 30000) - Test: `tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/ExecutionActorTests.cs` 1. Failing test (follow the existing `ExecutionActorTests` pattern for compiling a `Script` directly with `CSharpScript.Create(code, options, typeof(ScriptGlobals))` — test-compiled, so the trust validator is not in the path): ```csharp [Fact] public async Task TimedOutScript_WithBlockedThread_EmitsStuckThreadSiteEvent() { var gate = new SemaphoreSlim(0); StuckTestHooks.Gate = gate; // small public static test hook class in the test assembly var script = CompileRaw("ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.Actors.StuckTestHooks.Gate.Wait(); return null;"); var siteEvents = new RecordingSiteEventLogger(); // reuse/extend the existing fake in TestSupport var options = new SiteRuntimeOptions { ScriptExecutionTimeoutSeconds = 1, StuckScriptGraceMs = 200 }; SpawnExecutionActor(script, options, siteEvents); // existing harness helper await WaitUntilAsync(() => siteEvents.Events.Any(e => e.Category == "script" && e.Message.Contains("still executing", StringComparison.OrdinalIgnoreCase)), timeout: TimeSpan.FromSeconds(10)); gate.Release(); // free the scheduler thread so the test run stays clean } ``` 2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~TimedOutScript_WithBlockedThread_EmitsStuckThreadSiteEvent"` → **FAIL**. 3. Implement inside the execution lambda in `ScriptExecutionActor`: - `var completed = 0;` before the `try`; `Interlocked.Exchange(ref completed, 1);` in the existing `finally` (add one if missing — the scope-dispose finally exists). - Immediately after `using var cts = new CancellationTokenSource(timeout);`: ```csharp // Cooperative-only timeout containment (S2): the CTS firing does NOT free a // thread blocked in synchronous I/O. When the timeout elapses, wait a grace // period on the thread pool and, if the body still hasn't returned, name the // script loudly — this is the only signal an operator gets that one of the // bounded script-execution threads is gone. var graceMs = options.StuckScriptGraceMs; cts.Token.Register(() => _ = Task.Run(async () => { await Task.Delay(graceMs); if (Volatile.Read(ref completed) == 0) { var msg = $"Script '{scriptName}' on instance '{instanceName}' exceeded its " + $"{timeout.TotalSeconds:F0}s timeout and is STILL EXECUTING — its dedicated " + "script-execution thread is blocked (cooperative cancellation not observed)."; logger.LogError(msg); _ = siteEventLogger?.LogEventAsync("script", "Error", instanceName, $"ScriptActor:{scriptName}", msg); } })); ``` - Note: `Register` callback fires on cancellation only; normal completion disposes the CTS with no callback. `Task.Run`/`Task.Delay` run on the thread pool, not the (possibly saturated) script scheduler — deliberate. 4. Run → **PASS**; run `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~ExecutionActorTests"`. 5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/ScriptExecutionActor.cs src/ZB.MOM.WW.ScadaBridge.SiteRuntime/SiteRuntimeOptions.cs tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/ExecutionActorTests.cs && git commit -m "feat(site-runtime): stuck-script watchdog names the script holding a scheduler thread past timeout (S2)"` ### Task 9: ScriptActor — trigger-expression evaluation off the dispatcher (coalesced, actor-confined edge state) **Classification:** high-risk (actor concurrency, trigger semantics) **Estimated implement time:** ~5 min **Parallelizable with:** 1, 2, 3, 5, 10, 12, 13, 19, 22, 24 **Files:** - Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/ScriptActor.cs` (~lines 255–307) - Test: `tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/ScriptActorTests.cs` 1. Failing test (expression that only returns true on a scheduler thread — proves evaluation left the dispatcher; reuse the ScriptActorTests harness that builds an Expression-triggered ScriptActor with a compiled trigger expression and a script body observable from the test): ```csharp [Fact] public void ExpressionTrigger_EvaluatesOnScriptSchedulerThread_AndStillFires() { // Expression is TRUE only when evaluated on a script-execution thread. var expr = CompileTriggerExpression( "System.Threading.Thread.CurrentThread.Name != null && " + "System.Threading.Thread.CurrentThread.Name.StartsWith(\"script-execution-\")"); var fired = CreateExpressionScriptActorWithObservableBody(expr); // harness helper returning a signal _scriptActor.Tell(new AttributeValueChanged("inst", "A", "A", 1, "Good", DateTimeOffset.UtcNow)); AwaitAssert(() => Assert.True(fired.IsSet), TimeSpan.FromSeconds(10)); } [Fact] public void ExpressionTrigger_OnTrue_StillFiresOncePerFalseTrueEdge() { // Regression: async evaluation must preserve edge semantics. // expr: (int)Attributes["A"] > 5, OnTrue mode. SendChange("A", 1); SendChange("A", 10); SendChange("A", 11); // true stays true AwaitAssert(() => Assert.Equal(1, ExecutionCount()), TimeSpan.FromSeconds(10)); SendChange("A", 1); SendChange("A", 9); AwaitAssert(() => Assert.Equal(2, ExecutionCount()), TimeSpan.FromSeconds(10)); } ``` 2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~ExpressionTrigger_EvaluatesOnScriptSchedulerThread"` → **FAIL** (evaluation currently happens synchronously on the dispatcher thread, whose name is not `script-execution-*`). 3. Implement: - New internal message: `private sealed record ExpressionEvalResult(bool Result);` + `Receive(HandleExpressionEvalResult);` in the ctor. - Fields: `private bool _evalInFlight; private bool _evalPending;` - Replace the body of `EvaluateExpressionTrigger()` with `StartExpressionEvaluation()`: ```csharp private void StartExpressionEvaluation() { if (_compiledTriggerExpression == null || _triggerConfig is not ExpressionTriggerConfig) return; if (_evalInFlight) { _evalPending = true; return; } // coalesce bursts: one in flight, one pending _evalInFlight = true; var snapshot = new Dictionary(_attributeSnapshot); // point-in-time copy, actor thread var expression = _compiledTriggerExpression; var self = Self; Task.Factory.StartNew(async () => { try { using var cts = new CancellationTokenSource(TimeSpan.FromSeconds(2)); var state = await expression.RunAsync(new TriggerExpressionGlobals(snapshot), cancellationToken: cts.Token); return state.ReturnValue is bool b && b; } catch (Exception ex) { LogExpressionError(ex); return false; } }, CancellationToken.None, TaskCreationOptions.DenyChildAttach, ScriptExecutionScheduler.Shared(_options)).Unwrap().PipeTo(self, success: r => new ExpressionEvalResult(r)); } private void HandleExpressionEvalResult(ExpressionEvalResult msg) { _evalInFlight = false; if (_triggerConfig is ExpressionTriggerConfig exprConfig) { if (exprConfig.Mode == TriggerMode.WhileTrue) HandleWhileTrueTransition(msg.Result, _lastExpressionResult); else if (msg.Result && !_lastExpressionResult) TrySpawnExecution(null); _lastExpressionResult = msg.Result; } if (_evalPending) { _evalPending = false; StartExpressionEvaluation(); } } ``` - Edge state (`_lastExpressionResult`, timers) is touched only on the actor thread — the concurrency contract holds. `LogExpressionError` is called from the background task: it only touches `_healthCollector`/`_logger`/`_serviceProvider` (thread-safe singletons) — verify, and if in doubt route the error through the result message instead. 4. Run both new tests + `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~ScriptActorTests"` → **PASS** (existing expression tests must survive; they may need `AwaitAssert` in place of synchronous asserts since evaluation is now async). 5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/ScriptActor.cs tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/ScriptActorTests.cs && git commit -m "perf(site-runtime): ScriptActor trigger expressions evaluate on the script scheduler, coalesced, edge state actor-confined (P1)"` ### Task 10: AlarmActor — same async trigger-expression evaluation **Classification:** high-risk (actor concurrency, alarm state machine) **Estimated implement time:** ~5 min **Parallelizable with:** 1, 2, 3, 5, 9, 12, 13, 19, 22, 24 **Files:** - Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/AlarmActor.cs` (~lines 210–240, 491–519) - Test: `tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/AlarmActorTests.cs` 1. Failing test (mirror Task 9's thread-name expression trick with an Expression-triggered alarm; assert the alarm transitions to Active via the existing `AlarmStateChanged` parent-probe pattern in AlarmActorTests): ```csharp [Fact] public void ExpressionAlarm_EvaluatesOnSchedulerThread_AndActivates() { var expr = CompileTriggerExpression( "System.Threading.Thread.CurrentThread.Name != null && " + "System.Threading.Thread.CurrentThread.Name.StartsWith(\"script-execution-\")"); var alarm = CreateExpressionAlarmActor(expr, parentProbe); // existing harness alarm.Tell(new AttributeValueChanged("inst", "A", "A", 1, "Good", DateTimeOffset.UtcNow)); var change = parentProbe.ExpectMsg(TimeSpan.FromSeconds(10)); Assert.Equal(AlarmState.Active, change.State); } ``` 2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~ExpressionAlarm_EvaluatesOnSchedulerThread"` → **FAIL**. 3. Implement, structurally identical to Task 9: - Extract the existing Normal↔Active transition block in `HandleAttributeValueChanged` (the `isTriggered && _currentState == AlarmState.Normal` / clear branches) into `private void ApplyTriggeredState(bool isTriggered)`. - In `HandleAttributeValueChanged`, the `AlarmTriggerType.Expression` arm no longer computes `EvaluateExpression()` inline — it calls `StartExpressionEvaluation()` (in-flight + pending coalescing, snapshot copy, run on `ScriptExecutionScheduler.Shared(_options)`, `PipeTo` self as `ExpressionEvalResult`). `Receive` → `ApplyTriggeredState(msg.Result)` + drain pending. - Non-expression trigger types keep their synchronous path untouched (they are cheap numeric comparisons). - Delete the now-unused synchronous `EvaluateExpression()` (or keep as the body invoked inside the task). 4. Run new test + `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~AlarmActorTests"` → **PASS** (existing expression-alarm tests may need `AwaitAssert`). 5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/AlarmActor.cs tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/AlarmActorTests.cs && git commit -m "perf(site-runtime): AlarmActor trigger expressions evaluate on the script scheduler (P1)"` ### Task 11: InstanceActor — retry failed/lost tag subscriptions per connection **Classification:** high-risk (actor timers, DCL handshake) **Estimated implement time:** ~5 min **Parallelizable with:** 1, 2, 4, 9, 10, 12, 13, 19, 22, 24 **Files:** - Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/InstanceActor.cs` (line 207 no-op handler; `SubscribeToDcl` ~962–1010) - Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/SiteRuntimeOptions.cs` (new `TagSubscribeRetryIntervalMs`, default 5000) - Test: `tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/InstanceActorTests.cs` 1. Failing tests (TestKit; the InstanceActor harness passes a `TestProbe` as `dclManager`): ```csharp [Fact] public void FailedSubscribeResponse_IsRetried() { var options = new SiteRuntimeOptions { TagSubscribeRetryIntervalMs = 200 }; var (instance, dclProbe) = CreateInstanceWithDataSourcedAttribute(options); // existing harness var first = dclProbe.ExpectMsg(); instance.Tell(new SubscribeTagsResponse(first.CorrelationId, first.InstanceUniqueName, false, "Unknown connection: conn-1", DateTimeOffset.UtcNow), dclProbe.Ref); var retry = dclProbe.ExpectMsg(TimeSpan.FromSeconds(5)); // S4 core Assert.Equal(first.ConnectionName, retry.ConnectionName); Assert.Equal(first.TagPaths, retry.TagPaths); } [Fact] public void LostSubscribeResponse_IsRetried_AndSuccessStopsRetrying() { var options = new SiteRuntimeOptions { TagSubscribeRetryIntervalMs = 200 }; var (instance, dclProbe) = CreateInstanceWithDataSourcedAttribute(options); var first = dclProbe.ExpectMsg(); // No reply at all (dead-lettered response) -> must re-send. var retry = dclProbe.ExpectMsg(TimeSpan.FromSeconds(5)); instance.Tell(new SubscribeTagsResponse(retry.CorrelationId, retry.InstanceUniqueName, true, null, DateTimeOffset.UtcNow), dclProbe.Ref); dclProbe.ExpectNoMsg(TimeSpan.FromMilliseconds(600)); // retry timer cancelled } ``` 2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~FailedSubscribeResponse_IsRetried|FullyQualifiedName~LostSubscribeResponse_IsRetried"` → **FAIL** (response is a no-op; no re-send). 3. Implement: - Refactor `SubscribeToDcl` to build the per-connection groups once into a field `Dictionary> _tagsByConnection`, then call `SendTagSubscribe(connectionName)` per connection. - `SendTagSubscribe(string conn)`: mint a correlation id, record `_pendingTagSubscribes[correlationId] = conn`, Tell the DCL manager, and (re)arm a per-connection retry: `_tagSubscribeRetryTimers[conn]?.Cancel(); _tagSubscribeRetryTimers[conn] = Context.System.Scheduler.ScheduleTellOnceCancelable(TimeSpan.FromMilliseconds(_options.TagSubscribeRetryIntervalMs), Self, new RetryTagSubscribe(conn), Self);` — arming on **send** covers both the failure reply AND the lost reply with one mechanism (mirrors `NativeAlarmActor`'s timer but closes its lost-response gap too). - Replace `Receive(_ => { })` with a real handler: resolve `conn` from `_pendingTagSubscribes` (remove the entry); on `Success` → cancel the connection's retry timer, `LogDebug`; on failure → `LogWarning` + fire-and-forget an `instance_lifecycle`/`connection` site event ("tag subscribe failed … will retry") and leave the armed timer to re-send. Unknown correlation id (stale retry's superseded reply) → ignore. - `Receive(msg => SendTagSubscribe(msg.ConnectionName));` (internal record). `PostStop`: cancel all retry timers. - Note the DCL's connection-level-failure reply (`Success=false, "connection unavailable — will re-subscribe on reconnect"`) now ALSO retriggers a subscribe — harmless: the DCL treats already-subscribed tags as `AlreadySubscribed` and its `_subscriptionsByInstance` update is idempotent (verified `HandleSubscribe`/`HandleSubscribeCompleted`). 4. Run → **PASS**; run `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~InstanceActor"` → green. 5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/InstanceActor.cs src/ZB.MOM.WW.ScadaBridge.SiteRuntime/SiteRuntimeOptions.cs tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/InstanceActorTests.cs && git commit -m "fix(site-runtime): InstanceActor retries failed/lost tag subscriptions per connection (S4/UA6) — closes the unknown-connection ordering race"` ### Task 12: NativeAlarmActor — retry on lost subscribe response **Classification:** small **Estimated implement time:** ~3 min **Parallelizable with:** 1, 2, 3, 4, 5, 9, 10, 11, 13, 19, 22, 24 **Files:** - Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/NativeAlarmActor.cs` (`SendSubscribe` ~123–132, `HandleSubscribeResponse` ~269–286) - Test: `tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/NativeAlarmActorTests.cs` 1. Failing test: ```csharp [Fact] public void LostSubscribeResponse_ResendsSubscribe() { var options = new SiteRuntimeOptions { NativeAlarmRetryIntervalMs = 200 }; var (actor, dclProbe) = CreateNativeAlarmActor(options); // existing harness dclProbe.ExpectMsg(); // Swallow the response entirely — today the actor never retries. dclProbe.ExpectMsg(TimeSpan.FromSeconds(5)); } ``` 2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~LostSubscribeResponse_ResendsSubscribe"` → **FAIL**. 3. Implement: in `SendSubscribe()`, after the Tell, arm the response-timeout retry using the existing `_retryTimer` field: `_retryTimer?.Cancel(); _retryTimer = Context.System.Scheduler.ScheduleTellOnceCancelable(TimeSpan.FromMilliseconds(_options.NativeAlarmRetryIntervalMs * 2), Self, RetrySubscribe.Instance, Self);`. In `HandleSubscribeResponse` success branch, add `_retryTimer?.Cancel();` before returning. The failure branch already re-arms (keep it — it re-arms at 1× interval, which now simply overrides the 2× response-timeout arm). 4. Run → **PASS**; run `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~NativeAlarmActorTests"`. 5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/NativeAlarmActor.cs tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/NativeAlarmActorTests.cs && git commit -m "fix(site-runtime): NativeAlarmActor retries a lost subscribe response, not only a failed one (S4)"` ### Task 13: DeploymentManager — retry the startup config load **Classification:** standard **Estimated implement time:** ~4 min **Parallelizable with:** 1, 2, 3, 5, 9, 10, 11, 12, 19, 22, 24 **Files:** - Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/DeploymentManagerActor.cs` (PreStart ~240–253, `HandleStartupConfigsLoaded` ~291–297) - Test: `tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/DeploymentManagerActorTests.cs` 1. Failing test: ```csharp [Fact] public void StartupLoadFailure_IsRetried_UntilSuccess() { var calls = 0; _storage.GetAllDeployedConfigsAsync().Returns(_ => ++calls == 1 ? Task.FromException>(new IOException("db locked")) : Task.FromResult(new List { NewDeployedInstance("inst-1", ValidConfigJson()) })); // harness helpers var dm = CreateDeploymentManager(); // retry interval overridable to ~200ms for the test AwaitAssert(() => { Assert.True(calls >= 2); Sys.ActorSelection(dm.Path / "inst-1").Tell(new Identify(1)); Assert.NotNull(ExpectMsg().Subject); }, TimeSpan.FromSeconds(10)); } ``` (If `_storage` is the concrete `SiteStorageService` in the harness rather than a substitute, follow the harness's existing seam — several DM tests already fake persistence failures for `DeployPersistenceResult`; reuse that approach.) 2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~StartupLoadFailure_IsRetried"` → **FAIL** (one attempt, then dead). 3. Implement: - Extract the PreStart load block into `private void LoadDeployedConfigs()` and call it from `PreStart`. - In `HandleStartupConfigsLoaded`, replace the error `return;` with: log the error, then `Timers.StartSingleTimer("startup-load-retry", RetryStartupLoad.Instance, StartupLoadRetryInterval);` (private static readonly `TimeSpan StartupLoadRetryInterval = TimeSpan.FromSeconds(5)` — internal-settable or ctor-overridable for the test, matching how other DM test knobs are exposed). Fixed-interval, unlimited — consistent with the system's fixed-interval retry philosophy; each failure logs. - `Receive(_ => LoadDeployedConfigs());` (internal singleton message). Guard: if configs already loaded (`_deployedInstanceNames.Count > 0` after a successful load — add a `_startupLoadCompleted` bool set in the success path), ignore stale retries. 4. Run → **PASS**; run the DM suite. 5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/DeploymentManagerActor.cs tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/DeploymentManagerActorTests.cs && git commit -m "fix(site-runtime): retry the startup deployed-config load on SQLite failure (S5) — no more silent zero-instance node"` ### Task 14: Watch all Instance Actors; clean the map on unexpected termination **Classification:** high-risk (supervision/termination flow interacts with redeploy buffering) **Estimated implement time:** ~5 min **Parallelizable with:** 1, 2, 5, 9, 10, 12, 19, 22, 24 **Files:** - Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/DeploymentManagerActor.cs` (`CreateInstanceActor` ~1605–1632, `HandleTerminated`) - Test: `tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/DeploymentManagerActorTests.cs` 1. Failing test: ```csharp [Fact] public void InstanceActorInitFailure_RemovesDeadRefFromMap() { var poisonConfig = "{ not json"; // InstanceActor ctor JsonSerializer.Deserialize throws var dm = CreateDeploymentManager(); var healthProbe = _healthCollector; // fake collector already captured by the harness dm.Tell(new DeployInstanceCommand(Guid.NewGuid().ToString(), "poison", poisonConfig, "r1" /*…*/)); // After the child Stops (decider: ActorInitializationException -> Stop), the map // must not hold a dead ref: instance counts must return to 0 deployed. AwaitAssert(() => Assert.Equal(0, healthProbe.LastDeployedCount), TimeSpan.FromSeconds(10)); } ``` **Note:** with Task 4 in place, `"{ not json"` is rejected by the compile validator (deserialize error) before an actor is created — so drive this test through a config that *deserializes and validates* but makes the ctor throw (e.g. a config whose `Attributes` list contains an entry that trips `DecodeAttributeValue`'s non-guarded path, or temporarily bypass via the startup path: seed storage with the poison config and let startup create the actor). Prefer the **startup path** (storage returns the poison row): it does not pass through the deploy validation gate and is exactly the "dead ref" scenario S6 describes. 2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~InstanceActorInitFailure_RemovesDeadRefFromMap"` → **FAIL** (map keeps the dead ref; counts stay 1). 3. Implement: - `CreateInstanceActor`: after `Context.ActorOf`, add `Context.Watch(actorRef);`. - `HandleTerminated(Terminated t)`: keep the existing redeploy-buffer logic FIRST (it matches on its own bookkeeping). Add a fallback branch at the end: if `_instanceActors.TryGetValue(t.ActorRef.Path.Name, out var mapped) && mapped.Equals(t.ActorRef)` → remove from `_instanceActors`, `UpdateInstanceCounts()`, `LogError` + fire-and-forget a `deployment` site event (`"Instance {name} terminated unexpectedly (init failure or crash) — removed from routing"`). Guard: expected stops (disable/delete/redeploy) remove or replace the map entry *before* the child stops, so `mapped.Equals(t.ActorRef)` is false for them and the fallback is inert — verify against `HandleDisable`/`HandleDelete`/the redeploy shadow map and adjust ordering only if a path removes after stop. - Redeploy path already `Context.Watch`es; double-watch is idempotent in Akka.NET (single Terminated), but confirm the redeploy handler consumes its Terminated *before* the fallback branch runs (order the if/else so the buffered-redeploy match wins). 4. Run → **PASS**; run `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~DeploymentManager"` (the redeploy race suites are the regression canary here — all must stay green). 5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/DeploymentManagerActor.cs tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/DeploymentManagerActorTests.cs && git commit -m "fix(site-runtime): watch every Instance Actor; unexpected termination evicts the dead ref from routing (S6)"` ### Task 15: Fail the in-flight deployment when its Instance Actor dies during init **Classification:** high-risk (deployment status protocol) **Estimated implement time:** ~5 min **Parallelizable with:** 1, 2, 5, 9, 10, 12, 19, 22, 24 **Files:** - Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/DeploymentManagerActor.cs` (`ApplyDeployment` ~538–587, `HandleDeployPersistenceResult` ~596–636, `HandleTerminated`) - Test: `tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/DeploymentManagerActorTests.cs` 1. Failing test: ```csharp [Fact] public void Deploy_WhoseActorDiesDuringInit_ReportsFailed_NotSuccess() { // Config that passes compile validation but whose InstanceActor ctor throws — // reuse/extend the poison-config helper from Task 14 (ctor-throwing but valid JSON), // or a config helper stub if none exists (e.g. attribute row that trips ctor indexing). var dm = CreateDeploymentManager(); dm.Tell(new DeployInstanceCommand(Guid.NewGuid().ToString(), "dies-on-init", CtorThrowingConfig(), "r1" /*…*/)); var resp = ExpectMsg(TimeSpan.FromSeconds(10)); Assert.Equal(DeploymentStatus.Failed, resp.Status); } ``` 2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~Deploy_WhoseActorDiesDuringInit"` → **FAIL** (Success is reported once SQLite persists). 3. Implement: - Field: `private readonly Dictionary _inFlightDeployments = new();` keyed by instance name — store a small record `InFlightDeploy(string DeploymentId, IActorRef ReplyTo)` instead (clearer): `private readonly Dictionary _inFlightDeployments = new();` - `ApplyDeployment`: before spawning the persistence task, `_inFlightDeployments[instanceName] = new InFlightDeploy(command.DeploymentId, sender);` - Task 14's unexpected-`Terminated` fallback branch: after evicting the map entry, if `_inFlightDeployments.Remove(name, out var inflight)` → `inflight.ReplyTo.Tell(new DeploymentStatusResponse(inflight.DeploymentId, name, DeploymentStatus.Failed, "Instance actor failed during initialization — see site event log", DateTimeOffset.UtcNow));` and remove the optimistically-persisted row (`Task.Run(() => _storage.RemoveDeployedConfigAsync(name))` fire-and-forget with fault log) + `_deployedInstanceNames.Remove(name)` + `UpdateInstanceCounts()`. - `HandleDeployPersistenceResult`: only reply if `_inFlightDeployments.Remove(result.InstanceName)` returns true (both success and failure branches) — if the Terminated path already failed the deployment, swallow the late persistence Success (log Debug). Both orders (Terminated-then-persistence, persistence-then-Terminated) now produce exactly one reply. 4. Run → **PASS**; re-run the full DM suite plus Task 4's test (the validation-gate reply path must not double-reply): `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~DeploymentManager"`. 5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/DeploymentManagerActor.cs tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/DeploymentManagerActorTests.cs && git commit -m "fix(site-runtime): deployment whose Instance Actor dies during init reports Failed, exactly-once reply (S6)"` ### Task 16: DataConnectionActor — capture `_adapter` into locals before every background task **Classification:** high-risk (actor-state confinement) **Estimated implement time:** ~4 min **Parallelizable with:** 3, 5, 9, 10, 11, 12, 13, 19, 21, 22, 24 **Files:** - Modify: `src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Actors/DataConnectionActor.cs` (`HandleSubscribe` ~707–758, `HandleWriteBatch` ~1155–1202, `HandleWrite`, `ReSubscribeAll` ~1594–1605; audit every `Task.Run`/local-async-function for `_adapter` reads) - Test: `tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests/DataConnectionActorTests.cs` 1. Failing test (deterministic via a gated adapter — the in-flight subscribe must finish against the adapter that STARTED it, even after a failover swaps `_adapter`): ```csharp [Fact] public async Task InFlightSubscribe_ContinuesOnOriginalAdapter_AfterFailover() { var subscribeGate = new TaskCompletionSource(); var primaryAdapter = Substitute.For(); primaryAdapter.ConnectAsync(default!).ReturnsForAnyArgs(Task.CompletedTask); primaryAdapter.SubscribeAsync(default!, default!).ReturnsForAnyArgs( _ => subscribeGate.Task, // tag 1 blocks mid-flight _ => Task.FromResult("sub-2")); // tag 2 resumes after failover var backupAdapter = Substitute.For(); var factory = Substitute.For(); factory.Create(default!, default!).ReturnsForAnyArgs(backupAdapter); var actor = CreateConnectedActor(primaryAdapter, factory, backupConfig: SomeBackupConfig()); // harness actor.Tell(new SubscribeTagsRequest(NewCorrelation(), "inst", "conn", new[] { "t1", "t2" }, Now())); TriggerFailoverToBackup(actor); // drive AdapterDisconnected + failed reconnects past retry count subscribeGate.SetResult("sub-1"); // release the in-flight background loop AwaitAssert(async () => { await primaryAdapter.ReceivedWithAnyArgs(2).SubscribeAsync(default!, default!); await backupAdapter.DidNotReceiveWithAnyArgs().SubscribeAsync(default!, default!); }); } ``` (Build on the existing `DataConnectionActorTests` failover harness — it already drives `CountFailureAndMaybeFailover` via consecutive `ConnectResult` failures.) 2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests --filter "FullyQualifiedName~InFlightSubscribe_ContinuesOnOriginalAdapter"` → **FAIL** (later loop iterations read the swapped `_adapter` field and hit the backup with the old generation). 3. Implement — one pattern, four sites: - `HandleSubscribe`: `var adapter = _adapter;` on the actor thread before `Task.Run`; use `adapter.SubscribeAsync(...)` and `SeedTagsAsync(adapter, tagsToSeed)` inside the task. Update the confinement comment at ~689 to note the adapter is captured too. - `HandleWriteBatch`: `var adapter = _adapter;` before `RunAsync()`; use `adapter.WriteBatchAsync` / `adapter.WriteAsync` (the second read currently happens after the first `await`, off-thread). - `HandleWrite`: same audit/fix (the single-write path has the same local-async-function shape). - `ReSubscribeAll`: the per-tag `SubscribeAsync` loop runs on the actor thread (safe), but capture `var subscribeAdapter = _adapter;` and use it in the loop anyway for symmetry with `reseedAdapter` — zero cost, removes the last field read reachable from this method's continuations. 4. Run → **PASS**; full DCL actor suite: `dotnet test tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests --filter "FullyQualifiedName~DataConnectionActor"`. 5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Actors/DataConnectionActor.cs tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests/DataConnectionActorTests.cs && git commit -m "fix(dcl): capture adapter into locals before background tasks — no off-thread reads of the mutable _adapter field (S7)"` ### Task 17: DCL — tag→instances reverse index for the value fan-out hot path **Classification:** high-risk (hottest DCL loop, index-consistency invariants) **Estimated implement time:** ~5 min **Parallelizable with:** 5, 9, 10, 11, 12, 13, 19, 21, 22, 24 **Files:** - Modify: `src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Actors/DataConnectionActor.cs` (`HandleTagValueReceived` ~1693–1738; every `_subscriptionsByInstance` mutation site) - Test: `tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests/DataConnectionActorTests.cs` 1. Failing test (behavioral — index correctness across unsubscribe, which a naive index gets wrong): ```csharp [Fact] public void TagFanOut_AfterOneInstanceUnsubscribes_OnlyRemainingSubscriberReceives() { var actor = CreateConnectedActorWithInstantSubscribes(); // adapter substitute returns sub ids immediately var probeA = CreateTestProbe(); var probeB = CreateTestProbe(); actor.Tell(SubscribeRequest("inst-A", new[] { "shared-tag" }), probeA.Ref); actor.Tell(SubscribeRequest("inst-B", new[] { "shared-tag" }), probeB.Ref); probeA.ExpectMsg(); probeB.ExpectMsg(); probeA.FishForMessage(m => m is TagValueUpdate, hint: "seed"); // drain seeds as the harness does actor.Tell(UnsubscribeRequest("inst-A", new[] { "shared-tag" }), probeA.Ref); DeliverTagValue(actor, "shared-tag", 42); // helper: Tell TagValueReceived with current generation probeB.ExpectMsg(u => u.TagPath == "shared-tag"); probeA.ExpectNoMsg(TimeSpan.FromMilliseconds(300)); } ``` This passes today via the O(N) scan — write it FIRST and confirm it passes (pin the behavior), then convert `HandleTagValueReceived` to the index and confirm it STILL passes. Add one more assertion-style test that also passes pre-change (two instances, disjoint tags, each receives only its own) — these are the regression net for the index bookkeeping. 2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests --filter "FullyQualifiedName~TagFanOut_"` → expect **PASS** pre-change (documented pin), then proceed. 3. Implement: - Field: `private readonly Dictionary> _instancesByTag = new();` with XML doc mirroring `_tagSubscriberCount`'s rationale. - Add/remove alongside **every** `_subscriptionsByInstance[inst]` tag-set mutation. Grep `_subscriptionsByInstance` and cover: `HandleSubscribeCompleted` (tags added), `HandleUnsubscribe` (tags removed + instance entry removal), the orphan-release guards (~836–985, 1795–1858 region), and any instance-removal path. Write two private helpers `IndexTag(string tag, string inst)` / `UnindexTag(string tag, string inst)` (removing the tag key when its set empties) and use them at every site — no inline mutation. - `HandleTagValueReceived`: replace the `_subscriptionsByInstance` scan with `if (_instancesByTag.TryGetValue(msg.TagPath, out var insts)) foreach (var inst in insts) if (_subscribers.TryGetValue(inst, out var s)) s.Tell(...)`. - `ReSubscribeAll` preserves `_subscriptionsByInstance` — so it must NOT clear `_instancesByTag` (it is derived from the same preserved state). Note this in a comment. 4. Run the fan-out tests + full DCL suite → **PASS**. 5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Actors/DataConnectionActor.cs tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests/DataConnectionActorTests.cs && git commit -m "perf(dcl): tag->instances reverse index replaces per-update all-instances scan (P3)"` ### Task 18: InstanceActor — per-attribute child-subscription routing **Classification:** high-risk (hot-path routing semantics for scripts AND alarms) **Estimated implement time:** ~5 min **Parallelizable with:** 1, 2, 5, 10, 12, 13, 17, 19, 22, 24 **Files:** - Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/InstanceActor.cs` (`CreateChildActors` ~1348–1488, `PublishAndNotifyChildren` ~1179–1209) - Test: `tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/InstanceActorChildAttributeRaceTests.cs` (or a new `InstanceActorChildRoutingTests.cs`) 1. Failing test: ```csharp [Fact] public void AttributeChange_IsRoutedOnlyToInterestedChildren() { // Instance with: script S_A (ValueChange on "A"), script S_B (ValueChange on "B"), // script S_expr (Expression trigger). Script bodies record executions via the // static-recording pattern the existing tests use. var instance = CreateInstance(ScriptsAB_and_Expression()); SetStatic(instance, "A", 1); // publishes AttributeValueChanged for A AwaitAssert(() => { Assert.Equal(1, Executions("S_A")); Assert.Equal(0, Executions("S_B")); // FAILS pre-change only via mailbox counting… }); } ``` `S_B` never *executes* today either (its own trigger check discards) — so assert on **message delivery**, not execution: give the routing seam a test hook. Concretely: make the routing maps `internal` and unit-test map construction directly (`InstanceActor` exposes `internal IReadOnlyDictionary> ChildrenByMonitoredAttribute` / `internal IReadOnlyList AllChangeChildren` next to the existing `ScriptActorCount` diagnostics), plus a behavioral test that the Expression script still fires on ANY attribute change and the ValueChange script still fires on its own attribute: ```csharp [Fact] public void RoutingMaps_AreBuiltFromTriggerConfigs() { var actor = CreateInstanceActorRef(ScriptsAB_and_Expression()); var underlying = GetUnderlyingActor(actor); // TestActorRef pattern used elsewhere in this file Assert.Single(underlying.ChildrenByMonitoredAttribute["A"]); Assert.Single(underlying.ChildrenByMonitoredAttribute["B"]); Assert.Single(underlying.AllChangeChildren); // the Expression script } ``` 2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~RoutingMaps_AreBuiltFromTriggerConfigs"` → **FAIL**. 3. Implement: - Fields: `private readonly Dictionary> _childrenByMonitoredAttribute = new(); private readonly List _allChangeChildren = new();` - In `CreateChildActors`, after creating each script/alarm actor, classify: - Script trigger `valuechange`/`conditional` → parse `attributeName` from `TriggerConfiguration` JSON (reuse/extract the parse `ScriptActor.ParseTriggerConfig` does — lift a tiny static helper `TriggerRouting.TryGetMonitoredAttribute(triggerType, triggerConfigJson, out string attr)` into `Scripts/` so ScriptActor and InstanceActor share one parser). Add to `_childrenByMonitoredAttribute[attr]`. - Script trigger `expression` → `_allChangeChildren` (its snapshot must stay current). - Script trigger `interval`/`call`/unparseable → **`_allChangeChildren` if unparseable (fail-open, log warning), else no routing** (interval/call scripts never react to attribute changes; their `_attributeSnapshot` is only read by expression evaluation, which they don't have — verify this before excluding, it is true today at `ScriptActor` where the snapshot is read only in `EvaluateExpressionTrigger`). - Alarms: same classification via the alarm's trigger type — Expression → all bucket; HiLo/ValueMatch/RangeViolation/RateOfChange → monitored attribute (parse from the alarm config the same way `AlarmActor.IsMonitoredAttribute` derives it); unparseable → all bucket. - `PublishAndNotifyChildren`: replace the two full loops with `foreach (var c in _allChangeChildren) c.Tell(changed);` + `if (_childrenByMonitoredAttribute.TryGetValue(changed.AttributeName, out var interested)) foreach (var c in interested) c.Tell(changed);`. Children keep their own trigger gates (defense in depth — routing is an optimization, not the correctness gate). - Note: `changed.AttributeName` on the DCL path is the canonical name (`HandleTagValueUpdate` maps tag→attributes before publishing) — key the map by canonical name; confirm against `HandleTagValueUpdate`. 4. Run new tests + the whole InstanceActor/ScriptActor/AlarmActor suites: `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~InstanceActor|FullyQualifiedName~ScriptActor|FullyQualifiedName~AlarmActor"` → **PASS**. 5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/InstanceActor.cs src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/ScriptActor.cs src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Scripts tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests && git commit -m "perf(site-runtime): per-attribute child routing in InstanceActor — attribute changes reach only interested children (P2)"` ### Task 19: SiteStorageService — WAL journal mode + busy timeout **Classification:** small **Estimated implement time:** ~3 min **Parallelizable with:** 1, 2, 3, 4, 5, 9, 10, 11, 12, 13, 22, 24 **Files:** - Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Persistence/SiteStorageService.cs` (ctor ~21–25, `InitializeAsync` ~41–132) - Test: `tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Persistence/` (add to the existing SiteStorageService test file) 1. Failing test: ```csharp [Fact] public async Task Initialize_EnablesWalJournalMode() { var service = CreateTempFileStorageService(); // existing temp-SQLite fixture pattern await service.InitializeAsync(); await using var conn = service.CreateConnection(); await conn.OpenAsync(); await using var cmd = conn.CreateCommand(); cmd.CommandText = "PRAGMA journal_mode;"; Assert.Equal("wal", ((string)(await cmd.ExecuteScalarAsync())!).ToLowerInvariant()); } ``` Note: WAL requires a file-backed DB — if the existing fixture uses `:memory:`, use the temp-file variant (several persistence tests already do). 2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~Initialize_EnablesWalJournalMode"` → **FAIL** (`delete`). 3. Implement: - In `InitializeAsync`, before creating tables: execute `PRAGMA journal_mode=WAL;` (persistent, database-level — one-time). Log the resulting mode; `:memory:`/unsupported filesystems fall back gracefully (don't throw if the pragma returns non-wal, just warn). - In the ctor, normalize the connection string through `SqliteConnectionStringBuilder` and ensure `DefaultTimeout` is at least 5 s (Microsoft.Data.Sqlite uses the command timeout as its busy handler — an explicit floor documents intent even though the library default is 30): only raise it if the caller set something lower. Add an XML remark on `CreateConnection()` documenting WAL + busy-handling so the repositories layered on it inherit the story. 4. Run → **PASS**; run the persistence folder suite: `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~SiteStorageService"`. 5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Persistence/SiteStorageService.cs tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Persistence && git commit -m "fix(site-runtime): site SQLite runs WAL with an explicit busy-timeout floor (S8) — concurrent writers stop serializing on the rollback journal"` ### Task 20: NativeAlarmActor — coalesce per-transition SQLite upserts **Classification:** standard **Estimated implement time:** ~5 min **Parallelizable with:** 1, 2, 4, 5, 9, 10, 11, 13, 17, 22, 24 **Files:** - Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/NativeAlarmActor.cs` (`PersistUpsert` ~420–428, timer plumbing) - Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Persistence/SiteStorageService.cs` (new `UpsertNativeAlarmsAsync` batch — one connection + transaction) - Test: `tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/NativeAlarmActorTests.cs` 1. Failing test (against the temp-SQLite storage the existing NativeAlarmActor tests use): ```csharp [Fact] public async Task AlarmStorm_CoalescesPersistence_LatestPerSourceRefWins() { var (actor, storage) = CreateNativeAlarmActorWithTempStorage(flushIntervalMs: 100); for (var i = 0; i < 50; i++) DeliverLiveTransition(actor, sourceRef: $"ref-{i % 5}", severity: i); // 50 transitions, 5 refs await Task.Delay(500); // > flush interval var rows = await storage.GetNativeAlarmsAsync("inst", "src"); Assert.Equal(5, rows.Count); // Latest transition per ref persisted (severity of the LAST write for ref-4 is 49): Assert.Contains(rows, r => r.ConditionJson.Contains("49")); } ``` 2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~AlarmStorm_CoalescesPersistence"` → **FAIL** only on the *mechanism* (row content passes either way) — so also assert the batch API is used: add `UpsertNativeAlarmsAsync` first (it won't exist → compile-fail = the failing signal), or count connections via a storage wrapper. Simplest honest failing signal: write the test calling `storage.UpsertNativeAlarmsAsync(...)` shape via the actor and assert rows — compile failure is the RED step. 3. Implement: - `SiteStorageService.UpsertNativeAlarmsAsync(string instance, string source, IReadOnlyList<(string SourceRef, string ConditionJson, DateTimeOffset At)> rows)` — one connection, one transaction, one prepared command re-bound per row. - `NativeAlarmActor`: replace direct `PersistUpsert(t)` calls (in `ApplyLiveTransition` and `ApplySnapshotSwap`) with `_dirtyUpserts[t.SourceReference] = t;` + arm a 100 ms single-shot flush timer if not armed (`ScheduleTellOnceCancelable`, internal `FlushPersistence` message). The flush handler snapshots + clears `_dirtyUpserts`, calls the batch API fire-and-forget with the existing `OnlyOnFaulted` logging. `PersistDelete` stays immediate (rare; also remove the ref from `_dirtyUpserts` so a delete isn't resurrected by a pending upsert). `PostStop`: best-effort final flush (fire-and-forget). - Rehydration semantics unchanged: worst case a crash loses ≤100 ms of mirror writes; the (re)subscribe snapshot swap reconciles — note this in the class doc. 4. Run → **PASS**; full NativeAlarmActor suite green. 5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/NativeAlarmActor.cs src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Persistence/SiteStorageService.cs tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/NativeAlarmActorTests.cs && git commit -m "perf(site-runtime): coalesce native-alarm mirror persistence into batched flushes (P4)"` ### Task 21: Persist native-alarm display metadata for rehydration **Classification:** standard (SQLite schema migration — additive column) **Estimated implement time:** ~5 min **Parallelizable with:** 1, 2, 4, 5, 9, 10, 11, 13, 22, 24 **Files:** - Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Persistence/SiteStorageService.cs` (`native_alarm_state` schema ~117–124, `MigrateSchemaAsync`, upsert/read methods) - Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/NativeAlarmActor.cs` (`PersistUpsert`/flush ~420–428, `HandleRehydration` ~134–174) - Test: `tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/NativeAlarmActorTests.cs` 1. Failing test: ```csharp [Fact] public async Task Rehydration_RestoresDisplayMetadata() { var (actor1, storage) = CreateNativeAlarmActorWithTempStorage(); DeliverLiveTransition(actor1, sourceRef: "ref-1", alarmTypeName: "HighLevelAlarm", category: "Process", message: "Tank overflow"); await FlushAndStop(actor1); var (actor2, parentProbe) = RecreateActorOnSameStorage(storage); // rehydrates in PreStart var emitted = parentProbe.FishForMessage(m => m.SourceReference == "ref-1"); Assert.Equal("HighLevelAlarm", emitted.AlarmTypeName); // empty string today Assert.Equal("Process", emitted.Category); Assert.Equal("Tank overflow", emitted.Message); } ``` 2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~Rehydration_RestoresDisplayMetadata"` → **FAIL** (metadata comes back `string.Empty`). 3. Implement: - Schema: add `metadata_json TEXT` to the `CREATE TABLE native_alarm_state` DDL AND `await TryAddColumnAsync(connection, "native_alarm_state", "metadata_json", "TEXT");` in `MigrateSchemaAsync` (existing-DB upgrade). - Persist: serialize a small record `NativeAlarmMetadata(string AlarmTypeName, string Category, string Message, string CurrentValue, string LimitValue)` from the transition into `metadata_json` in the upsert (single + Task 20 batch). `NativeAlarmRow` gains `string? MetadataJson`. - Rehydrate: in `HandleRehydration`, deserialize `MetadataJson` (null/malformed → empty strings, current behavior) and build the placeholder `NativeAlarmTransition` with the restored fields instead of `string.Empty`. 4. Run → **PASS**; full NativeAlarmActor + persistence suites. 5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Persistence/SiteStorageService.cs src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/NativeAlarmActor.cs tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/NativeAlarmActorTests.cs && git commit -m "feat(site-runtime): persist native-alarm display metadata so rehydrated conditions render fully (UA4)"` ### Task 22: CertStoreActor — export trusted certs (thumbprint + DER) for reconciliation **Classification:** standard (additive Commons message contract) **Estimated implement time:** ~4 min **Parallelizable with:** 1, 2, 3, 4, 5, 9, 10, 11, 12, 13, 16, 17, 19, 24 **Files:** - Modify: `src/ZB.MOM.WW.ScadaBridge.Commons/Messages/Management/` (the file holding `WriteCertToLocalStore`/`ListLocalCerts` — add `ExportLocalTrustedCerts` + `LocalCertExport`) - Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/CertStoreActor.cs` (new handler) - Test: `tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/CertStoreActorTests.cs` 1. Failing test (reuse the temp-dir PKI fixture in `CertStoreActorTests`): ```csharp [Fact] public void ExportLocalTrustedCerts_ReturnsThumbprintAndDerBytes() { var (actor, testCertDer, thumbprint) = CreateCertStoreWithOneTrustedCert(); // existing fixture helpers actor.Tell(new ExportLocalTrustedCerts()); var export = ExpectMsg(); var entry = Assert.Single(export.Certs); Assert.Equal(thumbprint, entry.Thumbprint); Assert.Equal(testCertDer, entry.DerBytes); } ``` 2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~ExportLocalTrustedCerts_ReturnsThumbprintAndDerBytes"` → **FAIL** (types don't exist). 3. Implement: - Commons (additive, next to the existing cert messages): `public sealed record ExportLocalTrustedCerts(); public sealed record ExportedCert(string Thumbprint, byte[] DerBytes); public sealed record LocalCertExport(bool Success, string? Error, IReadOnlyList Certs);` - `CertStoreActor`: `Receive` — enumerate `*.der` in `_trustedStoreDir`, load each via `X509CertificateLoader`/`new X509Certificate2(bytes)` (match how `HandleList` reads them today) to compute the thumbprint, reply `LocalCertExport(true, null, list)`; any IO error → `LocalCertExport(false, ex.Message, [])`. 4. Run → **PASS**. 5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.Commons src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/CertStoreActor.cs tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/CertStoreActorTests.cs && git commit -m "feat(site-runtime): CertStoreActor exports trusted certs (thumbprint+DER) for node reconciliation (UA1 groundwork)"` ### Task 23: Cert-trust reconcile-on-join — singleton pushes its trusted set to a (re)joining site node **Classification:** high-risk (cluster membership events) **Estimated implement time:** ~5 min **Parallelizable with:** 1, 2, 5, 9, 10, 12, 17, 19, 24 **Files:** - Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/DeploymentManagerActor.cs` (PreStart/PostStop cluster subscription; new handlers near the cert section ~900–1007) - Test: `tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/DeploymentManagerActorTests.cs` 1. Failing test — test the reconcile flow via the internal message (constructing a real `ClusterEvent.MemberUp` in a unit test is impractical; the MemberUp handler itself is a two-liner producing this message). Use the same "forwarding actor named `cert-store` → TestProbe" trick the existing cert broadcast tests use: ```csharp [Fact] public void SiteNodeJoined_PushesLocalTrustedCertsToJoinedNode() { var certStoreProbe = InstallCertStoreForwarder(); // /user/cert-store -> probe (existing pattern) var dm = CreateDeploymentManager(); dm.Tell(new DeploymentManagerActor.SiteNodeJoined(Cluster.Get(Sys).SelfAddress)); certStoreProbe.ExpectMsg(); // 1) read own store dm.Tell(new LocalCertExport(true, null, new[] { new ExportedCert("ABC123", new byte[] { 1, 2, 3 }) }), certStoreProbe.Ref); // 2) pushed to the joined node's cert-store (self address in-test -> same selection): var write = certStoreProbe.ExpectMsg(TimeSpan.FromSeconds(5)); Assert.Equal("ABC123", write.Thumbprint); } ``` (Match `WriteCertToLocalStore`'s real field names/args from Commons when writing the assertion.) 2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~SiteNodeJoined_PushesLocalTrustedCerts"` → **FAIL**. 3. Implement: - `PreStart`: `Cluster.Get(Context.System).Subscribe(Self, ClusterEvent.SubscriptionInitialStateMode.InitialStateAsEvents, typeof(ClusterEvent.MemberUp));` `PostStop`: `Cluster.Get(Context.System).Unsubscribe(Self);` - `Receive`: if the member has the site role and is not the singleton's own node → `Self.Tell(new SiteNodeJoined(member.Address))` (public/internal record for testability). - `Receive`: capture the address; ask the LOCAL cert store (`Context.ActorSelection($"/user/{CertStoreActor.WellKnownName}")`) with `ExportLocalTrustedCerts` (short `CertBroadcastTimeout` Ask, `ContinueWith` → pipe a wrapper carrying the target address). On the piped `LocalCertExport` (wrap it: `CertReconcileExport(Address Target, LocalCertExport Export)`): for each cert, `Tell` the target node's cert store (`new RootActorPath(target) / "user" / CertStoreActor.WellKnownName`) a `WriteCertToLocalStore(...)` — fire-and-forget, additive union (documented: removals do NOT reconcile until central persistence lands; the write is idempotent — `CertStoreActor.HandleWrite` overwrites by thumbprint). - Log one info line per reconcile (`"Cert-trust reconcile: pushed {N} trusted cert(s) to joining node {Address}"`); export failure → warning, no retry (the next MemberUp or manual trust re-broadcast heals). - Simplification for the test flow: since the test drives `LocalCertExport` directly at the actor, have `Receive` be the handler and also accept a bare `LocalCertExport` only if you keep a single pending target field — prefer the wrapper message; adjust the test to send the wrapper if needed. 4. Run → **PASS**; full DM suite green. 5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/DeploymentManagerActor.cs tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/DeploymentManagerActorTests.cs && git commit -m "feat(site-runtime): cert-trust reconcile-on-join — singleton pushes trusted certs to (re)joining site nodes (UA1)"` ### Task 24: Wire `ConfigFetchRetryCount` into the standby replication fetch **Classification:** small **Estimated implement time:** ~4 min **Parallelizable with:** 1, 2, 3, 4, 5, 9, 10, 12, 13, 16, 17, 19, 22 **Files:** - Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/SiteReplicationActor.cs` (`HandleApplyConfigDeploy` ~153–217; ctor gains `SiteRuntimeOptions? options = null`) - Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/SiteRuntimeOptions.cs` (fix the "Reserved — not yet wired" XML doc, lines 63–67) - Modify: the `SiteReplicationActor` Props construction site (Host/ServiceCollectionExtensions — grep `new SiteReplicationActor(` / `Props.Create(() => new SiteReplicationActor`) to pass options - Test: `tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/SiteReplicationActorTests.cs` 1. Failing test: ```csharp [Fact] public void ReplicatedFetch_RetriesUpToConfigFetchRetryCount() { var attempts = 0; var fetcher = Substitute.For(); fetcher.FetchAsync(default!, default!, default!, default).ReturnsForAnyArgs(_ => ++attempts < 3 ? Task.FromException(new HttpRequestException("central hiccup")) : Task.FromResult("{\"config\":true}")); var actor = CreateReplicationActor(fetcher, new SiteRuntimeOptions { ConfigFetchRetryCount = 3 }); // existing harness + new arg actor.Tell(new ApplyConfigDeploy("inst", "dep-1", "r1", true, "http://central", "tok")); AwaitAssert(() => { Assert.Equal(3, attempts); _storage.Received(1).StoreDeployedConfigIfNewerAsync("inst", "{\"config\":true}", "dep-1", "r1", true); }, TimeSpan.FromSeconds(10)); } ``` 2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~ReplicatedFetch_RetriesUpToConfigFetchRetryCount"` → **FAIL** (single attempt). 3. Implement: replace the single `FetchAsync(...).ContinueWith(...)` with a local async function launched fire-and-forget (same best-effort model): ```csharp var maxAttempts = Math.Max(1, _options?.ConfigFetchRetryCount ?? 1); _ = FetchWithRetryAsync(); // errors fully handled inside; observes every fault async Task FetchWithRetryAsync() { for (var attempt = 1; attempt <= maxAttempts; attempt++) { try { var json = await _configFetcher.FetchAsync(msg.CentralFetchBaseUrl, msg.DeploymentId, msg.FetchToken, CancellationToken.None); await _storage.StoreDeployedConfigIfNewerAsync(msg.InstanceName, json, msg.DeploymentId, msg.RevisionHash, msg.IsEnabled); return; } catch (DeploymentConfigFetchException fex) when (fex.IsSuperseded) { /* keep the existing "superseded" info log */ return; } catch (Exception ex) { // keep existing per-failure log wording; add attempt counters if (attempt < maxAttempts) await Task.Delay(TimeSpan.FromSeconds(2)); else _logger.LogError(ex, "Replicated config fetch failed after {Attempts} attempt(s) for {Instance} (deployment {DeploymentId}) — reconciliation will backstop", maxAttempts, msg.InstanceName, msg.DeploymentId); } } } ``` Update the `SiteRuntimeOptions.ConfigFetchRetryCount` XML doc: remove "Reserved … not yet wired"; state "Bounded attempt count (including the first) for the standby's replicated-config fetch; 2 s fixed delay between attempts; superseded fetches never retry." 4. Run → **PASS**; run the replication suite. 5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/SiteReplicationActor.cs src/ZB.MOM.WW.ScadaBridge.SiteRuntime/SiteRuntimeOptions.cs tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/SiteReplicationActorTests.cs && git commit -m "feat(site-runtime): wire ConfigFetchRetryCount into the standby replicated-config fetch (UA2)"` (include the Props-site file in the `git add` if touched) ### Task 25: Low-severity code cleanups — S9, S10, C2, C6 **Classification:** small (batched Low findings; concrete edits below) **Estimated implement time:** ~5 min **Parallelizable with:** 13, 19, 21, 22, 24 **Files:** - Modify: `src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Actors/DataConnectionActor.cs` (PostStop ~216–221; `HandleSubscribeAlarms` ~1763) - Modify: `src/ZB.MOM.WW.ScadaBridge.SiteEventLogging/SiteEventLogger.cs` (comment ~215–218) - Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/ScriptActor.cs` (`EvaluateCondition` fallback ~474–477) - Test: `tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/ScriptActorTests.cs`, `tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests/DataConnectionActorAlarmTests.cs` 1. **C6 failing test** (culture-independence of the non-numeric fallback): ```csharp [Fact] public void ConditionalTrigger_NonNumericFallback_IsCultureInvariant() { var original = CultureInfo.CurrentCulture; try { CultureInfo.CurrentCulture = new CultureInfo("de-DE"); // 1.5 -> "1,5" // Conditional trigger: operator "==", Threshold = 1.5; value "1,5" fails // invariant numeric parse -> string fallback. Old code compares against // Threshold.ToString() == "1,5" under de-DE and FIRES; invariant must NOT. var fired = DriveConditionalTrigger(threshold: 1.5, op: "==", value: "1,5"); // harness helper Assert.False(fired); } finally { CultureInfo.CurrentCulture = original; } } ``` 2. **S10 failing test** (EventFilter on the warning): ```csharp [Fact] public void SecondAlarmSubscriber_WithDifferentFilter_LogsOverwriteWarning() { var actor = CreateConnectedAlarmCapableActor(); // existing alarm-test harness actor.Tell(AlarmSubscribe("src-1", filter: "HighAlarm"), probeA.Ref); EventFilter.Warning(contains: "condition filter").ExpectOne(() => actor.Tell(AlarmSubscribe("src-1", filter: "LowAlarm"), probeB.Ref)); } ``` 3. Run both → **FAIL** (`dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~ConditionalTrigger_NonNumericFallback"` and `dotnet test tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests --filter "FullyQualifiedName~SecondAlarmSubscriber_WithDifferentFilter"`). 4. Implement all four edits: - **C6** (`ScriptActor.cs:476`): `return string.Equals(value.ToString(), config.Threshold.ToString(CultureInfo.InvariantCulture), StringComparison.Ordinal);` — matches the invariant-culture handling two lines up. - **S10** (`DataConnectionActor.cs`, before line 1763's overwrite): `if (_alarmSourceFilter.TryGetValue(request.SourceReference, out var existingFilter) && !string.Equals(existingFilter, request.ConditionFilter, StringComparison.Ordinal)) _log.Warning("[{0}] Alarm condition filter for source {1} overwritten: '{2}' -> '{3}' (last subscriber wins; co-subscribers must agree)", _connectionName, request.SourceReference, existingFilter, request.ConditionFilter);` - **S9** (`DataConnectionActor.PostStop`): replace `_ = _adapter.DisposeAsync().AsTask();` with `_adapter.DisposeAsync().AsTask().ContinueWith(t => _log.Warning("[{0}] Adapter dispose faulted on stop: {1}", _connectionName, t.Exception?.GetBaseException().Message), TaskContinuationOptions.OnlyOnFaulted);` - **C2** (`SiteEventLogger.cs:215-218`): rewrite the comment: "The channel is bounded with DropOldest, so TryWrite only returns false once the channel is completed (shutdown) — full-buffer pressure evicts the oldest pending event instead of failing the write." 5. Run both tests → **PASS**; run both touched projects' suites. 6. Commit: `git add src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Actors/DataConnectionActor.cs src/ZB.MOM.WW.ScadaBridge.SiteEventLogging/SiteEventLogger.cs src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/ScriptActor.cs tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/ScriptActorTests.cs tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests/DataConnectionActorAlarmTests.cs && git commit -m "fix(site-runtime,dcl): low-severity cleanups — observed adapter dispose (S9), alarm-filter overwrite warning (S10), bounded-channel comment (C2), invariant-culture condition fallback (C6)"` ### Task 26: Design-doc updates — C3, C4, C5 + behavior notes for this plan's changes **Classification:** trivial (docs only; repo rule: docs travel with code) **Estimated implement time:** ~5 min **Parallelizable with:** none (write after the behavior tasks land so the docs describe reality) **Files:** - Modify: `docs/requirements/Component-DataConnectionLayer.md` - Modify: `docs/requirements/Component-SiteRuntime.md` 1. `Component-DataConnectionLayer.md`: - **C3** (line ~303): change "The retry interval is defined **per data connection**" to "The retry interval is a **shared setting for all data connections** (`ReconnectInterval` in the Shared Settings table)" — matching the table at lines 170–177 and `DataConnectionOptions`. - **C4** (line ~302, item 1): qualify — "Instance Actors see the staleness immediately and publish it to the site stream/debug view; **script and alarm actors are deliberately NOT re-notified on a quality-only change** (no value changed — re-evaluating triggers would cause spurious firings). Scripts observe quality on their next attribute read; a computed alarm holds its last-known state until fresh values arrive." - **S1 note** (connection lifecycle section): add "On every (re)connect the adapter detaches and disposes the previous protocol client (session, keep-alive timers, event loop) before creating a new one — a stale client can neither leak nor signal a disconnect against the new session." - **S10 note**: add one line documenting the last-subscriber-wins alarm filter + the logged warning on mismatch. 2. `Component-SiteRuntime.md`: - **C5** (Shared Script Library section): "Shared scripts always execute with **Root scope** — `Attributes[...]` inside a shared script addresses root attributes even when invoked from a composed-module script. Callers needing module-scoped access must pass values as parameters." - **S2/UA5** (Script Trust Model / execution-timeout paragraph ~409–412): qualify "Exceeding the timeout cancels the script" → "…cancels the script **cooperatively**: a script blocked in synchronous I/O or a CPU loop does not observe cancellation and continues to occupy its dedicated script-execution thread. A watchdog logs the script by name when its thread has not returned within a grace period after timeout, and the scheduler's queue depth / busy-thread-age gauges are reported through Health Monitoring." - **S3** (Script Compilation Errors section ~474–477): add "Enforced at the site by a pre-compile validation gate in the Deployment Manager: all scripts, alarm on-trigger scripts, and trigger expressions are compiled off-thread before the Instance Actor is created or the config persisted; any failure rejects the deployment with the compile errors." - **S4/UA6** (Instance Actor / DCL interaction): add "A failed or unanswered tag-subscription handshake is retried on a fixed interval per connection (default 5 s), mirroring the native-alarm subscribe retry." - **S5**: note the fixed-interval retry of the startup deployed-config load. - **S6**: note that all Instance Actors are watched and an unexpected termination evicts the ref and fails any in-flight deployment. **Deviation from the plan (2026-07-09, see master-tracker "Plan deviations during Wave 1 PLAN-03 execution"):** deploy-success is NOT reported on persistence alone — the store can commit before the actor's async `PreStart` throws, so the design uses a **two-signal join**: the Instance Actor sends a new `InstanceActorInitialized` readiness message at the end of a successful `PreStart`, and the Deployment Manager reports `Success` only once BOTH persistence has committed AND that signal has arrived. An actor that dies during init never signals readiness → the `Terminated` fallback replies `Failed` and rolls the optimistic state back (the persisted-row rollback is deferred until the store commits so it cannot race the write). Document the readiness message + the join. - **UA1** (cert trust section ~125): add "On a site node (re)joining the cluster, the Deployment Manager singleton pushes its trusted-cert set to the joining node's CertStore (additive union). Removal reconciliation and central persistence of trust decisions remain the documented follow-up." - **UA4** (native alarm mirror section): "Persisted mirror rows include display metadata (type/category/message), so rehydrated conditions render fully before the first source snapshot." - **P2** note (attribute change fan-out ~191): "The Instance Actor routes attribute changes per monitored attribute (Expression-trigger children receive all changes); child actors keep their own trigger gates as defense in depth." - **P6 follow-up note** (deployment/startup section): "Follow-up: per-instance Roslyn compiles still run inside Instance Actor start during staggered startup; moving them off-thread is deferred (affects failover time-to-recover only)." 3. Review with `git diff docs/` — verify no stale cross-references (e.g., the C3 sentence is the only "per data connection" retry claim). 4. Commit: `git add docs/requirements/Component-DataConnectionLayer.md docs/requirements/Component-SiteRuntime.md && git commit -m "docs(requirements): sync SiteRuntime + DCL specs with hardening changes (C3/C4/C5 + S1-S6, P2, UA1/UA4 behavior notes)"` --- ## Dependencies on other plans - **Plan 05 (Templates/Deployment/Transport)** owns script-trust-policy and Roslyn compile **caching** in central validation paths. Task 3/4 here adds a *site-side* compile gate that duplicates compile work per deploy — if plan 05 introduces a shared compile-cache/collectible-ALC abstraction, the `DeployCompileValidator` should adopt it later; no ordering dependency (this plan's gate is self-contained). - **Plan 02 (Communication/S&F)** owns the S&F store SQLite (`StoreAndForwardStorage`) — Task 19 deliberately touches only `SiteStorageService`. If plan 02 also enables WAL on the S&F store, the two changes are independent files. - **Plan 06 (Edge integrations)** owns inbound script execution (DI-scope use-after-dispose). Task 8's watchdog pattern (`completed` flag + grace check) is reusable there; no ordering dependency. - **Plan 01 (Cluster/Host/Failover)** owns SBR enablement. Task 23 (reconcile-on-join) becomes *more* valuable once real failover works, but functions independently. - Health-report additive fields (Task 6) touch `Commons/Messages/Health/SiteHealthReport.cs` — coordinate with any other plan touching that record (additive-only on both sides makes merges trivial). ## Execution order **P0 (do first, in order):** Task 1, Task 2 (S1 — the single highest-value fix: stops the leak *and* the healthy-connection flapping), then Task 3 → Task 4 (S3 — deployment honesty, matches OVERALL P1 recommendation #8). **P1:** Tasks 5 → 6 → 7 (scheduler observability chain), Task 8 (watchdog), Tasks 9, 10 (P1 dispatcher relief), Task 11 → 12 (subscription retries), Task 13 (startup retry), Task 14 → 15 (dead-child bookkeeping — 15 depends on 14's Terminated fallback). **P2:** Task 16 → 17 (both edit `DataConnectionActor.cs`, sequential), Task 18 (after 11 — both edit `InstanceActor.cs`), Task 19 → 20 → 21 (19 and 21 both edit `SiteStorageService.cs`; 20 and 21 both edit `NativeAlarmActor.cs` — run 19, 20, 21 in that order; 12 must precede 20 for `NativeAlarmActor.cs`), Task 22 → 23 (23 needs 22's messages and follows 15 in `DeploymentManagerActor.cs`), Task 24. **Last:** Task 25 (after 9 and 17 — shared files), then Task 26 (docs describe the landed behavior). Same-file sequencing summary: `DeploymentManagerActor.cs`: 4 → 13 → 14 → 15 → 23. `InstanceActor.cs`: 4 (doc comment) → 11 → 18. `NativeAlarmActor.cs`: 12 → 20 → 21. `SiteStorageService.cs`: 19 → 20 → 21. `DataConnectionActor.cs`: 16 → 17 → 25. `ScriptActor.cs`: 9 → 18 (helper extraction) → 25. `SiteRuntimeOptions.cs`: 8 → 11 → 24. After the final task: `dotnet build ZB.MOM.WW.ScadaBridge.slnx && dotnet test ZB.MOM.WW.ScadaBridge.slnx`, and for cluster-runtime verification rebuild the docker image (`bash docker/deploy.sh`).