Joseph Doherty
1298 lines
98 KiB
Markdown
1298 lines
98 KiB
Markdown
# Site Runtime & DCL Hardening Implementation Plan
|
||
|
||
> **For Claude:** REQUIRED SUB-SKILL: Use superpowers-extended-cc:executing-plans to implement this plan task-by-task.
|
||
|
||
**Goal:** Fix every finding in `archreview/03-site-runtime-dcl.md` — adapter reconnect lifecycle leaks, script-containment observability, deployment honesty (site-side compile failure must reject the deployment per spec), subscription/startup retry holes, dead-child bookkeeping, hot-path fan-out cost, and site SQLite hardening — with design docs updated alongside the code.
|
||
|
||
**Architecture:** All changes are site-side, inside `ZB.MOM.WW.ScadaBridge.SiteRuntime` and `ZB.MOM.WW.ScadaBridge.DataConnectionLayer` (plus additive message/report contracts in Commons/HealthMonitoring). The actor model is preserved: every fix keeps state mutation on the actor thread (background work returns via `PipeTo`), retries use the existing fixed-interval timer idiom, and new health metrics ride the existing `ISiteHealthCollector` → `SiteHealthReport` path with additive-only contract changes. The deploy compile gate runs off-thread before actor creation so the singleton mailbox never blocks on Roslyn.
|
||
|
||
**Tech Stack:** C#/.NET, Akka.NET (ReceiveActor/UntypedActor, TestKit.Xunit2), Roslyn scripting, Microsoft.Data.Sqlite, xunit + NSubstitute.
|
||
|
||
Build: `dotnet build ZB.MOM.WW.ScadaBridge.slnx`. Test per-project: `dotnet test tests/<project> --filter <name>`.
|
||
|
||
## Findings Coverage
|
||
|
||
| Finding | Severity | Task(s) / Disposition |
|
||
|---|---|---|
|
||
| S1 — reconnect abandons previous client, stale `ConnectionLost` flaps healthy connection | High | 1 (OPC UA), 2 (MxGateway) |
|
||
| S2 — cooperative-only timeout; no stuck-thread observability | High | 5, 6, 7, 8. Sub-suggestion "inject ADO.NET command timeouts into Database.Connection" **deferred**: scripts own their ADO.NET commands (`SqlCommand.CommandTimeout` defaults to 30 s already); the watchdog + gauges make the residual case visible. |
|
||
| S3 — site compile failure doesn't reject deployment (spec drift) | High | 3, 4 |
|
||
| S4 — no retry on failed tag subscription; NativeAlarm lost-response gap | Medium | 11 (InstanceActor), 12 (NativeAlarmActor) |
|
||
| S5 — startup SQLite load failure aborts silently, no retry | Medium | 13 |
|
||
| S6 — dead child ref in `_instanceActors` + Success reported for dead instance | Medium | 14, 15 |
|
||
| S7 — background tasks read mutable `_adapter` field off-thread | Medium | 16 |
|
||
| S8 — site SQLite: no WAL/busy-timeout | Low | 19 |
|
||
| S9 — fire-and-forget adapter dispose in `PostStop` unobserved | Low | 25 |
|
||
| S10 — alarm condition filter last-subscriber-wins, silent overwrite | Low | 25 |
|
||
| P1 — trigger expressions block dispatcher threads per attribute change | High | 9 (ScriptActor), 10 (AlarmActor) |
|
||
| P2 — attribute-change fan-out broadcast to every child | Medium | 18 |
|
||
| P3 — DCL tag-value fan-out scans all instances per update | Medium | 17 |
|
||
| P4 — per-transition SQLite upserts on native-alarm mirror | Low | 20 |
|
||
| P5 — per-attribute script read is an Ask round-trip | Low | **Deferred** — correct and spec-consistent; no profiling evidence yet; batch `GetAttributes` is a follow-up when profiling flags it. |
|
||
| P6 — Roslyn compiles inside `InstanceActor.PreStart` during staggered startup | Low | **Deferred** (partially mitigated) — Task 3/4 moves *deploy-path* compiles off the singleton mailbox; moving *startup-path* compiles out of `PreStart` requires an async child-creation phase (messages arriving before children exist) and is out of scope; risk is failover time-to-recover only. Logged as follow-up in Task 26 doc note. |
|
||
| C1 — positive (Akka discipline) | — | No action. |
|
||
| C2 — stale "unbounded channel" comment in SiteEventLogger | Low | 25 |
|
||
| C3 — spec self-contradiction on reconnect interval scope | Low | 26 |
|
||
| C4 — quality-only change not delivered to children vs. spec wording | Low | 26 |
|
||
| C5 — shared scripts run Root scope, undocumented | Low | 26 |
|
||
| C6 — `EvaluateCondition` fallback uses current-culture `ToString` | Low | 25 |
|
||
| UA1 — cert-trust convergence after node downtime | Underdev. | 22, 23 (+ doc in 26). List-aggregation across both nodes and removal-reconcile **deferred** to the documented central-persistence follow-up (`Component-SiteRuntime.md:125`). |
|
||
| UA2 — `ConfigFetchRetryCount` dead option | Underdev. | 24 |
|
||
| UA3 — no aggregated live alarm stream | Underdev. | **Deferred** — documented M7 follow-up, central-side; out of this plan's site-runtime scope. |
|
||
| UA4 — native-alarm rehydration loses display metadata | Underdev. | 21 |
|
||
| UA5 — no scheduler stuck-thread/queue-depth observability | Underdev. | 5, 6, 7, 8 (same fix as S2) |
|
||
| UA6 — tag-subscription failure path has no retry/site event | Underdev. | 11, 12 (same fix as S4) |
|
||
| UA7 — test gaps (reconnect-stale-handler, compile-fail deploy status, subscribe race) | Underdev. | Covered by the TDD tests in Tasks 1, 2, 4, 11 |
|
||
|
||
---
|
||
|
||
### Task 1: OpcUaDataConnection — dispose + detach the previous client on reconnect
|
||
|
||
**Classification:** high-risk (adapter lifecycle, concurrency with SDK callback threads)
|
||
**Estimated implement time:** ~5 min
|
||
**Parallelizable with:** 2, 3, 5, 9, 10, 12, 13, 19, 22, 24
|
||
**Files:**
|
||
- Modify: `src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Adapters/OpcUaDataConnection.cs` (ConnectAsync, ~lines 97–108)
|
||
- Test: `tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests/OpcUaDataConnectionTests.cs`
|
||
|
||
1. Add failing tests (reuse the existing `Substitute.For<IOpcUaClient>()` / `IOpcUaClientFactory` fixture; note the fixture returns one `_mockClient` — for these tests configure the factory to return a **fresh substitute per `Create()` call**):
|
||
|
||
```csharp
|
||
[Fact]
|
||
public async Task Reconnect_DisposesAndDetachesPreviousClient()
|
||
{
|
||
var clients = new List<IOpcUaClient>();
|
||
_mockFactory.Create().Returns(_ =>
|
||
{
|
||
var c = Substitute.For<IOpcUaClient>();
|
||
clients.Add(c);
|
||
return c;
|
||
});
|
||
var details = new Dictionary<string, string> { ["EndpointUrl"] = "opc.tcp://x:4840" };
|
||
|
||
await _adapter.ConnectAsync(details); // client[0]
|
||
await _adapter.ConnectAsync(details); // reconnect -> client[1]
|
||
|
||
Assert.Equal(2, clients.Count);
|
||
await clients[0].Received(1).DisposeAsync();
|
||
await clients[1].DidNotReceive().DisposeAsync();
|
||
}
|
||
|
||
[Fact]
|
||
public async Task StaleClientConnectionLost_DoesNotRaiseDisconnected_AfterReconnect()
|
||
{
|
||
var clients = new List<IOpcUaClient>();
|
||
_mockFactory.Create().Returns(_ =>
|
||
{
|
||
var c = Substitute.For<IOpcUaClient>();
|
||
clients.Add(c);
|
||
return c;
|
||
});
|
||
var details = new Dictionary<string, string> { ["EndpointUrl"] = "opc.tcp://x:4840" };
|
||
var disconnects = 0;
|
||
_adapter.Disconnected += () => Interlocked.Increment(ref disconnects);
|
||
|
||
await _adapter.ConnectAsync(details);
|
||
await _adapter.ConnectAsync(details); // reconnect resets _disconnectFired
|
||
|
||
// The OLD client's keep-alive dies later — must be a no-op now.
|
||
clients[0].ConnectionLost += Raise.Event<Action>();
|
||
Assert.Equal(0, disconnects);
|
||
|
||
// The CURRENT client's loss must still signal.
|
||
clients[1].ConnectionLost += Raise.Event<Action>();
|
||
Assert.Equal(1, disconnects);
|
||
}
|
||
```
|
||
|
||
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests --filter "FullyQualifiedName~Reconnect_DisposesAndDetachesPreviousClient|FullyQualifiedName~StaleClientConnectionLost_DoesNotRaiseDisconnected"` → expect **FAIL** (old client never disposed; stale handler still wired).
|
||
3. Implement in `ConnectAsync`, immediately before `_client = _clientFactory.Create();`:
|
||
|
||
```csharp
|
||
// Reconnect reuses this adapter instance: detach + dispose the previous
|
||
// client so (a) its session/keep-alive/socket do not leak per flap and
|
||
// (b) its late ConnectionLost cannot flap the NEW healthy session after
|
||
// _disconnectFired is reset below. Dispose is best-effort fire-and-forget
|
||
// (CloseAsync against a dead server can block for the operation timeout);
|
||
// faults are logged, never thrown into the connect path.
|
||
if (_client is { } previousClient)
|
||
{
|
||
previousClient.ConnectionLost -= OnClientConnectionLost;
|
||
_ = previousClient.DisposeAsync().AsTask().ContinueWith(
|
||
t => _logger.LogWarning(t.Exception?.GetBaseException(),
|
||
"Disposing previous OPC UA client failed for {Endpoint}", _endpointUrl),
|
||
TaskContinuationOptions.OnlyOnFaulted);
|
||
}
|
||
```
|
||
|
||
Also stop the previous heartbeat monitor if active (call the existing `StopHeartbeatMonitor()` before creating the new client) so the old `StaleTagMonitor` cannot fire against the new session.
|
||
4. Run the two tests → expect **PASS**. Run the full adapter suite: `dotnet test tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests --filter "FullyQualifiedName~OpcUaDataConnectionTests"` → all green.
|
||
5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Adapters/OpcUaDataConnection.cs tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests/OpcUaDataConnectionTests.cs && git commit -m "fix(dcl): dispose+detach previous OPC UA client on reconnect (S1) — stops session leak and stale ConnectionLost flapping"`
|
||
|
||
### Task 2: MxGatewayDataConnection — dispose previous client + cancel previous event loop on reconnect
|
||
|
||
**Classification:** high-risk (adapter lifecycle)
|
||
**Estimated implement time:** ~4 min
|
||
**Parallelizable with:** 1, 3, 5, 9, 10, 12, 13, 19, 22, 24
|
||
**Files:**
|
||
- Modify: `src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Adapters/MxGatewayDataConnection.cs` (ConnectAsync, ~lines 67–88)
|
||
- Test: create `tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests/Adapters/MxGatewayDataConnectionReconnectTests.cs`
|
||
|
||
1. Failing test (mirror the OPC UA fixture shape with `Substitute.For<IMxGatewayClient>()` / `IMxGatewayClientFactory`; capture the `CancellationToken` handed to `RunEventLoopAsync`):
|
||
|
||
```csharp
|
||
[Fact]
|
||
public async Task Reconnect_DisposesPreviousClient_AndCancelsPreviousEventLoop()
|
||
{
|
||
var clients = new List<IMxGatewayClient>();
|
||
var loopTokens = new List<CancellationToken>();
|
||
var factory = Substitute.For<IMxGatewayClientFactory>();
|
||
factory.Create().Returns(_ =>
|
||
{
|
||
var c = Substitute.For<IMxGatewayClient>();
|
||
c.RunEventLoopAsync(Arg.Any<Action<MxGatewayValueUpdate>>(), Arg.Do<CancellationToken>(t => loopTokens.Add(t)))
|
||
.Returns(Task.Delay(Timeout.Infinite));
|
||
clients.Add(c);
|
||
return c;
|
||
});
|
||
var adapter = new MxGatewayDataConnection(factory, NullLogger<MxGatewayDataConnection>.Instance);
|
||
var details = new Dictionary<string, string> { ["Endpoint"] = "localhost:5000", ["ApiKey"] = "k" };
|
||
|
||
await adapter.ConnectAsync(details);
|
||
await adapter.ConnectAsync(details); // reconnect
|
||
|
||
await clients[0].Received(1).DisposeAsync();
|
||
Assert.True(loopTokens[0].IsCancellationRequested); // old loop cancelled
|
||
Assert.False(loopTokens[1].IsCancellationRequested); // new loop live
|
||
}
|
||
```
|
||
|
||
(Adjust the `RunEventLoopAsync` signature/`MxGatewayValueUpdate` names to the real `IMxGatewayClient` members — check `src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Adapters/IMxGatewayClient.cs`.)
|
||
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests --filter "FullyQualifiedName~MxGatewayDataConnectionReconnectTests"` → **FAIL**.
|
||
3. Implement at the top of `ConnectAsync` (before `_client = _clientFactory.Create();`):
|
||
|
||
```csharp
|
||
// Reconnect reuses this adapter: cancel the previous event loop and dispose
|
||
// the previous client so a flapping gateway cannot accumulate orphaned
|
||
// streams/sockets, and the old loop's fault path cannot signal Disconnected
|
||
// against the new session after the guard resets below.
|
||
if (_eventLoopCts is { } previousLoopCts)
|
||
{
|
||
previousLoopCts.Cancel();
|
||
previousLoopCts.Dispose();
|
||
_eventLoopCts = null;
|
||
}
|
||
if (_client is { } previousClient)
|
||
{
|
||
_ = previousClient.DisposeAsync().AsTask().ContinueWith(
|
||
t => _logger.LogWarning(t.Exception?.GetBaseException(),
|
||
"Disposing previous MxGateway client failed"),
|
||
TaskContinuationOptions.OnlyOnFaulted);
|
||
}
|
||
```
|
||
|
||
Note ordering: keep `Interlocked.Exchange(ref _disconnectFired, 0)` AFTER the cancel/dispose, so a fault raised by the dying loop during teardown is absorbed by the *old* guard cycle. Also clear `_subs`/`_tagToSub` is NOT needed (re-subscribe repopulates via the actor), but verify `RunEventLoopAsync`'s catch respects the cancelled token → `OperationCanceledException` path, which it already does.
|
||
4. Run → **PASS**; then full project: `dotnet test tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests` → green.
|
||
5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Adapters/MxGatewayDataConnection.cs tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests/Adapters/MxGatewayDataConnectionReconnectTests.cs && git commit -m "fix(dcl): cancel old event loop + dispose old MxGateway client on reconnect (S1)"`
|
||
|
||
### Task 3: Pure deploy-time compile validator for a flattened config
|
||
|
||
**Classification:** standard
|
||
**Estimated implement time:** ~5 min
|
||
**Parallelizable with:** 1, 2, 5, 9, 10, 12, 19, 22, 24
|
||
**Files:**
|
||
- Create: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Deployment/DeployCompileValidator.cs`
|
||
- Test: create `tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Deployment/DeployCompileValidatorTests.cs`
|
||
|
||
1. Failing test:
|
||
|
||
```csharp
|
||
public class DeployCompileValidatorTests
|
||
{
|
||
private static string ConfigWith(string scriptCode) => JsonSerializer.Serialize(new FlattenedConfiguration
|
||
{
|
||
Scripts = new List<ResolvedScript>
|
||
{
|
||
new() { CanonicalName = "S1", Code = scriptCode, TriggerType = "Call" }
|
||
},
|
||
Attributes = new List<ResolvedAttribute>(),
|
||
Alarms = new List<ResolvedAlarm>(),
|
||
NativeAlarmSources = new List<ResolvedNativeAlarmSource>()
|
||
});
|
||
|
||
[Fact]
|
||
public void BrokenScript_ReturnsErrors()
|
||
{
|
||
var validator = new DeployCompileValidator(new ScriptCompilationService(/* mirror ctor args used in ScriptCompilationServiceTests */));
|
||
var errors = validator.Validate(ConfigWith("this is not C# ~~~"));
|
||
Assert.NotEmpty(errors);
|
||
Assert.Contains(errors, e => e.Contains("S1"));
|
||
}
|
||
|
||
[Fact]
|
||
public void ValidScript_ReturnsNoErrors()
|
||
{
|
||
var validator = new DeployCompileValidator(new ScriptCompilationService(/* as above */));
|
||
Assert.Empty(validator.Validate(ConfigWith("return 1 + 1;")));
|
||
}
|
||
|
||
[Fact]
|
||
public void BrokenTriggerExpression_ReturnsErrors()
|
||
{
|
||
var cfg = /* config with one Expression-triggered script whose TriggerConfiguration
|
||
JSON carries expression: "1 +" (mirror shape used in ScriptActorTests) */;
|
||
var validator = new DeployCompileValidator(new ScriptCompilationService(/* as above */));
|
||
Assert.NotEmpty(validator.Validate(cfg));
|
||
}
|
||
}
|
||
```
|
||
|
||
(Mirror `FlattenedConfiguration`/`ResolvedScript` construction and the `ScriptCompilationService` ctor from the existing `ScriptCompilationServiceTests` / `InstanceActorTests` fixtures.)
|
||
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~DeployCompileValidatorTests"` → **FAIL** (type does not exist).
|
||
3. Implement `DeployCompileValidator`: a small class taking the same compilation-service type `InstanceActor` uses; method `IReadOnlyList<string> Validate(string configJson)`:
|
||
- Deserialize `FlattenedConfiguration`; malformed JSON → single error `"Configuration JSON failed to deserialize: …"` (this also fronts Task 15's poison-config case).
|
||
- For each `config.Scripts`: `_compilationService.Compile(script.CanonicalName, script.Code)`; on failure add `$"Script '{script.CanonicalName}': {string.Join("; ", result.Errors)}"`.
|
||
- For each alarm with `OnTriggerScriptCanonicalName`: resolve the script def (missing def → error) and compile it (mirroring `InstanceActor.CreateChildActors` lines 1404–1424).
|
||
- For each Expression-trigger script/alarm: extract via `TriggerExpressionGlobals.ExtractExpression` and `CompileTriggerExpression`; failures are errors (mirroring `InstanceActor.CompileTriggerExpression`).
|
||
- Pure and thread-safe (no actor state) — callable from `Task.Run`.
|
||
4. Run → **PASS**.
|
||
5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Deployment/DeployCompileValidator.cs tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Deployment/DeployCompileValidatorTests.cs && git commit -m "feat(site-runtime): pure deploy-time compile validator for flattened configs (S3 groundwork)"`
|
||
|
||
### Task 4: Reject the deployment on site-side compile failure (per spec)
|
||
|
||
**Classification:** high-risk (deployment pipeline, singleton actor)
|
||
**Estimated implement time:** ~5 min
|
||
**Parallelizable with:** 1, 2, 12, 19, 22, 24
|
||
**Files:**
|
||
- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/DeploymentManagerActor.cs` (HandleDeploy entry + new piped message)
|
||
- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/InstanceActor.cs` (fix the contradictory XML doc at line 1339)
|
||
- Test: `tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/DeploymentManagerActorTests.cs`
|
||
|
||
1. Failing test (reuse the existing DeploymentManagerActor TestKit harness/fixture in that file):
|
||
|
||
```csharp
|
||
[Fact]
|
||
public void Deploy_WithNonCompilingScript_ReportsFailed_AndCreatesNoInstanceActor()
|
||
{
|
||
var badConfig = /* FlattenedConfiguration JSON with one script whose Code = "not C# ~~~" —
|
||
reuse the config-builder helper the existing deploy tests use */;
|
||
var dm = CreateDeploymentManager(); // existing harness factory
|
||
|
||
dm.Tell(new DeployInstanceCommand(/* deploymentId */ Guid.NewGuid().ToString(),
|
||
"bad-instance", badConfig, /* revisionHash */ "r1" /*, …existing args*/));
|
||
|
||
var resp = ExpectMsg<DeploymentStatusResponse>(TimeSpan.FromSeconds(10));
|
||
Assert.Equal(DeploymentStatus.Failed, resp.Status);
|
||
Assert.Contains("compile", resp.ErrorMessage, StringComparison.OrdinalIgnoreCase);
|
||
|
||
// No partial state applied: instance actor must not exist, config must not persist.
|
||
Sys.ActorSelection(dm.Path / "bad-instance").Tell(new Identify(1));
|
||
Assert.Null(ExpectMsg<ActorIdentity>().Subject);
|
||
_storage.DidNotReceive().StoreDeployedConfigAsync("bad-instance",
|
||
Arg.Any<string>(), Arg.Any<string>(), Arg.Any<string>(), Arg.Any<bool>());
|
||
}
|
||
```
|
||
|
||
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~Deploy_WithNonCompilingScript"` → **FAIL** (today it reports Success).
|
||
3. Implement:
|
||
- Add internal message `private sealed record DeployValidationResult(DeployInstanceCommand Cmd, IActorRef ReplyTo, IReadOnlyList<string> Errors);` and `Receive<DeployValidationResult>(HandleDeployValidationResult);`.
|
||
- Rename the current `HandleDeploy(DeployInstanceCommand)` body to `ProceedWithDeploy(DeployInstanceCommand command, IActorRef sender)` (this is the code that decides redeploy-buffer vs `ApplyDeployment` — keep it byte-identical).
|
||
- New `HandleDeploy`: capture `Sender`; run `Task.Run(() => _deployCompileValidator.Validate(command.FlattenedConfigurationJson))` → `ContinueWith` into `DeployValidationResult` → `PipeTo(Self)`. Validation is pure, so the singleton mailbox never blocks on Roslyn, and results are applied on the actor thread in arrival order (the existing redeploy-supersede handling still governs ordering).
|
||
- `HandleDeployValidationResult`: if `Errors.Count > 0` → `LogDeploymentEvent("Error", …, "compile validation failed")`, reply `DeploymentStatusResponse(DeploymentId, Instance, DeploymentStatus.Failed, "Script compilation failed on site: " + string.Join(" | ", errors), UtcNow)` and **return** (no actor creation, no persistence — spec: "No partial state is applied"). Else call `ProceedWithDeploy(msg.Cmd, msg.ReplyTo)`.
|
||
- Construct `_deployCompileValidator = new DeployCompileValidator(_compilationService)` in the ctor.
|
||
- Fix `InstanceActor.cs:1339` XML doc: replace "Compilation errors reject entire instance deployment (logged but actor still starts)" with "Compile failures are rejected at deploy time by the Deployment Manager's validation gate; a failure reaching this point (startup load of a legacy config) is logged and the script skipped."
|
||
4. Run the new test → **PASS**; then the deploy suites: `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~DeploymentManager"` → green (existing deploy/redeploy/disable-supersede tests must still pass — they exercise `ProceedWithDeploy` through the new gate with compiling configs).
|
||
5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/DeploymentManagerActor.cs src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/InstanceActor.cs tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/DeploymentManagerActorTests.cs && git commit -m "fix(site-runtime): site-side compile failure now rejects the deployment per spec (S3) — off-thread validation gate before actor creation"`
|
||
|
||
### Task 5: ScriptExecutionScheduler gauges — queue depth, busy threads, oldest-busy age
|
||
|
||
**Classification:** small (concurrency-adjacent but additive instrumentation)
|
||
**Estimated implement time:** ~4 min
|
||
**Parallelizable with:** 1, 2, 3, 9, 10, 11, 12, 13, 19, 22, 24
|
||
**Files:**
|
||
- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Scripts/ScriptExecutionScheduler.cs`
|
||
- Test: `tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Scripts/ScriptExecutionSchedulerTests.cs`
|
||
|
||
1. Failing test:
|
||
|
||
```csharp
|
||
[Fact]
|
||
public async Task Gauges_ReflectBusyThreadAndQueueDepth()
|
||
{
|
||
using var scheduler = new ScriptExecutionScheduler(1);
|
||
using var gate = new ManualResetEventSlim(false);
|
||
|
||
var blocking = Task.Factory.StartNew(() => gate.Wait(),
|
||
CancellationToken.None, TaskCreationOptions.None, scheduler);
|
||
var queued = Task.Factory.StartNew(() => { },
|
||
CancellationToken.None, TaskCreationOptions.None, scheduler);
|
||
|
||
await WaitUntilAsync(() => scheduler.BusyThreadCount == 1);
|
||
Assert.Equal(1, scheduler.QueueDepth); // second task waiting
|
||
Assert.True(scheduler.OldestBusyAge > TimeSpan.Zero);
|
||
|
||
gate.Set();
|
||
await Task.WhenAll(blocking, queued);
|
||
await WaitUntilAsync(() => scheduler.BusyThreadCount == 0);
|
||
Assert.Null(scheduler.OldestBusyAge);
|
||
}
|
||
// WaitUntilAsync: small poll helper (100×50ms) — add locally if the test file has none.
|
||
```
|
||
|
||
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~Gauges_ReflectBusyThreadAndQueueDepth"` → **FAIL** (members don't exist).
|
||
3. Implement:
|
||
- `private readonly long[] _busySinceTicks;` sized `threadCount`, initialized 0. Give each worker its index (change `new Thread(WorkerLoop)` to `new Thread(() => WorkerLoop(index))` with a captured local).
|
||
- In `WorkerLoop(int index)`: around `TryExecuteTask(task)` do `Volatile.Write(ref _busySinceTicks[index], Environment.TickCount64);` before and `Volatile.Write(ref _busySinceTicks[index], 0);` in a `finally`.
|
||
- Public gauges: `public int QueueDepth => _queue.Count;` `public int BusyThreadCount` (count of non-zero slots), `public TimeSpan? OldestBusyAge` (max of `now - since` over non-zero slots, else null).
|
||
4. Run → **PASS**. Run full scheduler suite: `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~ScriptExecutionScheduler"`.
|
||
5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Scripts/ScriptExecutionScheduler.cs tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Scripts/ScriptExecutionSchedulerTests.cs && git commit -m "feat(site-runtime): script scheduler gauges — queue depth, busy threads, oldest-busy age (S2/UA5)"`
|
||
|
||
### Task 6: Surface scheduler stats through ISiteHealthCollector + SiteHealthReport (additive)
|
||
|
||
**Classification:** small
|
||
**Estimated implement time:** ~4 min
|
||
**Parallelizable with:** 1, 2, 3, 4, 9, 10, 11, 12, 13, 19, 22, 24
|
||
**Files:**
|
||
- Modify: `src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/ISiteHealthCollector.cs` (new default-no-op method)
|
||
- Modify: `src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/SiteHealthCollector.cs` (store + include in `CollectReport`)
|
||
- Modify: `src/ZB.MOM.WW.ScadaBridge.Commons/Messages/Health/SiteHealthReport.cs` (additive optional properties)
|
||
- Test: `tests/ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests/` (add to the existing SiteHealthCollector test file)
|
||
|
||
1. Failing test (in the existing collector test class):
|
||
|
||
```csharp
|
||
[Fact]
|
||
public void CollectReport_CarriesScriptSchedulerStats()
|
||
{
|
||
var collector = CreateCollector(); // existing fixture helper
|
||
collector.SetScriptSchedulerStats(queueDepth: 7, busyThreads: 3, oldestBusyAgeSeconds: 42.5);
|
||
var report = collector.CollectReport("site-1");
|
||
Assert.Equal(7, report.ScriptQueueDepth);
|
||
Assert.Equal(3, report.ScriptBusyThreads);
|
||
Assert.Equal(42.5, report.ScriptOldestBusyAgeSeconds);
|
||
}
|
||
```
|
||
|
||
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests --filter "FullyQualifiedName~CollectReport_CarriesScriptSchedulerStats"` → **FAIL**.
|
||
3. Implement:
|
||
- Interface: `void SetScriptSchedulerStats(int queueDepth, int busyThreads, double? oldestBusyAgeSeconds) { }` (default no-op, matching the `SetSiteEventLogWriteFailures` pattern so test fakes keep compiling).
|
||
- Collector: store via `Interlocked`/volatile fields; include in `CollectReport`.
|
||
- `SiteHealthReport`: **additive** optional init properties with safe defaults (`public int ScriptQueueDepth { get; init; }`, `public int ScriptBusyThreads { get; init; }`, `public double? ScriptOldestBusyAgeSeconds { get; init; }`) — do NOT change the positional parameter list (message-contract additive-only rule).
|
||
4. Run → **PASS**; run full projects: `dotnet test tests/ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests` and `dotnet test tests/ZB.MOM.WW.ScadaBridge.Commons.Tests`.
|
||
5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/ISiteHealthCollector.cs src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/SiteHealthCollector.cs src/ZB.MOM.WW.ScadaBridge.Commons/Messages/Health/SiteHealthReport.cs tests/ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests && git commit -m "feat(health): additive script-scheduler stats on ISiteHealthCollector + SiteHealthReport (S2/UA5)"`
|
||
|
||
### Task 7: Periodic scheduler-stats reporter (hosted service)
|
||
|
||
**Classification:** small
|
||
**Estimated implement time:** ~4 min
|
||
**Parallelizable with:** 1, 2, 4, 9, 10, 11, 12, 13, 19, 22, 24 (needs 5 + 6 merged)
|
||
**Files:**
|
||
- Create: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Scripts/ScriptSchedulerStatsReporter.cs`
|
||
- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/ServiceCollectionExtensions.cs` (register hosted service on site hosts)
|
||
- Test: create `tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Scripts/ScriptSchedulerStatsReporterTests.cs`
|
||
|
||
1. Failing test:
|
||
|
||
```csharp
|
||
[Fact]
|
||
public async Task Reporter_PushesStatsToCollector()
|
||
{
|
||
var collector = Substitute.For<ISiteHealthCollector>();
|
||
var options = new SiteRuntimeOptions { ScriptExecutionThreadCount = 1 };
|
||
using var reporter = new ScriptSchedulerStatsReporter(
|
||
collector, options, NullLogger<ScriptSchedulerStatsReporter>.Instance,
|
||
pollInterval: TimeSpan.FromMilliseconds(50));
|
||
|
||
await reporter.StartAsync(CancellationToken.None);
|
||
await Task.Delay(300);
|
||
await reporter.StopAsync(CancellationToken.None);
|
||
|
||
collector.ReceivedWithAnyArgs().SetScriptSchedulerStats(default, default, default);
|
||
}
|
||
```
|
||
|
||
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~ScriptSchedulerStatsReporterTests"` → **FAIL**.
|
||
3. Implement a `BackgroundService` mirroring `SiteEventLogFailureCountReporter` (`src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/SiteEventLogFailureCountReporter.cs`): every 10 s (test-overridable ctor param) read `ScriptExecutionScheduler.Shared(options)` gauges and call `collector.SetScriptSchedulerStats(...)`. Register it in the SiteRuntime `ServiceCollectionExtensions` next to the existing site-role hosted services. Never throw from the loop (catch + log, matching the sibling reporter).
|
||
4. Run → **PASS**. Build the solution once (`dotnet build ZB.MOM.WW.ScadaBridge.slnx`) to confirm Host wiring compiles.
|
||
5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Scripts/ScriptSchedulerStatsReporter.cs src/ZB.MOM.WW.ScadaBridge.SiteRuntime/ServiceCollectionExtensions.cs tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Scripts/ScriptSchedulerStatsReporterTests.cs && git commit -m "feat(site-runtime): periodic scheduler-stats reporter feeding site health (S2/UA5)"`
|
||
|
||
### Task 8: Stuck-script watchdog — name the script whose thread never came back
|
||
|
||
**Classification:** high-risk (script execution path)
|
||
**Estimated implement time:** ~5 min
|
||
**Parallelizable with:** 1, 2, 4, 9, 10, 12, 13, 19, 22, 24
|
||
**Files:**
|
||
- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/ScriptExecutionActor.cs` (~lines 122–260)
|
||
- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/SiteRuntimeOptions.cs` (new `StuckScriptGraceMs`, default 30000)
|
||
- Test: `tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/ExecutionActorTests.cs`
|
||
|
||
1. Failing test (follow the existing `ExecutionActorTests` pattern for compiling a `Script<object?>` directly with `CSharpScript.Create<object>(code, options, typeof(ScriptGlobals))` — test-compiled, so the trust validator is not in the path):
|
||
|
||
```csharp
|
||
[Fact]
|
||
public async Task TimedOutScript_WithBlockedThread_EmitsStuckThreadSiteEvent()
|
||
{
|
||
var gate = new SemaphoreSlim(0);
|
||
StuckTestHooks.Gate = gate; // small public static test hook class in the test assembly
|
||
var script = CompileRaw("ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.Actors.StuckTestHooks.Gate.Wait(); return null;");
|
||
var siteEvents = new RecordingSiteEventLogger(); // reuse/extend the existing fake in TestSupport
|
||
var options = new SiteRuntimeOptions { ScriptExecutionTimeoutSeconds = 1, StuckScriptGraceMs = 200 };
|
||
|
||
SpawnExecutionActor(script, options, siteEvents); // existing harness helper
|
||
|
||
await WaitUntilAsync(() => siteEvents.Events.Any(e =>
|
||
e.Category == "script" && e.Message.Contains("still executing", StringComparison.OrdinalIgnoreCase)),
|
||
timeout: TimeSpan.FromSeconds(10));
|
||
|
||
gate.Release(); // free the scheduler thread so the test run stays clean
|
||
}
|
||
```
|
||
|
||
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~TimedOutScript_WithBlockedThread_EmitsStuckThreadSiteEvent"` → **FAIL**.
|
||
3. Implement inside the execution lambda in `ScriptExecutionActor`:
|
||
- `var completed = 0;` before the `try`; `Interlocked.Exchange(ref completed, 1);` in the existing `finally` (add one if missing — the scope-dispose finally exists).
|
||
- Immediately after `using var cts = new CancellationTokenSource(timeout);`:
|
||
|
||
```csharp
|
||
// Cooperative-only timeout containment (S2): the CTS firing does NOT free a
|
||
// thread blocked in synchronous I/O. When the timeout elapses, wait a grace
|
||
// period on the thread pool and, if the body still hasn't returned, name the
|
||
// script loudly — this is the only signal an operator gets that one of the
|
||
// bounded script-execution threads is gone.
|
||
var graceMs = options.StuckScriptGraceMs;
|
||
cts.Token.Register(() => _ = Task.Run(async () =>
|
||
{
|
||
await Task.Delay(graceMs);
|
||
if (Volatile.Read(ref completed) == 0)
|
||
{
|
||
var msg = $"Script '{scriptName}' on instance '{instanceName}' exceeded its " +
|
||
$"{timeout.TotalSeconds:F0}s timeout and is STILL EXECUTING — its dedicated " +
|
||
"script-execution thread is blocked (cooperative cancellation not observed).";
|
||
logger.LogError(msg);
|
||
_ = siteEventLogger?.LogEventAsync("script", "Error", instanceName,
|
||
$"ScriptActor:{scriptName}", msg);
|
||
}
|
||
}));
|
||
```
|
||
|
||
- Note: `Register` callback fires on cancellation only; normal completion disposes the CTS with no callback. `Task.Run`/`Task.Delay` run on the thread pool, not the (possibly saturated) script scheduler — deliberate.
|
||
4. Run → **PASS**; run `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~ExecutionActorTests"`.
|
||
5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/ScriptExecutionActor.cs src/ZB.MOM.WW.ScadaBridge.SiteRuntime/SiteRuntimeOptions.cs tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/ExecutionActorTests.cs && git commit -m "feat(site-runtime): stuck-script watchdog names the script holding a scheduler thread past timeout (S2)"`
|
||
|
||
### Task 9: ScriptActor — trigger-expression evaluation off the dispatcher (coalesced, actor-confined edge state)
|
||
|
||
**Classification:** high-risk (actor concurrency, trigger semantics)
|
||
**Estimated implement time:** ~5 min
|
||
**Parallelizable with:** 1, 2, 3, 5, 10, 12, 13, 19, 22, 24
|
||
**Files:**
|
||
- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/ScriptActor.cs` (~lines 255–307)
|
||
- Test: `tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/ScriptActorTests.cs`
|
||
|
||
1. Failing test (expression that only returns true on a scheduler thread — proves evaluation left the dispatcher; reuse the ScriptActorTests harness that builds an Expression-triggered ScriptActor with a compiled trigger expression and a script body observable from the test):
|
||
|
||
```csharp
|
||
[Fact]
|
||
public void ExpressionTrigger_EvaluatesOnScriptSchedulerThread_AndStillFires()
|
||
{
|
||
// Expression is TRUE only when evaluated on a script-execution thread.
|
||
var expr = CompileTriggerExpression(
|
||
"System.Threading.Thread.CurrentThread.Name != null && " +
|
||
"System.Threading.Thread.CurrentThread.Name.StartsWith(\"script-execution-\")");
|
||
var fired = CreateExpressionScriptActorWithObservableBody(expr); // harness helper returning a signal
|
||
|
||
_scriptActor.Tell(new AttributeValueChanged("inst", "A", "A", 1, "Good", DateTimeOffset.UtcNow));
|
||
|
||
AwaitAssert(() => Assert.True(fired.IsSet), TimeSpan.FromSeconds(10));
|
||
}
|
||
|
||
[Fact]
|
||
public void ExpressionTrigger_OnTrue_StillFiresOncePerFalseTrueEdge()
|
||
{
|
||
// Regression: async evaluation must preserve edge semantics.
|
||
// expr: (int)Attributes["A"] > 5, OnTrue mode.
|
||
SendChange("A", 1); SendChange("A", 10); SendChange("A", 11); // true stays true
|
||
AwaitAssert(() => Assert.Equal(1, ExecutionCount()), TimeSpan.FromSeconds(10));
|
||
SendChange("A", 1); SendChange("A", 9);
|
||
AwaitAssert(() => Assert.Equal(2, ExecutionCount()), TimeSpan.FromSeconds(10));
|
||
}
|
||
```
|
||
|
||
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~ExpressionTrigger_EvaluatesOnScriptSchedulerThread"` → **FAIL** (evaluation currently happens synchronously on the dispatcher thread, whose name is not `script-execution-*`).
|
||
3. Implement:
|
||
- New internal message: `private sealed record ExpressionEvalResult(bool Result);` + `Receive<ExpressionEvalResult>(HandleExpressionEvalResult);` in the ctor.
|
||
- Fields: `private bool _evalInFlight; private bool _evalPending;`
|
||
- Replace the body of `EvaluateExpressionTrigger()` with `StartExpressionEvaluation()`:
|
||
|
||
```csharp
|
||
private void StartExpressionEvaluation()
|
||
{
|
||
if (_compiledTriggerExpression == null || _triggerConfig is not ExpressionTriggerConfig) return;
|
||
if (_evalInFlight) { _evalPending = true; return; } // coalesce bursts: one in flight, one pending
|
||
_evalInFlight = true;
|
||
|
||
var snapshot = new Dictionary<string, object?>(_attributeSnapshot); // point-in-time copy, actor thread
|
||
var expression = _compiledTriggerExpression;
|
||
var self = Self;
|
||
Task.Factory.StartNew(async () =>
|
||
{
|
||
try
|
||
{
|
||
using var cts = new CancellationTokenSource(TimeSpan.FromSeconds(2));
|
||
var state = await expression.RunAsync(new TriggerExpressionGlobals(snapshot), cancellationToken: cts.Token);
|
||
return state.ReturnValue is bool b && b;
|
||
}
|
||
catch (Exception ex) { LogExpressionError(ex); return false; }
|
||
}, CancellationToken.None, TaskCreationOptions.DenyChildAttach,
|
||
ScriptExecutionScheduler.Shared(_options)).Unwrap().PipeTo(self,
|
||
success: r => new ExpressionEvalResult(r));
|
||
}
|
||
|
||
private void HandleExpressionEvalResult(ExpressionEvalResult msg)
|
||
{
|
||
_evalInFlight = false;
|
||
if (_triggerConfig is ExpressionTriggerConfig exprConfig)
|
||
{
|
||
if (exprConfig.Mode == TriggerMode.WhileTrue) HandleWhileTrueTransition(msg.Result, _lastExpressionResult);
|
||
else if (msg.Result && !_lastExpressionResult) TrySpawnExecution(null);
|
||
_lastExpressionResult = msg.Result;
|
||
}
|
||
if (_evalPending) { _evalPending = false; StartExpressionEvaluation(); }
|
||
}
|
||
```
|
||
|
||
- Edge state (`_lastExpressionResult`, timers) is touched only on the actor thread — the concurrency contract holds. `LogExpressionError` is called from the background task: it only touches `_healthCollector`/`_logger`/`_serviceProvider` (thread-safe singletons) — verify, and if in doubt route the error through the result message instead.
|
||
4. Run both new tests + `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~ScriptActorTests"` → **PASS** (existing expression tests must survive; they may need `AwaitAssert` in place of synchronous asserts since evaluation is now async).
|
||
5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/ScriptActor.cs tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/ScriptActorTests.cs && git commit -m "perf(site-runtime): ScriptActor trigger expressions evaluate on the script scheduler, coalesced, edge state actor-confined (P1)"`
|
||
|
||
### Task 10: AlarmActor — same async trigger-expression evaluation
|
||
|
||
**Classification:** high-risk (actor concurrency, alarm state machine)
|
||
**Estimated implement time:** ~5 min
|
||
**Parallelizable with:** 1, 2, 3, 5, 9, 12, 13, 19, 22, 24
|
||
**Files:**
|
||
- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/AlarmActor.cs` (~lines 210–240, 491–519)
|
||
- Test: `tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/AlarmActorTests.cs`
|
||
|
||
1. Failing test (mirror Task 9's thread-name expression trick with an Expression-triggered alarm; assert the alarm transitions to Active via the existing `AlarmStateChanged` parent-probe pattern in AlarmActorTests):
|
||
|
||
```csharp
|
||
[Fact]
|
||
public void ExpressionAlarm_EvaluatesOnSchedulerThread_AndActivates()
|
||
{
|
||
var expr = CompileTriggerExpression(
|
||
"System.Threading.Thread.CurrentThread.Name != null && " +
|
||
"System.Threading.Thread.CurrentThread.Name.StartsWith(\"script-execution-\")");
|
||
var alarm = CreateExpressionAlarmActor(expr, parentProbe); // existing harness
|
||
|
||
alarm.Tell(new AttributeValueChanged("inst", "A", "A", 1, "Good", DateTimeOffset.UtcNow));
|
||
|
||
var change = parentProbe.ExpectMsg<AlarmStateChanged>(TimeSpan.FromSeconds(10));
|
||
Assert.Equal(AlarmState.Active, change.State);
|
||
}
|
||
```
|
||
|
||
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~ExpressionAlarm_EvaluatesOnSchedulerThread"` → **FAIL**.
|
||
3. Implement, structurally identical to Task 9:
|
||
- Extract the existing Normal↔Active transition block in `HandleAttributeValueChanged` (the `isTriggered && _currentState == AlarmState.Normal` / clear branches) into `private void ApplyTriggeredState(bool isTriggered)`.
|
||
- In `HandleAttributeValueChanged`, the `AlarmTriggerType.Expression` arm no longer computes `EvaluateExpression()` inline — it calls `StartExpressionEvaluation()` (in-flight + pending coalescing, snapshot copy, run on `ScriptExecutionScheduler.Shared(_options)`, `PipeTo` self as `ExpressionEvalResult`). `Receive<ExpressionEvalResult>` → `ApplyTriggeredState(msg.Result)` + drain pending.
|
||
- Non-expression trigger types keep their synchronous path untouched (they are cheap numeric comparisons).
|
||
- Delete the now-unused synchronous `EvaluateExpression()` (or keep as the body invoked inside the task).
|
||
4. Run new test + `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~AlarmActorTests"` → **PASS** (existing expression-alarm tests may need `AwaitAssert`).
|
||
5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/AlarmActor.cs tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/AlarmActorTests.cs && git commit -m "perf(site-runtime): AlarmActor trigger expressions evaluate on the script scheduler (P1)"`
|
||
|
||
### Task 11: InstanceActor — retry failed/lost tag subscriptions per connection
|
||
|
||
**Classification:** high-risk (actor timers, DCL handshake)
|
||
**Estimated implement time:** ~5 min
|
||
**Parallelizable with:** 1, 2, 4, 9, 10, 12, 13, 19, 22, 24
|
||
**Files:**
|
||
- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/InstanceActor.cs` (line 207 no-op handler; `SubscribeToDcl` ~962–1010)
|
||
- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/SiteRuntimeOptions.cs` (new `TagSubscribeRetryIntervalMs`, default 5000)
|
||
- Test: `tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/InstanceActorTests.cs`
|
||
|
||
1. Failing tests (TestKit; the InstanceActor harness passes a `TestProbe` as `dclManager`):
|
||
|
||
```csharp
|
||
[Fact]
|
||
public void FailedSubscribeResponse_IsRetried()
|
||
{
|
||
var options = new SiteRuntimeOptions { TagSubscribeRetryIntervalMs = 200 };
|
||
var (instance, dclProbe) = CreateInstanceWithDataSourcedAttribute(options); // existing harness
|
||
|
||
var first = dclProbe.ExpectMsg<SubscribeTagsRequest>();
|
||
instance.Tell(new SubscribeTagsResponse(first.CorrelationId, first.InstanceUniqueName,
|
||
false, "Unknown connection: conn-1", DateTimeOffset.UtcNow), dclProbe.Ref);
|
||
|
||
var retry = dclProbe.ExpectMsg<SubscribeTagsRequest>(TimeSpan.FromSeconds(5)); // S4 core
|
||
Assert.Equal(first.ConnectionName, retry.ConnectionName);
|
||
Assert.Equal(first.TagPaths, retry.TagPaths);
|
||
}
|
||
|
||
[Fact]
|
||
public void LostSubscribeResponse_IsRetried_AndSuccessStopsRetrying()
|
||
{
|
||
var options = new SiteRuntimeOptions { TagSubscribeRetryIntervalMs = 200 };
|
||
var (instance, dclProbe) = CreateInstanceWithDataSourcedAttribute(options);
|
||
|
||
var first = dclProbe.ExpectMsg<SubscribeTagsRequest>();
|
||
// No reply at all (dead-lettered response) -> must re-send.
|
||
var retry = dclProbe.ExpectMsg<SubscribeTagsRequest>(TimeSpan.FromSeconds(5));
|
||
|
||
instance.Tell(new SubscribeTagsResponse(retry.CorrelationId, retry.InstanceUniqueName,
|
||
true, null, DateTimeOffset.UtcNow), dclProbe.Ref);
|
||
dclProbe.ExpectNoMsg(TimeSpan.FromMilliseconds(600)); // retry timer cancelled
|
||
}
|
||
```
|
||
|
||
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~FailedSubscribeResponse_IsRetried|FullyQualifiedName~LostSubscribeResponse_IsRetried"` → **FAIL** (response is a no-op; no re-send).
|
||
3. Implement:
|
||
- Refactor `SubscribeToDcl` to build the per-connection groups once into a field `Dictionary<string, List<string>> _tagsByConnection`, then call `SendTagSubscribe(connectionName)` per connection.
|
||
- `SendTagSubscribe(string conn)`: mint a correlation id, record `_pendingTagSubscribes[correlationId] = conn`, Tell the DCL manager, and (re)arm a per-connection retry: `_tagSubscribeRetryTimers[conn]?.Cancel(); _tagSubscribeRetryTimers[conn] = Context.System.Scheduler.ScheduleTellOnceCancelable(TimeSpan.FromMilliseconds(_options.TagSubscribeRetryIntervalMs), Self, new RetryTagSubscribe(conn), Self);` — arming on **send** covers both the failure reply AND the lost reply with one mechanism (mirrors `NativeAlarmActor`'s timer but closes its lost-response gap too).
|
||
- Replace `Receive<SubscribeTagsResponse>(_ => { })` with a real handler: resolve `conn` from `_pendingTagSubscribes` (remove the entry); on `Success` → cancel the connection's retry timer, `LogDebug`; on failure → `LogWarning` + fire-and-forget an `instance_lifecycle`/`connection` site event ("tag subscribe failed … will retry") and leave the armed timer to re-send. Unknown correlation id (stale retry's superseded reply) → ignore.
|
||
- `Receive<RetryTagSubscribe>(msg => SendTagSubscribe(msg.ConnectionName));` (internal record). `PostStop`: cancel all retry timers.
|
||
- Note the DCL's connection-level-failure reply (`Success=false, "connection unavailable — will re-subscribe on reconnect"`) now ALSO retriggers a subscribe — harmless: the DCL treats already-subscribed tags as `AlreadySubscribed` and its `_subscriptionsByInstance` update is idempotent (verified `HandleSubscribe`/`HandleSubscribeCompleted`).
|
||
4. Run → **PASS**; run `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~InstanceActor"` → green.
|
||
5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/InstanceActor.cs src/ZB.MOM.WW.ScadaBridge.SiteRuntime/SiteRuntimeOptions.cs tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/InstanceActorTests.cs && git commit -m "fix(site-runtime): InstanceActor retries failed/lost tag subscriptions per connection (S4/UA6) — closes the unknown-connection ordering race"`
|
||
|
||
### Task 12: NativeAlarmActor — retry on lost subscribe response
|
||
|
||
**Classification:** small
|
||
**Estimated implement time:** ~3 min
|
||
**Parallelizable with:** 1, 2, 3, 4, 5, 9, 10, 11, 13, 19, 22, 24
|
||
**Files:**
|
||
- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/NativeAlarmActor.cs` (`SendSubscribe` ~123–132, `HandleSubscribeResponse` ~269–286)
|
||
- Test: `tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/NativeAlarmActorTests.cs`
|
||
|
||
1. Failing test:
|
||
|
||
```csharp
|
||
[Fact]
|
||
public void LostSubscribeResponse_ResendsSubscribe()
|
||
{
|
||
var options = new SiteRuntimeOptions { NativeAlarmRetryIntervalMs = 200 };
|
||
var (actor, dclProbe) = CreateNativeAlarmActor(options); // existing harness
|
||
|
||
dclProbe.ExpectMsg<SubscribeAlarmsRequest>();
|
||
// Swallow the response entirely — today the actor never retries.
|
||
dclProbe.ExpectMsg<SubscribeAlarmsRequest>(TimeSpan.FromSeconds(5));
|
||
}
|
||
```
|
||
|
||
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~LostSubscribeResponse_ResendsSubscribe"` → **FAIL**.
|
||
3. Implement: in `SendSubscribe()`, after the Tell, arm the response-timeout retry using the existing `_retryTimer` field: `_retryTimer?.Cancel(); _retryTimer = Context.System.Scheduler.ScheduleTellOnceCancelable(TimeSpan.FromMilliseconds(_options.NativeAlarmRetryIntervalMs * 2), Self, RetrySubscribe.Instance, Self);`. In `HandleSubscribeResponse` success branch, add `_retryTimer?.Cancel();` before returning. The failure branch already re-arms (keep it — it re-arms at 1× interval, which now simply overrides the 2× response-timeout arm).
|
||
4. Run → **PASS**; run `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~NativeAlarmActorTests"`.
|
||
5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/NativeAlarmActor.cs tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/NativeAlarmActorTests.cs && git commit -m "fix(site-runtime): NativeAlarmActor retries a lost subscribe response, not only a failed one (S4)"`
|
||
|
||
### Task 13: DeploymentManager — retry the startup config load
|
||
|
||
**Classification:** standard
|
||
**Estimated implement time:** ~4 min
|
||
**Parallelizable with:** 1, 2, 3, 5, 9, 10, 11, 12, 19, 22, 24
|
||
**Files:**
|
||
- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/DeploymentManagerActor.cs` (PreStart ~240–253, `HandleStartupConfigsLoaded` ~291–297)
|
||
- Test: `tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/DeploymentManagerActorTests.cs`
|
||
|
||
1. Failing test:
|
||
|
||
```csharp
|
||
[Fact]
|
||
public void StartupLoadFailure_IsRetried_UntilSuccess()
|
||
{
|
||
var calls = 0;
|
||
_storage.GetAllDeployedConfigsAsync().Returns(_ =>
|
||
++calls == 1
|
||
? Task.FromException<List<DeployedInstance>>(new IOException("db locked"))
|
||
: Task.FromResult(new List<DeployedInstance>
|
||
{ NewDeployedInstance("inst-1", ValidConfigJson()) })); // harness helpers
|
||
|
||
var dm = CreateDeploymentManager(); // retry interval overridable to ~200ms for the test
|
||
|
||
AwaitAssert(() =>
|
||
{
|
||
Assert.True(calls >= 2);
|
||
Sys.ActorSelection(dm.Path / "inst-1").Tell(new Identify(1));
|
||
Assert.NotNull(ExpectMsg<ActorIdentity>().Subject);
|
||
}, TimeSpan.FromSeconds(10));
|
||
}
|
||
```
|
||
|
||
(If `_storage` is the concrete `SiteStorageService` in the harness rather than a substitute, follow the harness's existing seam — several DM tests already fake persistence failures for `DeployPersistenceResult`; reuse that approach.)
|
||
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~StartupLoadFailure_IsRetried"` → **FAIL** (one attempt, then dead).
|
||
3. Implement:
|
||
- Extract the PreStart load block into `private void LoadDeployedConfigs()` and call it from `PreStart`.
|
||
- In `HandleStartupConfigsLoaded`, replace the error `return;` with: log the error, then `Timers.StartSingleTimer("startup-load-retry", RetryStartupLoad.Instance, StartupLoadRetryInterval);` (private static readonly `TimeSpan StartupLoadRetryInterval = TimeSpan.FromSeconds(5)` — internal-settable or ctor-overridable for the test, matching how other DM test knobs are exposed). Fixed-interval, unlimited — consistent with the system's fixed-interval retry philosophy; each failure logs.
|
||
- `Receive<RetryStartupLoad>(_ => LoadDeployedConfigs());` (internal singleton message). Guard: if configs already loaded (`_deployedInstanceNames.Count > 0` after a successful load — add a `_startupLoadCompleted` bool set in the success path), ignore stale retries.
|
||
4. Run → **PASS**; run the DM suite.
|
||
5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/DeploymentManagerActor.cs tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/DeploymentManagerActorTests.cs && git commit -m "fix(site-runtime): retry the startup deployed-config load on SQLite failure (S5) — no more silent zero-instance node"`
|
||
|
||
### Task 14: Watch all Instance Actors; clean the map on unexpected termination
|
||
|
||
**Classification:** high-risk (supervision/termination flow interacts with redeploy buffering)
|
||
**Estimated implement time:** ~5 min
|
||
**Parallelizable with:** 1, 2, 5, 9, 10, 12, 19, 22, 24
|
||
**Files:**
|
||
- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/DeploymentManagerActor.cs` (`CreateInstanceActor` ~1605–1632, `HandleTerminated`)
|
||
- Test: `tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/DeploymentManagerActorTests.cs`
|
||
|
||
1. Failing test:
|
||
|
||
```csharp
|
||
[Fact]
|
||
public void InstanceActorInitFailure_RemovesDeadRefFromMap()
|
||
{
|
||
var poisonConfig = "{ not json"; // InstanceActor ctor JsonSerializer.Deserialize throws
|
||
var dm = CreateDeploymentManager();
|
||
var healthProbe = _healthCollector; // fake collector already captured by the harness
|
||
|
||
dm.Tell(new DeployInstanceCommand(Guid.NewGuid().ToString(), "poison", poisonConfig, "r1" /*…*/));
|
||
|
||
// After the child Stops (decider: ActorInitializationException -> Stop), the map
|
||
// must not hold a dead ref: instance counts must return to 0 deployed.
|
||
AwaitAssert(() => Assert.Equal(0, healthProbe.LastDeployedCount), TimeSpan.FromSeconds(10));
|
||
}
|
||
```
|
||
|
||
**Note:** with Task 4 in place, `"{ not json"` is rejected by the compile validator (deserialize error) before an actor is created — so drive this test through a config that *deserializes and validates* but makes the ctor throw (e.g. a config whose `Attributes` list contains an entry that trips `DecodeAttributeValue`'s non-guarded path, or temporarily bypass via the startup path: seed storage with the poison config and let startup create the actor). Prefer the **startup path** (storage returns the poison row): it does not pass through the deploy validation gate and is exactly the "dead ref" scenario S6 describes.
|
||
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~InstanceActorInitFailure_RemovesDeadRefFromMap"` → **FAIL** (map keeps the dead ref; counts stay 1).
|
||
3. Implement:
|
||
- `CreateInstanceActor`: after `Context.ActorOf`, add `Context.Watch(actorRef);`.
|
||
- `HandleTerminated(Terminated t)`: keep the existing redeploy-buffer logic FIRST (it matches on its own bookkeeping). Add a fallback branch at the end: if `_instanceActors.TryGetValue(t.ActorRef.Path.Name, out var mapped) && mapped.Equals(t.ActorRef)` → remove from `_instanceActors`, `UpdateInstanceCounts()`, `LogError` + fire-and-forget a `deployment` site event (`"Instance {name} terminated unexpectedly (init failure or crash) — removed from routing"`). Guard: expected stops (disable/delete/redeploy) remove or replace the map entry *before* the child stops, so `mapped.Equals(t.ActorRef)` is false for them and the fallback is inert — verify against `HandleDisable`/`HandleDelete`/the redeploy shadow map and adjust ordering only if a path removes after stop.
|
||
- Redeploy path already `Context.Watch`es; double-watch is idempotent in Akka.NET (single Terminated), but confirm the redeploy handler consumes its Terminated *before* the fallback branch runs (order the if/else so the buffered-redeploy match wins).
|
||
4. Run → **PASS**; run `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~DeploymentManager"` (the redeploy race suites are the regression canary here — all must stay green).
|
||
5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/DeploymentManagerActor.cs tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/DeploymentManagerActorTests.cs && git commit -m "fix(site-runtime): watch every Instance Actor; unexpected termination evicts the dead ref from routing (S6)"`
|
||
|
||
### Task 15: Fail the in-flight deployment when its Instance Actor dies during init
|
||
|
||
**Classification:** high-risk (deployment status protocol)
|
||
**Estimated implement time:** ~5 min
|
||
**Parallelizable with:** 1, 2, 5, 9, 10, 12, 19, 22, 24
|
||
**Files:**
|
||
- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/DeploymentManagerActor.cs` (`ApplyDeployment` ~538–587, `HandleDeployPersistenceResult` ~596–636, `HandleTerminated`)
|
||
- Test: `tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/DeploymentManagerActorTests.cs`
|
||
|
||
1. Failing test:
|
||
|
||
```csharp
|
||
[Fact]
|
||
public void Deploy_WhoseActorDiesDuringInit_ReportsFailed_NotSuccess()
|
||
{
|
||
// Config that passes compile validation but whose InstanceActor ctor throws —
|
||
// reuse/extend the poison-config helper from Task 14 (ctor-throwing but valid JSON),
|
||
// or a config helper stub if none exists (e.g. attribute row that trips ctor indexing).
|
||
var dm = CreateDeploymentManager();
|
||
dm.Tell(new DeployInstanceCommand(Guid.NewGuid().ToString(), "dies-on-init", CtorThrowingConfig(), "r1" /*…*/));
|
||
|
||
var resp = ExpectMsg<DeploymentStatusResponse>(TimeSpan.FromSeconds(10));
|
||
Assert.Equal(DeploymentStatus.Failed, resp.Status);
|
||
}
|
||
```
|
||
|
||
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~Deploy_WhoseActorDiesDuringInit"` → **FAIL** (Success is reported once SQLite persists).
|
||
3. Implement:
|
||
- Field: `private readonly Dictionary<string, DeployPersistenceResult?> _inFlightDeployments = new();` keyed by instance name — store a small record `InFlightDeploy(string DeploymentId, IActorRef ReplyTo)` instead (clearer): `private readonly Dictionary<string, InFlightDeploy> _inFlightDeployments = new();`
|
||
- `ApplyDeployment`: before spawning the persistence task, `_inFlightDeployments[instanceName] = new InFlightDeploy(command.DeploymentId, sender);`
|
||
- Task 14's unexpected-`Terminated` fallback branch: after evicting the map entry, if `_inFlightDeployments.Remove(name, out var inflight)` → `inflight.ReplyTo.Tell(new DeploymentStatusResponse(inflight.DeploymentId, name, DeploymentStatus.Failed, "Instance actor failed during initialization — see site event log", DateTimeOffset.UtcNow));` and remove the optimistically-persisted row (`Task.Run(() => _storage.RemoveDeployedConfigAsync(name))` fire-and-forget with fault log) + `_deployedInstanceNames.Remove(name)` + `UpdateInstanceCounts()`.
|
||
- `HandleDeployPersistenceResult`: only reply if `_inFlightDeployments.Remove(result.InstanceName)` returns true (both success and failure branches) — if the Terminated path already failed the deployment, swallow the late persistence Success (log Debug). Both orders (Terminated-then-persistence, persistence-then-Terminated) now produce exactly one reply.
|
||
4. Run → **PASS**; re-run the full DM suite plus Task 4's test (the validation-gate reply path must not double-reply): `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~DeploymentManager"`.
|
||
5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/DeploymentManagerActor.cs tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/DeploymentManagerActorTests.cs && git commit -m "fix(site-runtime): deployment whose Instance Actor dies during init reports Failed, exactly-once reply (S6)"`
|
||
|
||
### Task 16: DataConnectionActor — capture `_adapter` into locals before every background task
|
||
|
||
**Classification:** high-risk (actor-state confinement)
|
||
**Estimated implement time:** ~4 min
|
||
**Parallelizable with:** 3, 5, 9, 10, 11, 12, 13, 19, 21, 22, 24
|
||
**Files:**
|
||
- Modify: `src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Actors/DataConnectionActor.cs` (`HandleSubscribe` ~707–758, `HandleWriteBatch` ~1155–1202, `HandleWrite`, `ReSubscribeAll` ~1594–1605; audit every `Task.Run`/local-async-function for `_adapter` reads)
|
||
- Test: `tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests/DataConnectionActorTests.cs`
|
||
|
||
1. Failing test (deterministic via a gated adapter — the in-flight subscribe must finish against the adapter that STARTED it, even after a failover swaps `_adapter`):
|
||
|
||
```csharp
|
||
[Fact]
|
||
public async Task InFlightSubscribe_ContinuesOnOriginalAdapter_AfterFailover()
|
||
{
|
||
var subscribeGate = new TaskCompletionSource<string>();
|
||
var primaryAdapter = Substitute.For<IDataConnection>();
|
||
primaryAdapter.ConnectAsync(default!).ReturnsForAnyArgs(Task.CompletedTask);
|
||
primaryAdapter.SubscribeAsync(default!, default!).ReturnsForAnyArgs(
|
||
_ => subscribeGate.Task, // tag 1 blocks mid-flight
|
||
_ => Task.FromResult("sub-2")); // tag 2 resumes after failover
|
||
var backupAdapter = Substitute.For<IDataConnection>();
|
||
var factory = Substitute.For<IDataConnectionFactory>();
|
||
factory.Create(default!, default!).ReturnsForAnyArgs(backupAdapter);
|
||
|
||
var actor = CreateConnectedActor(primaryAdapter, factory, backupConfig: SomeBackupConfig()); // harness
|
||
actor.Tell(new SubscribeTagsRequest(NewCorrelation(), "inst", "conn", new[] { "t1", "t2" }, Now()));
|
||
|
||
TriggerFailoverToBackup(actor); // drive AdapterDisconnected + failed reconnects past retry count
|
||
|
||
subscribeGate.SetResult("sub-1"); // release the in-flight background loop
|
||
AwaitAssert(async () =>
|
||
{
|
||
await primaryAdapter.ReceivedWithAnyArgs(2).SubscribeAsync(default!, default!);
|
||
await backupAdapter.DidNotReceiveWithAnyArgs().SubscribeAsync(default!, default!);
|
||
});
|
||
}
|
||
```
|
||
|
||
(Build on the existing `DataConnectionActorTests` failover harness — it already drives `CountFailureAndMaybeFailover` via consecutive `ConnectResult` failures.)
|
||
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests --filter "FullyQualifiedName~InFlightSubscribe_ContinuesOnOriginalAdapter"` → **FAIL** (later loop iterations read the swapped `_adapter` field and hit the backup with the old generation).
|
||
3. Implement — one pattern, four sites:
|
||
- `HandleSubscribe`: `var adapter = _adapter;` on the actor thread before `Task.Run`; use `adapter.SubscribeAsync(...)` and `SeedTagsAsync(adapter, tagsToSeed)` inside the task. Update the confinement comment at ~689 to note the adapter is captured too.
|
||
- `HandleWriteBatch`: `var adapter = _adapter;` before `RunAsync()`; use `adapter.WriteBatchAsync` / `adapter.WriteAsync` (the second read currently happens after the first `await`, off-thread).
|
||
- `HandleWrite`: same audit/fix (the single-write path has the same local-async-function shape).
|
||
- `ReSubscribeAll`: the per-tag `SubscribeAsync` loop runs on the actor thread (safe), but capture `var subscribeAdapter = _adapter;` and use it in the loop anyway for symmetry with `reseedAdapter` — zero cost, removes the last field read reachable from this method's continuations.
|
||
4. Run → **PASS**; full DCL actor suite: `dotnet test tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests --filter "FullyQualifiedName~DataConnectionActor"`.
|
||
5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Actors/DataConnectionActor.cs tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests/DataConnectionActorTests.cs && git commit -m "fix(dcl): capture adapter into locals before background tasks — no off-thread reads of the mutable _adapter field (S7)"`
|
||
|
||
### Task 17: DCL — tag→instances reverse index for the value fan-out hot path
|
||
|
||
**Classification:** high-risk (hottest DCL loop, index-consistency invariants)
|
||
**Estimated implement time:** ~5 min
|
||
**Parallelizable with:** 5, 9, 10, 11, 12, 13, 19, 21, 22, 24
|
||
**Files:**
|
||
- Modify: `src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Actors/DataConnectionActor.cs` (`HandleTagValueReceived` ~1693–1738; every `_subscriptionsByInstance` mutation site)
|
||
- Test: `tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests/DataConnectionActorTests.cs`
|
||
|
||
1. Failing test (behavioral — index correctness across unsubscribe, which a naive index gets wrong):
|
||
|
||
```csharp
|
||
[Fact]
|
||
public void TagFanOut_AfterOneInstanceUnsubscribes_OnlyRemainingSubscriberReceives()
|
||
{
|
||
var actor = CreateConnectedActorWithInstantSubscribes(); // adapter substitute returns sub ids immediately
|
||
var probeA = CreateTestProbe(); var probeB = CreateTestProbe();
|
||
|
||
actor.Tell(SubscribeRequest("inst-A", new[] { "shared-tag" }), probeA.Ref);
|
||
actor.Tell(SubscribeRequest("inst-B", new[] { "shared-tag" }), probeB.Ref);
|
||
probeA.ExpectMsg<SubscribeTagsResponse>(); probeB.ExpectMsg<SubscribeTagsResponse>();
|
||
probeA.FishForMessage(m => m is TagValueUpdate, hint: "seed"); // drain seeds as the harness does
|
||
|
||
actor.Tell(UnsubscribeRequest("inst-A", new[] { "shared-tag" }), probeA.Ref);
|
||
DeliverTagValue(actor, "shared-tag", 42); // helper: Tell TagValueReceived with current generation
|
||
|
||
probeB.ExpectMsg<TagValueUpdate>(u => u.TagPath == "shared-tag");
|
||
probeA.ExpectNoMsg(TimeSpan.FromMilliseconds(300));
|
||
}
|
||
```
|
||
|
||
This passes today via the O(N) scan — write it FIRST and confirm it passes (pin the behavior), then convert `HandleTagValueReceived` to the index and confirm it STILL passes. Add one more assertion-style test that also passes pre-change (two instances, disjoint tags, each receives only its own) — these are the regression net for the index bookkeeping.
|
||
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests --filter "FullyQualifiedName~TagFanOut_"` → expect **PASS** pre-change (documented pin), then proceed.
|
||
3. Implement:
|
||
- Field: `private readonly Dictionary<string, HashSet<string>> _instancesByTag = new();` with XML doc mirroring `_tagSubscriberCount`'s rationale.
|
||
- Add/remove alongside **every** `_subscriptionsByInstance[inst]` tag-set mutation. Grep `_subscriptionsByInstance` and cover: `HandleSubscribeCompleted` (tags added), `HandleUnsubscribe` (tags removed + instance entry removal), the orphan-release guards (~836–985, 1795–1858 region), and any instance-removal path. Write two private helpers `IndexTag(string tag, string inst)` / `UnindexTag(string tag, string inst)` (removing the tag key when its set empties) and use them at every site — no inline mutation.
|
||
- `HandleTagValueReceived`: replace the `_subscriptionsByInstance` scan with `if (_instancesByTag.TryGetValue(msg.TagPath, out var insts)) foreach (var inst in insts) if (_subscribers.TryGetValue(inst, out var s)) s.Tell(...)`.
|
||
- `ReSubscribeAll` preserves `_subscriptionsByInstance` — so it must NOT clear `_instancesByTag` (it is derived from the same preserved state). Note this in a comment.
|
||
4. Run the fan-out tests + full DCL suite → **PASS**.
|
||
5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Actors/DataConnectionActor.cs tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests/DataConnectionActorTests.cs && git commit -m "perf(dcl): tag->instances reverse index replaces per-update all-instances scan (P3)"`
|
||
|
||
### Task 18: InstanceActor — per-attribute child-subscription routing
|
||
|
||
**Classification:** high-risk (hot-path routing semantics for scripts AND alarms)
|
||
**Estimated implement time:** ~5 min
|
||
**Parallelizable with:** 1, 2, 5, 10, 12, 13, 17, 19, 22, 24
|
||
**Files:**
|
||
- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/InstanceActor.cs` (`CreateChildActors` ~1348–1488, `PublishAndNotifyChildren` ~1179–1209)
|
||
- Test: `tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/InstanceActorChildAttributeRaceTests.cs` (or a new `InstanceActorChildRoutingTests.cs`)
|
||
|
||
1. Failing test:
|
||
|
||
```csharp
|
||
[Fact]
|
||
public void AttributeChange_IsRoutedOnlyToInterestedChildren()
|
||
{
|
||
// Instance with: script S_A (ValueChange on "A"), script S_B (ValueChange on "B"),
|
||
// script S_expr (Expression trigger). Script bodies record executions via the
|
||
// static-recording pattern the existing tests use.
|
||
var instance = CreateInstance(ScriptsAB_and_Expression());
|
||
|
||
SetStatic(instance, "A", 1); // publishes AttributeValueChanged for A
|
||
|
||
AwaitAssert(() =>
|
||
{
|
||
Assert.Equal(1, Executions("S_A"));
|
||
Assert.Equal(0, Executions("S_B")); // FAILS pre-change only via mailbox counting…
|
||
});
|
||
}
|
||
```
|
||
|
||
`S_B` never *executes* today either (its own trigger check discards) — so assert on **message delivery**, not execution: give the routing seam a test hook. Concretely: make the routing maps `internal` and unit-test map construction directly (`InstanceActor` exposes `internal IReadOnlyDictionary<string, List<IActorRef>> ChildrenByMonitoredAttribute` / `internal IReadOnlyList<IActorRef> AllChangeChildren` next to the existing `ScriptActorCount` diagnostics), plus a behavioral test that the Expression script still fires on ANY attribute change and the ValueChange script still fires on its own attribute:
|
||
|
||
```csharp
|
||
[Fact]
|
||
public void RoutingMaps_AreBuiltFromTriggerConfigs()
|
||
{
|
||
var actor = CreateInstanceActorRef(ScriptsAB_and_Expression());
|
||
var underlying = GetUnderlyingActor(actor); // TestActorRef pattern used elsewhere in this file
|
||
Assert.Single(underlying.ChildrenByMonitoredAttribute["A"]);
|
||
Assert.Single(underlying.ChildrenByMonitoredAttribute["B"]);
|
||
Assert.Single(underlying.AllChangeChildren); // the Expression script
|
||
}
|
||
```
|
||
|
||
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~RoutingMaps_AreBuiltFromTriggerConfigs"` → **FAIL**.
|
||
3. Implement:
|
||
- Fields: `private readonly Dictionary<string, List<IActorRef>> _childrenByMonitoredAttribute = new(); private readonly List<IActorRef> _allChangeChildren = new();`
|
||
- In `CreateChildActors`, after creating each script/alarm actor, classify:
|
||
- Script trigger `valuechange`/`conditional` → parse `attributeName` from `TriggerConfiguration` JSON (reuse/extract the parse `ScriptActor.ParseTriggerConfig` does — lift a tiny static helper `TriggerRouting.TryGetMonitoredAttribute(triggerType, triggerConfigJson, out string attr)` into `Scripts/` so ScriptActor and InstanceActor share one parser). Add to `_childrenByMonitoredAttribute[attr]`.
|
||
- Script trigger `expression` → `_allChangeChildren` (its snapshot must stay current).
|
||
- Script trigger `interval`/`call`/unparseable → **`_allChangeChildren` if unparseable (fail-open, log warning), else no routing** (interval/call scripts never react to attribute changes; their `_attributeSnapshot` is only read by expression evaluation, which they don't have — verify this before excluding, it is true today at `ScriptActor` where the snapshot is read only in `EvaluateExpressionTrigger`).
|
||
- Alarms: same classification via the alarm's trigger type — Expression → all bucket; HiLo/ValueMatch/RangeViolation/RateOfChange → monitored attribute (parse from the alarm config the same way `AlarmActor.IsMonitoredAttribute` derives it); unparseable → all bucket.
|
||
- `PublishAndNotifyChildren`: replace the two full loops with `foreach (var c in _allChangeChildren) c.Tell(changed);` + `if (_childrenByMonitoredAttribute.TryGetValue(changed.AttributeName, out var interested)) foreach (var c in interested) c.Tell(changed);`. Children keep their own trigger gates (defense in depth — routing is an optimization, not the correctness gate).
|
||
- Note: `changed.AttributeName` on the DCL path is the canonical name (`HandleTagValueUpdate` maps tag→attributes before publishing) — key the map by canonical name; confirm against `HandleTagValueUpdate`.
|
||
4. Run new tests + the whole InstanceActor/ScriptActor/AlarmActor suites: `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~InstanceActor|FullyQualifiedName~ScriptActor|FullyQualifiedName~AlarmActor"` → **PASS**.
|
||
5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/InstanceActor.cs src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/ScriptActor.cs src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Scripts tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests && git commit -m "perf(site-runtime): per-attribute child routing in InstanceActor — attribute changes reach only interested children (P2)"`
|
||
|
||
### Task 19: SiteStorageService — WAL journal mode + busy timeout
|
||
|
||
**Classification:** small
|
||
**Estimated implement time:** ~3 min
|
||
**Parallelizable with:** 1, 2, 3, 4, 5, 9, 10, 11, 12, 13, 22, 24
|
||
**Files:**
|
||
- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Persistence/SiteStorageService.cs` (ctor ~21–25, `InitializeAsync` ~41–132)
|
||
- Test: `tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Persistence/` (add to the existing SiteStorageService test file)
|
||
|
||
1. Failing test:
|
||
|
||
```csharp
|
||
[Fact]
|
||
public async Task Initialize_EnablesWalJournalMode()
|
||
{
|
||
var service = CreateTempFileStorageService(); // existing temp-SQLite fixture pattern
|
||
await service.InitializeAsync();
|
||
|
||
await using var conn = service.CreateConnection();
|
||
await conn.OpenAsync();
|
||
await using var cmd = conn.CreateCommand();
|
||
cmd.CommandText = "PRAGMA journal_mode;";
|
||
Assert.Equal("wal", ((string)(await cmd.ExecuteScalarAsync())!).ToLowerInvariant());
|
||
}
|
||
```
|
||
|
||
Note: WAL requires a file-backed DB — if the existing fixture uses `:memory:`, use the temp-file variant (several persistence tests already do).
|
||
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~Initialize_EnablesWalJournalMode"` → **FAIL** (`delete`).
|
||
3. Implement:
|
||
- In `InitializeAsync`, before creating tables: execute `PRAGMA journal_mode=WAL;` (persistent, database-level — one-time). Log the resulting mode; `:memory:`/unsupported filesystems fall back gracefully (don't throw if the pragma returns non-wal, just warn).
|
||
- In the ctor, normalize the connection string through `SqliteConnectionStringBuilder` and ensure `DefaultTimeout` is at least 5 s (Microsoft.Data.Sqlite uses the command timeout as its busy handler — an explicit floor documents intent even though the library default is 30): only raise it if the caller set something lower. Add an XML remark on `CreateConnection()` documenting WAL + busy-handling so the repositories layered on it inherit the story.
|
||
4. Run → **PASS**; run the persistence folder suite: `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~SiteStorageService"`.
|
||
5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Persistence/SiteStorageService.cs tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Persistence && git commit -m "fix(site-runtime): site SQLite runs WAL with an explicit busy-timeout floor (S8) — concurrent writers stop serializing on the rollback journal"`
|
||
|
||
### Task 20: NativeAlarmActor — coalesce per-transition SQLite upserts
|
||
|
||
**Classification:** standard
|
||
**Estimated implement time:** ~5 min
|
||
**Parallelizable with:** 1, 2, 4, 5, 9, 10, 11, 13, 17, 22, 24
|
||
**Files:**
|
||
- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/NativeAlarmActor.cs` (`PersistUpsert` ~420–428, timer plumbing)
|
||
- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Persistence/SiteStorageService.cs` (new `UpsertNativeAlarmsAsync` batch — one connection + transaction)
|
||
- Test: `tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/NativeAlarmActorTests.cs`
|
||
|
||
1. Failing test (against the temp-SQLite storage the existing NativeAlarmActor tests use):
|
||
|
||
```csharp
|
||
[Fact]
|
||
public async Task AlarmStorm_CoalescesPersistence_LatestPerSourceRefWins()
|
||
{
|
||
var (actor, storage) = CreateNativeAlarmActorWithTempStorage(flushIntervalMs: 100);
|
||
|
||
for (var i = 0; i < 50; i++)
|
||
DeliverLiveTransition(actor, sourceRef: $"ref-{i % 5}", severity: i); // 50 transitions, 5 refs
|
||
|
||
await Task.Delay(500); // > flush interval
|
||
var rows = await storage.GetNativeAlarmsAsync("inst", "src");
|
||
Assert.Equal(5, rows.Count);
|
||
// Latest transition per ref persisted (severity of the LAST write for ref-4 is 49):
|
||
Assert.Contains(rows, r => r.ConditionJson.Contains("49"));
|
||
}
|
||
```
|
||
|
||
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~AlarmStorm_CoalescesPersistence"` → **FAIL** only on the *mechanism* (row content passes either way) — so also assert the batch API is used: add `UpsertNativeAlarmsAsync` first (it won't exist → compile-fail = the failing signal), or count connections via a storage wrapper. Simplest honest failing signal: write the test calling `storage.UpsertNativeAlarmsAsync(...)` shape via the actor and assert rows — compile failure is the RED step.
|
||
3. Implement:
|
||
- `SiteStorageService.UpsertNativeAlarmsAsync(string instance, string source, IReadOnlyList<(string SourceRef, string ConditionJson, DateTimeOffset At)> rows)` — one connection, one transaction, one prepared command re-bound per row.
|
||
- `NativeAlarmActor`: replace direct `PersistUpsert(t)` calls (in `ApplyLiveTransition` and `ApplySnapshotSwap`) with `_dirtyUpserts[t.SourceReference] = t;` + arm a 100 ms single-shot flush timer if not armed (`ScheduleTellOnceCancelable`, internal `FlushPersistence` message). The flush handler snapshots + clears `_dirtyUpserts`, calls the batch API fire-and-forget with the existing `OnlyOnFaulted` logging. `PersistDelete` stays immediate (rare; also remove the ref from `_dirtyUpserts` so a delete isn't resurrected by a pending upsert). `PostStop`: best-effort final flush (fire-and-forget).
|
||
- Rehydration semantics unchanged: worst case a crash loses ≤100 ms of mirror writes; the (re)subscribe snapshot swap reconciles — note this in the class doc.
|
||
4. Run → **PASS**; full NativeAlarmActor suite green.
|
||
5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/NativeAlarmActor.cs src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Persistence/SiteStorageService.cs tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/NativeAlarmActorTests.cs && git commit -m "perf(site-runtime): coalesce native-alarm mirror persistence into batched flushes (P4)"`
|
||
|
||
### Task 21: Persist native-alarm display metadata for rehydration
|
||
|
||
**Classification:** standard (SQLite schema migration — additive column)
|
||
**Estimated implement time:** ~5 min
|
||
**Parallelizable with:** 1, 2, 4, 5, 9, 10, 11, 13, 22, 24
|
||
**Files:**
|
||
- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Persistence/SiteStorageService.cs` (`native_alarm_state` schema ~117–124, `MigrateSchemaAsync`, upsert/read methods)
|
||
- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/NativeAlarmActor.cs` (`PersistUpsert`/flush ~420–428, `HandleRehydration` ~134–174)
|
||
- Test: `tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/NativeAlarmActorTests.cs`
|
||
|
||
1. Failing test:
|
||
|
||
```csharp
|
||
[Fact]
|
||
public async Task Rehydration_RestoresDisplayMetadata()
|
||
{
|
||
var (actor1, storage) = CreateNativeAlarmActorWithTempStorage();
|
||
DeliverLiveTransition(actor1, sourceRef: "ref-1",
|
||
alarmTypeName: "HighLevelAlarm", category: "Process", message: "Tank overflow");
|
||
await FlushAndStop(actor1);
|
||
|
||
var (actor2, parentProbe) = RecreateActorOnSameStorage(storage); // rehydrates in PreStart
|
||
var emitted = parentProbe.FishForMessage<AlarmStateChanged>(m => m.SourceReference == "ref-1");
|
||
Assert.Equal("HighLevelAlarm", emitted.AlarmTypeName); // empty string today
|
||
Assert.Equal("Process", emitted.Category);
|
||
Assert.Equal("Tank overflow", emitted.Message);
|
||
}
|
||
```
|
||
|
||
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~Rehydration_RestoresDisplayMetadata"` → **FAIL** (metadata comes back `string.Empty`).
|
||
3. Implement:
|
||
- Schema: add `metadata_json TEXT` to the `CREATE TABLE native_alarm_state` DDL AND `await TryAddColumnAsync(connection, "native_alarm_state", "metadata_json", "TEXT");` in `MigrateSchemaAsync` (existing-DB upgrade).
|
||
- Persist: serialize a small record `NativeAlarmMetadata(string AlarmTypeName, string Category, string Message, string CurrentValue, string LimitValue)` from the transition into `metadata_json` in the upsert (single + Task 20 batch). `NativeAlarmRow` gains `string? MetadataJson`.
|
||
- Rehydrate: in `HandleRehydration`, deserialize `MetadataJson` (null/malformed → empty strings, current behavior) and build the placeholder `NativeAlarmTransition` with the restored fields instead of `string.Empty`.
|
||
4. Run → **PASS**; full NativeAlarmActor + persistence suites.
|
||
5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Persistence/SiteStorageService.cs src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/NativeAlarmActor.cs tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/NativeAlarmActorTests.cs && git commit -m "feat(site-runtime): persist native-alarm display metadata so rehydrated conditions render fully (UA4)"`
|
||
|
||
### Task 22: CertStoreActor — export trusted certs (thumbprint + DER) for reconciliation
|
||
|
||
**Classification:** standard (additive Commons message contract)
|
||
**Estimated implement time:** ~4 min
|
||
**Parallelizable with:** 1, 2, 3, 4, 5, 9, 10, 11, 12, 13, 16, 17, 19, 24
|
||
**Files:**
|
||
- Modify: `src/ZB.MOM.WW.ScadaBridge.Commons/Messages/Management/` (the file holding `WriteCertToLocalStore`/`ListLocalCerts` — add `ExportLocalTrustedCerts` + `LocalCertExport`)
|
||
- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/CertStoreActor.cs` (new handler)
|
||
- Test: `tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/CertStoreActorTests.cs`
|
||
|
||
1. Failing test (reuse the temp-dir PKI fixture in `CertStoreActorTests`):
|
||
|
||
```csharp
|
||
[Fact]
|
||
public void ExportLocalTrustedCerts_ReturnsThumbprintAndDerBytes()
|
||
{
|
||
var (actor, testCertDer, thumbprint) = CreateCertStoreWithOneTrustedCert(); // existing fixture helpers
|
||
actor.Tell(new ExportLocalTrustedCerts());
|
||
var export = ExpectMsg<LocalCertExport>();
|
||
var entry = Assert.Single(export.Certs);
|
||
Assert.Equal(thumbprint, entry.Thumbprint);
|
||
Assert.Equal(testCertDer, entry.DerBytes);
|
||
}
|
||
```
|
||
|
||
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~ExportLocalTrustedCerts_ReturnsThumbprintAndDerBytes"` → **FAIL** (types don't exist).
|
||
3. Implement:
|
||
- Commons (additive, next to the existing cert messages): `public sealed record ExportLocalTrustedCerts(); public sealed record ExportedCert(string Thumbprint, byte[] DerBytes); public sealed record LocalCertExport(bool Success, string? Error, IReadOnlyList<ExportedCert> Certs);`
|
||
- `CertStoreActor`: `Receive<ExportLocalTrustedCerts>` — enumerate `*.der` in `_trustedStoreDir`, load each via `X509CertificateLoader`/`new X509Certificate2(bytes)` (match how `HandleList` reads them today) to compute the thumbprint, reply `LocalCertExport(true, null, list)`; any IO error → `LocalCertExport(false, ex.Message, [])`.
|
||
4. Run → **PASS**.
|
||
5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.Commons src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/CertStoreActor.cs tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/CertStoreActorTests.cs && git commit -m "feat(site-runtime): CertStoreActor exports trusted certs (thumbprint+DER) for node reconciliation (UA1 groundwork)"`
|
||
|
||
### Task 23: Cert-trust reconcile-on-join — singleton pushes its trusted set to a (re)joining site node
|
||
|
||
**Classification:** high-risk (cluster membership events)
|
||
**Estimated implement time:** ~5 min
|
||
**Parallelizable with:** 1, 2, 5, 9, 10, 12, 17, 19, 24
|
||
**Files:**
|
||
- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/DeploymentManagerActor.cs` (PreStart/PostStop cluster subscription; new handlers near the cert section ~900–1007)
|
||
- Test: `tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/DeploymentManagerActorTests.cs`
|
||
|
||
1. Failing test — test the reconcile flow via the internal message (constructing a real `ClusterEvent.MemberUp` in a unit test is impractical; the MemberUp handler itself is a two-liner producing this message). Use the same "forwarding actor named `cert-store` → TestProbe" trick the existing cert broadcast tests use:
|
||
|
||
```csharp
|
||
[Fact]
|
||
public void SiteNodeJoined_PushesLocalTrustedCertsToJoinedNode()
|
||
{
|
||
var certStoreProbe = InstallCertStoreForwarder(); // /user/cert-store -> probe (existing pattern)
|
||
var dm = CreateDeploymentManager();
|
||
|
||
dm.Tell(new DeploymentManagerActor.SiteNodeJoined(Cluster.Get(Sys).SelfAddress));
|
||
|
||
certStoreProbe.ExpectMsg<ExportLocalTrustedCerts>(); // 1) read own store
|
||
dm.Tell(new LocalCertExport(true, null,
|
||
new[] { new ExportedCert("ABC123", new byte[] { 1, 2, 3 }) }), certStoreProbe.Ref);
|
||
// 2) pushed to the joined node's cert-store (self address in-test -> same selection):
|
||
var write = certStoreProbe.ExpectMsg<WriteCertToLocalStore>(TimeSpan.FromSeconds(5));
|
||
Assert.Equal("ABC123", write.Thumbprint);
|
||
}
|
||
```
|
||
|
||
(Match `WriteCertToLocalStore`'s real field names/args from Commons when writing the assertion.)
|
||
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~SiteNodeJoined_PushesLocalTrustedCerts"` → **FAIL**.
|
||
3. Implement:
|
||
- `PreStart`: `Cluster.Get(Context.System).Subscribe(Self, ClusterEvent.SubscriptionInitialStateMode.InitialStateAsEvents, typeof(ClusterEvent.MemberUp));` `PostStop`: `Cluster.Get(Context.System).Unsubscribe(Self);`
|
||
- `Receive<ClusterEvent.MemberUp>`: if the member has the site role and is not the singleton's own node → `Self.Tell(new SiteNodeJoined(member.Address))` (public/internal record for testability).
|
||
- `Receive<SiteNodeJoined>`: capture the address; ask the LOCAL cert store (`Context.ActorSelection($"/user/{CertStoreActor.WellKnownName}")`) with `ExportLocalTrustedCerts` (short `CertBroadcastTimeout` Ask, `ContinueWith` → pipe a wrapper carrying the target address). On the piped `LocalCertExport` (wrap it: `CertReconcileExport(Address Target, LocalCertExport Export)`): for each cert, `Tell` the target node's cert store (`new RootActorPath(target) / "user" / CertStoreActor.WellKnownName`) a `WriteCertToLocalStore(...)` — fire-and-forget, additive union (documented: removals do NOT reconcile until central persistence lands; the write is idempotent — `CertStoreActor.HandleWrite` overwrites by thumbprint).
|
||
- Log one info line per reconcile (`"Cert-trust reconcile: pushed {N} trusted cert(s) to joining node {Address}"`); export failure → warning, no retry (the next MemberUp or manual trust re-broadcast heals).
|
||
- Simplification for the test flow: since the test drives `LocalCertExport` directly at the actor, have `Receive<CertReconcileExport>` be the handler and also accept a bare `LocalCertExport` only if you keep a single pending target field — prefer the wrapper message; adjust the test to send the wrapper if needed.
|
||
4. Run → **PASS**; full DM suite green.
|
||
5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/DeploymentManagerActor.cs tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/DeploymentManagerActorTests.cs && git commit -m "feat(site-runtime): cert-trust reconcile-on-join — singleton pushes trusted certs to (re)joining site nodes (UA1)"`
|
||
|
||
### Task 24: Wire `ConfigFetchRetryCount` into the standby replication fetch
|
||
|
||
**Classification:** small
|
||
**Estimated implement time:** ~4 min
|
||
**Parallelizable with:** 1, 2, 3, 4, 5, 9, 10, 12, 13, 16, 17, 19, 22
|
||
**Files:**
|
||
- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/SiteReplicationActor.cs` (`HandleApplyConfigDeploy` ~153–217; ctor gains `SiteRuntimeOptions? options = null`)
|
||
- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/SiteRuntimeOptions.cs` (fix the "Reserved — not yet wired" XML doc, lines 63–67)
|
||
- Modify: the `SiteReplicationActor` Props construction site (Host/ServiceCollectionExtensions — grep `new SiteReplicationActor(` / `Props.Create(() => new SiteReplicationActor`) to pass options
|
||
- Test: `tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/SiteReplicationActorTests.cs`
|
||
|
||
1. Failing test:
|
||
|
||
```csharp
|
||
[Fact]
|
||
public void ReplicatedFetch_RetriesUpToConfigFetchRetryCount()
|
||
{
|
||
var attempts = 0;
|
||
var fetcher = Substitute.For<IDeploymentConfigFetcher>();
|
||
fetcher.FetchAsync(default!, default!, default!, default).ReturnsForAnyArgs(_ =>
|
||
++attempts < 3
|
||
? Task.FromException<string>(new HttpRequestException("central hiccup"))
|
||
: Task.FromResult("{\"config\":true}"));
|
||
var actor = CreateReplicationActor(fetcher,
|
||
new SiteRuntimeOptions { ConfigFetchRetryCount = 3 }); // existing harness + new arg
|
||
|
||
actor.Tell(new ApplyConfigDeploy("inst", "dep-1", "r1", true, "http://central", "tok"));
|
||
|
||
AwaitAssert(() =>
|
||
{
|
||
Assert.Equal(3, attempts);
|
||
_storage.Received(1).StoreDeployedConfigIfNewerAsync("inst", "{\"config\":true}", "dep-1", "r1", true);
|
||
}, TimeSpan.FromSeconds(10));
|
||
}
|
||
```
|
||
|
||
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~ReplicatedFetch_RetriesUpToConfigFetchRetryCount"` → **FAIL** (single attempt).
|
||
3. Implement: replace the single `FetchAsync(...).ContinueWith(...)` with a local async function launched fire-and-forget (same best-effort model):
|
||
|
||
```csharp
|
||
var maxAttempts = Math.Max(1, _options?.ConfigFetchRetryCount ?? 1);
|
||
_ = FetchWithRetryAsync(); // errors fully handled inside; observes every fault
|
||
async Task FetchWithRetryAsync()
|
||
{
|
||
for (var attempt = 1; attempt <= maxAttempts; attempt++)
|
||
{
|
||
try
|
||
{
|
||
var json = await _configFetcher.FetchAsync(msg.CentralFetchBaseUrl, msg.DeploymentId, msg.FetchToken, CancellationToken.None);
|
||
await _storage.StoreDeployedConfigIfNewerAsync(msg.InstanceName, json, msg.DeploymentId, msg.RevisionHash, msg.IsEnabled);
|
||
return;
|
||
}
|
||
catch (DeploymentConfigFetchException fex) when (fex.IsSuperseded)
|
||
{ /* keep the existing "superseded" info log */ return; }
|
||
catch (Exception ex)
|
||
{
|
||
// keep existing per-failure log wording; add attempt counters
|
||
if (attempt < maxAttempts) await Task.Delay(TimeSpan.FromSeconds(2));
|
||
else _logger.LogError(ex, "Replicated config fetch failed after {Attempts} attempt(s) for {Instance} (deployment {DeploymentId}) — reconciliation will backstop", maxAttempts, msg.InstanceName, msg.DeploymentId);
|
||
}
|
||
}
|
||
}
|
||
```
|
||
|
||
Update the `SiteRuntimeOptions.ConfigFetchRetryCount` XML doc: remove "Reserved … not yet wired"; state "Bounded attempt count (including the first) for the standby's replicated-config fetch; 2 s fixed delay between attempts; superseded fetches never retry."
|
||
4. Run → **PASS**; run the replication suite.
|
||
5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/SiteReplicationActor.cs src/ZB.MOM.WW.ScadaBridge.SiteRuntime/SiteRuntimeOptions.cs tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/SiteReplicationActorTests.cs && git commit -m "feat(site-runtime): wire ConfigFetchRetryCount into the standby replicated-config fetch (UA2)"` (include the Props-site file in the `git add` if touched)
|
||
|
||
### Task 25: Low-severity code cleanups — S9, S10, C2, C6
|
||
|
||
**Classification:** small (batched Low findings; concrete edits below)
|
||
**Estimated implement time:** ~5 min
|
||
**Parallelizable with:** 13, 19, 21, 22, 24
|
||
**Files:**
|
||
- Modify: `src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Actors/DataConnectionActor.cs` (PostStop ~216–221; `HandleSubscribeAlarms` ~1763)
|
||
- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteEventLogging/SiteEventLogger.cs` (comment ~215–218)
|
||
- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/ScriptActor.cs` (`EvaluateCondition` fallback ~474–477)
|
||
- Test: `tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/ScriptActorTests.cs`, `tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests/DataConnectionActorAlarmTests.cs`
|
||
|
||
1. **C6 failing test** (culture-independence of the non-numeric fallback):
|
||
|
||
```csharp
|
||
[Fact]
|
||
public void ConditionalTrigger_NonNumericFallback_IsCultureInvariant()
|
||
{
|
||
var original = CultureInfo.CurrentCulture;
|
||
try
|
||
{
|
||
CultureInfo.CurrentCulture = new CultureInfo("de-DE"); // 1.5 -> "1,5"
|
||
// Conditional trigger: operator "==", Threshold = 1.5; value "1,5" fails
|
||
// invariant numeric parse -> string fallback. Old code compares against
|
||
// Threshold.ToString() == "1,5" under de-DE and FIRES; invariant must NOT.
|
||
var fired = DriveConditionalTrigger(threshold: 1.5, op: "==", value: "1,5"); // harness helper
|
||
Assert.False(fired);
|
||
}
|
||
finally { CultureInfo.CurrentCulture = original; }
|
||
}
|
||
```
|
||
|
||
2. **S10 failing test** (EventFilter on the warning):
|
||
|
||
```csharp
|
||
[Fact]
|
||
public void SecondAlarmSubscriber_WithDifferentFilter_LogsOverwriteWarning()
|
||
{
|
||
var actor = CreateConnectedAlarmCapableActor(); // existing alarm-test harness
|
||
actor.Tell(AlarmSubscribe("src-1", filter: "HighAlarm"), probeA.Ref);
|
||
EventFilter.Warning(contains: "condition filter").ExpectOne(() =>
|
||
actor.Tell(AlarmSubscribe("src-1", filter: "LowAlarm"), probeB.Ref));
|
||
}
|
||
```
|
||
|
||
3. Run both → **FAIL** (`dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~ConditionalTrigger_NonNumericFallback"` and `dotnet test tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests --filter "FullyQualifiedName~SecondAlarmSubscriber_WithDifferentFilter"`).
|
||
4. Implement all four edits:
|
||
- **C6** (`ScriptActor.cs:476`): `return string.Equals(value.ToString(), config.Threshold.ToString(CultureInfo.InvariantCulture), StringComparison.Ordinal);` — matches the invariant-culture handling two lines up.
|
||
- **S10** (`DataConnectionActor.cs`, before line 1763's overwrite): `if (_alarmSourceFilter.TryGetValue(request.SourceReference, out var existingFilter) && !string.Equals(existingFilter, request.ConditionFilter, StringComparison.Ordinal)) _log.Warning("[{0}] Alarm condition filter for source {1} overwritten: '{2}' -> '{3}' (last subscriber wins; co-subscribers must agree)", _connectionName, request.SourceReference, existingFilter, request.ConditionFilter);`
|
||
- **S9** (`DataConnectionActor.PostStop`): replace `_ = _adapter.DisposeAsync().AsTask();` with `_adapter.DisposeAsync().AsTask().ContinueWith(t => _log.Warning("[{0}] Adapter dispose faulted on stop: {1}", _connectionName, t.Exception?.GetBaseException().Message), TaskContinuationOptions.OnlyOnFaulted);`
|
||
- **C2** (`SiteEventLogger.cs:215-218`): rewrite the comment: "The channel is bounded with DropOldest, so TryWrite only returns false once the channel is completed (shutdown) — full-buffer pressure evicts the oldest pending event instead of failing the write."
|
||
5. Run both tests → **PASS**; run both touched projects' suites.
|
||
6. Commit: `git add src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Actors/DataConnectionActor.cs src/ZB.MOM.WW.ScadaBridge.SiteEventLogging/SiteEventLogger.cs src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/ScriptActor.cs tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/ScriptActorTests.cs tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests/DataConnectionActorAlarmTests.cs && git commit -m "fix(site-runtime,dcl): low-severity cleanups — observed adapter dispose (S9), alarm-filter overwrite warning (S10), bounded-channel comment (C2), invariant-culture condition fallback (C6)"`
|
||
|
||
### Task 26: Design-doc updates — C3, C4, C5 + behavior notes for this plan's changes
|
||
|
||
**Classification:** trivial (docs only; repo rule: docs travel with code)
|
||
**Estimated implement time:** ~5 min
|
||
**Parallelizable with:** none (write after the behavior tasks land so the docs describe reality)
|
||
**Files:**
|
||
- Modify: `docs/requirements/Component-DataConnectionLayer.md`
|
||
- Modify: `docs/requirements/Component-SiteRuntime.md`
|
||
|
||
1. `Component-DataConnectionLayer.md`:
|
||
- **C3** (line ~303): change "The retry interval is defined **per data connection**" to "The retry interval is a **shared setting for all data connections** (`ReconnectInterval` in the Shared Settings table)" — matching the table at lines 170–177 and `DataConnectionOptions`.
|
||
- **C4** (line ~302, item 1): qualify — "Instance Actors see the staleness immediately and publish it to the site stream/debug view; **script and alarm actors are deliberately NOT re-notified on a quality-only change** (no value changed — re-evaluating triggers would cause spurious firings). Scripts observe quality on their next attribute read; a computed alarm holds its last-known state until fresh values arrive."
|
||
- **S1 note** (connection lifecycle section): add "On every (re)connect the adapter detaches and disposes the previous protocol client (session, keep-alive timers, event loop) before creating a new one — a stale client can neither leak nor signal a disconnect against the new session."
|
||
- **S10 note**: add one line documenting the last-subscriber-wins alarm filter + the logged warning on mismatch.
|
||
2. `Component-SiteRuntime.md`:
|
||
- **C5** (Shared Script Library section): "Shared scripts always execute with **Root scope** — `Attributes[...]` inside a shared script addresses root attributes even when invoked from a composed-module script. Callers needing module-scoped access must pass values as parameters."
|
||
- **S2/UA5** (Script Trust Model / execution-timeout paragraph ~409–412): qualify "Exceeding the timeout cancels the script" → "…cancels the script **cooperatively**: a script blocked in synchronous I/O or a CPU loop does not observe cancellation and continues to occupy its dedicated script-execution thread. A watchdog logs the script by name when its thread has not returned within a grace period after timeout, and the scheduler's queue depth / busy-thread-age gauges are reported through Health Monitoring."
|
||
- **S3** (Script Compilation Errors section ~474–477): add "Enforced at the site by a pre-compile validation gate in the Deployment Manager: all scripts, alarm on-trigger scripts, and trigger expressions are compiled off-thread before the Instance Actor is created or the config persisted; any failure rejects the deployment with the compile errors."
|
||
- **S4/UA6** (Instance Actor / DCL interaction): add "A failed or unanswered tag-subscription handshake is retried on a fixed interval per connection (default 5 s), mirroring the native-alarm subscribe retry."
|
||
- **S5**: note the fixed-interval retry of the startup deployed-config load.
|
||
- **S6**: note that all Instance Actors are watched and an unexpected termination evicts the ref and fails any in-flight deployment. **Deviation from the plan (2026-07-09, see master-tracker "Plan deviations during Wave 1 PLAN-03 execution"):** deploy-success is NOT reported on persistence alone — the store can commit before the actor's async `PreStart` throws, so the design uses a **two-signal join**: the Instance Actor sends a new `InstanceActorInitialized` readiness message at the end of a successful `PreStart`, and the Deployment Manager reports `Success` only once BOTH persistence has committed AND that signal has arrived. An actor that dies during init never signals readiness → the `Terminated` fallback replies `Failed` and rolls the optimistic state back (the persisted-row rollback is deferred until the store commits so it cannot race the write). Document the readiness message + the join.
|
||
- **UA1** (cert trust section ~125): add "On a site node (re)joining the cluster, the Deployment Manager singleton pushes its trusted-cert set to the joining node's CertStore (additive union). Removal reconciliation and central persistence of trust decisions remain the documented follow-up."
|
||
- **UA4** (native alarm mirror section): "Persisted mirror rows include display metadata (type/category/message), so rehydrated conditions render fully before the first source snapshot."
|
||
- **P2** note (attribute change fan-out ~191): "The Instance Actor routes attribute changes per monitored attribute (Expression-trigger children receive all changes); child actors keep their own trigger gates as defense in depth."
|
||
- **P6 follow-up note** (deployment/startup section): "Follow-up: per-instance Roslyn compiles still run inside Instance Actor start during staggered startup; moving them off-thread is deferred (affects failover time-to-recover only)."
|
||
3. Review with `git diff docs/` — verify no stale cross-references (e.g., the C3 sentence is the only "per data connection" retry claim).
|
||
4. Commit: `git add docs/requirements/Component-DataConnectionLayer.md docs/requirements/Component-SiteRuntime.md && git commit -m "docs(requirements): sync SiteRuntime + DCL specs with hardening changes (C3/C4/C5 + S1-S6, P2, UA1/UA4 behavior notes)"`
|
||
|
||
---
|
||
|
||
## Dependencies on other plans
|
||
|
||
- **Plan 05 (Templates/Deployment/Transport)** owns script-trust-policy and Roslyn compile **caching** in central validation paths. Task 3/4 here adds a *site-side* compile gate that duplicates compile work per deploy — if plan 05 introduces a shared compile-cache/collectible-ALC abstraction, the `DeployCompileValidator` should adopt it later; no ordering dependency (this plan's gate is self-contained).
|
||
- **Plan 02 (Communication/S&F)** owns the S&F store SQLite (`StoreAndForwardStorage`) — Task 19 deliberately touches only `SiteStorageService`. If plan 02 also enables WAL on the S&F store, the two changes are independent files.
|
||
- **Plan 06 (Edge integrations)** owns inbound script execution (DI-scope use-after-dispose). Task 8's watchdog pattern (`completed` flag + grace check) is reusable there; no ordering dependency.
|
||
- **Plan 01 (Cluster/Host/Failover)** owns SBR enablement. Task 23 (reconcile-on-join) becomes *more* valuable once real failover works, but functions independently.
|
||
- Health-report additive fields (Task 6) touch `Commons/Messages/Health/SiteHealthReport.cs` — coordinate with any other plan touching that record (additive-only on both sides makes merges trivial).
|
||
|
||
## Execution order
|
||
|
||
**P0 (do first, in order):** Task 1, Task 2 (S1 — the single highest-value fix: stops the leak *and* the healthy-connection flapping), then Task 3 → Task 4 (S3 — deployment honesty, matches OVERALL P1 recommendation #8).
|
||
|
||
**P1:** Tasks 5 → 6 → 7 (scheduler observability chain), Task 8 (watchdog), Tasks 9, 10 (P1 dispatcher relief), Task 11 → 12 (subscription retries), Task 13 (startup retry), Task 14 → 15 (dead-child bookkeeping — 15 depends on 14's Terminated fallback).
|
||
|
||
**P2:** Task 16 → 17 (both edit `DataConnectionActor.cs`, sequential), Task 18 (after 11 — both edit `InstanceActor.cs`), Task 19 → 20 → 21 (19 and 21 both edit `SiteStorageService.cs`; 20 and 21 both edit `NativeAlarmActor.cs` — run 19, 20, 21 in that order; 12 must precede 20 for `NativeAlarmActor.cs`), Task 22 → 23 (23 needs 22's messages and follows 15 in `DeploymentManagerActor.cs`), Task 24.
|
||
|
||
**Last:** Task 25 (after 9 and 17 — shared files), then Task 26 (docs describe the landed behavior).
|
||
|
||
Same-file sequencing summary: `DeploymentManagerActor.cs`: 4 → 13 → 14 → 15 → 23. `InstanceActor.cs`: 4 (doc comment) → 11 → 18. `NativeAlarmActor.cs`: 12 → 20 → 21. `SiteStorageService.cs`: 19 → 20 → 21. `DataConnectionActor.cs`: 16 → 17 → 25. `ScriptActor.cs`: 9 → 18 (helper extraction) → 25. `SiteRuntimeOptions.cs`: 8 → 11 → 24.
|
||
|
||
After the final task: `dotnet build ZB.MOM.WW.ScadaBridge.slnx && dotnet test ZB.MOM.WW.ScadaBridge.slnx`, and for cluster-runtime verification rebuild the docker image (`bash docker/deploy.sh`).
|