docs(archreview): round-2 re-review (2026-07-12) + 8 fix plans (86 tasks)
Re-ran all 8 domain reviews at HEAD8c888f13against theb910f5ebbaseline: every round-1 finding source-verified (168 fixed, 0 regressions, 0 false claims); 56 new findings (1 Critical / 4 High / 15 Medium / 36 Low), concentrated in post-baseline code (anti-entropy resync, KPI rollup backfill, live alarm stream) and seams the fixes exposed. Headliners: S&F resync predicate inversion can wipe the delivering node's buffer (02-N1 Critical); resync snapshot exceeds the Akka remoting frame size (02-N2); failover drill kills the one node keep-oldest can't survive (01-N1); unbounded rollup backfill per failover (04-R1); live production API key in untracked test.txt (08-NF1). Adds PLAN-R2-01..08 + .tasks.json manifests and the Round-2 board, P0 list, cross-plan mutexes, and wave order in 00-MASTER-TRACKER.
This commit is contained in:
Joseph Doherty
@@ -0,0 +1,485 @@
|
||||
# PLAN-R2-03 — Site Runtime & DCL Round-2 Hardening Implementation Plan
|
||||
|
||||
> **For Claude:** REQUIRED SUB-SKILL: Use superpowers-extended-cc:executing-plans to implement this plan task-by-task.
|
||||
|
||||
**Goal:** Fix the round-2 findings in `archreview/03-site-runtime-dcl.md` (dated 2026-07-12) — the one Medium residual on the MxGateway reconnect path (a stale event loop's non-OCE fault can still flap the fresh connection through the generic catch) plus five Low polish items: missing failure mapping on the expression-eval `PipeTo`, two PLAN-03 option knobs absent from the eager options validator, the deploy-path double-compile (adopt the earmarked compile cache), the reintroduced "off-thread" spec-drift sentence, and the undocumented Expression-trigger/script-scheduler coupling. Round-1 deferrals P5 and P6 stay deferred (coverage-table rows only; Tasks 5/6's compile cache shrinks P6's warm-process cost as a side effect).
|
||||
|
||||
**Architecture:** All changes are site-side, inside `ZB.MOM.WW.ScadaBridge.SiteRuntime` and `ZB.MOM.WW.ScadaBridge.DataConnectionLayer`, plus their design doc. The established discipline is preserved: adapter lifecycle faults are classified on the *loop's own* cancellation token (never the mutable fields a newer generation may have replaced), every off-thread result — including a *faulted* task — returns to the actor via `PipeTo` and is applied on the actor thread, and options fail fast at boot with key-naming messages per the PLAN-08 §1.5 convention. The N4 compile cache mirrors TemplateEngine's `ScriptCompileVerdictCache` (SHA-256-keyed, bounded, clear-on-overflow) but is site-local and stores the full `ScriptCompilationResult` — sound because Roslyn `Script<T>` objects are immutable and thread-safe and the result's error text is name-free — so the synchronous deploy gate (deliberately on the singleton mailbox, see N5) and the `InstanceActor.PreStart` recompile share one Roslyn compile per script body per process lifetime. Docs travel with code.
|
||||
|
||||
**Tech Stack:** C#/.NET, Akka.NET (ReceiveActor, TestKit.Xunit2), Roslyn scripting, gRPC (Grpc.Core exception shapes via the `ZB.MOM.WW.MxGateway.Client` package), xunit + NSubstitute.
|
||||
|
||||
Build: `dotnet build ZB.MOM.WW.ScadaBridge.slnx`. Test per-project: `dotnet test tests/<project> --filter <name>`.
|
||||
|
||||
---
|
||||
|
||||
### Task 1: MxGatewayDataConnection — a stale event loop's fault on a cancelled token must not signal Disconnected
|
||||
|
||||
**Classification:** high-risk (adapter reconnect lifecycle, concurrency with the dying loop)
|
||||
**Estimated implement time:** ~5 min
|
||||
**Parallelizable with:** 2, 3, 4, 5
|
||||
**Files:**
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Adapters/MxGatewayDataConnection.cs` (`ConnectAsync` ~lines 106–108, `RunEventLoopAsync` ~lines 111–130)
|
||||
- Test: `tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests/Adapters/MxGatewayDataConnectionReconnectTests.cs`
|
||||
|
||||
The S1 fix cancels the previous loop's CTS and disposes the previous client (`:77–89`), then resets `_disconnectFired` (`:91`) — but `Cancel()` is not synchronous with the loop's exit. The old loop's `await foreach` over the gRPC stream surfaces its failure milliseconds later, and a cancelled/disposed Grpc.Net stream commonly faults with `RpcException(StatusCode.Cancelled)` (the default without `ThrowOperationCanceledOnCancellation`) or `ObjectDisposedException` from the concurrent `DisposeAsync` — neither is an `OperationCanceledException`, so both fall into the generic `catch` (`:125–129`) → `RaiseDisconnected()` against the *reset* guard → the exact S1 scenario-2 flap, potentially self-sustaining.
|
||||
|
||||
1. Add failing tests to the existing reconnect test class (extend its per-client substitute fixture so each client's `RunEventLoopAsync` returns a controllable `TaskCompletionSource` task instead of `Task.Delay(Timeout.Infinite)`):
|
||||
|
||||
```csharp
|
||||
[Fact]
|
||||
public async Task StaleEventLoopRpcFault_AfterReconnect_DoesNotSignalDisconnected()
|
||||
{
|
||||
var clients = new List<IMxGatewayClient>();
|
||||
var loops = new List<TaskCompletionSource>();
|
||||
var loopTokens = new List<CancellationToken>();
|
||||
var factory = Substitute.For<IMxGatewayClientFactory>();
|
||||
factory.Create().Returns(_ =>
|
||||
{
|
||||
var c = Substitute.For<IMxGatewayClient>();
|
||||
var tcs = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously);
|
||||
loops.Add(tcs);
|
||||
c.RunEventLoopAsync(Arg.Any<Action<MxValueUpdate>>(), Arg.Do<CancellationToken>(t => loopTokens.Add(t)))
|
||||
.Returns(tcs.Task);
|
||||
clients.Add(c);
|
||||
return c;
|
||||
});
|
||||
var adapter = new MxGatewayDataConnection(factory, NullLogger<MxGatewayDataConnection>.Instance);
|
||||
var disconnects = 0;
|
||||
adapter.Disconnected += () => Interlocked.Increment(ref disconnects);
|
||||
var details = new Dictionary<string, string> { ["Endpoint"] = "http://gw:5000", ["ApiKey"] = "k" };
|
||||
|
||||
await adapter.ConnectAsync(details);
|
||||
await WaitUntilAsync(() => loopTokens.Count == 1);
|
||||
await adapter.ConnectAsync(details); // reconnect: cancels loop #0, resets _disconnectFired
|
||||
await WaitUntilAsync(() => loopTokens.Count == 2);
|
||||
|
||||
// The OLD loop observes its cancellation as a gRPC fault, not an OCE —
|
||||
// the Grpc.Net default without ThrowOperationCanceledOnCancellation.
|
||||
loops[0].SetException(new RpcException(new Status(StatusCode.Cancelled, "call cancelled")));
|
||||
await Task.Delay(200);
|
||||
Assert.Equal(0, disconnects); // stale fault absorbed — the fresh connection must not flap
|
||||
|
||||
// The CURRENT loop's genuine fault must still signal, exactly once.
|
||||
loops[1].SetException(new RpcException(new Status(StatusCode.Unavailable, "gateway gone")));
|
||||
await WaitUntilAsync(() => disconnects == 1);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task StaleEventLoopObjectDisposedFault_AfterReconnect_DoesNotSignalDisconnected()
|
||||
{
|
||||
// Identical drive, but the old loop faults with the concurrent-DisposeAsync shape.
|
||||
// … same fixture …
|
||||
loops[0].SetException(new ObjectDisposedException("MxGatewayClient"));
|
||||
await Task.Delay(200);
|
||||
Assert.Equal(0, disconnects);
|
||||
}
|
||||
```
|
||||
|
||||
(`RpcException`/`Status`/`StatusCode` come from `Grpc.Core`, transitively via the `ZB.MOM.WW.MxGateway.Client` package the test project already references — `RealMxGatewayClient.cs:227` catches `RpcException` in the main project. If the test assembly cannot resolve them, add a `Grpc.Net.Client` PackageReference; the version is already pinned in `Directory.Packages.props`.)
|
||||
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests --filter "FullyQualifiedName~StaleEventLoopRpcFault_AfterReconnect|FullyQualifiedName~StaleEventLoopObjectDisposedFault_AfterReconnect"` → expect **FAIL** (the generic catch raises `Disconnected` against the reset guard: `disconnects == 1` after the stale fault).
|
||||
3. Implement:
|
||||
- Change `RunEventLoopAsync` to take the client AND token as parameters — `private async Task RunEventLoopAsync(IMxGatewayClient client, CancellationToken ct)` — and use `client.RunEventLoopAsync(…)` instead of the `_client!` field read (the old loop must never bind a newer generation's client).
|
||||
- Add a filtered catch **before** the generic one:
|
||||
|
||||
```csharp
|
||||
catch (Exception ex) when (ct.IsCancellationRequested)
|
||||
{
|
||||
// Teardown fault of a superseded loop (N1): Cancel() is not synchronous with
|
||||
// the loop's exit — a cancelled/disposed gRPC stream commonly surfaces as
|
||||
// RpcException(StatusCode.Cancelled) (Grpc.Net default without
|
||||
// ThrowOperationCanceledOnCancellation) or ObjectDisposedException from the
|
||||
// concurrent DisposeAsync, milliseconds AFTER ConnectAsync has already reset
|
||||
// _disconnectFired. Any fault on a cancelled token is normal shutdown;
|
||||
// raising Disconnected here would flap the NEW healthy session.
|
||||
_logger.LogDebug(ex,
|
||||
"MxGateway event loop faulted after cancellation — superseded loop teardown; not signalling disconnect");
|
||||
}
|
||||
```
|
||||
|
||||
- In `ConnectAsync`, capture the token and client into locals before the lambda (a rapid double-reconnect must not hand loop #1 loop #2's token/client — the current `Task.Run(() => RunEventLoopAsync(_eventLoopCts.Token))` reads the *field* lazily):
|
||||
|
||||
```csharp
|
||||
_eventLoopCts = new CancellationTokenSource();
|
||||
var loopToken = _eventLoopCts.Token; // capture NOW: the field may be replaced by a
|
||||
var loopClient = _client; // subsequent reconnect before Task.Run executes (N1)
|
||||
_ = Task.Run(() => RunEventLoopAsync(loopClient, loopToken));
|
||||
```
|
||||
|
||||
4. Run the two new tests → **PASS**; then the full adapter suite: `dotnet test tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests --filter "FullyQualifiedName~MxGatewayDataConnection"` → all green (the existing `Reconnect_DisposesPreviousClient_AndCancelsPreviousEventLoop` test pins the S1 behavior and must survive the signature change).
|
||||
5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Adapters/MxGatewayDataConnection.cs tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests/Adapters/MxGatewayDataConnectionReconnectTests.cs && git commit -m "fix(dcl): stale MxGateway event-loop fault on a cancelled token no longer flaps the fresh connection (plan R2-03 T1)"`
|
||||
|
||||
### Task 2: ScriptActor — failure mapping on the expression-eval `PipeTo` (a faulted scheduler task must not kill the trigger)
|
||||
|
||||
**Classification:** high-risk (actor concurrency, trigger liveness)
|
||||
**Estimated implement time:** ~5 min
|
||||
**Parallelizable with:** 1, 3, 4, 5, 6
|
||||
**Files:**
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/ScriptActor.cs` (Receive block ~line 161, `StartExpressionEvaluation` PipeTo ~lines 311–313, internal messages ~line 623)
|
||||
- Test: `tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/ScriptActorTests.cs`
|
||||
|
||||
`StartExpressionEvaluation` pipes with `success:` only (`:312–313`). The inner body catches everything, so the only fault source is the outer `Task.Factory.StartNew` itself (e.g. `ObjectDisposedException` from a disposed `ScriptExecutionScheduler` during shutdown/test teardown) — but if it ever fires, the actor gets an unhandled `Status.Failure` and `_evalInFlight` stays `true` forever (`:284–285`): every subsequent change parks in `_evalPending` and the expression trigger is dead for the actor's lifetime with no log.
|
||||
|
||||
1. Failing test. Add a public static gate hook to the test file (the `CompileRawTriggerExpression` helper compiles *without* the trust validator, so the expression may reference it — same trick as the P1 thread-name tests):
|
||||
|
||||
```csharp
|
||||
public static class EvalGate
|
||||
{
|
||||
public static readonly SemaphoreSlim Gate = new(0);
|
||||
private static int _entries;
|
||||
public static int Entries => Volatile.Read(ref _entries);
|
||||
public static bool Block() { Interlocked.Increment(ref _entries); Gate.Wait(); return false; }
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ExpressionEvalTaskFault_ClearsInFlight_AndDrainsPendingEvaluation()
|
||||
{
|
||||
var expr = CompileRawTriggerExpression(
|
||||
"ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.Actors.EvalGate.Block()");
|
||||
var (actor, _) = CreateTriggeredActor(
|
||||
"ExprFault", "Expression", "{\"expression\":\"true\",\"mode\":\"OnTrue\"}", null, expr);
|
||||
try
|
||||
{
|
||||
actor.Tell(Change("A", "1")); // eval starts on the scheduler and BLOCKS → _evalInFlight = true
|
||||
AwaitAssert(() => Assert.Equal(1, EvalGate.Entries), TimeSpan.FromSeconds(10));
|
||||
actor.Tell(Change("A", "2")); // coalesces → _evalPending = true
|
||||
|
||||
// Simulate the faulted scheduler task the PipeTo failure mapping now surfaces
|
||||
// (e.g. ObjectDisposedException from a disposed ScriptExecutionScheduler).
|
||||
actor.Tell(new ScriptActor.ExpressionEvalFailed(new ObjectDisposedException("scheduler")));
|
||||
|
||||
// In-flight cleared + pending drained ⇒ a SECOND evaluation starts and blocks too.
|
||||
AwaitAssert(() => Assert.Equal(2, EvalGate.Entries), TimeSpan.FromSeconds(10));
|
||||
}
|
||||
finally
|
||||
{
|
||||
EvalGate.Gate.Release(EvalGate.Entries); // ALWAYS free the shared scheduler threads
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
(Note the expression must reference the gate's fully-qualified name; the raw compile references `typeof(object).Assembly` only, so also add the test assembly to `CompileRawTriggerExpression`'s `.WithReferences(...)` for this test — or add an overload taking extra references. The `finally` release is mandatory: the gate blocks threads of the *process-wide* `ScriptExecutionScheduler.Shared`.)
|
||||
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~ExpressionEvalTaskFault_ClearsInFlight"` → expect **FAIL** — the RED step is the compile failure (`ScriptActor.ExpressionEvalFailed` does not exist); behaviorally, without the fix the second evaluation never starts either.
|
||||
3. Implement:
|
||||
- New message next to `ExpressionEvalResult` (`:623`) — `internal` (not `private`) so the TestKit can Tell it, same visibility idiom as `IntervalTick`/`ScriptExecutionCompleted`:
|
||||
|
||||
```csharp
|
||||
/// <summary>
|
||||
/// Piped back to self when the off-dispatcher evaluation TASK itself faults
|
||||
/// (e.g. ObjectDisposedException from a disposed ScriptExecutionScheduler
|
||||
/// during shutdown) — the inner body's catch never sees that. Without this
|
||||
/// mapping the actor receives an unhandled Status.Failure and _evalInFlight
|
||||
/// stays true forever, permanently parking the expression trigger (N2).
|
||||
/// </summary>
|
||||
internal sealed record ExpressionEvalFailed(Exception Cause);
|
||||
```
|
||||
|
||||
- Ctor Receive block (after the `ExpressionEvalResult` registration at `:161`): `Receive<ExpressionEvalFailed>(HandleExpressionEvalFailed);`
|
||||
- PipeTo (`:312–313`): `….PipeTo(self, success: r => new ExpressionEvalResult(r), failure: ex => new ExpressionEvalFailed(ex));`
|
||||
- Handler — log via the existing `LogExpressionError` (`:397`, already increments the health counter), then reuse the result path with `false` (consistent with the inner catch's treated-as-false semantics — clears in-flight, applies the false edge, drains pending):
|
||||
|
||||
```csharp
|
||||
private void HandleExpressionEvalFailed(ExpressionEvalFailed msg)
|
||||
{
|
||||
LogExpressionError(msg.Cause);
|
||||
HandleExpressionEvalResult(new ExpressionEvalResult(false));
|
||||
}
|
||||
```
|
||||
|
||||
4. Run the new test → **PASS**; then `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~ScriptActorTests"` → all green (the P1 expression/WhileTrue suites are the regression canary).
|
||||
5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/ScriptActor.cs tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/ScriptActorTests.cs && git commit -m "fix(siteruntime): faulted expression-eval task clears _evalInFlight instead of killing the ScriptActor trigger (plan R2-03 T2)"`
|
||||
|
||||
### Task 3: AlarmActor — same failure mapping on the expression-eval `PipeTo`
|
||||
|
||||
**Classification:** high-risk (alarm state machine liveness)
|
||||
**Estimated implement time:** ~4 min
|
||||
**Parallelizable with:** 1, 2, 4, 5, 6
|
||||
**Files:**
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/AlarmActor.cs` (Receive block ~line 188, `StartExpressionEvaluation` PipeTo ~lines 552–554, internal messages ~line 797)
|
||||
- Test: `tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/AlarmActorTests.cs`
|
||||
|
||||
Structurally identical to Task 2 — `AlarmActor.cs:552–554` pipes success-only, `:523–524` sets `_evalInFlight`, so a faulted task permanently kills the Expression alarm.
|
||||
|
||||
1. Failing test (reuse the `ExpressionAlarm_EvaluatesOnSchedulerThread_AndActivates` harness in `AlarmActorTests.cs:1031` — same `ResolvedAlarm`/`instanceProbe` construction):
|
||||
|
||||
```csharp
|
||||
[Fact]
|
||||
public void ExpressionEvalTaskFault_DoesNotKillTheAlarmTrigger()
|
||||
{
|
||||
var expr = CompileRawTriggerExpression("true");
|
||||
var alarmConfig = new ResolvedAlarm
|
||||
{
|
||||
CanonicalName = "ExprFaultAlarm",
|
||||
TriggerType = "Expression",
|
||||
TriggerConfiguration = "{\"expression\":\"true\"}",
|
||||
PriorityLevel = 1
|
||||
};
|
||||
var instanceProbe = CreateTestProbe();
|
||||
var alarm = ActorOf(Props.Create(() => new AlarmActor(
|
||||
"ExprFaultAlarm", "Pump1", instanceProbe.Ref, alarmConfig,
|
||||
null, _sharedLibrary, _options,
|
||||
NullLogger<AlarmActor>.Instance, expr)));
|
||||
|
||||
// Simulate the faulted scheduler task: must be handled (logged, in-flight
|
||||
// cleared, false applied) — NOT an unhandled Status.Failure.
|
||||
alarm.Tell(new AlarmActor.ExpressionEvalFailed(new ObjectDisposedException("scheduler")));
|
||||
|
||||
// The trigger must still be alive: the next change evaluates and activates.
|
||||
alarm.Tell(new AttributeValueChanged("Pump1", "A", "A", 1, "Good", DateTimeOffset.UtcNow));
|
||||
var change = instanceProbe.ExpectMsg<AlarmStateChanged>(TimeSpan.FromSeconds(10));
|
||||
Assert.Equal(AlarmState.Active, change.State);
|
||||
}
|
||||
```
|
||||
|
||||
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~ExpressionEvalTaskFault_DoesNotKillTheAlarmTrigger"` → expect **FAIL** (compile error — `AlarmActor.ExpressionEvalFailed` does not exist).
|
||||
3. Implement, mirroring Task 2:
|
||||
- `internal sealed record ExpressionEvalFailed(Exception Cause);` next to the existing `ExpressionEvalResult` (`:797`), with the same N2 XML doc.
|
||||
- `Receive<ExpressionEvalFailed>(HandleExpressionEvalFailed);` after `:188`.
|
||||
- PipeTo (`:553–554`) gains `failure: ex => new ExpressionEvalFailed(ex)`.
|
||||
- Handler — same shape as the inner catch's error path, then delegate to the false result (the class doc already promises "a throwing … expression is treated as false so that the state machine still runs — an Active alarm correctly clears"):
|
||||
|
||||
```csharp
|
||||
private void HandleExpressionEvalFailed(ExpressionEvalFailed msg)
|
||||
{
|
||||
_healthCollector?.IncrementAlarmError();
|
||||
_logger.LogError(msg.Cause,
|
||||
"Alarm {Alarm} trigger-expression evaluation task faulted on {Instance}; treated as false",
|
||||
_alarmName, _instanceName);
|
||||
HandleExpressionEvalResult(new ExpressionEvalResult(false));
|
||||
}
|
||||
```
|
||||
|
||||
4. Run the new test → **PASS**; then `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~AlarmActorTests"` → all green.
|
||||
5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/AlarmActor.cs tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/AlarmActorTests.cs && git commit -m "fix(siteruntime): faulted expression-eval task clears _evalInFlight instead of killing the AlarmActor trigger (plan R2-03 T3)"`
|
||||
|
||||
### Task 4: Eagerly validate `TagSubscribeRetryIntervalMs` and `StuckScriptGraceMs`
|
||||
|
||||
**Classification:** small (additive validator rules per the PLAN-08 §1.5 convention)
|
||||
**Estimated implement time:** ~3 min
|
||||
**Parallelizable with:** 1, 2, 3, 5, 6
|
||||
**Files:**
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/SiteRuntimeOptionsValidator.cs` (append after the `ConfigFetchRetryCount` rule, lines 56–58)
|
||||
- Test: `tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/SiteRuntimeOptionsValidatorTests.cs`
|
||||
|
||||
The validator covers ten options but not the two PLAN-03 added: `TagSubscribeRetryIntervalMs` (`SiteRuntimeOptions.cs:76` — a 0 hot-loops the subscribe retry, a negative value throws inside `InstanceActor.SendTagSubscribe`'s `ScheduleTellOnceCancelable`) and `StuckScriptGraceMs` (`SiteRuntimeOptions.cs:86` — a negative value throws `ArgumentOutOfRangeException` inside the watchdog's `Task.Delay`, `ScriptExecutionActor.cs:144`).
|
||||
|
||||
1. Failing tests (mirror the existing `Validate(...)` helper and assertion shape in the test file):
|
||||
|
||||
```csharp
|
||||
[Fact]
|
||||
public void ZeroTagSubscribeRetryIntervalMs_IsRejected()
|
||||
{
|
||||
var result = Validate(new SiteRuntimeOptions { TagSubscribeRetryIntervalMs = 0 });
|
||||
|
||||
Assert.True(result.Failed);
|
||||
Assert.Contains("TagSubscribeRetryIntervalMs", result.FailureMessage);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void NegativeStuckScriptGraceMs_IsRejected()
|
||||
{
|
||||
var result = Validate(new SiteRuntimeOptions { StuckScriptGraceMs = -1 });
|
||||
|
||||
Assert.True(result.Failed);
|
||||
Assert.Contains("StuckScriptGraceMs", result.FailureMessage);
|
||||
}
|
||||
```
|
||||
|
||||
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~SiteRuntimeOptionsValidatorTests"` → the two new tests **FAIL** (validator succeeds on both).
|
||||
3. Implement — two `RequireThat` lines matching the file's key-naming message style:
|
||||
|
||||
```csharp
|
||||
builder.RequireThat(options.TagSubscribeRetryIntervalMs > 0,
|
||||
$"ScadaBridge:SiteRuntime:TagSubscribeRetryIntervalMs must be greater than 0 " +
|
||||
$"(was {options.TagSubscribeRetryIntervalMs}); a zero interval hot-loops the tag-subscribe " +
|
||||
"retry and a negative one throws inside the Instance Actor's retry scheduling.");
|
||||
|
||||
builder.RequireThat(options.StuckScriptGraceMs >= 0,
|
||||
$"ScadaBridge:SiteRuntime:StuckScriptGraceMs must be >= 0 " +
|
||||
$"(was {options.StuckScriptGraceMs}); a negative grace throws inside the stuck-script watchdog's delay.");
|
||||
```
|
||||
|
||||
4. Run → **PASS**, including `DefaultOptions_AreValid` (defaults 5000/30000 satisfy both rules).
|
||||
5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.SiteRuntime/SiteRuntimeOptionsValidator.cs tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/SiteRuntimeOptionsValidatorTests.cs && git commit -m "fix(siteruntime): eagerly validate TagSubscribeRetryIntervalMs + StuckScriptGraceMs at boot (plan R2-03 T4)"`
|
||||
|
||||
### Task 5: SiteScriptCompileCache — bounded process-wide compiled-script cache (N4 groundwork)
|
||||
|
||||
**Classification:** standard (pure new class, no consumers yet)
|
||||
**Estimated implement time:** ~5 min
|
||||
**Parallelizable with:** 1, 2, 3, 4
|
||||
**Files:**
|
||||
- Create: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Scripts/SiteScriptCompileCache.cs`
|
||||
- Test: create `tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Scripts/SiteScriptCompileCacheTests.cs`
|
||||
|
||||
This is the site-side adoption of the compile-cache abstraction the master tracker earmarked when PLAN-03's Task 3/4 shipped ("later refactor, not a blocker") — TemplateEngine's `ScriptCompileVerdictCache` pattern, but storing the full `ScriptCompilationResult` so both the deploy gate and the `PreStart` recompile reuse the compiled `Script<object?>` itself. Soundness: the result is a pure function of (code, globals type, compile-time-static trust policy); the stored error text is name-free (`Diagnostic.GetMessage()` / `"Compilation exception: …"` — callers prefix their own context); Roslyn `Script<T>` is immutable and thread-safe (`RunAsync` creates per-run state), so one instance may be shared across actors.
|
||||
|
||||
1. Failing tests (the type does not exist — compile failure is the RED step). The cache is process-wide static state: put this test class and Task 6's `ScriptCompilationServiceTests` additions in one xunit collection (`[Collection("SiteScriptCompileCache")]`) and call `Clear()` at the top of each test, so parallel classes cannot skew `Hits`/`Count`:
|
||||
|
||||
```csharp
|
||||
[Collection("SiteScriptCompileCache")]
|
||||
public class SiteScriptCompileCacheTests
|
||||
{
|
||||
private static ScriptCompilationResult Ok() =>
|
||||
ScriptCompilationResult.Succeeded(CSharpScript.Create<object?>("1", ScriptOptions.Default, typeof(ScriptGlobals)));
|
||||
|
||||
[Fact]
|
||||
public void GetOrAdd_SameCodeAndGlobals_ComputesOnce()
|
||||
{
|
||||
SiteScriptCompileCache.Clear();
|
||||
var factoryCalls = 0;
|
||||
ScriptCompilationResult Factory() { factoryCalls++; return Ok(); }
|
||||
|
||||
var r1 = SiteScriptCompileCache.GetOrAdd("return 1;", typeof(ScriptGlobals), Factory);
|
||||
var r2 = SiteScriptCompileCache.GetOrAdd("return 1;", typeof(ScriptGlobals), Factory);
|
||||
|
||||
Assert.Equal(1, factoryCalls);
|
||||
Assert.Same(r1, r2);
|
||||
Assert.Equal(1, SiteScriptCompileCache.Hits);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void GetOrAdd_SameCode_DifferentGlobals_AreSeparateEntries()
|
||||
{
|
||||
SiteScriptCompileCache.Clear();
|
||||
SiteScriptCompileCache.GetOrAdd("1 > 0", typeof(ScriptGlobals), Ok);
|
||||
SiteScriptCompileCache.GetOrAdd("1 > 0", typeof(TriggerExpressionGlobals), Ok);
|
||||
|
||||
Assert.Equal(2, SiteScriptCompileCache.Count);
|
||||
Assert.Equal(0, SiteScriptCompileCache.Hits); // no cross-globals hit
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Overflow_ClearsWholesale()
|
||||
{
|
||||
SiteScriptCompileCache.Clear();
|
||||
for (var i = 0; i <= SiteScriptCompileCache.MaxEntries; i++)
|
||||
SiteScriptCompileCache.GetOrAdd($"return {i};", typeof(ScriptGlobals), Ok);
|
||||
|
||||
Assert.True(SiteScriptCompileCache.Count <= SiteScriptCompileCache.MaxEntries);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~SiteScriptCompileCacheTests"` → **FAIL** (type does not exist).
|
||||
3. Implement, mirroring `ScriptCompileVerdictCache` (`src/ZB.MOM.WW.ScadaBridge.TemplateEngine/Validation/ScriptCompileVerdictCache.cs`): `internal static class SiteScriptCompileCache` (InternalsVisibleTo already covers the test assembly) with
|
||||
- `internal const int MaxEntries = 1024;` — deliberately smaller than the verdict cache's 4096: entries pin compiled assemblies, not just verdict strings; on overflow the cache is cleared wholesale (results are recomputable, coarse reset avoids eviction bookkeeping).
|
||||
- Key = `globalsType.FullName + ":" + Convert.ToHexString(SHA256.HashData(Encoding.UTF8.GetBytes(code)))` — the globals discriminator keeps `ScriptGlobals` and `TriggerExpressionGlobals` compiles of identical source apart.
|
||||
- `public static ScriptCompilationResult GetOrAdd(string code, Type globalsType, Func<ScriptCompilationResult> factory)` over a `ConcurrentDictionary<string, ScriptCompilationResult>`, incrementing `Hits` on a hit; `Hits`/`Count`/`Clear()` exposed for tests and diagnostics — same surface as the TemplateEngine cache.
|
||||
- XML doc stating the three soundness invariants above (pure function of code+globals+static policy; name-free errors; `Script<T>` immutability) and that **failures are cached too** — a bad script body fails identically for every caller.
|
||||
4. Run → **PASS**.
|
||||
5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Scripts/SiteScriptCompileCache.cs tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Scripts/SiteScriptCompileCacheTests.cs && git commit -m "feat(siteruntime): bounded process-wide compiled-script cache, keyed on code hash + globals type (plan R2-03 T5)"`
|
||||
|
||||
### Task 6: Wire the cache into ScriptCompilationService — deploy gate + PreStart share one compile per script per revision
|
||||
|
||||
**Classification:** standard (single choke point; all call sites inherit)
|
||||
**Estimated implement time:** ~4 min
|
||||
**Parallelizable with:** 1, 2, 3, 4 (needs 5 merged)
|
||||
**Files:**
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Scripts/ScriptCompilationService.cs` (`CompileCore`, ~lines 98–139)
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/DeploymentManagerActor.cs` (gate rationale comment ~lines 510–521 — comment only)
|
||||
- Test: `tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Scripts/ScriptCompilationServiceTests.cs`
|
||||
|
||||
`CompileCore` is the single funnel for every site-side compile — the `DeployCompileValidator` gate (`DeploymentManagerActor.cs:522`), the `InstanceActor.CreateChildActors` recompile (`InstanceActor.cs:1485, 1497–1498`), and `SharedScriptLibrary` all flow through it — so caching here collapses the 2×N-compiles-per-bulk-redeploy cost (N4) and shrinks P6's warm-process staggered-startup recompiles for free, with zero call-site changes.
|
||||
|
||||
1. Failing test (add to `ScriptCompilationServiceTests`, tagged `[Collection("SiteScriptCompileCache")]` per Task 5's isolation note):
|
||||
|
||||
```csharp
|
||||
[Fact]
|
||||
public void Compile_SameCodeTwice_SharesOneRoslynCompile()
|
||||
{
|
||||
SiteScriptCompileCache.Clear();
|
||||
var r1 = _service.Compile("deploy-gate-copy", "return 1 + 1;");
|
||||
var r2 = _service.Compile("prestart-copy", "return 1 + 1;");
|
||||
|
||||
Assert.True(r1.IsSuccess);
|
||||
Assert.Same(r1.CompiledScript, r2.CompiledScript); // one compile, shared Script<T> (N4)
|
||||
Assert.Equal(1, SiteScriptCompileCache.Hits);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Compile_ScriptAndTriggerExpression_DoNotCrossContaminate()
|
||||
{
|
||||
SiteScriptCompileCache.Clear();
|
||||
var script = _service.Compile("s", "1 > 0");
|
||||
var trigger = _service.CompileTriggerExpression("t", "1 > 0");
|
||||
|
||||
Assert.NotSame(script.CompiledScript, trigger.CompiledScript); // different globals surfaces
|
||||
}
|
||||
```
|
||||
|
||||
2. Run: `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~Compile_SameCodeTwice_SharesOneRoslynCompile|FullyQualifiedName~Compile_ScriptAndTriggerExpression_DoNotCrossContaminate"` → **FAIL** (`Assert.Same` fails: two independent compiles today).
|
||||
3. Implement:
|
||||
- Rename the current `CompileCore` body to `private ScriptCompilationResult CompileUncached(string name, string code, Type globalsType)` (byte-identical, including the trust-model check — the verdict is deterministic on code, so it is cacheable with the compile).
|
||||
- New `CompileCore`:
|
||||
|
||||
```csharp
|
||||
private ScriptCompilationResult CompileCore(string name, string code, Type globalsType)
|
||||
{
|
||||
var result = SiteScriptCompileCache.GetOrAdd(code, globalsType,
|
||||
() => CompileUncached(name, code, globalsType));
|
||||
if (!result.IsSuccess)
|
||||
_logger.LogDebug(
|
||||
"Script {Script}: returning cached compile failure ({ErrorCount} error(s))",
|
||||
name, result.Errors.Count); // miss-path logs name the FIRST caller; keep this caller visible
|
||||
return result;
|
||||
}
|
||||
```
|
||||
|
||||
- Update the class XML doc: results (success AND failure) are memoised process-wide in `SiteScriptCompileCache`; error text is name-free by construction, so cached failures render correctly for every caller.
|
||||
- Append one sentence to the `HandleDeploy` gate comment (`DeploymentManagerActor.cs:519–521`): "Redeploys and multi-instance deploys of unchanged scripts hit the process-wide compile cache (`SiteScriptCompileCache`), so the synchronous gate recompiles only genuinely new code and the Instance Actor's PreStart reuses the gate's compile (N4)."
|
||||
4. Run the two new tests → **PASS**; then the funnel's consumer suites: `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests --filter "FullyQualifiedName~ScriptCompilationServiceTests|FullyQualifiedName~DeployCompileValidatorTests|FullyQualifiedName~SharedScriptLibraryTests|FullyQualifiedName~TrustModelSemanticTests"` → all green (these pin that caching changed no verdicts).
|
||||
5. Commit: `git add src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Scripts/ScriptCompilationService.cs src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/DeploymentManagerActor.cs tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Scripts/ScriptCompilationServiceTests.cs && git commit -m "perf(siteruntime): deploy gate + InstanceActor PreStart share one Roslyn compile via the compile cache (plan R2-03 T6)"`
|
||||
|
||||
### Task 7: Design-doc sync — synchronous compile gate (N5), Expression/scheduler coupling (N6), P6/N4 note refresh
|
||||
|
||||
**Classification:** trivial (docs + one stale XML doc; repo rule: docs travel with code)
|
||||
**Estimated implement time:** ~4 min
|
||||
**Parallelizable with:** none (write after the behavior tasks land so the docs describe reality)
|
||||
**Files:**
|
||||
- Modify: `docs/requirements/Component-SiteRuntime.md` (lines ~489, ~234–246, ~99)
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Deployment/DeployCompileValidator.cs` (class XML doc, ~lines 16–18)
|
||||
|
||||
1. **N5** — `Component-SiteRuntime.md:489`: the sentence claims the gate compiles "**off-thread**"; the shipped gate is deliberately synchronous (`DeploymentManagerActor.cs:510–522`). Replace the off-thread clause so the bullet reads: "…the Deployment Manager compiles **all** of the instance's scripts, alarm on-trigger scripts, and trigger expressions **synchronously, as a pure prefix step of the deploy handler on the singleton's actor thread** — deliberately not off-thread, because piping the verdict back to self would reorder concurrent deploys against delete/disable and break the redeploy-supersede mailbox-FIFO ordering (see the in-code rationale in `DeploymentManagerActor.HandleDeploy`); unchanged script bodies hit the process-wide compile cache, so the gate recompiles only genuinely new code. Any compile failure rejects the deployment with the collected compile errors…" (keep the existing honesty tail verbatim).
|
||||
2. **N6** — `Component-SiteRuntime.md` Alarm Evaluation section (~line 236, after the trigger-type list): add one bullet: "**Expression** trigger evaluation runs on the bounded script-execution thread pool shared with script bodies (and Expression *script* triggers): scheduler saturation — e.g. stuck scripts occupying all threads — delays Expression alarm transitions site-wide until a thread frees. This is a deliberate trade-off (evaluation no longer blocks dispatcher threads; per-actor coalescing bounds queue growth) and the scheduler gauges + stuck-script watchdog make the saturated state visible."
|
||||
3. **P6/N4 note refresh** — `Component-SiteRuntime.md:99` (the existing P6 follow-up note): append "As of the round-2 hardening, a process-wide compile cache dedupes identical script bodies within a node's process lifetime (the deploy gate's compile is reused by Instance Actor start), shrinking the recompile cost; the *first* compile after process start still runs inside Instance Actor start, so the deferral stands."
|
||||
4. **N5 (code-comment drift)** — `DeployCompileValidator.cs:16–18`: the XML doc still says "it is safe to run off the singleton mailbox on a `Task.Run` thread — the deploy path never blocks on Roslyn", which describes the plan's original shape, not the shipped synchronous gate. Reword: "Because it holds no actor state and only calls the (stateless) `ScriptCompilationService`, it is pure and thread-safe; the Deployment Manager nevertheless invokes it *synchronously on the singleton's actor thread* as a pure prefix step, preserving mailbox-FIFO deploy ordering (see `HandleDeploy`). Compiles of unchanged script bodies are deduped by the process-wide compile cache."
|
||||
5. Review with `git diff docs/requirements/Component-SiteRuntime.md src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Deployment/DeployCompileValidator.cs` — verify no other "off-thread" claim about the gate remains (`grep -n "off-thread" docs/requirements/Component-SiteRuntime.md` should return only the line-99 P6 note, which refers to the *startup* compiles, not the gate).
|
||||
6. Commit: `git add docs/requirements/Component-SiteRuntime.md src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Deployment/DeployCompileValidator.cs && git commit -m "docs(siteruntime): compile gate is synchronous by design; Expression alarms share the bounded script pool (plan R2-03 T7)"`
|
||||
|
||||
---
|
||||
|
||||
## Dependencies on other plans
|
||||
|
||||
- **R2-05 (Templates/Deployment/Transport round 2)** owns the Transport options-validator knob; **R2-08** owns the AuditLog/Host validator gaps — Task 4 here covers ONLY the two SiteRuntime knobs (`TagSubscribeRetryIntervalMs`, `StuckScriptGraceMs`), per the cross-plan ruling.
|
||||
- **TemplateEngine's `ScriptCompileVerdictCache`** stays untouched: it caches *verdicts* for central validation; Task 5/6's `SiteScriptCompileCache` caches *compiled scripts* for site execution. The two are intentionally separate (different result shapes, different assemblies); no shared abstraction is introduced.
|
||||
- The site-side live-alarm-stream code (`SiteStreamManager.SubscribeSiteAlarms`) was reviewed clean in round 2 — nothing from the aggregated-live-alarm-stream plan lands here.
|
||||
|
||||
## Execution order
|
||||
|
||||
**P0 (do first):** Task 1 (N1 — the only Medium; closes the last open S1 mechanism).
|
||||
|
||||
**P1 (parallel, file-disjoint):** Tasks 2, 3 (N2 — liveness of every Expression trigger/alarm), Task 4 (N3), Task 5 (N4 groundwork).
|
||||
|
||||
**P2:** Task 6 (after 5 — consumes the cache; comment-only touch on `DeploymentManagerActor.cs`).
|
||||
|
||||
**Last:** Task 7 (docs describe the landed behavior).
|
||||
|
||||
Same-file sequencing summary: `ScriptCompilationService.cs`: 6 only. `DeploymentManagerActor.cs`: 6 only (comment). `DeployCompileValidator.cs`: 7 only (XML doc). No file is touched by two tasks.
|
||||
|
||||
After the final task: `dotnet build ZB.MOM.WW.ScadaBridge.slnx && dotnet test ZB.MOM.WW.ScadaBridge.slnx`, and for cluster-runtime verification rebuild the docker image (`bash docker/deploy.sh`).
|
||||
|
||||
---
|
||||
|
||||
## Findings Coverage
|
||||
|
||||
| Finding | Severity | Task(s) / Disposition |
|
||||
|---|---|---|
|
||||
| N1 — MxGateway: stale event loop's non-OCE fault (`RpcException(Cancelled)`/`ObjectDisposedException`) can flap the fresh connection through the generic catch; lazy `_eventLoopCts` field read in the `Task.Run` lambda | Medium | 1 (ct-guarded catch + token/client captured into locals; churn scenario regression-tested with both fault shapes) |
|
||||
| N2 — expression-eval `PipeTo` has a success mapping only; a faulted scheduler task permanently kills the trigger with `_evalInFlight` stuck | Low | 2 (ScriptActor), 3 (AlarmActor) |
|
||||
| N3 — `TagSubscribeRetryIntervalMs` / `StuckScriptGraceMs` missing from the eager options validator | Low | 4 (SiteRuntime knobs only — Transport → R2-05, AuditLog/Host → R2-08 per the cross-plan ruling) |
|
||||
| N4 — every deploy compiles each script twice (synchronous gate + `PreStart` recompile), no compile cache | Low | 5, 6 — **full adoption ships here** (one compile per script body per process; gate, PreStart, and SharedScriptLibrary all funnel through the cached `CompileCore`), so no deferred-register remainder. The gate itself stays synchronous by design (N5). |
|
||||
| N5 — spec drift (reintroduced): `Component-SiteRuntime.md:489` says the gate runs "off-thread"; the shipped gate is deliberately synchronous | Low | 7 (spec sentence + the matching stale `DeployCompileValidator` XML doc) |
|
||||
| N6 — Expression triggers/alarms share the bounded script scheduler; saturation silently stalls all Expression evaluation site-wide | Low | 7 (documented as the accepted trade-off the report judged it to be — gauges/watchdog already make it visible; no behavior change) |
|
||||
| P5 — per-attribute script read is an Ask round-trip (round-1 deferral) | Low | **Deferred (accepted, unchanged)** — spec-consistent, no profiling evidence; batch `GetAttributes` remains the profiling-gated follow-up. No task. |
|
||||
| P6 — Roslyn compiles inside `InstanceActor.PreStart` during staggered startup (round-1 deferral) | Low | **Deferred (accepted, further mitigated)** — Tasks 5/6's cache dedupes identical bodies within a warm process (the report's own observation); the first compile after process start still runs in `PreStart`, and moving it out stays out of scope. Doc note refreshed in 7. |
|
||||
| UA1 removal-reconcile / cross-node list-aggregation sub-scope (round-1 residue) | — | **Deferred** behind the documented central-persistence-of-cert-trust follow-up (`Component-SiteRuntime.md:125`). No task. |
|
||||
| S1–S10, P1–P4, C2–C6, UA1–UA7 (round-1 findings) | — | **Verified fixed** by the round-2 report's disposition table (26 fixed, evidence-read) — no action. |
|
||||
Reference in New Issue
Block a user