docs(archreview): round-2 re-review (2026-07-12) + 8 fix plans (86 tasks)
Re-ran all 8 domain reviews at HEAD8c888f13against theb910f5ebbaseline: every round-1 finding source-verified (168 fixed, 0 regressions, 0 false claims); 56 new findings (1 Critical / 4 High / 15 Medium / 36 Low), concentrated in post-baseline code (anti-entropy resync, KPI rollup backfill, live alarm stream) and seams the fixes exposed. Headliners: S&F resync predicate inversion can wipe the delivering node's buffer (02-N1 Critical); resync snapshot exceeds the Akka remoting frame size (02-N2); failover drill kills the one node keep-oldest can't survive (01-N1); unbounded rollup backfill per failover (04-R1); live production API key in untracked test.txt (08-NF1). Adds PLAN-R2-01..08 + .tasks.json manifests and the Round-2 board, P0 list, cross-plan mutexes, and wave order in 00-MASTER-TRACKER.
This commit is contained in:
Joseph Doherty
@@ -1,155 +1,94 @@
|
||||
# Architecture Review 03 — Site Runtime & Data Connection Layer
|
||||
# Architecture Review 03 — Site Runtime & Data Connection Layer (Round 2)
|
||||
|
||||
Reviewer domain: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime`, `src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer`, `src/ZB.MOM.WW.ScadaBridge.SiteEventLogging`, matched against `docs/requirements/Component-SiteRuntime.md`, `Component-DataConnectionLayer.md`, `Component-SiteEventLogging.md`, and `HighLevelReqs.md`. All findings are grounded in direct reads of the cited files.
|
||||
|
||||
## Scope
|
||||
|
||||
- **Site Runtime actors**: `DeploymentManagerActor` (1,720 LOC), `InstanceActor` (1,572), `ScriptActor` (594), `ScriptExecutionActor` (305), `AlarmActor` (780), `AlarmExecutionActor`, `NativeAlarmActor` (448), `CertStoreActor` (189), `SiteReplicationActor` (292), `SiteReconciliationActor`.
|
||||
- **Script subsystem**: `ScriptCompilationService`, `SharedScriptLibrary`, `ScriptExecutionScheduler`, `ScriptRuntimeContext` (2,384 — skimmed for Ask/blocking patterns), trigger expressions.
|
||||
- **Streaming**: `SiteStreamManager` (BroadcastHub + per-subscriber buffers).
|
||||
- **Persistence**: `SiteStorageService` (site SQLite: deployed configs, static overrides, artifacts, `native_alarm_state`), `OperationTrackingStore` (not deep-read).
|
||||
- **DCL**: `DataConnectionManagerActor` (345), `DataConnectionActor` (2,000), `OpcUaDataConnection` (401), `RealOpcUaClient` (1,313 — lifecycle sections), `MxGatewayDataConnection` (297 — connect path), alarm mappers, browse/search/verify paths.
|
||||
- **Site Event Logging**: `SiteEventLogger` (326), purge/query services.
|
||||
- **Tests**: inventoried `tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests`, `...DataConnectionLayer.Tests`, `...SiteEventLogging.Tests`.
|
||||
|
||||
## Maturity Verdict
|
||||
|
||||
This is a mature, unusually carefully engineered subsystem — the Become/Stash state machine, in-flight subscribe bookkeeping, adapter-generation guards, seed-read retry, redeploy termination-watch buffering, and per-source native-alarm cap/eviction all show evidence of multiple hardening passes, and the code matches the design docs closely with extensive rationale comments and broad targeted test coverage. The residual risk is concentrated in three places: **adapter lifecycle on reconnect** (the previous OPC UA / MxGateway client object is silently abandoned on every `ConnectAsync`, leaking sessions and leaving a stale `ConnectionLost` handler that can flap a healthy connection), **script containment** (the execution timeout is cooperative-only, so a hung blocking script permanently consumes one of the 8 dedicated script threads; trigger expressions block actor threads on the default dispatcher), and **deployment honesty** (a site-side script compile failure does not reject the deployment as the spec requires — the instance starts with the script silently missing and central is told Success). None of these break the happy path; all of them are the kind of defect that surfaces weeks into production under flaky networks or a bad script.
|
||||
**Date:** 2026-07-12
|
||||
**Scope:** `src/ZB.MOM.WW.ScadaBridge.SiteRuntime` (#3: DeploymentManager singleton, Instance/Script/Alarm/NativeAlarm actors, script execution scheduler, site-wide Akka stream), `src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer` (#4: OPC UA + MxGateway adapters, connection actors, alarm subscription seam, CertStoreActor), `src/ZB.MOM.WW.ScadaBridge.SiteEventLogging` (#12); matched against `docs/requirements/Component-SiteRuntime.md`, `Component-DataConnectionLayer.md`, `Component-SiteEventLogging.md`.
|
||||
**Method:** Re-verified every round-1 finding against current source (HEAD `8c888f13`) with file:line evidence — plan claims were not trusted; each fix was read in the code. Fresh sweep over `git diff b910f5eb..HEAD` scoped to the three projects (38 files, +2,205/−640) covering the PLAN-03 fix commits, the PLAN-08 options-validator/notification-excision commits, and the 2026-07-10 aggregated-live-alarm-stream site-side piece (`SubscribeSiteAlarms`).
|
||||
**Round-1 vs round-2 verdict:** Round 1 found a mature subsystem with three concentrated risks (adapter reconnect lifecycle, script containment, deployment honesty); round 2 confirms all three are genuinely fixed in code — 26 of 28 actionable findings verified fixed, 2 deferred with sound rationale — leaving only one Medium residual (a stale MxGateway event-loop fault can still flap a fresh connection through the generic catch) and a handful of Low polish items.
|
||||
|
||||
---
|
||||
|
||||
## Findings — Stability
|
||||
## Round-1 Finding Disposition
|
||||
|
||||
### S1. [High] Reconnect abandons the previous protocol client undisposed, and its stale `ConnectionLost` handler can flap a healthy connection
|
||||
`OpcUaDataConnection.ConnectAsync` (`src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Adapters/OpcUaDataConnection.cs:99-101`) does:
|
||||
Every disposition below was verified by direct read of the cited current source, not the plan's claims.
|
||||
|
||||
```csharp
|
||||
_client = _clientFactory.Create();
|
||||
_client.ConnectionLost += OnClientConnectionLost;
|
||||
await _client.ConnectAsync(...);
|
||||
```
|
||||
| Finding | R1 severity | Disposition | Evidence (file:line) |
|
||||
|---|---|---|---|
|
||||
| S1 — reconnect abandons previous client; stale `ConnectionLost` flaps healthy connection | High | **Fixed (verified)** — with one Medium residual on the MxGateway fault path (new finding N1) | `OpcUaDataConnection.cs:99-118` detaches `ConnectionLost`, fire-and-forget-disposes the previous client with fault logging, and `StopHeartbeatMonitor()` (:107) kills the old `StaleTagMonitor`; `MxGatewayDataConnection.cs:71-91` cancels+disposes the previous event loop/client before the guard reset. Tests exist: `OpcUaDataConnectionTests.cs` (`Reconnect_DisposesAndDetachesPreviousClient`, `StaleClientConnectionLost_…`), `Adapters/MxGatewayDataConnectionReconnectTests.cs` |
|
||||
| S2 — cooperative-only timeout; no stuck-thread observability | High | **Fixed (verified)** (observability shipped; ADO.NET command-timeout sub-suggestion deferred with rationale — scripts own their commands, 30 s default) | Gauges: `ScriptExecutionScheduler.cs:29,80-113` (`QueueDepth`/`BusyThreadCount`/`OldestBusyAge` via per-worker `_busySinceTicks`, volatile writes in `WorkerLoop` :121-128). Watchdog: `ScriptExecutionActor.cs:131-154` names the script whose thread never came back after timeout+`StuckScriptGraceMs`, logs + site event; `completed` flipped in the finally (:326). Reporter: `ScriptSchedulerStatsReporter.cs:71-74`, registered `ServiceCollectionExtensions.cs:76-79`; health surface `SiteHealthCollector.cs:151-162,260-262`; additive `SiteHealthReport.cs:65-75` |
|
||||
| S3 — site compile failure doesn't reject deployment (spec drift) | High | **Fixed (verified)** — deliberate deviation: gate is synchronous on the singleton, not off-thread (see N4/N5) | `DeployCompileValidator.cs:37-116` compiles every script, alarm on-trigger script, and Expression trigger (malformed JSON is itself an error, fronting the poison-config case); gate at `DeploymentManagerActor.cs:508-537` replies `DeploymentStatus.Failed` with the collected errors and returns before any actor creation or persistence. Synchronous-on-mailbox rationale documented in code (:510-521): off-thread piping would reorder deploys vs. delete/disable and break the redeploy-supersede FIFO. Test: `DeploymentManagerActorTests.cs` (`Deploy_WithNonCompilingScript…`) |
|
||||
| S4 — no retry on failed tag subscription; NativeAlarm lost-response gap | Medium | **Fixed (verified)** — stronger than asked: retry arms on *send*, so lost responses are covered too | `InstanceActor.cs:1047-1068` `SendTagSubscribe` arms a per-connection `TagSubscribeRetryIntervalMs` timer on every send with correlation-id supersede; `HandleSubscribeTagsResponse` (:1075-1099) cancels on Success, logs + emits a site event and leaves the timer armed on failure; timers cancelled in PostStop (:296-298). `NativeAlarmActor.cs:114,128,147,319` — response-timeout retry armed in `SendSubscribe`, cancelled on Success |
|
||||
| S5 — startup SQLite load failure aborts silently, no retry | Medium | **Fixed (verified)** | `DeploymentManagerActor.cs:286-290,345-356,400-415` — failed load schedules `RetryStartupLoad` at a fixed interval (default 5 s, ctor-overridable), unlimited, with a startup-complete flag so a stale retry cannot re-run after success |
|
||||
| S6 — dead child ref in `_instanceActors` + Success reported for dead instance | Medium | **Fixed (verified)** — redesigned beyond the plan: two-signal deploy join, not swallow-only | Every Instance Actor is watched (`DeploymentManagerActor.cs:1995`); unexpected `Terminated` evicts the dead ref (:618+). Deploy `Success` requires `InFlightDeploy.Persisted && Initialized` (:872-892, class :2093-2104) where `Initialized` is the new `InstanceActorInitialized` readiness message sent at the end of `InstanceActor.PreStart` (`InstanceActor.cs:280-286`, `Messages/InstanceActorInitialized.cs`); init-failure row rollback is deferred until the store commits (`_initFailedPendingRowRemoval`, :135,657,824) so it can never race the optimistic write. Design-doc capture: `Component-SiteRuntime.md:109` |
|
||||
| S7 — background tasks read mutable `_adapter` field off-thread | Medium | **Fixed (verified)** | Adapter captured into a local on the actor thread before every background task: `DataConnectionActor.cs:704` (subscribe), `:1155` (write), `:1199` (trigger write), `:1638` (re-subscribe, symmetric with the pre-existing `reseedAdapter`) |
|
||||
| S8 — site SQLite: no WAL/busy-timeout | Low | **Fixed (verified)** | `SiteStorageService.cs:23` (busy-timeout floor pinned in the normalized connection string), `:63-74` (`PRAGMA journal_mode=WAL` set once in `InitializeAsync`, warning logged on fallback) |
|
||||
| S9 — fire-and-forget adapter dispose in `PostStop` unobserved | Low | **Fixed (verified)** | `DataConnectionActor.cs:232-237` — `ContinueWith(OnlyOnFaulted, log)` on the dispose task |
|
||||
| S10 — alarm condition filter last-subscriber-wins, silent overwrite | Low | **Fixed (verified)** | `DataConnectionActor.cs:1840-1851` — warning logged when an incoming filter differs from the stored one; filter also parsed once into `_alarmSourceFilterPredicate` (:126) |
|
||||
| P1 — trigger expressions block dispatcher threads per attribute change | High | **Fixed (verified)** — with one Low residual (new finding N2) | `ScriptActor.cs:281-334` and `AlarmActor.cs:520-577`: evaluation runs on `ScriptExecutionScheduler.Shared` against a point-in-time snapshot, result piped back as `ExpressionEvalResult`, edge state (`_lastExpressionResult`, WhileTrue timers) applied only on the actor thread, bursts coalesced (one in flight + one pending) |
|
||||
| P2 — attribute-change fan-out broadcast to every child | Medium | **Fixed (verified)** | New `Scripts/TriggerRouting.cs` (three-way monitored/all-changes/none classification, fail-open on unparseable config); `InstanceActor.PublishAndNotifyChildren` (:1270-1289) routes to `_allChangeChildren` + `_childrenByMonitoredAttribute[attr]` only; `RouteChild` files children at creation (:1451-1469, wired at :1516/:1576) |
|
||||
| P3 — DCL tag-value fan-out scans all instances per update | Medium | **Fixed (verified)** | `DataConnectionActor.cs:1745-1764` `IndexTag`/`UnindexTag` maintain the `_instancesByTag` reverse index at every subscription mutation site (incl. instance release, :1076); `HandleTagValueReceived` (:1778-1790) fans out O(subscribers-of-tag) |
|
||||
| P4 — per-transition SQLite upserts on native-alarm mirror | Low | **Fixed (verified)** | `NativeAlarmActor.cs:57-70,115,469-476` — latest-transition-per-source coalescing map, batched flush on a 100 ms timer (`FlushPersistence`), best-effort final flush in PostStop (:139-143) |
|
||||
| P5 — per-attribute script read is an Ask round-trip | Low | **Deferred (accepted)** | `ScriptRuntimeContext` unchanged on this path; plan rationale (spec-consistent, no profiling evidence, batch `GetAttributes` as follow-up) holds |
|
||||
| P6 — Roslyn compiles inside `InstanceActor.PreStart` during staggered startup | Low | **Deferred (accepted, partially mitigated)** | Compiles still run in `PreStart` → `CreateChildActors` (`InstanceActor.cs:260-280,1472-1499`); the deploy-path gate (S3) validates earlier but PreStart recompiles — see new finding N4. Risk remains failover time-to-recover only |
|
||||
| C1 — positive: Akka discipline | — | Still true; the new code keeps the discipline (PipeTo everywhere, captured `Self`, actor-confined state) | e.g. `ScriptActor.cs:289` captured `self`; `DataConnectionActor.cs:704` captured adapter |
|
||||
| C2 — stale "unbounded channel" comment in SiteEventLogger | Low | **Fixed (verified)** | `SiteEventLogger.cs:21-22,69,215-216` — comments now correctly describe bounded/DropOldest semantics |
|
||||
| C3 — spec self-contradiction on reconnect-interval scope | Low | **Fixed (verified)** | `Component-DataConnectionLayer.md:304` — "shared setting for all data connections (`ReconnectInterval` in the Shared Settings table)" |
|
||||
| C4 — quality-only change not delivered to children vs. spec wording | Low | **Fixed (verified)** | `Component-DataConnectionLayer.md:303` — spec now states children are "deliberately NOT re-notified on a quality-only change" with the spurious-firing rationale; divergence is now a documented decision |
|
||||
| C5 — shared scripts run Root scope, undocumented | Low | **Fixed (verified)** | `Component-SiteRuntime.md:344` — "Shared scripts always execute with Root scope (C5)… Callers needing module-scoped access must pass the values in as parameters" |
|
||||
| C6 — `EvaluateCondition` fallback uses current-culture `ToString` | Low | **Fixed (verified)** | `ScriptActor.cs:513-516` — threshold rendered with `CultureInfo.InvariantCulture`, `StringComparison.Ordinal` |
|
||||
| UA1 — cert-trust convergence after node downtime | Underdev. | **Fixed (verified)** — additive reconcile-on-join; removal-reconcile + cross-node list aggregation remain deferred to the documented central-persistence follow-up | `CertStoreActor.cs:139+` exports the trusted set (thumbprint+DER); `DeploymentManagerActor.cs:277-281,321-328` singleton subscribes to `ClusterEvent.MemberUp`; `:1296-1365` re-reads the local store and pushes each cert to the joiner's CertStoreActor (idempotent by thumbprint, additive union only, fire-and-forget with heal-on-next-join). Divergence direction is always singleton→joiner since trust commands flow through the singleton, so the additive push closes the real hole |
|
||||
| UA2 — `ConfigFetchRetryCount` dead option | Underdev. | **Fixed (verified)** | `SiteRuntimeOptions.cs:68` (default 3), consumed at `SiteReplicationActor.cs:86` (floor 1) with a fixed 2 s inter-attempt delay (:270); validated ≥ 0 (`SiteRuntimeOptionsValidator.cs:56-58`) |
|
||||
| UA3 — no aggregated live alarm stream | Underdev. | **No longer applicable — shipped post-plan (2026-07-10)** | Deferred by PLAN-03 as central-scope; the aggregated-live-alarm-stream plan delivered it. Site-side piece reviewed: `SiteStreamManager.SubscribeSiteAlarms` (`SiteStreamManager.cs:134-159`) — alarm-only filter off the same BroadcastHub, per-subscriber DropHead buffer, KillSwitch teardown shared with `Unsubscribe`/`RemoveSubscriber`; clean |
|
||||
| UA4 — native-alarm rehydration loses display metadata | Underdev. | **Fixed (verified)** | `SiteStorageService.cs:174-175,975` — `MetadataJson` (type/category/message/values) persisted alongside `condition_json` and restored on rehydration |
|
||||
| UA5 — no scheduler stuck-thread/queue-depth observability | Underdev. | **Fixed (verified)** (same fix as S2) | See S2 row |
|
||||
| UA6 — tag-subscription failure path has no retry/site event | Underdev. | **Fixed (verified)** (same fix as S4; site event emitted at `InstanceActor.cs:1093-1096`) | See S4 row |
|
||||
| UA7 — test gaps (reconnect-stale-handler, compile-fail deploy status, subscribe race) | Underdev. | **Fixed (verified)** | `OpcUaDataConnectionTests.cs` (both S1 tests present), `Adapters/MxGatewayDataConnectionReconnectTests.cs`, `DeploymentManagerActorTests.cs` (compile-fail deploy + `InstanceActorInitialized` handshake tests) |
|
||||
|
||||
The **previous** `_client` is neither `DisposeAsync`'d nor detached. `DataConnectionActor`'s reconnect loop calls `_adapter.ConnectAsync` repeatedly on the *same adapter* (`DataConnectionActor.cs:537-547`); only the failover branch replaces the adapter. `MxGatewayDataConnection.ConnectAsync` has the same pattern (`MxGatewayDataConnection.cs:70-88`) and additionally overwrites `_eventLoopCts` without cancelling the previous event loop.
|
||||
|
||||
Failure scenarios:
|
||||
1. **Session/socket leak per disconnect cycle.** After a keep-alive-detected drop, the old `RealOpcUaClient` still holds its `ISession` + subscription (`RealOpcUaClient.cs:133-160`); `Session.CloseAsync` only runs from `DisconnectAsync` (`RealOpcUaClient.cs:392-396`), which nothing calls on this path. A site whose PLC link flaps hourly accumulates orphaned SDK session objects, keep-alive timers, and sockets indefinitely.
|
||||
2. **Spurious disconnect of a healthy connection.** If the drop was detected *reactively* (a `ReadAsync` failure → `RaiseDisconnected`, `OpcUaDataConnection.cs:222-227`), the old client's own once-only guard (`_connectionLostFired`) has **not** fired. After a successful reconnect resets the adapter's `_disconnectFired` (`OpcUaDataConnection.cs:104`), the abandoned session's keep-alive eventually fails, fires the *old* client's `ConnectionLost`, which is still wired to `OnClientConnectionLost` → `RaiseDisconnected()` → `AdapterDisconnected` → `BecomeReconnecting()` on a perfectly healthy new session. Each such cycle abandons another client — self-sustaining reconnect churn.
|
||||
|
||||
Fix shape: in both adapters' `ConnectAsync`, detach + dispose the old client before creating the new one (mirroring what the actor's failover branch already does at the adapter level, `DataConnectionActor.cs:421-431`).
|
||||
|
||||
### S2. [High] Script execution timeout is cooperative-only; hung scripts permanently consume the bounded script-thread pool
|
||||
`ScriptExecutionActor` runs the body on the dedicated `ScriptExecutionScheduler` (default **8 threads**, `SiteRuntimeOptions.cs:46`) and enforces the timeout via `CancellationTokenSource(timeout)` passed to `compiledScript.RunAsync` (`ScriptExecutionActor.cs:129, 246`). Roslyn only observes the token between statements; a script blocked in synchronous I/O (`Database.Connection` ADO.NET call, a socket in a wedged state) or a CPU loop never observes cancellation. The `OperationCanceledException` path replies/logs, but the *thread* is only released when the blocking call itself returns. `ScriptExecutionScheduler.WorkerLoop` (`ScriptExecutionScheduler.cs:71-84`) has no watchdog, no thread replacement, no "N tasks stuck > timeout" metric.
|
||||
|
||||
Failure scenario: a site DB connection black-holes (firewall drops packets, no RST). Eight interval-triggered scripts each block ~30+ min in a synchronous read. All subsequent script *and alarm on-trigger* executions site-wide queue in `_queue` forever — alarms still transition (evaluation is on actor threads) but no on-trigger scripts run, and nothing on the health dashboard says why. The spec's claim that "Exceeding the timeout cancels the script" (`Component-SiteRuntime.md:411`) overstates what is enforced.
|
||||
|
||||
Mitigations worth considering: a scheduler gauge (queue depth + busy-thread age) surfaced through `ISiteHealthCollector`; per-script-thread stuck detection that logs the script name; ADO.NET command timeouts injected into `Database.Connection`.
|
||||
|
||||
### S3. [High] Spec drift: site-side script compile failure does not reject the deployment — central is told Success with the script silently missing
|
||||
`Component-SiteRuntime.md:474-477`: "If script compilation fails when a deployment is received, the entire deployment for that instance is **rejected**. No partial state is applied. The failure is reported back to central as a failed deployment."
|
||||
|
||||
Implementation: `InstanceActor.CreateChildActors` logs and `continue`s past a failed compile (`InstanceActor.cs:1362-1369`; same for alarm on-trigger scripts at 1418-1423 and trigger expressions at 1524-1531). The instance starts with the failing script absent; `DeploymentManagerActor.HandleDeployPersistenceResult` reports `DeploymentStatus.Success` purely on SQLite persistence (`DeploymentManagerActor.cs:596-610`). The XML doc on `CreateChildActors` even states the contradiction: "Compilation errors reject entire instance deployment (logged but actor still starts)" (`InstanceActor.cs:1339`).
|
||||
|
||||
The design-time gate in Template Engine makes this rare, but the spec explicitly calls the site the last line ("Site-side compilation failures indicate an unexpected issue") — and version skew between central's Roslyn surface and the site's is exactly the case where the gates disagree. Today the operator sees a green deploy and a dead script discoverable only in the site event log.
|
||||
|
||||
### S4. [Medium] Instance actor never retries a failed tag subscription; a subscribe/create ordering race strands attributes at Uncertain forever
|
||||
`InstanceActor` registers `Receive<SubscribeTagsResponse>(_ => { })` — an explicit no-op (`InstanceActor.cs:207`). If the DCL replies `Success=false` there is **no retry and no logging**. Two ways to get there:
|
||||
|
||||
- `DataConnectionManagerActor.HandleRoute` replies "Unknown connection" when the `SubscribeTagsRequest` arrives before the `CreateConnectionCommand` has been processed (`DataConnectionManagerActor.cs:104-115`). The Deployment Manager Tells `CreateConnectionCommand` and then creates the Instance Actor (`DeploymentManagerActor.cs:543-546`), whose `PreStart` Tells `SubscribeTagsRequest` — but the two messages come from **different senders**, so Akka's per-sender-pair ordering guarantee does not formally order them at the manager's mailbox. If the race fires, the instance's data-sourced attributes stay Uncertain until redeploy/restart.
|
||||
- The connection-level-failure reply ("connection unavailable — will re-subscribe on reconnect", `DataConnectionActor.cs:1009-1012`) is self-healing because `_subscriptionsByInstance` was already updated and `ReSubscribeAll` re-derives from it — that case is fine. The unknown-connection case leaves **no state anywhere**.
|
||||
|
||||
Contrast: `NativeAlarmActor` retries a failed subscribe on a timer (`NativeAlarmActor.cs:269-286`). The tag path deserves the same treatment — at minimum, log the failure and schedule a bounded re-subscribe.
|
||||
|
||||
Related sub-gap: `NativeAlarmActor`'s retry only arms on a *failure response*. A subscribe whose response is **lost** (DCL manager restarted between request and reply; message dead-lettered) never retries — `SendSubscribe` is a bare `Tell` with no response timeout (`NativeAlarmActor.cs:123-131`). Rehydrated SQLite state masks the gap in the debug view while the mirror is actually dead.
|
||||
|
||||
### S5. [Medium] Deployment Manager startup aborts silently on a SQLite read failure — site runs zero instances with no retry
|
||||
`HandleStartupConfigsLoaded`: on error, `_logger.LogError(...); return;` (`DeploymentManagerActor.cs:291-297`). One transient SQLite failure (locked file during a crash-recovery race, disk hiccup) at singleton start/failover leaves the active node with **no Instance Actors** until a human restarts the process. There is no retry timer, no health flag beyond the log line (health instance counts will read 0 deployed, which at least shows on the dashboard, but nothing distinguishes "genuinely nothing deployed" from "startup load failed"). A bounded retry (the same `Timers.StartSingleTimer` idiom used two methods later) would close this.
|
||||
|
||||
### S6. [Medium] A throwing `InstanceActor` constructor leaves a dead ref in `_instanceActors` and still reports the deployment Success
|
||||
`InstanceActor`'s constructor deserializes the config JSON (`InstanceActor.cs:138`) and builds the resolved-attribute index; a malformed/oversized config throws → `ActorInitializationException` → Deployment Manager's decider Stops the child (`DeploymentManagerActor.cs:268-274`). But:
|
||||
- `CreateInstanceActor` already stored the ref in `_instanceActors` (`DeploymentManagerActor.cs:1628-1629`) and nothing watches for this failure mode (the `Context.Watch` only happens on the redeploy path), so the map holds a dead ref — debug view, inbound-API routes, etc. dead-letter with 30s Ask timeouts instead of "not found".
|
||||
- `ApplyDeployment`'s persistence continues and `HandleDeployPersistenceResult` reports `Success` (`DeploymentManagerActor.cs:596-610`) — central shows a healthy deployment for an instance that is not running.
|
||||
|
||||
A `Context.Watch` on every created child (removing from `_instanceActors` on unexpected `Terminated`) would fix both the stale map and give a hook to fail the pending deployment.
|
||||
|
||||
### S7. [Medium] Subscribe background task reads the mutable `_adapter` field off-thread, violating its own confinement contract
|
||||
`HandleSubscribe`'s `Task.Run` body awaits `_adapter.SubscribeAsync(...)` per tag and calls `SeedTagsAsync(_adapter, ...)` (`DataConnectionActor.cs:707-758`) — reading the actor field from a thread-pool thread, despite the comment eight lines earlier: "The background task below must NOT read or mutate actor state — these partitioned lists are the only state it sees" (`DataConnectionActor.cs:689-690`). If a failover swaps `_adapter` mid-loop (`CountFailureAndMaybeFailover`, `DataConnectionActor.cs:610-635`), later iterations subscribe on the **new** adapter while stamping the **old** generation — the resulting values are dropped by the generation guard and the monitored items are orphaned on the new adapter (never in `_subscriptionIds` for that generation… actually worse: `SubscribeCompleted` will store them against current state, mixing generations). `ReSubscribeAll` (`DataConnectionActor.cs:1596`) and `HandleWriteBatch`'s local async function (`DataConnectionActor.cs:1161-1201`) have the same field-read pattern. Capture the adapter into a local before spawning the task (as the re-seed block at `DataConnectionActor.cs:1620` correctly does with `reseedAdapter`).
|
||||
|
||||
### S8. [Low] Site SQLite: connection-per-call, no WAL / busy-timeout tuning, many concurrent writers
|
||||
`SiteStorageService` opens a fresh `SqliteConnection` per operation with a bare `Data Source=` string (`SiteStorageService.cs:34-44, 163-166`) — no `journal_mode=WAL`, relying on Microsoft.Data.Sqlite's default command-timeout-as-busy-handler. Concurrent writers against the same file include: deploy persistence, static-override fire-and-forget writes (`InstanceActor.cs:433-441`), per-transition native-alarm upserts (`NativeAlarmActor.cs:420-428`), replication applies (`SiteReplicationActor.cs:182-216`), and reconciliation. Under a native-alarm storm plus a deploy, rollback-journal locking serializes everything and a slow write can push others toward `SQLITE_BUSY`. WAL + explicit busy_timeout is a one-line hardening. (Contrast: `SiteEventLogger` got the full treatment — single owned connection, lock, bounded write queue.)
|
||||
|
||||
### S9. [Low] Fire-and-forget adapter disposal in `PostStop`
|
||||
`DataConnectionActor.PostStop` discards the `DisposeAsync` task (`DataConnectionActor.cs:218-221`); a hanging/faulting dispose (OPC UA `CloseAsync` against a dead server can block for the operation timeout) is unobserved. Acceptable trade-off (documented elsewhere in the file), but pair it with a `ContinueWith(OnlyOnFaulted, log)` like the other fire-and-forget paths in this codebase do.
|
||||
|
||||
### S10. [Low] Alarm condition filter is last-subscriber-wins across co-subscribers of one source
|
||||
`_alarmSourceFilter[request.SourceReference] = request.ConditionFilter` overwrites unconditionally (`DataConnectionActor.cs:104-108, 1763-1766`); two instances binding the same source with different filters silently re-gate each other's transitions. The field's XML doc honestly declares the sharp edge, but nothing *validates* agreement or logs the overwrite — a mismatch is undiagnosable in the field. Log a warning when the incoming filter differs from the stored one.
|
||||
**Disposition counts:** 26 Fixed (verified) · 2 Deferred (accepted: P5, P6) · 0 Not fixed · 0 Regressed. (UA3 counted Fixed via the post-plan 2026-07-10 delivery; UA1's removal-reconcile sub-scope remains an explicitly documented deferral inside an otherwise-fixed item.)
|
||||
|
||||
---
|
||||
|
||||
## Findings — Performance
|
||||
## New Findings (Round 2)
|
||||
|
||||
### P1. [High] Trigger expressions block actor threads on the default dispatcher — per attribute change
|
||||
`ScriptActor.EvaluateExpressionTrigger` runs the compiled Roslyn expression with `.GetAwaiter().GetResult()` under a 2-second CTS (`ScriptActor.cs:283-287`); `AlarmActor.EvaluateExpression` is identical (`AlarmActor.cs:503-506`). Both execute **on the actor's default-dispatcher thread**, and both run on *every* `AttributeValueChanged` the actor receives (Expression triggers have no monitored-attribute gate — `ScriptActor.cs:255-258`, `AlarmActor.cs:215-218`). A site with a handful of Expression-triggered scripts/alarms on a chatty instance (e.g., 50 tags at 1 Hz) synchronously executes 100s of Roslyn script invocations per second on dispatcher threads; one pathological expression (2 s timeout, treated-as-false) stalls its actor's mailbox and steals a dispatcher thread for the duration. The `ScriptExecutionScheduler` exists precisely to keep script code off these threads — trigger expressions bypass it. The in-code justification ("trusted Design-role users, compile-checked") addresses malice, not cost.
|
||||
### N1. [Medium] MxGateway: a stale event loop's *non-OCE* fault can still flap the fresh connection — the S1 fix closed the OPC UA path but left this one open
|
||||
The S1 fix in `MxGatewayDataConnection.ConnectAsync` cancels the previous loop's CTS and disposes the previous client (`MxGatewayDataConnection.cs:77-89`), then resets the disconnect guard on the very next line (`Interlocked.Exchange(ref _disconnectFired, 0)`, `:91`) — *before* the old loop has actually observed the cancellation. The comment claims a teardown fault "is absorbed by the old guard cycle", but `Cancel()` is not synchronous with the loop's exit: the old loop's `await foreach (… _session!.StreamEventsAsync(_lastSeq, ct))` (`RealMxGatewayClient.cs:278-294`) surfaces its failure milliseconds later, while the new `ConnectAsync` is still awaiting. `RunEventLoopAsync`'s catch (`MxGatewayDataConnection.cs:121-129`) only absorbs `OperationCanceledException`; a cancelled/disposed gRPC stream commonly faults with `RpcException(StatusCode.Cancelled)` (Grpc.Net default without `ThrowOperationCanceledOnCancellation`, which the wrapped `MxGatewayClient` package is not shown to set) or `ObjectDisposedException` from the concurrent `DisposeAsync` — both fall into the generic `catch` → `RaiseDisconnected()` against the *reset* guard → `AdapterDisconnected` → reconnect → which cancels/disposes again → potentially self-sustaining churn, the exact S1 scenario-2 mechanism. Unit tests can't catch it (the reconnect test substitutes the client and never faults the loop), and there is no real gateway in the local env.
|
||||
**Fix shape:** in the generic catch, treat `ct.IsCancellationRequested == true` as normal shutdown (no `RaiseDisconnected`). Secondary hardening in the same method: `Task.Run(() => RunEventLoopAsync(_eventLoopCts.Token))` (`:108`) reads the `_eventLoopCts` *field* lazily — a rapid double-reconnect can hand loop #1 loop #2's token; capture the token into a local before the lambda.
|
||||
|
||||
### P2. [Medium] Attribute-change fan-out is broadcast to every child, filtering in the receiver
|
||||
`InstanceActor.PublishAndNotifyChildren` Tells **every** Script Actor and **every** Alarm Actor for every attribute change (`InstanceActor.cs:1184-1194`); `Component-SiteRuntime.md:191` describes children subscribing "for the specific monitored attribute". An instance with 30 scripts + 20 alarms and 100 Hz aggregate tag updates generates 5,000 child messages/sec, ~98% of which are discarded by the child's trigger check (`ScriptActor.cs:229-233`, `AlarmActor.cs:219-222`) — pure mailbox/dispatch overhead, plus each `ScriptActor` maintains an `_attributeSnapshot` copy of every value (`ScriptActor.cs:227`). A per-attribute subscription map on the Instance Actor (attributeName → interested children; Expression-trigger children in an "all" bucket) would cut this by an order of magnitude on realistic configs.
|
||||
### N2. [Low] Trigger-expression `PipeTo` has a success mapping only — a faulted scheduler task permanently kills the actor's expression trigger
|
||||
`ScriptActor.StartExpressionEvaluation` (`ScriptActor.cs:311-313`) and `AlarmActor.StartExpressionEvaluation` (`AlarmActor.cs:552-554`) pipe the evaluation back with `PipeTo(self, success: r => new ExpressionEvalResult(r))` and no `failure:` mapping. The inner body catches all exceptions and returns `false`, so the only fault sources are the outer `Task.Factory.StartNew`/`QueueTask` itself (e.g. `ObjectDisposedException` from a disposed `ScriptExecutionScheduler` during shutdown or in test teardown). If that ever fires, the actor receives an unhandled `Status.Failure` and — critically — `_evalInFlight` stays `true` forever (`ScriptActor.cs:284-285`, `AlarmActor.cs:523-524`), so every subsequent attribute change parks in `_evalPending` and the expression trigger is dead for the actor's lifetime with no log. One-line fix: add `failure: ex => new ExpressionEvalResult(false)` (or a dedicated failure message that logs).
|
||||
|
||||
### P3. [Medium] DCL tag-value fan-out scans all instances per update
|
||||
`HandleTagValueReceived` iterates `_subscriptionsByInstance` (every instance) with a per-instance `HashSet.Contains` for every incoming tag value (`DataConnectionActor.cs:1706-1716`). The actor already maintains `_tagSubscriberCount` (tag → count) for O(1) unsubscribe; the matching tag → *instances* reverse index is missing. With 500 instances on one connection at high sample rates this is the hottest loop in the DCL. Also `_healthCollector.UpdateTagQuality` is invoked on **every** update (`DataConnectionActor.cs:1737`) — cheap if the collector is a plain field write, but worth confirming it isn't lock-contended.
|
||||
### N3. [Low] New PLAN-03 option knobs are missing from the eager options validator this same initiative introduced
|
||||
`SiteRuntimeOptionsValidator.cs:20-58` validates ten options, but not the two PLAN-03 added: `TagSubscribeRetryIntervalMs` (`SiteRuntimeOptions.cs:76`, drives `ScheduleTellOnceCancelable` in `InstanceActor.SendTagSubscribe` `:1065-1067` — a 0 hot-loops the subscribe retry, a negative value throws inside the actor) and `StuckScriptGraceMs` (`SiteRuntimeOptions.cs:86`, a negative value throws `ArgumentOutOfRangeException` inside the watchdog's `Task.Delay`, `ScriptExecutionActor.cs:144`). Inconsistent with the PLAN-08 §1.5 "fail fast at boot with a key-naming message" convention; two `RequireThat` lines close it.
|
||||
|
||||
### P4. [Low] Per-transition SQLite upserts on the native-alarm mirror
|
||||
`NativeAlarmActor.PersistUpsert` issues one async SQLite write per accepted transition (`NativeAlarmActor.cs:420-428`), each opening a fresh connection (S8). An A&C alarm storm (hundreds of transitions/sec across sources) turns into hundreds of single-row upsert connections/sec. Bounded by `MirroredAlarmCapPerSource` (1000) per source in state size but not in write rate. Batching/coalescing (persist latest per SourceReference on a short timer) would be a proportionate hardening if storms are expected.
|
||||
### N4. [Low] Every deploy now compiles each script twice — synchronously on the singleton mailbox at the gate, then again in `InstanceActor.PreStart` — with no compile cache
|
||||
The S3 gate runs `_deployCompileValidator.Validate` on the singleton's actor thread (`DeploymentManagerActor.cs:522`; deliberate and well-argued for mailbox-FIFO ordering, :510-521), and the created Instance Actor then recompiles the identical scripts/expressions in `CreateChildActors` (`InstanceActor.cs:1485,1497-1498`). `ScriptCompilationService` has no compile cache, so a site-wide redeploy of N instances serializes roughly 2×N script-set Roslyn compiles through the singleton + instance start path (each 50–300 ms). Single-instance deploys are fine; bulk redeploy latency and central Ask timeouts are the exposure. The master tracker already earmarks adopting PLAN-05's compile-cache abstraction for `DeployCompileValidator` ("later refactor, not a blocker") — this finding is the concrete reason to actually do it, and a keyed cache would also shrink the P6 failover-recompile cost for free.
|
||||
|
||||
### P5. [Low] Every script attribute read is an actor Ask round-trip
|
||||
`ScriptRuntimeContext` Asks the Instance Actor per `GetAttribute` (`ScriptRuntimeContext.cs:375`) — a temp actor allocation + two mailbox hops per read. Correct (serialized through the source of truth) and consistent with the spec, but a script reading 50 attributes in a loop pays 50 round-trips; a `GetAttributes` batch message (the DeploymentManager routing path already fans out N Asks — `DeploymentManagerActor.cs:1360-1367` — which would also benefit) is an easy win if profiling ever flags it.
|
||||
### N5. [Low] Spec drift (reintroduced): `Component-SiteRuntime.md` says the compile gate runs "off-thread" — the shipped gate is deliberately synchronous
|
||||
`Component-SiteRuntime.md:489`: "…the Deployment Manager compiles all of the instance's scripts … **off-thread**; any compile failure rejects the deployment…". The implementation intentionally validates *on* the singleton thread (`DeploymentManagerActor.cs:510-522`), a deviation captured in the master tracker and the in-code comment but never folded back into the component doc. Given this repo's docs-travel-with-code rule — and that round 1's S3 was itself a doc-vs-code honesty finding — the sentence should say "synchronously, as a pure prefix step of the deploy handler (preserving mailbox FIFO ordering)".
|
||||
|
||||
### P6. [Low] Roslyn compiles run inside `InstanceActor.PreStart` on the default dispatcher during staggered startup
|
||||
Each instance compiles all its scripts + alarm scripts + trigger expressions synchronously in `CreateChildActors` (called from `PreStart`, `InstanceActor.cs:250-255, 1348+`). The startup stagger (20 instances / 100 ms, `SiteRuntimeOptions.cs:13-19`) is tuned for OPC UA connection storms, not compile cost — 500 instances × several scripts each at ~50–300 ms/compile will occupy default-dispatcher threads for minutes after failover. The Deployment Manager already moved *shared*-script compilation off-thread (`DeploymentManagerActor.cs:1168-1191`); per-instance compiles did not get the same treatment. Failover time-to-recover is the metric at risk.
|
||||
### N6. [Low] Observation: trigger expressions now share the 8-thread bounded script scheduler with (possibly stuck) script bodies — saturation silently stalls all Expression triggers and Expression alarms site-wide
|
||||
The P1 fix routes every expression evaluation through `ScriptExecutionScheduler.Shared` (`ScriptActor.cs:312`, `AlarmActor.cs:553`). Before, a pathological expression stalled a dispatcher thread (bad) but expressions always ran; now, eight stuck script bodies (the S2 scenario) also mean **no Expression-triggered script or alarm evaluates anywhere on the site** until a thread frees — alarm activation latency becomes unbounded during scheduler saturation. The coalescing (one in-flight + one pending per actor) correctly bounds queue growth, and the new gauges/watchdog make the state visible, so this is an acceptable and arguably correct trade-off — but it is a behavioral coupling the component doc's Alarm section should state (one sentence: "Expression alarm evaluation shares the bounded script-execution pool; scheduler saturation delays alarm transitions").
|
||||
|
||||
---
|
||||
|
||||
## Findings — Conventions
|
||||
## What's Genuinely Good
|
||||
|
||||
### C1. [Positive] Akka.NET discipline is exemplary
|
||||
Worth stating because it is the dominant impression: Become/Stash exactly as the spec draws it (`DataConnectionActor.cs:232-533`); Tell on hot paths, Ask only at boundaries (`Component-SiteRuntime.md:430-442` ↔ implementation); `Sender`/`Self` captured before every continuation (e.g., `DeploymentManagerActor.cs:464-470`, `DataConnectionActor.cs:149-153`); all async results marshalled via `PipeTo`; supervision matches the spec table precisely — DM `OneForOne` Resume/Stop-on-init (`DeploymentManagerActor.cs:263-279`), Instance Resume (`InstanceActor.cs:281-293`), Script/Alarm coordinators Stop their execution children (`ScriptActor.cs:180-192`, `AlarmActor.cs:192-204`). The redeploy Terminated-watch buffering with the name-shadow map and its disable/delete supersede handling (`DeploymentManagerActor.cs:60-90, 389-435, 641-698, 762-831`) is genuinely hard concurrency done correctly. The in-flight subscribe/orphan-release guards in the DCL (`DataConnectionActor.cs:77-90, 836-985, 1795-1858`) close race classes most codebases never notice.
|
||||
|
||||
### C2. [Low] Stale comment: SiteEventLogger claims its channel is unbounded
|
||||
`SiteEventLogger.cs:216-218` — "The channel is unbounded, so the only way TryWrite fails is…" — the channel became bounded (`Channel.CreateBounded`, `SiteEventLogger.cs:74-88`), and `TryWrite` on a bounded DropOldest channel still only fails when completed, so the *behavior* is right but the stated reason is wrong. Trivial fix; flagged because this file's comments are otherwise load-bearing.
|
||||
|
||||
### C3. [Low] Spec is internally inconsistent on the reconnect interval
|
||||
`Component-DataConnectionLayer.md:303` says "The retry interval is defined **per data connection**", but the same document's Shared Settings table (`Component-DataConnectionLayer.md:170-177`) and the implementation (`DataConnectionOptions.ReconnectInterval`, one value for all connections — `DataConnectionActor.cs:463`) make it global. Fix the sentence at :303 (per the CLAUDE.md rule that the design doc travels with the code).
|
||||
|
||||
### C4. [Low] Quality-only change on disconnect is not delivered to Alarm/Script actors — deliberate, but diverges from the spec's wording
|
||||
`HandleConnectionQualityChanged` updates qualities and the stream but explicitly does **not** notify children ("firing scripts/alarms would cause spurious evaluations", `InstanceActor.cs:924-956`). The DCL spec promises "Instance Actors and their downstream consumers (**alarms, scripts checking quality**) see the staleness immediately" (`Component-DataConnectionLayer.md:302`). Practical consequence: an Active alarm computed from a now-dead connection stays Active (arguably correct — last-known-state), and a Conditional/Expression script never learns quality went Bad. The implementation choice is defensible; the spec sentence should be qualified so the divergence is a decision, not drift.
|
||||
|
||||
### C5. [Low] Shared scripts execute without the caller's scope and without a `Scope` assignment
|
||||
`SharedScriptLibrary.ExecuteAsync` builds `ScriptGlobals` without setting `Scope` (`SharedScriptLibrary.cs:98-103`), so a shared script always runs `ScriptScope.Root` (`ScriptCompilationService.cs:200-201`) even when invoked from a composed-module script. If a shared script uses `Attributes[...]`, it addresses **root** attributes regardless of caller scope. Possibly intended (shared scripts are instance-agnostic), but it is undocumented in `Component-SiteRuntime.md`'s Shared Script Library section and is a surprise waiting for a template author.
|
||||
|
||||
### C6. [Low] `EvaluateCondition` fallback compares `value.ToString()` to `Threshold.ToString()` with current-culture double formatting
|
||||
`ScriptActor.cs:474-477`: the non-numeric fallback stringifies the double threshold with the host culture (the numeric path was carefully invariant-culture'd; the fallback wasn't). A `"1.5"` threshold on a de-DE host renders `"1,5"`. Edge-case only (non-parseable values), but inconsistent with the deliberate invariant-culture work two lines up.
|
||||
- **The fix quality matches the codebase's standard.** Every PLAN-03 fix reads like the surrounding hardened code: retries use the existing fixed-interval cancelable-timer idiom with supersede correlation ids (`InstanceActor.cs:1047-1068`), all off-thread work returns via `PipeTo` with state applied on the actor thread, and health metrics ride the additive-only `SiteHealthReport` contract.
|
||||
- **Two fixes shipped *better* than the plan asked.** S4's retry arms on *send* (covering lost responses, not just failure replies — the plan's sub-gap), and S6 was redesigned mid-execution when the implementer discovered the store commits before `PreStart` throws: the `InstanceActorInitialized` two-signal join (`DeploymentManagerActor.cs:2085-2104`) plus commit-ordered rollback is deterministic where the plan's swallow-only guard would have silently kept the bug. The deviation is documented in the master tracker *and* the component doc (`Component-SiteRuntime.md:109`).
|
||||
- **The S3 synchronous-gate decision is correctly reasoned.** The in-code comment (`DeploymentManagerActor.cs:510-521`) explains precisely why off-thread validation would break the redeploy-supersede/delete ordering that depends on mailbox FIFO — the kind of concurrency reasoning this file consistently gets right. (It just needs the doc sentence fixed — N5.)
|
||||
- **Instrumentation is lock-free and cheap.** The scheduler gauges use per-worker `Volatile` slots read without locks (`ScriptExecutionScheduler.cs:29,83-113`); the stuck-watchdog deliberately runs its grace delay on the thread pool, not the possibly-saturated script scheduler (`ScriptExecutionActor.cs:138-139`).
|
||||
- **The reverse indexes are maintained at every mutation site.** Both P2's child-routing maps (filed at child creation, `InstanceActor.cs:1451-1469`) and P3's `_instancesByTag` (add/remove helpers called from subscribe, unsubscribe, and instance-release paths incl. `:1076`) show the discipline that makes index-based fan-out safe.
|
||||
- **`TriggerRouting` fails open.** Unparseable trigger configs route to the all-changes bucket rather than silencing a child (`TriggerRouting.cs:81-105`) — routing is an optimization, the child's own gate stays the correctness check, exactly the right layering.
|
||||
- **Docs kept pace.** C3/C4/C5 spec fixes landed, the S6 deviation is in the component doc, and the shared-script Root-scope surprise is now a documented contract (`Component-SiteRuntime.md:344`).
|
||||
|
||||
---
|
||||
|
||||
## Underdeveloped Areas
|
||||
## Severity Tally (new findings only)
|
||||
|
||||
1. **Cert-trust convergence after node downtime.** The trust broadcast reaches only currently-Up nodes (`DeploymentManagerActor.BroadcastToSiteCertStores`, :962-1007); a node that is down during the broadcast rejoins with a divergent PKI store and there is no reconcile-on-join, no periodic store diff, and `ListServerCertsCommand` reads only the singleton's own node (:936-953) so the divergence is invisible from central. Central persistence of trust decisions is the documented follow-up (`Component-SiteRuntime.md:125`); until then a join-time pull from the peer's `CertStoreActor` would close the failover hole.
|
||||
2. **Dead option**: `SiteRuntimeOptions.ConfigFetchRetryCount` — "Reserved — consumed by the standby replication fetch in a later task; not yet wired" (`SiteRuntimeOptions.cs:63-67`). The standby fetch is genuinely single-attempt (`SiteReplicationActor.cs:178-216`), leaning entirely on reconciliation as backstop.
|
||||
3. **No aggregated live alarm stream** — the M7 operator Alarm Summary polls per-instance `DebugViewSnapshot` Asks; documented as a follow-up in CLAUDE.md/M7 plan. Fine for now; scale limit is the SemaphoreSlim-capped fan-out on the central side.
|
||||
4. **Native alarm rehydration loses display metadata** — only `condition_json` + timestamps persist (`SiteStorageService` `native_alarm_state`, :117-124), so a rehydrated condition renders with empty type/category/message until the first source snapshot (`NativeAlarmActor.cs:161-167`). Acknowledged in comments; worth a line in the component doc's operator-facing section.
|
||||
5. **No stuck-thread/queue-depth observability for the script scheduler** (see S2) — `ScriptExecutionScheduler` exposes nothing to `ISiteHealthCollector`.
|
||||
6. **Tag-subscription failure path has no retry or site-event entry** (see S4) — the spec's Tag Path Resolution retry covers resolution failures inside the DCL, but the instance-side response handling is a hole.
|
||||
7. **Tests**: coverage breadth is very good (redeploy races, waiter races, native-alarm swap/cap, seed reads, scheduler, trust model — 17 actor test files, 17 script test files, dedicated DCL alarm/browse/verify suites, plus `OpcUaAlarmLiveSmokeTests`). Gaps that match the findings above: no test exercises reconnect-then-stale-`ConnectionLost` (S1), compile-failure-deploy-status (S3), the unknown-connection subscribe race (S4), or sustained-churn fan-out cost (the `PerformanceTests` project was not examined for these paths).
|
||||
| Severity | Count | Findings |
|
||||
|---|---|---|
|
||||
| Critical | 0 | — |
|
||||
| High | 0 | — |
|
||||
| Medium | 1 | N1 |
|
||||
| Low | 5 | N2, N3, N4, N5, N6 |
|
||||
|
||||
---
|
||||
|
||||
## Prioritized Recommendations
|
||||
|
||||
1. **(S1)** Dispose + detach the previous client in `OpcUaDataConnection.ConnectAsync` / `MxGatewayDataConnection.ConnectAsync` (and cancel the old MxGateway event loop). This is the single highest-value fix: it removes both a per-flap resource leak and a healthy-connection flapping mechanism.
|
||||
2. **(S3)** Make site-side compile failure fail the deployment, per spec: have `CreateChildActors` report compile outcomes to the Deployment Manager (or pre-compile in `ApplyDeployment` before actor creation, which also fixes P6 by moving compiles off the instance-start path) and return `DeploymentStatus.Failed` with the errors.
|
||||
3. **(S2)** Add stuck-execution observability (scheduler queue depth + oldest-busy-thread age as health metrics; log the script name on timeout-with-thread-still-busy), and inject command timeouts into script-facing DB connections.
|
||||
4. **(P1)** Move trigger-expression evaluation off the actor thread — evaluate on the `ScriptExecutionScheduler` and post the boolean result back to the actor (edge-state stays actor-confined), or precompile expressions to delegates rather than re-running Roslyn scripts.
|
||||
5. **(S4)** Handle `SubscribeTagsResponse.Success == false` in `InstanceActor` with a logged, bounded re-subscribe timer (mirror `NativeAlarmActor`'s retry); add a response-timeout retry to `NativeAlarmActor.SendSubscribe`.
|
||||
6. **(S5/S6)** Retry the startup config load on failure; `Context.Watch` all Instance Actors so an init-failure removes the dead ref and fails the in-flight deployment.
|
||||
7. **(P2/P3)** Introduce per-attribute child-subscription routing in `InstanceActor` and a tag→instances reverse index in `DataConnectionActor` before scaling to the 500-instance design point.
|
||||
8. **(S8)** Enable WAL + explicit busy timeout on the site SQLite stores.
|
||||
9. **(S7)** Capture `_adapter` into locals before every background task in `DataConnectionActor` (three sites).
|
||||
10. **(C3/C4/C5)** Doc fixes: reconcile the reconnect-interval wording, qualify the disconnect-quality-visibility promise, and document shared-script scope semantics — per the repo rule that design docs and code travel together.
|
||||
Round-1 residue: P5 and P6 remain deferred with accepted rationale (P6's cost is now partially addressable via N4's compile cache); UA1's removal-reconcile/list-aggregation sub-scope stays parked behind the documented central-persistence-of-cert-trust follow-up (`Component-SiteRuntime.md:125`).
|
||||
|
||||
Reference in New Issue
Block a user