docs(archreview): add architecture-review fix plans (P0 initiative baseline)

archreview/03-site-runtime-dcl.md

@@ -0,0 +1,155 @@
+# Architecture Review 03 — Site Runtime & Data Connection Layer
+
+Reviewer domain: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime`, `src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer`, `src/ZB.MOM.WW.ScadaBridge.SiteEventLogging`, matched against `docs/requirements/Component-SiteRuntime.md`, `Component-DataConnectionLayer.md`, `Component-SiteEventLogging.md`, and `HighLevelReqs.md`. All findings are grounded in direct reads of the cited files.
+
+## Scope
+
+- **Site Runtime actors**: `DeploymentManagerActor` (1,720 LOC), `InstanceActor` (1,572), `ScriptActor` (594), `ScriptExecutionActor` (305), `AlarmActor` (780), `AlarmExecutionActor`, `NativeAlarmActor` (448), `CertStoreActor` (189), `SiteReplicationActor` (292), `SiteReconciliationActor`.
+- **Script subsystem**: `ScriptCompilationService`, `SharedScriptLibrary`, `ScriptExecutionScheduler`, `ScriptRuntimeContext` (2,384 — skimmed for Ask/blocking patterns), trigger expressions.
+- **Streaming**: `SiteStreamManager` (BroadcastHub + per-subscriber buffers).
+- **Persistence**: `SiteStorageService` (site SQLite: deployed configs, static overrides, artifacts, `native_alarm_state`), `OperationTrackingStore` (not deep-read).
+- **DCL**: `DataConnectionManagerActor` (345), `DataConnectionActor` (2,000), `OpcUaDataConnection` (401), `RealOpcUaClient` (1,313 — lifecycle sections), `MxGatewayDataConnection` (297 — connect path), alarm mappers, browse/search/verify paths.
+- **Site Event Logging**: `SiteEventLogger` (326), purge/query services.
+- **Tests**: inventoried `tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests`, `...DataConnectionLayer.Tests`, `...SiteEventLogging.Tests`.
+
+## Maturity Verdict
+
+This is a mature, unusually carefully engineered subsystem — the Become/Stash state machine, in-flight subscribe bookkeeping, adapter-generation guards, seed-read retry, redeploy termination-watch buffering, and per-source native-alarm cap/eviction all show evidence of multiple hardening passes, and the code matches the design docs closely with extensive rationale comments and broad targeted test coverage. The residual risk is concentrated in three places: **adapter lifecycle on reconnect** (the previous OPC UA / MxGateway client object is silently abandoned on every `ConnectAsync`, leaking sessions and leaving a stale `ConnectionLost` handler that can flap a healthy connection), **script containment** (the execution timeout is cooperative-only, so a hung blocking script permanently consumes one of the 8 dedicated script threads; trigger expressions block actor threads on the default dispatcher), and **deployment honesty** (a site-side script compile failure does not reject the deployment as the spec requires — the instance starts with the script silently missing and central is told Success). None of these break the happy path; all of them are the kind of defect that surfaces weeks into production under flaky networks or a bad script.
+
+---
+
+## Findings — Stability
+
+### S1. [High] Reconnect abandons the previous protocol client undisposed, and its stale `ConnectionLost` handler can flap a healthy connection
+`OpcUaDataConnection.ConnectAsync` (`src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Adapters/OpcUaDataConnection.cs:99-101`) does:
+
+```csharp
+_client = _clientFactory.Create();
+_client.ConnectionLost += OnClientConnectionLost;
+await _client.ConnectAsync(...);
+```
+
+The **previous** `_client` is neither `DisposeAsync`'d nor detached. `DataConnectionActor`'s reconnect loop calls `_adapter.ConnectAsync` repeatedly on the *same adapter* (`DataConnectionActor.cs:537-547`); only the failover branch replaces the adapter. `MxGatewayDataConnection.ConnectAsync` has the same pattern (`MxGatewayDataConnection.cs:70-88`) and additionally overwrites `_eventLoopCts` without cancelling the previous event loop.
+
+Failure scenarios:
+1. **Session/socket leak per disconnect cycle.** After a keep-alive-detected drop, the old `RealOpcUaClient` still holds its `ISession` + subscription (`RealOpcUaClient.cs:133-160`); `Session.CloseAsync` only runs from `DisconnectAsync` (`RealOpcUaClient.cs:392-396`), which nothing calls on this path. A site whose PLC link flaps hourly accumulates orphaned SDK session objects, keep-alive timers, and sockets indefinitely.
+2. **Spurious disconnect of a healthy connection.** If the drop was detected *reactively* (a `ReadAsync` failure → `RaiseDisconnected`, `OpcUaDataConnection.cs:222-227`), the old client's own once-only guard (`_connectionLostFired`) has **not** fired. After a successful reconnect resets the adapter's `_disconnectFired` (`OpcUaDataConnection.cs:104`), the abandoned session's keep-alive eventually fails, fires the *old* client's `ConnectionLost`, which is still wired to `OnClientConnectionLost` → `RaiseDisconnected()` → `AdapterDisconnected` → `BecomeReconnecting()` on a perfectly healthy new session. Each such cycle abandons another client — self-sustaining reconnect churn.
+
+Fix shape: in both adapters' `ConnectAsync`, detach + dispose the old client before creating the new one (mirroring what the actor's failover branch already does at the adapter level, `DataConnectionActor.cs:421-431`).
+
+### S2. [High] Script execution timeout is cooperative-only; hung scripts permanently consume the bounded script-thread pool
+`ScriptExecutionActor` runs the body on the dedicated `ScriptExecutionScheduler` (default **8 threads**, `SiteRuntimeOptions.cs:46`) and enforces the timeout via `CancellationTokenSource(timeout)` passed to `compiledScript.RunAsync` (`ScriptExecutionActor.cs:129, 246`). Roslyn only observes the token between statements; a script blocked in synchronous I/O (`Database.Connection` ADO.NET call, a socket in a wedged state) or a CPU loop never observes cancellation. The `OperationCanceledException` path replies/logs, but the *thread* is only released when the blocking call itself returns. `ScriptExecutionScheduler.WorkerLoop` (`ScriptExecutionScheduler.cs:71-84`) has no watchdog, no thread replacement, no "N tasks stuck > timeout" metric.
+
+Failure scenario: a site DB connection black-holes (firewall drops packets, no RST). Eight interval-triggered scripts each block ~30+ min in a synchronous read. All subsequent script *and alarm on-trigger* executions site-wide queue in `_queue` forever — alarms still transition (evaluation is on actor threads) but no on-trigger scripts run, and nothing on the health dashboard says why. The spec's claim that "Exceeding the timeout cancels the script" (`Component-SiteRuntime.md:411`) overstates what is enforced.
+
+Mitigations worth considering: a scheduler gauge (queue depth + busy-thread age) surfaced through `ISiteHealthCollector`; per-script-thread stuck detection that logs the script name; ADO.NET command timeouts injected into `Database.Connection`.
+
+### S3. [High] Spec drift: site-side script compile failure does not reject the deployment — central is told Success with the script silently missing
+`Component-SiteRuntime.md:474-477`: "If script compilation fails when a deployment is received, the entire deployment for that instance is **rejected**. No partial state is applied. The failure is reported back to central as a failed deployment."
+
+Implementation: `InstanceActor.CreateChildActors` logs and `continue`s past a failed compile (`InstanceActor.cs:1362-1369`; same for alarm on-trigger scripts at 1418-1423 and trigger expressions at 1524-1531). The instance starts with the failing script absent; `DeploymentManagerActor.HandleDeployPersistenceResult` reports `DeploymentStatus.Success` purely on SQLite persistence (`DeploymentManagerActor.cs:596-610`). The XML doc on `CreateChildActors` even states the contradiction: "Compilation errors reject entire instance deployment (logged but actor still starts)" (`InstanceActor.cs:1339`).
+
+The design-time gate in Template Engine makes this rare, but the spec explicitly calls the site the last line ("Site-side compilation failures indicate an unexpected issue") — and version skew between central's Roslyn surface and the site's is exactly the case where the gates disagree. Today the operator sees a green deploy and a dead script discoverable only in the site event log.
+
+### S4. [Medium] Instance actor never retries a failed tag subscription; a subscribe/create ordering race strands attributes at Uncertain forever
+`InstanceActor` registers `Receive<SubscribeTagsResponse>(_ => { })` — an explicit no-op (`InstanceActor.cs:207`). If the DCL replies `Success=false` there is **no retry and no logging**. Two ways to get there:
+
+- `DataConnectionManagerActor.HandleRoute` replies "Unknown connection" when the `SubscribeTagsRequest` arrives before the `CreateConnectionCommand` has been processed (`DataConnectionManagerActor.cs:104-115`). The Deployment Manager Tells `CreateConnectionCommand` and then creates the Instance Actor (`DeploymentManagerActor.cs:543-546`), whose `PreStart` Tells `SubscribeTagsRequest` — but the two messages come from **different senders**, so Akka's per-sender-pair ordering guarantee does not formally order them at the manager's mailbox. If the race fires, the instance's data-sourced attributes stay Uncertain until redeploy/restart.
+- The connection-level-failure reply ("connection unavailable — will re-subscribe on reconnect", `DataConnectionActor.cs:1009-1012`) is self-healing because `_subscriptionsByInstance` was already updated and `ReSubscribeAll` re-derives from it — that case is fine. The unknown-connection case leaves **no state anywhere**.
+
+Contrast: `NativeAlarmActor` retries a failed subscribe on a timer (`NativeAlarmActor.cs:269-286`). The tag path deserves the same treatment — at minimum, log the failure and schedule a bounded re-subscribe.
+
+Related sub-gap: `NativeAlarmActor`'s retry only arms on a *failure response*. A subscribe whose response is **lost** (DCL manager restarted between request and reply; message dead-lettered) never retries — `SendSubscribe` is a bare `Tell` with no response timeout (`NativeAlarmActor.cs:123-131`). Rehydrated SQLite state masks the gap in the debug view while the mirror is actually dead.
+
+### S5. [Medium] Deployment Manager startup aborts silently on a SQLite read failure — site runs zero instances with no retry
+`HandleStartupConfigsLoaded`: on error, `_logger.LogError(...); return;` (`DeploymentManagerActor.cs:291-297`). One transient SQLite failure (locked file during a crash-recovery race, disk hiccup) at singleton start/failover leaves the active node with **no Instance Actors** until a human restarts the process. There is no retry timer, no health flag beyond the log line (health instance counts will read 0 deployed, which at least shows on the dashboard, but nothing distinguishes "genuinely nothing deployed" from "startup load failed"). A bounded retry (the same `Timers.StartSingleTimer` idiom used two methods later) would close this.
+
+### S6. [Medium] A throwing `InstanceActor` constructor leaves a dead ref in `_instanceActors` and still reports the deployment Success
+`InstanceActor`'s constructor deserializes the config JSON (`InstanceActor.cs:138`) and builds the resolved-attribute index; a malformed/oversized config throws → `ActorInitializationException` → Deployment Manager's decider Stops the child (`DeploymentManagerActor.cs:268-274`). But:
+- `CreateInstanceActor` already stored the ref in `_instanceActors` (`DeploymentManagerActor.cs:1628-1629`) and nothing watches for this failure mode (the `Context.Watch` only happens on the redeploy path), so the map holds a dead ref — debug view, inbound-API routes, etc. dead-letter with 30s Ask timeouts instead of "not found".
+- `ApplyDeployment`'s persistence continues and `HandleDeployPersistenceResult` reports `Success` (`DeploymentManagerActor.cs:596-610`) — central shows a healthy deployment for an instance that is not running.
+
+A `Context.Watch` on every created child (removing from `_instanceActors` on unexpected `Terminated`) would fix both the stale map and give a hook to fail the pending deployment.
+
+### S7. [Medium] Subscribe background task reads the mutable `_adapter` field off-thread, violating its own confinement contract
+`HandleSubscribe`'s `Task.Run` body awaits `_adapter.SubscribeAsync(...)` per tag and calls `SeedTagsAsync(_adapter, ...)` (`DataConnectionActor.cs:707-758`) — reading the actor field from a thread-pool thread, despite the comment eight lines earlier: "The background task below must NOT read or mutate actor state — these partitioned lists are the only state it sees" (`DataConnectionActor.cs:689-690`). If a failover swaps `_adapter` mid-loop (`CountFailureAndMaybeFailover`, `DataConnectionActor.cs:610-635`), later iterations subscribe on the **new** adapter while stamping the **old** generation — the resulting values are dropped by the generation guard and the monitored items are orphaned on the new adapter (never in `_subscriptionIds` for that generation… actually worse: `SubscribeCompleted` will store them against current state, mixing generations). `ReSubscribeAll` (`DataConnectionActor.cs:1596`) and `HandleWriteBatch`'s local async function (`DataConnectionActor.cs:1161-1201`) have the same field-read pattern. Capture the adapter into a local before spawning the task (as the re-seed block at `DataConnectionActor.cs:1620` correctly does with `reseedAdapter`).
+
+### S8. [Low] Site SQLite: connection-per-call, no WAL / busy-timeout tuning, many concurrent writers
+`SiteStorageService` opens a fresh `SqliteConnection` per operation with a bare `Data Source=` string (`SiteStorageService.cs:34-44, 163-166`) — no `journal_mode=WAL`, relying on Microsoft.Data.Sqlite's default command-timeout-as-busy-handler. Concurrent writers against the same file include: deploy persistence, static-override fire-and-forget writes (`InstanceActor.cs:433-441`), per-transition native-alarm upserts (`NativeAlarmActor.cs:420-428`), replication applies (`SiteReplicationActor.cs:182-216`), and reconciliation. Under a native-alarm storm plus a deploy, rollback-journal locking serializes everything and a slow write can push others toward `SQLITE_BUSY`. WAL + explicit busy_timeout is a one-line hardening. (Contrast: `SiteEventLogger` got the full treatment — single owned connection, lock, bounded write queue.)
+
+### S9. [Low] Fire-and-forget adapter disposal in `PostStop`
+`DataConnectionActor.PostStop` discards the `DisposeAsync` task (`DataConnectionActor.cs:218-221`); a hanging/faulting dispose (OPC UA `CloseAsync` against a dead server can block for the operation timeout) is unobserved. Acceptable trade-off (documented elsewhere in the file), but pair it with a `ContinueWith(OnlyOnFaulted, log)` like the other fire-and-forget paths in this codebase do.
+
+### S10. [Low] Alarm condition filter is last-subscriber-wins across co-subscribers of one source
+`_alarmSourceFilter[request.SourceReference] = request.ConditionFilter` overwrites unconditionally (`DataConnectionActor.cs:104-108, 1763-1766`); two instances binding the same source with different filters silently re-gate each other's transitions. The field's XML doc honestly declares the sharp edge, but nothing *validates* agreement or logs the overwrite — a mismatch is undiagnosable in the field. Log a warning when the incoming filter differs from the stored one.
+
+---
+
+## Findings — Performance
+
+### P1. [High] Trigger expressions block actor threads on the default dispatcher — per attribute change
+`ScriptActor.EvaluateExpressionTrigger` runs the compiled Roslyn expression with `.GetAwaiter().GetResult()` under a 2-second CTS (`ScriptActor.cs:283-287`); `AlarmActor.EvaluateExpression` is identical (`AlarmActor.cs:503-506`). Both execute **on the actor's default-dispatcher thread**, and both run on *every* `AttributeValueChanged` the actor receives (Expression triggers have no monitored-attribute gate — `ScriptActor.cs:255-258`, `AlarmActor.cs:215-218`). A site with a handful of Expression-triggered scripts/alarms on a chatty instance (e.g., 50 tags at 1 Hz) synchronously executes 100s of Roslyn script invocations per second on dispatcher threads; one pathological expression (2 s timeout, treated-as-false) stalls its actor's mailbox and steals a dispatcher thread for the duration. The `ScriptExecutionScheduler` exists precisely to keep script code off these threads — trigger expressions bypass it. The in-code justification ("trusted Design-role users, compile-checked") addresses malice, not cost.
+
+### P2. [Medium] Attribute-change fan-out is broadcast to every child, filtering in the receiver
+`InstanceActor.PublishAndNotifyChildren` Tells **every** Script Actor and **every** Alarm Actor for every attribute change (`InstanceActor.cs:1184-1194`); `Component-SiteRuntime.md:191` describes children subscribing "for the specific monitored attribute". An instance with 30 scripts + 20 alarms and 100 Hz aggregate tag updates generates 5,000 child messages/sec, ~98% of which are discarded by the child's trigger check (`ScriptActor.cs:229-233`, `AlarmActor.cs:219-222`) — pure mailbox/dispatch overhead, plus each `ScriptActor` maintains an `_attributeSnapshot` copy of every value (`ScriptActor.cs:227`). A per-attribute subscription map on the Instance Actor (attributeName → interested children; Expression-trigger children in an "all" bucket) would cut this by an order of magnitude on realistic configs.
+
+### P3. [Medium] DCL tag-value fan-out scans all instances per update
+`HandleTagValueReceived` iterates `_subscriptionsByInstance` (every instance) with a per-instance `HashSet.Contains` for every incoming tag value (`DataConnectionActor.cs:1706-1716`). The actor already maintains `_tagSubscriberCount` (tag → count) for O(1) unsubscribe; the matching tag → *instances* reverse index is missing. With 500 instances on one connection at high sample rates this is the hottest loop in the DCL. Also `_healthCollector.UpdateTagQuality` is invoked on **every** update (`DataConnectionActor.cs:1737`) — cheap if the collector is a plain field write, but worth confirming it isn't lock-contended.
+
+### P4. [Low] Per-transition SQLite upserts on the native-alarm mirror
+`NativeAlarmActor.PersistUpsert` issues one async SQLite write per accepted transition (`NativeAlarmActor.cs:420-428`), each opening a fresh connection (S8). An A&C alarm storm (hundreds of transitions/sec across sources) turns into hundreds of single-row upsert connections/sec. Bounded by `MirroredAlarmCapPerSource` (1000) per source in state size but not in write rate. Batching/coalescing (persist latest per SourceReference on a short timer) would be a proportionate hardening if storms are expected.
+
+### P5. [Low] Every script attribute read is an actor Ask round-trip
+`ScriptRuntimeContext` Asks the Instance Actor per `GetAttribute` (`ScriptRuntimeContext.cs:375`) — a temp actor allocation + two mailbox hops per read. Correct (serialized through the source of truth) and consistent with the spec, but a script reading 50 attributes in a loop pays 50 round-trips; a `GetAttributes` batch message (the DeploymentManager routing path already fans out N Asks — `DeploymentManagerActor.cs:1360-1367` — which would also benefit) is an easy win if profiling ever flags it.
+
+### P6. [Low] Roslyn compiles run inside `InstanceActor.PreStart` on the default dispatcher during staggered startup
+Each instance compiles all its scripts + alarm scripts + trigger expressions synchronously in `CreateChildActors` (called from `PreStart`, `InstanceActor.cs:250-255, 1348+`). The startup stagger (20 instances / 100 ms, `SiteRuntimeOptions.cs:13-19`) is tuned for OPC UA connection storms, not compile cost — 500 instances × several scripts each at ~50–300 ms/compile will occupy default-dispatcher threads for minutes after failover. The Deployment Manager already moved *shared*-script compilation off-thread (`DeploymentManagerActor.cs:1168-1191`); per-instance compiles did not get the same treatment. Failover time-to-recover is the metric at risk.
+
+---
+
+## Findings — Conventions
+
+### C1. [Positive] Akka.NET discipline is exemplary
+Worth stating because it is the dominant impression: Become/Stash exactly as the spec draws it (`DataConnectionActor.cs:232-533`); Tell on hot paths, Ask only at boundaries (`Component-SiteRuntime.md:430-442` ↔ implementation); `Sender`/`Self` captured before every continuation (e.g., `DeploymentManagerActor.cs:464-470`, `DataConnectionActor.cs:149-153`); all async results marshalled via `PipeTo`; supervision matches the spec table precisely — DM `OneForOne` Resume/Stop-on-init (`DeploymentManagerActor.cs:263-279`), Instance Resume (`InstanceActor.cs:281-293`), Script/Alarm coordinators Stop their execution children (`ScriptActor.cs:180-192`, `AlarmActor.cs:192-204`). The redeploy Terminated-watch buffering with the name-shadow map and its disable/delete supersede handling (`DeploymentManagerActor.cs:60-90, 389-435, 641-698, 762-831`) is genuinely hard concurrency done correctly. The in-flight subscribe/orphan-release guards in the DCL (`DataConnectionActor.cs:77-90, 836-985, 1795-1858`) close race classes most codebases never notice.
+
+### C2. [Low] Stale comment: SiteEventLogger claims its channel is unbounded
+`SiteEventLogger.cs:216-218` — "The channel is unbounded, so the only way TryWrite fails is…" — the channel became bounded (`Channel.CreateBounded`, `SiteEventLogger.cs:74-88`), and `TryWrite` on a bounded DropOldest channel still only fails when completed, so the *behavior* is right but the stated reason is wrong. Trivial fix; flagged because this file's comments are otherwise load-bearing.
+
+### C3. [Low] Spec is internally inconsistent on the reconnect interval
+`Component-DataConnectionLayer.md:303` says "The retry interval is defined **per data connection**", but the same document's Shared Settings table (`Component-DataConnectionLayer.md:170-177`) and the implementation (`DataConnectionOptions.ReconnectInterval`, one value for all connections — `DataConnectionActor.cs:463`) make it global. Fix the sentence at :303 (per the CLAUDE.md rule that the design doc travels with the code).
+
+### C4. [Low] Quality-only change on disconnect is not delivered to Alarm/Script actors — deliberate, but diverges from the spec's wording
+`HandleConnectionQualityChanged` updates qualities and the stream but explicitly does **not** notify children ("firing scripts/alarms would cause spurious evaluations", `InstanceActor.cs:924-956`). The DCL spec promises "Instance Actors and their downstream consumers (**alarms, scripts checking quality**) see the staleness immediately" (`Component-DataConnectionLayer.md:302`). Practical consequence: an Active alarm computed from a now-dead connection stays Active (arguably correct — last-known-state), and a Conditional/Expression script never learns quality went Bad. The implementation choice is defensible; the spec sentence should be qualified so the divergence is a decision, not drift.
+
+### C5. [Low] Shared scripts execute without the caller's scope and without a `Scope` assignment
+`SharedScriptLibrary.ExecuteAsync` builds `ScriptGlobals` without setting `Scope` (`SharedScriptLibrary.cs:98-103`), so a shared script always runs `ScriptScope.Root` (`ScriptCompilationService.cs:200-201`) even when invoked from a composed-module script. If a shared script uses `Attributes[...]`, it addresses **root** attributes regardless of caller scope. Possibly intended (shared scripts are instance-agnostic), but it is undocumented in `Component-SiteRuntime.md`'s Shared Script Library section and is a surprise waiting for a template author.
+
+### C6. [Low] `EvaluateCondition` fallback compares `value.ToString()` to `Threshold.ToString()` with current-culture double formatting
+`ScriptActor.cs:474-477`: the non-numeric fallback stringifies the double threshold with the host culture (the numeric path was carefully invariant-culture'd; the fallback wasn't). A `"1.5"` threshold on a de-DE host renders `"1,5"`. Edge-case only (non-parseable values), but inconsistent with the deliberate invariant-culture work two lines up.
+
+---
+
+## Underdeveloped Areas
+
+1. **Cert-trust convergence after node downtime.** The trust broadcast reaches only currently-Up nodes (`DeploymentManagerActor.BroadcastToSiteCertStores`, :962-1007); a node that is down during the broadcast rejoins with a divergent PKI store and there is no reconcile-on-join, no periodic store diff, and `ListServerCertsCommand` reads only the singleton's own node (:936-953) so the divergence is invisible from central. Central persistence of trust decisions is the documented follow-up (`Component-SiteRuntime.md:125`); until then a join-time pull from the peer's `CertStoreActor` would close the failover hole.
+2. **Dead option**: `SiteRuntimeOptions.ConfigFetchRetryCount` — "Reserved — consumed by the standby replication fetch in a later task; not yet wired" (`SiteRuntimeOptions.cs:63-67`). The standby fetch is genuinely single-attempt (`SiteReplicationActor.cs:178-216`), leaning entirely on reconciliation as backstop.
+3. **No aggregated live alarm stream** — the M7 operator Alarm Summary polls per-instance `DebugViewSnapshot` Asks; documented as a follow-up in CLAUDE.md/M7 plan. Fine for now; scale limit is the SemaphoreSlim-capped fan-out on the central side.
+4. **Native alarm rehydration loses display metadata** — only `condition_json` + timestamps persist (`SiteStorageService` `native_alarm_state`, :117-124), so a rehydrated condition renders with empty type/category/message until the first source snapshot (`NativeAlarmActor.cs:161-167`). Acknowledged in comments; worth a line in the component doc's operator-facing section.
+5. **No stuck-thread/queue-depth observability for the script scheduler** (see S2) — `ScriptExecutionScheduler` exposes nothing to `ISiteHealthCollector`.
+6. **Tag-subscription failure path has no retry or site-event entry** (see S4) — the spec's Tag Path Resolution retry covers resolution failures inside the DCL, but the instance-side response handling is a hole.
+7. **Tests**: coverage breadth is very good (redeploy races, waiter races, native-alarm swap/cap, seed reads, scheduler, trust model — 17 actor test files, 17 script test files, dedicated DCL alarm/browse/verify suites, plus `OpcUaAlarmLiveSmokeTests`). Gaps that match the findings above: no test exercises reconnect-then-stale-`ConnectionLost` (S1), compile-failure-deploy-status (S3), the unknown-connection subscribe race (S4), or sustained-churn fan-out cost (the `PerformanceTests` project was not examined for these paths).
+
+---
+
+## Prioritized Recommendations
+
+1. **(S1)** Dispose + detach the previous client in `OpcUaDataConnection.ConnectAsync` / `MxGatewayDataConnection.ConnectAsync` (and cancel the old MxGateway event loop). This is the single highest-value fix: it removes both a per-flap resource leak and a healthy-connection flapping mechanism.
+2. **(S3)** Make site-side compile failure fail the deployment, per spec: have `CreateChildActors` report compile outcomes to the Deployment Manager (or pre-compile in `ApplyDeployment` before actor creation, which also fixes P6 by moving compiles off the instance-start path) and return `DeploymentStatus.Failed` with the errors.
+3. **(S2)** Add stuck-execution observability (scheduler queue depth + oldest-busy-thread age as health metrics; log the script name on timeout-with-thread-still-busy), and inject command timeouts into script-facing DB connections.
+4. **(P1)** Move trigger-expression evaluation off the actor thread — evaluate on the `ScriptExecutionScheduler` and post the boolean result back to the actor (edge-state stays actor-confined), or precompile expressions to delegates rather than re-running Roslyn scripts.
+5. **(S4)** Handle `SubscribeTagsResponse.Success == false` in `InstanceActor` with a logged, bounded re-subscribe timer (mirror `NativeAlarmActor`'s retry); add a response-timeout retry to `NativeAlarmActor.SendSubscribe`.
+6. **(S5/S6)** Retry the startup config load on failure; `Context.Watch` all Instance Actors so an init-failure removes the dead ref and fails the in-flight deployment.
+7. **(P2/P3)** Introduce per-attribute child-subscription routing in `InstanceActor` and a tag→instances reverse index in `DataConnectionActor` before scaling to the 500-instance design point.
+8. **(S8)** Enable WAL + explicit busy timeout on the site SQLite stores.
+9. **(S7)** Capture `_adapter` into locals before every background task in `DataConnectionActor` (three sites).
+10. **(C3/C4/C5)** Doc fixes: reconcile the reconnect-interval wording, qualify the disconnect-quality-visibility promise, and document shared-script scope semantics — per the repo rule that design docs and code travel together.