ScadaBridge/archreview/03-site-runtime-dcl.md

Architecture Review 03 — Site Runtime & Data Connection Layer

Reviewer domain: src/ZB.MOM.WW.ScadaBridge.SiteRuntime, src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer, src/ZB.MOM.WW.ScadaBridge.SiteEventLogging, matched against docs/requirements/Component-SiteRuntime.md, Component-DataConnectionLayer.md, Component-SiteEventLogging.md, and HighLevelReqs.md. All findings are grounded in direct reads of the cited files.

Scope

• Site Runtime actors: DeploymentManagerActor (1,720 LOC), InstanceActor (1,572), ScriptActor (594), ScriptExecutionActor (305), AlarmActor (780), AlarmExecutionActor, NativeAlarmActor (448), CertStoreActor (189), SiteReplicationActor (292), SiteReconciliationActor.
• Script subsystem: ScriptCompilationService, SharedScriptLibrary, ScriptExecutionScheduler, ScriptRuntimeContext (2,384 — skimmed for Ask/blocking patterns), trigger expressions.
• Streaming: SiteStreamManager (BroadcastHub + per-subscriber buffers).
• Persistence: SiteStorageService (site SQLite: deployed configs, static overrides, artifacts, native_alarm_state), OperationTrackingStore (not deep-read).
• DCL: DataConnectionManagerActor (345), DataConnectionActor (2,000), OpcUaDataConnection (401), RealOpcUaClient (1,313 — lifecycle sections), MxGatewayDataConnection (297 — connect path), alarm mappers, browse/search/verify paths.
• Site Event Logging: SiteEventLogger (326), purge/query services.
• Tests: inventoried tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests, ...DataConnectionLayer.Tests, ...SiteEventLogging.Tests.

Maturity Verdict

This is a mature, unusually carefully engineered subsystem — the Become/Stash state machine, in-flight subscribe bookkeeping, adapter-generation guards, seed-read retry, redeploy termination-watch buffering, and per-source native-alarm cap/eviction all show evidence of multiple hardening passes, and the code matches the design docs closely with extensive rationale comments and broad targeted test coverage. The residual risk is concentrated in three places: adapter lifecycle on reconnect (the previous OPC UA / MxGateway client object is silently abandoned on every ConnectAsync, leaking sessions and leaving a stale ConnectionLost handler that can flap a healthy connection), script containment (the execution timeout is cooperative-only, so a hung blocking script permanently consumes one of the 8 dedicated script threads; trigger expressions block actor threads on the default dispatcher), and deployment honesty (a site-side script compile failure does not reject the deployment as the spec requires — the instance starts with the script silently missing and central is told Success). None of these break the happy path; all of them are the kind of defect that surfaces weeks into production under flaky networks or a bad script.

---

Findings — Stability

S1. [High] Reconnect abandons the previous protocol client undisposed, and its stale ConnectionLost handler can flap a healthy connection

OpcUaDataConnection.ConnectAsync (src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Adapters/OpcUaDataConnection.cs:99-101) does:

_client = _clientFactory.Create();
_client.ConnectionLost += OnClientConnectionLost;
await _client.ConnectAsync(...);

The previous _client is neither DisposeAsync'd nor detached. DataConnectionActor's reconnect loop calls _adapter.ConnectAsync repeatedly on the same adapter (DataConnectionActor.cs:537-547); only the failover branch replaces the adapter. MxGatewayDataConnection.ConnectAsync has the same pattern (MxGatewayDataConnection.cs:70-88) and additionally overwrites _eventLoopCts without cancelling the previous event loop.

Failure scenarios:

1. Session/socket leak per disconnect cycle. After a keep-alive-detected drop, the old RealOpcUaClient still holds its ISession + subscription (RealOpcUaClient.cs:133-160); Session.CloseAsync only runs from DisconnectAsync (RealOpcUaClient.cs:392-396), which nothing calls on this path. A site whose PLC link flaps hourly accumulates orphaned SDK session objects, keep-alive timers, and sockets indefinitely.
2. Spurious disconnect of a healthy connection. If the drop was detected reactively (a ReadAsync failure → RaiseDisconnected, OpcUaDataConnection.cs:222-227), the old client's own once-only guard (_connectionLostFired) has not fired. After a successful reconnect resets the adapter's _disconnectFired (OpcUaDataConnection.cs:104), the abandoned session's keep-alive eventually fails, fires the old client's ConnectionLost, which is still wired to OnClientConnectionLost → RaiseDisconnected() → AdapterDisconnected → BecomeReconnecting() on a perfectly healthy new session. Each such cycle abandons another client — self-sustaining reconnect churn.

Fix shape: in both adapters' ConnectAsync, detach + dispose the old client before creating the new one (mirroring what the actor's failover branch already does at the adapter level, DataConnectionActor.cs:421-431).

S2. [High] Script execution timeout is cooperative-only; hung scripts permanently consume the bounded script-thread pool

ScriptExecutionActor runs the body on the dedicated ScriptExecutionScheduler (default 8 threads, SiteRuntimeOptions.cs:46) and enforces the timeout via CancellationTokenSource(timeout) passed to compiledScript.RunAsync (ScriptExecutionActor.cs:129, 246). Roslyn only observes the token between statements; a script blocked in synchronous I/O (Database.Connection ADO.NET call, a socket in a wedged state) or a CPU loop never observes cancellation. The OperationCanceledException path replies/logs, but the thread is only released when the blocking call itself returns. ScriptExecutionScheduler.WorkerLoop (ScriptExecutionScheduler.cs:71-84) has no watchdog, no thread replacement, no "N tasks stuck > timeout" metric.

Failure scenario: a site DB connection black-holes (firewall drops packets, no RST). Eight interval-triggered scripts each block ~30+ min in a synchronous read. All subsequent script and alarm on-trigger executions site-wide queue in _queue forever — alarms still transition (evaluation is on actor threads) but no on-trigger scripts run, and nothing on the health dashboard says why. The spec's claim that "Exceeding the timeout cancels the script" (Component-SiteRuntime.md:411) overstates what is enforced.

Mitigations worth considering: a scheduler gauge (queue depth + busy-thread age) surfaced through ISiteHealthCollector; per-script-thread stuck detection that logs the script name; ADO.NET command timeouts injected into Database.Connection.

S3. [High] Spec drift: site-side script compile failure does not reject the deployment — central is told Success with the script silently missing

Component-SiteRuntime.md:474-477: "If script compilation fails when a deployment is received, the entire deployment for that instance is rejected. No partial state is applied. The failure is reported back to central as a failed deployment."

Implementation: InstanceActor.CreateChildActors logs and continues past a failed compile (InstanceActor.cs:1362-1369; same for alarm on-trigger scripts at 1418-1423 and trigger expressions at 1524-1531). The instance starts with the failing script absent; DeploymentManagerActor.HandleDeployPersistenceResult reports DeploymentStatus.Success purely on SQLite persistence (DeploymentManagerActor.cs:596-610). The XML doc on CreateChildActors even states the contradiction: "Compilation errors reject entire instance deployment (logged but actor still starts)" (InstanceActor.cs:1339).

The design-time gate in Template Engine makes this rare, but the spec explicitly calls the site the last line ("Site-side compilation failures indicate an unexpected issue") — and version skew between central's Roslyn surface and the site's is exactly the case where the gates disagree. Today the operator sees a green deploy and a dead script discoverable only in the site event log.

S4. [Medium] Instance actor never retries a failed tag subscription; a subscribe/create ordering race strands attributes at Uncertain forever

InstanceActor registers Receive<SubscribeTagsResponse>(_ => { }) — an explicit no-op (InstanceActor.cs:207). If the DCL replies Success=false there is no retry and no logging. Two ways to get there:

• DataConnectionManagerActor.HandleRoute replies "Unknown connection" when the SubscribeTagsRequest arrives before the CreateConnectionCommand has been processed (DataConnectionManagerActor.cs:104-115). The Deployment Manager Tells CreateConnectionCommand and then creates the Instance Actor (DeploymentManagerActor.cs:543-546), whose PreStart Tells SubscribeTagsRequest — but the two messages come from different senders, so Akka's per-sender-pair ordering guarantee does not formally order them at the manager's mailbox. If the race fires, the instance's data-sourced attributes stay Uncertain until redeploy/restart.
• The connection-level-failure reply ("connection unavailable — will re-subscribe on reconnect", DataConnectionActor.cs:1009-1012) is self-healing because _subscriptionsByInstance was already updated and ReSubscribeAll re-derives from it — that case is fine. The unknown-connection case leaves no state anywhere.

Contrast: NativeAlarmActor retries a failed subscribe on a timer (NativeAlarmActor.cs:269-286). The tag path deserves the same treatment — at minimum, log the failure and schedule a bounded re-subscribe.

Related sub-gap: NativeAlarmActor's retry only arms on a failure response. A subscribe whose response is lost (DCL manager restarted between request and reply; message dead-lettered) never retries — SendSubscribe is a bare Tell with no response timeout (NativeAlarmActor.cs:123-131). Rehydrated SQLite state masks the gap in the debug view while the mirror is actually dead.

S5. [Medium] Deployment Manager startup aborts silently on a SQLite read failure — site runs zero instances with no retry

HandleStartupConfigsLoaded: on error, _logger.LogError(...); return; (DeploymentManagerActor.cs:291-297). One transient SQLite failure (locked file during a crash-recovery race, disk hiccup) at singleton start/failover leaves the active node with no Instance Actors until a human restarts the process. There is no retry timer, no health flag beyond the log line (health instance counts will read 0 deployed, which at least shows on the dashboard, but nothing distinguishes "genuinely nothing deployed" from "startup load failed"). A bounded retry (the same Timers.StartSingleTimer idiom used two methods later) would close this.

S6. [Medium] A throwing InstanceActor constructor leaves a dead ref in _instanceActors and still reports the deployment Success

InstanceActor's constructor deserializes the config JSON (InstanceActor.cs:138) and builds the resolved-attribute index; a malformed/oversized config throws → ActorInitializationException → Deployment Manager's decider Stops the child (DeploymentManagerActor.cs:268-274). But:

• CreateInstanceActor already stored the ref in _instanceActors (DeploymentManagerActor.cs:1628-1629) and nothing watches for this failure mode (the Context.Watch only happens on the redeploy path), so the map holds a dead ref — debug view, inbound-API routes, etc. dead-letter with 30s Ask timeouts instead of "not found".
• ApplyDeployment's persistence continues and HandleDeployPersistenceResult reports Success (DeploymentManagerActor.cs:596-610) — central shows a healthy deployment for an instance that is not running.

A Context.Watch on every created child (removing from _instanceActors on unexpected Terminated) would fix both the stale map and give a hook to fail the pending deployment.

S7. [Medium] Subscribe background task reads the mutable _adapter field off-thread, violating its own confinement contract

HandleSubscribe's Task.Run body awaits _adapter.SubscribeAsync(...) per tag and calls SeedTagsAsync(_adapter, ...) (DataConnectionActor.cs:707-758) — reading the actor field from a thread-pool thread, despite the comment eight lines earlier: "The background task below must NOT read or mutate actor state — these partitioned lists are the only state it sees" (DataConnectionActor.cs:689-690). If a failover swaps _adapter mid-loop (CountFailureAndMaybeFailover, DataConnectionActor.cs:610-635), later iterations subscribe on the new adapter while stamping the old generation — the resulting values are dropped by the generation guard and the monitored items are orphaned on the new adapter (never in _subscriptionIds for that generation… actually worse: SubscribeCompleted will store them against current state, mixing generations). ReSubscribeAll (DataConnectionActor.cs:1596) and HandleWriteBatch's local async function (DataConnectionActor.cs:1161-1201) have the same field-read pattern. Capture the adapter into a local before spawning the task (as the re-seed block at DataConnectionActor.cs:1620 correctly does with reseedAdapter).

S8. [Low] Site SQLite: connection-per-call, no WAL / busy-timeout tuning, many concurrent writers

SiteStorageService opens a fresh SqliteConnection per operation with a bare Data Source= string (SiteStorageService.cs:34-44, 163-166) — no journal_mode=WAL, relying on Microsoft.Data.Sqlite's default command-timeout-as-busy-handler. Concurrent writers against the same file include: deploy persistence, static-override fire-and-forget writes (InstanceActor.cs:433-441), per-transition native-alarm upserts (NativeAlarmActor.cs:420-428), replication applies (SiteReplicationActor.cs:182-216), and reconciliation. Under a native-alarm storm plus a deploy, rollback-journal locking serializes everything and a slow write can push others toward SQLITE_BUSY. WAL + explicit busy_timeout is a one-line hardening. (Contrast: SiteEventLogger got the full treatment — single owned connection, lock, bounded write queue.)

S9. [Low] Fire-and-forget adapter disposal in PostStop

DataConnectionActor.PostStop discards the DisposeAsync task (DataConnectionActor.cs:218-221); a hanging/faulting dispose (OPC UA CloseAsync against a dead server can block for the operation timeout) is unobserved. Acceptable trade-off (documented elsewhere in the file), but pair it with a ContinueWith(OnlyOnFaulted, log) like the other fire-and-forget paths in this codebase do.

S10. [Low] Alarm condition filter is last-subscriber-wins across co-subscribers of one source

_alarmSourceFilter[request.SourceReference] = request.ConditionFilter overwrites unconditionally (DataConnectionActor.cs:104-108, 1763-1766); two instances binding the same source with different filters silently re-gate each other's transitions. The field's XML doc honestly declares the sharp edge, but nothing validates agreement or logs the overwrite — a mismatch is undiagnosable in the field. Log a warning when the incoming filter differs from the stored one.

---

Findings — Performance

P1. [High] Trigger expressions block actor threads on the default dispatcher — per attribute change

ScriptActor.EvaluateExpressionTrigger runs the compiled Roslyn expression with .GetAwaiter().GetResult() under a 2-second CTS (ScriptActor.cs:283-287); AlarmActor.EvaluateExpression is identical (AlarmActor.cs:503-506). Both execute on the actor's default-dispatcher thread, and both run on every AttributeValueChanged the actor receives (Expression triggers have no monitored-attribute gate — ScriptActor.cs:255-258, AlarmActor.cs:215-218). A site with a handful of Expression-triggered scripts/alarms on a chatty instance (e.g., 50 tags at 1 Hz) synchronously executes 100s of Roslyn script invocations per second on dispatcher threads; one pathological expression (2 s timeout, treated-as-false) stalls its actor's mailbox and steals a dispatcher thread for the duration. The ScriptExecutionScheduler exists precisely to keep script code off these threads — trigger expressions bypass it. The in-code justification ("trusted Design-role users, compile-checked") addresses malice, not cost.

P2. [Medium] Attribute-change fan-out is broadcast to every child, filtering in the receiver

InstanceActor.PublishAndNotifyChildren Tells every Script Actor and every Alarm Actor for every attribute change (InstanceActor.cs:1184-1194); Component-SiteRuntime.md:191 describes children subscribing "for the specific monitored attribute". An instance with 30 scripts + 20 alarms and 100 Hz aggregate tag updates generates 5,000 child messages/sec, ~98% of which are discarded by the child's trigger check (ScriptActor.cs:229-233, AlarmActor.cs:219-222) — pure mailbox/dispatch overhead, plus each ScriptActor maintains an _attributeSnapshot copy of every value (ScriptActor.cs:227). A per-attribute subscription map on the Instance Actor (attributeName → interested children; Expression-trigger children in an "all" bucket) would cut this by an order of magnitude on realistic configs.

P3. [Medium] DCL tag-value fan-out scans all instances per update

HandleTagValueReceived iterates _subscriptionsByInstance (every instance) with a per-instance HashSet.Contains for every incoming tag value (DataConnectionActor.cs:1706-1716). The actor already maintains _tagSubscriberCount (tag → count) for O(1) unsubscribe; the matching tag → instances reverse index is missing. With 500 instances on one connection at high sample rates this is the hottest loop in the DCL. Also _healthCollector.UpdateTagQuality is invoked on every update (DataConnectionActor.cs:1737) — cheap if the collector is a plain field write, but worth confirming it isn't lock-contended.

P4. [Low] Per-transition SQLite upserts on the native-alarm mirror

NativeAlarmActor.PersistUpsert issues one async SQLite write per accepted transition (NativeAlarmActor.cs:420-428), each opening a fresh connection (S8). An A&C alarm storm (hundreds of transitions/sec across sources) turns into hundreds of single-row upsert connections/sec. Bounded by MirroredAlarmCapPerSource (1000) per source in state size but not in write rate. Batching/coalescing (persist latest per SourceReference on a short timer) would be a proportionate hardening if storms are expected.

P5. [Low] Every script attribute read is an actor Ask round-trip

ScriptRuntimeContext Asks the Instance Actor per GetAttribute (ScriptRuntimeContext.cs:375) — a temp actor allocation + two mailbox hops per read. Correct (serialized through the source of truth) and consistent with the spec, but a script reading 50 attributes in a loop pays 50 round-trips; a GetAttributes batch message (the DeploymentManager routing path already fans out N Asks — DeploymentManagerActor.cs:1360-1367 — which would also benefit) is an easy win if profiling ever flags it.

P6. [Low] Roslyn compiles run inside InstanceActor.PreStart on the default dispatcher during staggered startup

Each instance compiles all its scripts + alarm scripts + trigger expressions synchronously in CreateChildActors (called from PreStart, InstanceActor.cs:250-255, 1348+). The startup stagger (20 instances / 100 ms, SiteRuntimeOptions.cs:13-19) is tuned for OPC UA connection storms, not compile cost — 500 instances × several scripts each at ~50–300 ms/compile will occupy default-dispatcher threads for minutes after failover. The Deployment Manager already moved shared-script compilation off-thread (DeploymentManagerActor.cs:1168-1191); per-instance compiles did not get the same treatment. Failover time-to-recover is the metric at risk.

---

Findings — Conventions

C1. [Positive] Akka.NET discipline is exemplary

Worth stating because it is the dominant impression: Become/Stash exactly as the spec draws it (DataConnectionActor.cs:232-533); Tell on hot paths, Ask only at boundaries (Component-SiteRuntime.md:430-442 ↔ implementation); Sender/Self captured before every continuation (e.g., DeploymentManagerActor.cs:464-470, DataConnectionActor.cs:149-153); all async results marshalled via PipeTo; supervision matches the spec table precisely — DM OneForOne Resume/Stop-on-init (DeploymentManagerActor.cs:263-279), Instance Resume (InstanceActor.cs:281-293), Script/Alarm coordinators Stop their execution children (ScriptActor.cs:180-192, AlarmActor.cs:192-204). The redeploy Terminated-watch buffering with the name-shadow map and its disable/delete supersede handling (DeploymentManagerActor.cs:60-90, 389-435, 641-698, 762-831) is genuinely hard concurrency done correctly. The in-flight subscribe/orphan-release guards in the DCL (DataConnectionActor.cs:77-90, 836-985, 1795-1858) close race classes most codebases never notice.

C2. [Low] Stale comment: SiteEventLogger claims its channel is unbounded

SiteEventLogger.cs:216-218 — "The channel is unbounded, so the only way TryWrite fails is…" — the channel became bounded (Channel.CreateBounded, SiteEventLogger.cs:74-88), and TryWrite on a bounded DropOldest channel still only fails when completed, so the behavior is right but the stated reason is wrong. Trivial fix; flagged because this file's comments are otherwise load-bearing.

C3. [Low] Spec is internally inconsistent on the reconnect interval

Component-DataConnectionLayer.md:303 says "The retry interval is defined per data connection", but the same document's Shared Settings table (Component-DataConnectionLayer.md:170-177) and the implementation (DataConnectionOptions.ReconnectInterval, one value for all connections — DataConnectionActor.cs:463) make it global. Fix the sentence at :303 (per the CLAUDE.md rule that the design doc travels with the code).

C4. [Low] Quality-only change on disconnect is not delivered to Alarm/Script actors — deliberate, but diverges from the spec's wording

HandleConnectionQualityChanged updates qualities and the stream but explicitly does not notify children ("firing scripts/alarms would cause spurious evaluations", InstanceActor.cs:924-956). The DCL spec promises "Instance Actors and their downstream consumers (alarms, scripts checking quality) see the staleness immediately" (Component-DataConnectionLayer.md:302). Practical consequence: an Active alarm computed from a now-dead connection stays Active (arguably correct — last-known-state), and a Conditional/Expression script never learns quality went Bad. The implementation choice is defensible; the spec sentence should be qualified so the divergence is a decision, not drift.

C5. [Low] Shared scripts execute without the caller's scope and without a Scope assignment

SharedScriptLibrary.ExecuteAsync builds ScriptGlobals without setting Scope (SharedScriptLibrary.cs:98-103), so a shared script always runs ScriptScope.Root (ScriptCompilationService.cs:200-201) even when invoked from a composed-module script. If a shared script uses Attributes[...], it addresses root attributes regardless of caller scope. Possibly intended (shared scripts are instance-agnostic), but it is undocumented in Component-SiteRuntime.md's Shared Script Library section and is a surprise waiting for a template author.

C6. [Low] EvaluateCondition fallback compares value.ToString() to Threshold.ToString() with current-culture double formatting

ScriptActor.cs:474-477: the non-numeric fallback stringifies the double threshold with the host culture (the numeric path was carefully invariant-culture'd; the fallback wasn't). A "1.5" threshold on a de-DE host renders "1,5". Edge-case only (non-parseable values), but inconsistent with the deliberate invariant-culture work two lines up.

---

Underdeveloped Areas

1. Cert-trust convergence after node downtime. The trust broadcast reaches only currently-Up nodes (DeploymentManagerActor.BroadcastToSiteCertStores, :962-1007); a node that is down during the broadcast rejoins with a divergent PKI store and there is no reconcile-on-join, no periodic store diff, and ListServerCertsCommand reads only the singleton's own node (:936-953) so the divergence is invisible from central. Central persistence of trust decisions is the documented follow-up (Component-SiteRuntime.md:125); until then a join-time pull from the peer's CertStoreActor would close the failover hole.
2. Dead option: SiteRuntimeOptions.ConfigFetchRetryCount — "Reserved — consumed by the standby replication fetch in a later task; not yet wired" (SiteRuntimeOptions.cs:63-67). The standby fetch is genuinely single-attempt (SiteReplicationActor.cs:178-216), leaning entirely on reconciliation as backstop.
3. No aggregated live alarm stream — the M7 operator Alarm Summary polls per-instance DebugViewSnapshot Asks; documented as a follow-up in CLAUDE.md/M7 plan. Fine for now; scale limit is the SemaphoreSlim-capped fan-out on the central side.
4. Native alarm rehydration loses display metadata — only condition_json + timestamps persist (SiteStorageService native_alarm_state, :117-124), so a rehydrated condition renders with empty type/category/message until the first source snapshot (NativeAlarmActor.cs:161-167). Acknowledged in comments; worth a line in the component doc's operator-facing section.
5. No stuck-thread/queue-depth observability for the script scheduler (see S2) — ScriptExecutionScheduler exposes nothing to ISiteHealthCollector.
6. Tag-subscription failure path has no retry or site-event entry (see S4) — the spec's Tag Path Resolution retry covers resolution failures inside the DCL, but the instance-side response handling is a hole.
7. Tests: coverage breadth is very good (redeploy races, waiter races, native-alarm swap/cap, seed reads, scheduler, trust model — 17 actor test files, 17 script test files, dedicated DCL alarm/browse/verify suites, plus OpcUaAlarmLiveSmokeTests). Gaps that match the findings above: no test exercises reconnect-then-stale-ConnectionLost (S1), compile-failure-deploy-status (S3), the unknown-connection subscribe race (S4), or sustained-churn fan-out cost (the PerformanceTests project was not examined for these paths).

---

Prioritized Recommendations

1. (S1) Dispose + detach the previous client in OpcUaDataConnection.ConnectAsync / MxGatewayDataConnection.ConnectAsync (and cancel the old MxGateway event loop). This is the single highest-value fix: it removes both a per-flap resource leak and a healthy-connection flapping mechanism.
2. (S3) Make site-side compile failure fail the deployment, per spec: have CreateChildActors report compile outcomes to the Deployment Manager (or pre-compile in ApplyDeployment before actor creation, which also fixes P6 by moving compiles off the instance-start path) and return DeploymentStatus.Failed with the errors.
3. (S2) Add stuck-execution observability (scheduler queue depth + oldest-busy-thread age as health metrics; log the script name on timeout-with-thread-still-busy), and inject command timeouts into script-facing DB connections.
4. (P1) Move trigger-expression evaluation off the actor thread — evaluate on the ScriptExecutionScheduler and post the boolean result back to the actor (edge-state stays actor-confined), or precompile expressions to delegates rather than re-running Roslyn scripts.
5. (S4) Handle SubscribeTagsResponse.Success == false in InstanceActor with a logged, bounded re-subscribe timer (mirror NativeAlarmActor's retry); add a response-timeout retry to NativeAlarmActor.SendSubscribe.
6. (S5/S6) Retry the startup config load on failure; Context.Watch all Instance Actors so an init-failure removes the dead ref and fails the in-flight deployment.
7. (P2/P3) Introduce per-attribute child-subscription routing in InstanceActor and a tag→instances reverse index in DataConnectionActor before scaling to the 500-instance design point.
8. (S8) Enable WAL + explicit busy timeout on the site SQLite stores.
9. (S7) Capture _adapter into locals before every background task in DataConnectionActor (three sites).
10. (C3/C4/C5) Doc fixes: reconcile the reconnect-interval wording, qualify the disconnect-quality-visibility promise, and document shared-script scope semantics — per the repo rule that design docs and code travel together.