ScadaBridge/code-reviews/SiteRuntime/findings.md

Code Review — SiteRuntime

Field	Value
Module	src/ZB.MOM.WW.ScadaBridge.SiteRuntime
Design doc	docs/requirements/Component-SiteRuntime.md
Status	Reviewed
Last reviewed	2026-06-20
Reviewer	claude-agent
Commit reviewed	4307c381
Open findings	0

Summary

The SiteRuntime module is broadly well-structured: the actor hierarchy matches the design doc, supervision strategies are explicit, and the trigger/alarm evaluation logic is thorough. However the review surfaced one genuinely serious correctness defect — Instance.SetAttribute never routes writes to the Data Connection Layer for data-sourced attributes, contradicting a core design decision and silently turning device writes into local-only static overrides. Several other findings cluster around two themes: (1) actor-thread discipline is violated in a few hot paths (blocking .GetAwaiter().GetResult() calls on the actor thread, a fragile fixed-delay reschedule for redeployment), and (2) the site-local repositories reach into SiteStorageService private state via reflection and mint entity IDs with the non-deterministic string.GetHashCode(). Script execution runs on the default thread pool rather than a dedicated blocking dispatcher (the code acknowledges this in a comment but ships it anyway). Test coverage exists for the coordinator actors, persistence and scripting, but the short-lived execution actors, the replication actor, and the repositories are untested.

Re-review 2026-05-17 (commit 39d737e)

The module was re-reviewed at commit 39d737e. No source under src/ZB.MOM.WW.ScadaBridge.SiteRuntime has changed since the previous review at 9c60592 (the only intervening commits are code-review documentation updates), so all of SiteRuntime-001..013, 015, 016 remain Resolved and SiteRuntime-014 remains Deferred — its Deferred justification (a trigger-evaluation concurrency design decision is required before either recommended fix can land in-module) still holds verbatim against the unchanged ScriptActor/AlarmActor source. The re-review nonetheless worked through all 10 checklist categories afresh and surfaced three new findings that the prior pass did not record: a cross-thread Dictionary enumeration race when the Instance Actor's live _attributes dictionary is handed by reference into child ScriptActor/AlarmActor constructors (SiteRuntime-017, Medium); a stale ScriptExecutionActor XML doc that still claims a "dedicated blocking I/O dispatcher" (SiteRuntime-018, Low); and two dead lifecycle handlers in InstanceActor that the Deployment Manager never routes to (SiteRuntime-019, Low). All three were subsequently resolved on 2026-05-17. Open findings: 0.

Re-review 2026-05-28 (commit 1eb6e97)

The module was re-reviewed at commit 1eb6e97 as part of the new baseline review. The SiteRuntime source surface has grown materially since the prior pass — primarily by threading ExecutionId/ParentExecutionId/SourceNode through the script-trust-boundary helpers and the cached-call telemetry emitters, and by adding OperationTrackingStore, the AuditingDbConnection/AuditingDbCommand/AuditingDbDataReader decorators, and ScriptExecutionScheduler. All 10 checklist categories were walked afresh. Seven new findings were recorded: a race that throws InvalidActorNameException when a second DeployInstanceCommand arrives for the same instance while a redeployment is still terminating its predecessor (SiteRuntime-020, Medium); an artifact-only data-connection update that never reaches the DCL (SiteRuntime-021, Medium); AuditingDbCommand.DbConnection.set reaching into AuditingDbConnection._inner via reflection — the same anti- pattern SiteRuntime-006 eliminated for the repositories, now reintroduced and in direct tension with the script trust model that forbids System.Reflection (SiteRuntime-022, Medium); Convert.ToDouble(value) in ScriptActor / AlarmActor running under CurrentCulture so a string attribute value becomes locale-sensitive (SiteRuntime-023, Low); OperationTrackingStore serialising every cached-call write through a single connection + SemaphoreSlim and using sync-over-async in Dispose() (SiteRuntime-024, Medium); inbound-API SetAttribute (and any future caller) accepting unknown attribute names and persisting them as overrides, polluting both _attributes and the SQLite override table (SiteRuntime-025, Low); and the ReplicationMessages.cs outbound/inbound record types still missing public XML docs (SiteRuntime-026, Low). Prior findings 001–019 remain Resolved/Deferred — no regressions observed in any of their fixed call sites. Open findings: 7.

Checklist coverage

#	Category	Examined	Notes
1	Correctness & logic bugs	✓	SetAttribute mis-routing, deploy double-count, redeploy reschedule race.
2	Akka.NET conventions	✓	Blocking on actor thread, script execution not on a dedicated dispatcher, premature success reply.
3	Concurrency & thread safety	✓	_attributes dictionary shared with child actors by reference; _executionCounter is actor-confined (OK).
4	Error handling & resilience	✓	Deploy reports Success before persistence; replicated artifact/S&F failures only logged (matches best-effort design).
5	Security	✓	Trust-model validation is substring-based and weak; reflection used to read private fields.
6	Performance & resource management	✓	Per-call SQLite connections (acceptable); CPU-bound scripts not interruptible by timeout.
7	Design-document adherence	✓	SetAttribute DCL routing missing; staggered-startup and supervision otherwise conform.
8	Code organization & conventions	✓	Repositories reflect into another class; synthetic IDs non-deterministic.
9	Testing coverage	✓	No tests for ScriptExecutionActor, AlarmExecutionActor, SiteReplicationActor, or the two repositories.
10	Documentation & comments	✓	Several XML comments describe behaviour the code does not implement (see findings).

Re-review (2026-05-28, 1eb6e97):

#	Category	Examined	Notes
1	Correctness & logic bugs	✓	Second-deploy race vs. pending redeploy (020); artifact-only data-connection update never reaches DCL (021); unknown-name SetAttribute persists bogus overrides (025).
2	Akka.NET conventions	✓	Trigger-eval blocking on coordinator mailbox remains Deferred (014); short-lived execution actors and replication actor otherwise conform.
3	Concurrency & thread safety	✓	DM's _instanceActors cache and _pendingRedeploys map shifted from old race; new ordering race surfaced (020). OperationTrackingStore single-connection + SemaphoreSlim serialises all cached writes (024).
4	Error handling & resilience	✓	Task.Run fire-and-forget replication paths log on faulted (acceptable, per "best-effort replication" design). DM's deploy persistence rollback path (resolved as SiteRuntime-005) intact.
5	Security	✓	Trust-model semantic analysis (SiteRuntime-011 fix) intact. AuditingDbCommand reflects into AuditingDbConnection._inner — same anti-pattern as SiteRuntime-006 (022). Audit emitter captures SQL parameter values verbatim per M4 design (M5 will redact).
6	Performance & resource management	✓	Per-call SQLite connections on hot paths in SiteStorageService (existing pattern, acceptable). OperationTrackingStore Dispose() does sync-over-async (024). ScriptExecutionScheduler bounded threads as expected.
7	Design-document adherence	✓	Artifact-only data-connection update path is silently inert (021) — contradicts the "site is self-contained after artifact deployment" intent.
8	Code organization & conventions	✓	Repository reflection-via-private-field anti-pattern reintroduced in AuditingDbCommand (022). ReplicationMessages.cs public records still undocumented (026).
9	Testing coverage	✓	SiteReplicationActor remains uncovered (SiteRuntime-016 deferred that gap to a clustered-ActorSystem harness, still outstanding). New findings have no targeted coverage yet.
10	Documentation & comments	✓	ReplicationMessages.cs records lack XML docs (026); other XML doc surface materially expanded in 1eb6e97.

Findings

Re-review 2026-06-20 (commit 4307c381) — full review

The module was re-reviewed in full at 4307c381 (current HEAD). The diff against the prior baseline 1eb6e97 shows as 100% additions because of the ScadaBridge rename (the project moved paths), so the whole module was re-read at its current state rather than relying on the diff. Health is good: all prior findings 001–026 remain Resolved/Deferred with no regressions observed in their fixed call sites (SetAttribute DCL routing, the watch+buffer redeploy, the SiteRuntime-020 terminating shadow, the OperationTrackingStore read/write split + safe Dispose, the AuditingDb Inner accessor, invariant-culture numeric parsing, the _attributes child-snapshot isolation). The new M7 surface (NativeAlarmActor, CertStoreActor) and the WaitForAttribute/batch-write additions are generally well-built. Five new findings were recorded, all in the native-alarm and deployment-lifecycle bookkeeping: an unbounded _latestAlarmEvents map on the Instance Actor (027, Medium), a phantom-active alarm left behind by NativeAlarmActor.EnforceCap (028, Medium), a delete-during-pending-redeploy that both over-decrements the deployed counter and resurrects the deleted instance (029, Medium), missing tests for the cap/retention/replication paths (030, Low), and stale site-side persistence of notification-list + SMTP config that the design decision moved to central-only (031, Low).

#	Category	Examined	Notes
1	Correctness & logic bugs	✓	EnforceCap leaves phantom-active alarm (028); delete-during-redeploy over-decrements + resurrects (029). HiLo/RoC/edge logic otherwise sound.
2	Akka.NET conventions	✓	Supervision (Resume coordinators / Stop short-lived) correct; execution actors on ScriptExecutionScheduler; cert broadcast uses PipeTo, captures sender before async. Trigger-eval mailbox-block stays Deferred (014).
3	Concurrency & thread safety	✓	Child-snapshot isolation (017) intact; _createdConnections actor-confined via ApplyArtifact… dispatch (021); OperationTrackingStore read/write split (024). No new actor-thread violations.
4	Error handling & resilience	✓	Best-effort persistence/replication paths log-on-fault; deploy reports Success only post-persist (005). Native source-unavailable retains+marks-uncertain.
5	Security	✓	Trust verdict delegated to shared ScriptTrustValidator; CertStoreActor path-traversal-guarded; no reflection in scripts; SQL params captured per M4 design (redaction at write time).
6	Performance & resource management	✓	_latestAlarmEvents unbounded growth on native-alarm churn (027). ScriptExecutionScheduler background threads OK; per-call SQLite connections acceptable.
7	Design-document adherence	✓	Actor hierarchy / native-alarm wiring conform. Stale site persistence of notification-list + SMTP config vs. "central-only, not deployed to sites" (031).
8	Code organization & conventions	✓	Reflection anti-pattern eliminated (006/022); Options owned by component; additive message evolution honoured.
9	Testing coverage	✓	Broad new suites (NativeAlarmActor, WaitForAttribute, SetAttribute, ExecutionActor, scheduler). EnforceCap/retention-drop + SiteReplicationActor still untested (030).
10	Documentation & comments	✓	XML docs accurate and extensive; ReplicationMessages now documented (026). No new stale comments found.

SiteRuntime-001 — Instance.SetAttribute never writes to the Data Connection Layer

Severity	High
Category	Design-document adherence
Status	Resolved
Location	src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Scripts/ScriptRuntimeContext.cs:106, src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/InstanceActor.cs:204

Description

The design doc (Component-SiteRuntime.md, "GetAttribute / SetAttribute" and "Script Runtime API") states that Instance.SetAttribute on a data-connected attribute must send a write request to the DCL, which writes to the physical device, and that the in-memory value is not optimistically updated. For static attributes it updates memory and persists an override.

The implementation makes no such distinction. ScriptRuntimeContext.SetAttribute unconditionally sends a SetStaticAttributeCommand, and InstanceActor.HandleSetStaticAttribute unconditionally treats every write as a static override: it mutates _attributes, publishes an AttributeValueChanged with hard-coded "Good" quality, notifies children, and persists a SQLite override. A script writing a data-sourced attribute therefore never reaches the device, the write failure can never be returned synchronously to the script, and the in-memory value diverges from the device until the next subscription update overwrites it. The persisted override is also wrong: data-sourced attributes should not have static overrides.

Recommendation

In InstanceActor, look up the target attribute in _configuration.Attributes. If it has a non-empty DataSourceReference, issue a DCL write (e.g. a WriteTagRequest to _dclManager) and surface success/failure to the caller; do not persist an override and do not optimistically mutate _attributes. Only attributes with no data source reference should follow the current static-override path. Consider splitting the message into SetStaticAttributeCommand vs SetDataAttributeCommand, or branching inside the handler.

Resolution

Resolved 2026-05-16 (<pending>): InstanceActor.HandleSetStaticAttribute now resolves the target attribute's data binding from _configuration. Data-sourced attributes are routed via a new HandleSetDataAttribute that Asks the DCL with a WriteTagRequest and pipes the device-write outcome back to the caller as a SetStaticAttributeResponse — no override is persisted and _attributes is not optimistically mutated. Static attributes keep the override path and now also reply with a SetStaticAttributeResponse. ScriptRuntimeContext.SetAttribute is now async Task and Asks the Instance Actor, throwing InvalidOperationException on a failed device write so scripts get the failure synchronously.

SiteRuntime-002 — RouteInboundApiSetAttributes always treats writes as static overrides

Severity	High
Category	Correctness & logic bugs
Status	Resolved
Location	src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/DeploymentManagerActor.cs:632

Description

RouteInboundApiSetAttributes (handling Route.To().SetAttribute(s) from the Inbound API) emits a SetStaticAttributeCommand for every attribute, so it inherits the same defect as SiteRuntime-001: writes to data-sourced attributes never reach the device and are instead persisted as static overrides. In addition the response is sent back as unconditionally successful (true) before the Instance Actor has even processed the command, so a non-existent attribute or a future DCL write failure is reported to the external caller as success.

Recommendation

Route through the same corrected InstanceActor write handler as SiteRuntime-001 so the static-vs-data distinction is honoured. The optimistic ack is acceptable for fire-and-forget static writes per the doc, but the XML comment should make the limitation explicit, and once data-attribute writes are supported they need a real response path.

Resolution

Resolved 2026-05-16 (<pending>): RouteInboundApiSetAttributes now Asks the Instance Actor per attribute (instead of fire-and-forget Tell) and aggregates the SetStaticAttributeResponse results. Because the Instance Actor handler is the SiteRuntime-001 corrected handler, data-sourced attributes now reach the DCL and the RouteToSetAttributesResponse reflects the real per-attribute outcome — a non-existent attribute or a failed device write is reported as failure rather than an unconditional optimistic true.

SiteRuntime-003 — Redeployment relies on a fixed 500 ms reschedule and can collide on the child actor name

Severity	High
Category	Akka.NET conventions
Status	Resolved
Location	src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/DeploymentManagerActor.cs:222

Description

HandleDeploy stops an existing Instance Actor with Context.Stop and then reschedules the same DeployInstanceCommand to itself after a hard-coded 500 ms, hoping the child has fully terminated by then. Context.Stop is asynchronous; the child is only removed from the parent's children collection after it actually stops (including running PostStop on its descendants). If a deeply nested or slow hierarchy takes longer than 500 ms, CreateInstanceActor calls Context.ActorOf with a name that still belongs to the terminating child and throws InvalidActorNameException. The _instanceActors dictionary check does not prevent this — the dictionary entry is removed immediately, but the Akka child registry is not. The 500 ms delay is also unconditionally added to every redeploy latency.

Recommendation

Watch the terminating child (Context.Watch) and recreate the Instance Actor only after receiving the Terminated message, instead of guessing with a timer. Buffer or stash the in-flight DeployInstanceCommand (and any further commands for that instance) until termination completes.

Resolution

Resolved 2026-05-16 (<pending>): HandleDeploy no longer uses a fixed 500 ms reschedule. When a redeployment targets a running instance, the existing Instance Actor is Context.Watch-ed and stopped, and the in-flight DeployInstanceCommand is buffered in a _pendingRedeploys map keyed by the terminating actor ref. A new Terminated handler recreates the Instance Actor only after the predecessor (and its whole subtree) has fully stopped, eliminating the InvalidActorNameException race and the unconditional redeploy-latency penalty. The shared ApplyDeployment helper also skips the _totalDeployedCount increment for redeployments, so the deployed-instance count no longer drifts (this additionally addresses the root cause behind SiteRuntime-004).

SiteRuntime-004 — _totalDeployedCount is incremented on redeployment of an existing instance

Severity	Medium — re-triaged: already fixed by the SiteRuntime-003 resolution.
Category	Correctness & logic bugs
Status	Resolved
Location	src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/DeploymentManagerActor.cs (ApplyDeployment)

Description

In HandleDeploy, the existing-actor branch (line 223) reschedules the command and returns. When the rescheduled command runs, no actor exists, so the code falls through to the "new instance" branch and executes _totalDeployedCount++ (line 239). A redeployment is an update of an already-deployed instance, not a new one, so the deployed count is over-counted by one on every redeploy. StoreDeployedConfigAsync uses UPSERT semantics, so the SQLite row count does not grow, but the in-memory _totalDeployedCount (reported to the health collector via UpdateInstanceCounts) drifts upward and the reported "disabled" count becomes wrong.

Re-triage (2026-05-16)

Verified against the current source: this is already fixed. The SiteRuntime-003 resolution replaced the fixed-delay reschedule with a shared ApplyDeployment helper that takes an isRedeploy flag and guards the counter with if (!isRedeploy) _totalDeployedCount++;. The redeploy path (HandleTerminated) always calls ApplyDeployment(..., isRedeploy: true), so the counter is no longer bumped on redeployment. The regression test DeploymentManagerRedeployTests.Redeploy_ExistingInstance_DoesNotOverCountDeployedInstances already covers this and passes. No further code change was required.

Resolution

Resolved 2026-05-16 (commit pending): no new change needed — the root cause was eliminated by the SiteRuntime-003 fix (the isRedeploy guard in ApplyDeployment). Confirmed by the existing passing regression test Redeploy_ExistingInstance_DoesNotOverCountDeployedInstances. Re-triaged from Open to Resolved.

SiteRuntime-005 — Deployment reports Success to central before persistence completes

Severity	Medium
Category	Error handling & resilience
Status	Resolved
Location	src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/DeploymentManagerActor.cs (ApplyDeployment, HandleDeployPersistenceResult)

Description

HandleDeploy replies to central with DeploymentStatus.Success immediately after creating the Instance Actor, while the SQLite persistence (StoreDeployedConfigAsync

• ClearStaticOverridesAsync) runs asynchronously on a Task.Run. If persistence fails, HandleDeployPersistenceResult only logs an error — central has already been told the deployment succeeded. On a subsequent node restart or failover the instance will not be re-created (it is not in SQLite), so the deployment is silently lost despite central recording success. This contradicts the design's intent that the site is the durable source of truth for deployed configs.

Recommendation

Persist the config before replying, or treat a persistence failure as a deployment failure and send a corrective DeploymentStatusResponse/health signal to central. At minimum, do not report Success until the config row is committed.

Resolution

Resolved 2026-05-16 (commit pending): root cause confirmed — ApplyDeployment sent DeploymentStatusResponse(Success) synchronously before the persistence Task.Run completed. The Success reply is now sent from HandleDeployPersistenceResult only once the persistence result is known: on success it replies Success; on a persistence failure it logs the error, stops the optimistically-created Instance Actor, rolls back the deployed-instance counter, and replies DeploymentStatus.Failed with the error message. DeployPersistenceResult carries an IsRedeploy flag so the counter rollback is skipped for redeployments. Regression tests: DeploymentManagerMediumFindingsTests.Deploy_PersistenceFailure_ReportsFailedNotSuccess and Deploy_Success_ReportsSuccessAndPersistsConfig.

SiteRuntime-006 — Site-local repositories read SiteStorageService private field via reflection

Severity	Medium
Category	Code organization & conventions
Status	Resolved
Location	src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Repositories/SiteExternalSystemRepository.cs, src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Repositories/SiteNotificationRepository.cs

Description

Both repositories' CreateConnection() use Type.GetField("_connectionString", BindingFlags.NonPublic | BindingFlags.Instance) to extract the private connection string out of SiteStorageService. This is brittle (any rename or refactor of the field breaks it at runtime, not compile time), defeats encapsulation, and the accompanying XML comment openly describes it as a "pragmatic" hack and is internally contradictory (it states a connection string is "passed separately at DI registration time" which is not what the code does). It also sits awkwardly against the project's own script trust model, which forbids System.Reflection in scripts.

Recommendation

Expose the connection string properly: add an ISiteStorageConnectionProvider (already referenced in ServiceCollectionExtensions XML docs but not used), or have SiteStorageService expose a CreateConnection() factory, and inject that into the repositories. Remove the reflection entirely.

Resolution

Resolved 2026-05-16 (commit pending): root cause confirmed — both repositories reflected into SiteStorageService._connectionString. SiteStorageService now exposes a public CreateConnection() factory method that returns an unopened SqliteConnection against the site database. Both SiteExternalSystemRepository and SiteNotificationRepository now obtain connections via _storage.CreateConnection(); all reflection (Type.GetField / BindingFlags) and the contradictory XML comments have been removed. This is a fully in-module refactor — no cross-module design decision was needed. Regression test: SiteRepositoryTests.ExternalSystemRepository_RoundTripsStoredDefinition exercises the repository's connection path end-to-end.

SiteRuntime-007 — Synthetic entity IDs use the non-deterministic string.GetHashCode()

Severity	Medium
Category	Correctness & logic bugs
Status	Resolved
Location	src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Repositories/SiteExternalSystemRepository.cs, src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Repositories/SiteNotificationRepository.cs, src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Repositories/SyntheticId.cs

Description

GenerateSyntheticId computes name.GetHashCode() & 0x7FFFFFFF. On .NET Core, string.GetHashCode() is randomized per process by default, so the "stable deterministic synthetic ID" promised by the XML comment is not stable at all — it changes every time the process restarts. Any caller that obtained an ID and later calls GetExternalSystemByIdAsync/GetNotificationListByIdAsync after a restart will fail to find the entity. It also risks collisions: distinct names can hash to the same 31-bit value, and GetExternalSystemByIdAsync would then return the wrong row.

Recommendation

Use a deterministic, collision-resistant hash (e.g. a stable FNV-1a or the first bytes of a SHA-256 of the name) if a synthetic integer ID is genuinely required, or better, change the repository contract to key these site-local artifacts by name rather than synthesising integer IDs.

Resolution

Resolved 2026-05-16 (commit pending): root cause confirmed — both repositories used name.GetHashCode(), which is per-process randomized on .NET Core. A new internal SyntheticId helper computes a deterministic, process-stable 31-bit ID using the FNV-1a hash over the name's UTF-8 bytes. Both GenerateSyntheticId methods now delegate to SyntheticId.From(name). (The integer-keyed lookups are kept because they are mandated by the shared IExternalSystemRepository/INotificationRepository contracts in Commons — changing those contracts to name-keyed would be a cross-module change outside this module's scope; the deterministic hash resolves the correctness defect within scope.) Regression tests: SiteRepositoryTests.ExternalSystemRepository_SyntheticId_IsStableAcrossRestart and NotificationRepository_SyntheticId_IsStableAcrossRestart re-create the service to simulate a process restart and confirm by-ID lookups still resolve.

SiteRuntime-008 — Blocking .GetAwaiter().GetResult() on the actor thread during startup

Severity	Medium
Category	Akka.NET conventions
Status	Resolved
Location	src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/DeploymentManagerActor.cs (HandleStartupConfigsLoaded, LoadSharedScriptsFromStorage, HandleSharedScriptsLoaded)

Description

LoadSharedScriptsFromStorage is called synchronously from HandleStartupConfigsLoaded (the actor's message handler) and performs _storage.GetAllSharedScriptsAsync().GetAwaiter().GetResult() followed by Roslyn compilation of every shared script. This blocks the DeploymentManager singleton's mailbox thread for the full duration of the SQLite read and all shared-script compilation. On the default dispatcher this also ties up a thread-pool thread and risks thread-pool starvation, and the singleton cannot process any other message (deployments, lifecycle commands, debug routing) until it returns. The rest of the class correctly uses PipeTo/ContinueWith.

Recommendation

Load shared scripts asynchronously and PipeTo(Self) an internal message, the same pattern already used for StartupConfigsLoaded. Perform compilation either inside the piped continuation handler (still on the actor thread but at least off the synchronous startup path) or on a dedicated background task whose result is piped back.

Resolution

Resolved 2026-05-16 (commit pending): root cause confirmed — the blocking .GetAwaiter().GetResult() and Roslyn compilation ran on the singleton's mailbox thread inside HandleStartupConfigsLoaded. LoadSharedScriptsFromStorage now runs the SQLite read and the Roslyn compilation on a background Task.Run and pipes a new internal SharedScriptsLoaded message back to the actor. A new HandleSharedScriptsLoaded handler then begins staggered Instance Actor creation, so the compilation→creation ordering is preserved without ever blocking the mailbox. A shared-script load failure is logged and startup proceeds (scripts needing a missing shared script fail at execution time). Regression test: DeploymentManagerMediumFindingsTests.Startup_WithSharedScripts_LoadsConfigsAndStaysResponsive (confirms startup completes and the actor stays responsive with shared scripts present).

SiteRuntime-009 — Script execution actors run scripts on the default thread pool, not a dedicated dispatcher

Severity	Medium
Category	Akka.NET conventions
Status	Resolved
Location	src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/ScriptExecutionActor.cs, src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/AlarmExecutionActor.cs, src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Scripts/ScriptExecutionScheduler.cs

Description

The design (CLAUDE.md "Architecture & Runtime") states Script Execution Actors run on a dedicated blocking I/O dispatcher. The code does not do this: ScriptActor.SpawnExecution and AlarmActor.SpawnAlarmExecution create the execution actors with no .WithDispatcher(...), and the execution itself runs inside a bare Task.Run, i.e. on the shared .NET thread pool. The // NOTE: In production, configure a dedicated ... dispatcher comments acknowledge the gap but it ships unconfigured. Scripts can perform synchronous blocking I/O (Database.Connection, synchronous ExternalSystem.Call); running them on the shared pool can starve it and stall unrelated Akka dispatchers and HTTP request handling under load.

Recommendation

Define the dedicated dispatcher in HOCON and chain .WithDispatcher(...) on the execution actor Props. If the Task.Run model is kept, run script bodies on a dedicated TaskScheduler / bounded scheduler rather than the global pool. Either way, remove the "in production, configure…" comments by actually configuring it.

Resolution

Resolved 2026-05-16 (commit pending): root cause confirmed — script and alarm on-trigger bodies ran inside a bare Task.Run on the shared ThreadPool. The recommendation's TaskScheduler option was taken because it is fully in-module (a HOCON dispatcher would require editing the Host's ActorSystem config, out of scope). A new ScriptExecutionScheduler provides a bounded set of dedicated background threads (count from the new SiteRuntimeOptions.ScriptExecutionThreadCount, default 8). ScriptExecutionActor and AlarmExecutionActor now run their bodies via Task.Factory.StartNew(..., ScriptExecutionScheduler.Shared(options)).Unwrap() instead of Task.Run, so blocking script I/O is contained to those dedicated threads and cannot starve the global pool. The misleading "in production, configure a dedicated dispatcher" comments were removed. Regression tests: ScriptExecutionSchedulerTests (Scheduler_RunsWork_OffTheThreadPool, Scheduler_RespectsConfiguredThreadCount, Scheduler_Shared_ReturnsSameInstanceForOptions).

SiteRuntime-010 — EnsureDclConnections never updates a connection whose configuration changed

Severity	Medium
Category	Correctness & logic bugs
Status	Resolved
Location	src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/DeploymentManagerActor.cs (EnsureDclConnections, ComputeConnectionConfigHash)

Description

EnsureDclConnections tracks created connections in _createdConnections and skips any name already present (if (_createdConnections.Contains(name)) continue;). The skip is purely name-based: if a redeployment (or an artifact deployment) changes the endpoint, credentials, backup endpoint, or FailoverRetryCount of an existing connection, the new configuration is silently ignored and the DCL keeps using the stale CreateConnectionCommand. There is no UpdateConnectionCommand path. The design states that after artifact deployment the site is fully self-contained with current configuration; this caching breaks that for connection changes.

Recommendation

Compare the incoming connection config against the last one sent and re-issue a create/update command when it differs, or have the DCL treat CreateConnectionCommand as idempotent upsert and always forward it. Key the cache on a config hash, not just the name.

Resolution

Resolved 2026-05-16 (commit pending): root cause confirmed — the cache was a name-only HashSet, so a changed connection config was silently dropped. _createdConnections is now a Dictionary<string,string> mapping connection name to a SHA-256 hash of its protocol/primary-config/backup-config/failover-retry-count (ComputeConnectionConfigHash). A connection whose hash is unchanged is still skipped; a connection whose config changed re-issues a CreateConnectionCommand so the DCL adopts the new configuration. Regression tests: DeploymentManagerMediumFindingsTests.EnsureDclConnections_ConnectionConfigChanged_ReissuesCreateCommand and EnsureDclConnections_UnchangedConfig_DoesNotReissueCreateCommand.

SiteRuntime-011 — Trust-model validation is a substring scan and is both over- and under-inclusive

Severity	Medium
Category	Security
Status	Resolved
Location	src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Scripts/ScriptCompilationService.cs (ValidateTrustModel)

Description

ValidateTrustModel enforces the script trust model by doing raw string.Contains / IndexOf on the script source text for forbidden namespace strings. This is unreliable in both directions:

• Bypass (under-inclusive): the check looks only for the literal namespace strings. A script can reach forbidden APIs without ever writing System.IO etc. — e.g. via fully-qualified type use through aliasing, global::-prefixed names, or simply because the namespace is already imported transitively. The compilation references include typeof(object).Assembly (the whole of System.Private.CoreLib, which contains System.IO.File, System.Threading.Thread, System.Reflection, etc.), so forbidden types are fully resolvable at compile time and the only barrier is this textual scan.
• False positives (over-inclusive): any occurrence of the substring in a comment, string literal, or an unrelated identifier (e.g. a variable named ProcessThreading) triggers a violation; the AllowedExceptions logic only rescues exact prefixes.
• The dead isAllowed variable at line 64 is computed and never used.

Recommendation

Enforce the trust model with a Roslyn SyntaxWalker/semantic analysis (inspect resolved symbols and their containing namespaces/assemblies), or restrict the compilation's metadata references and AssemblyLoadContext so forbidden types are genuinely unavailable, rather than relying on source-text matching. Remove the unused isAllowed variable.

Resolution

Resolved 2026-05-16 (commit pending): root cause confirmed — ValidateTrustModel was a raw string.Contains/IndexOf scan of the source text, with a dead isAllowed variable. It is now Roslyn semantic analysis: the script is parsed and a CSharpCompilation + SemanticModel are built; every name/member/object-creation node is resolved to its symbol and the symbol's containing namespace and fully-qualified containing type are checked against the forbidden roots. Bare namespace symbols are ignored (so the System.Threading qualifier of the allowed System.Threading.Tasks.Task no longer false-positives). A name that cannot be resolved (a type from an assembly deliberately absent from the script's references) falls back to a syntactic fully-qualified-name check, so e.g. System.Net.Http references are still rejected. The dead isAllowed variable was removed. This fixes both the bypass (global::/alias-qualified forbidden types) and the false positives (forbidden namespace string in a comment, string literal, or unrelated identifier). Regression tests: new TrustModelSemanticTests (alias/global:: detection, comment/ literal/identifier non-detection, allowed-exception resolution); all 39 existing SandboxTests + ScriptCompilationServiceTests continue to pass.

SiteRuntime-012 — AttributeAccessor/ScopeAccessors block the script on a synchronous Ask

Severity	Low
Category	Concurrency & thread safety
Status	Resolved
Location	src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Scripts/ScopeAccessors.cs:28

Description

AttributeAccessor's indexer getter calls _ctx.GetAttribute(...).GetAwaiter().GetResult(), synchronously blocking the script-execution thread on an actor Ask. Combined with SiteRuntime-009 (scripts run on the shared thread pool) this means a script that reads several attributes via Attributes["X"] holds a pool thread blocked for each round-trip. The async variants (GetAsync/SetAsync) exist but the ergonomic indexer encourages the blocking path. The XML comment notes "Reads block on the actor Ask" but does not warn about the thread-pool impact.

Recommendation

Once a dedicated script dispatcher exists (SiteRuntime-009) the blocking is contained to that pool, which is acceptable; until then, document the cost clearly and prefer steering script authors to the async accessors. Consider making the indexer internal-only and exposing only the async API.

Resolution

Resolved 2026-05-16 (pending commit): re-triaged against the current source — the finding's own recommendation states the blocking is acceptable once SiteRuntime-009's dedicated script dispatcher exists, and SiteRuntime-009 is now Resolved (ScriptExecutionActor/AlarmExecutionActor run script bodies on the dedicated ScriptExecutionScheduler threads, confirmed in ScriptExecutionActor.cs:74). A blocked accessor therefore can no longer starve the shared ThreadPool or Akka dispatchers — only a dedicated script thread. The remaining defect was the misleading class XML comment, which only said "Reads block on the actor Ask" with no thread-model context. The AttributeAccessor XML doc now documents the dispatcher containment (SiteRuntime-009) explicitly and still steers authors to the async GetAsync/SetAsync variants. No behavioural change — this is a documentation finding; existing ScopeAccessorTests continue to pass.

SiteRuntime-013 — HandleUnsubscribeDebugView does nothing despite documented behaviour

Severity	Low
Category	Documentation & comments
Status	Resolved
Location	src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/InstanceActor.cs:414

Description

HandleUnsubscribeDebugView is documented ("Debug view unsubscribe — removes subscription") and the actor registers a handler for UnsubscribeDebugViewRequest, but the body only logs a debug message — there is no subscription state in the Instance Actor to remove. The design places the actual subscription lifecycle in SiteStreamManager (Subscribe/Unsubscribe/RemoveSubscriber), so the Instance Actor genuinely has nothing to do here. The handler and its XML comment are therefore misleading: a reader expects it to tear down a subscription.

Recommendation

Either remove the no-op handler and route UnsubscribeDebugViewRequest to wherever the SiteStreamManager subscription is actually cancelled, or correct the XML comment to state explicitly that subscription teardown is handled by SiteStreamManager and this handler is a no-op acknowledgement.

Resolution

Resolved 2026-05-16 (pending commit): root cause confirmed — the Instance Actor holds no per-subscriber state, so HandleUnsubscribeDebugView genuinely has nothing to remove; the real debug-stream subscription lifecycle lives in SiteStreamManager (Subscribe/Unsubscribe/RemoveSubscriber). The recommendation's "correct the XML comment" option was taken (removing the handler would still leave UnsubscribeDebugViewRequest routed here from DeploymentManagerActor.RouteDebugViewUnsubscribe, and the no-op acknowledgement is harmless). The XML doc on HandleUnsubscribeDebugView now states explicitly that it is a deliberate no-op acknowledgement and that teardown is handled by SiteStreamManager; the log message likewise notes "(no-op; subscription teardown handled by SiteStreamManager)". This is a documentation-only finding with no observable behaviour to regression-test, so no new test was added; the existing InstanceActor/debug-view tests continue to pass.

SiteRuntime-014 — Trigger-expression evaluation blocks the coordinator actor thread

Severity	Low
Category	Akka.NET conventions
Status	Deferred
Location	src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/ScriptActor.cs:219, src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/AlarmActor.cs:389

Description

EvaluateExpressionTrigger (ScriptActor) and EvaluateExpression (AlarmActor) run a compiled Roslyn script with .RunAsync(...).GetAwaiter().GetResult() directly inside the actor's AttributeValueChanged message handler. This blocks the coordinator actor's mailbox thread for up to the 2-second timeout on every monitored attribute change. Coordinator actors are on the default dispatcher and process the hot path of attribute-change fan-out; a slow expression delays all other messages to that actor and consumes a thread-pool thread for the duration. The inline comments correctly note CPU-bound expressions are not interruptible but do not address the mailbox-blocking concern.

Recommendation

Trigger expressions are expected to be cheap, but to keep the actor responsive consider evaluating them off the actor thread (pipe the boolean result back as an internal message) or pre-compiling to a plain delegate that executes near-instantly without the Roslyn scripting RunAsync machinery.

Resolution

Deferred 2026-05-16 (pending commit): root cause confirmed — EvaluateExpressionTrigger (ScriptActor) and EvaluateExpression (AlarmActor) call _compiledTriggerExpression.RunAsync(...).GetAwaiter().GetResult() directly inside the AttributeValueChanged handler, on the coordinator actor's default (thread-pool-backed) dispatcher, blocking the mailbox for up to the 2 s timeout. Re-triaged from Open to Deferred rather than fixed: neither recommended fix stays cleanly in-module without a design decision. (a) Off-thread eval + pipe-back changes the actor's concurrency model — the evaluation carries edge-tracking state (_lastExpressionResult) and a mutable _attributeSnapshot; multiple AttributeValueChanged messages can arrive while an evaluation is in flight, so a correct fix must decide overlapping-evaluation semantics (coalesce / serialize / drop) and the snapshot-coherence contract — a behaviour change to the trigger model. (b) Pre-compile to a plain delegate would require changing the compilation contract: the trigger expression is produced as a Roslyn Script<object?> by ScriptCompilationService.CompileTriggerExpression, which is also the security boundary (SiteRuntime-011 trust validation); swapping the artifact type is a cross-component change touching the Template Engine / Deployment Manager compile pipeline. Given Low severity, a bounded 2 s worst case, and the inline note that trigger expressions are trusted, compile-checked, and expected to be cheap, this is left Deferred pending a design decision on trigger-evaluation concurrency rather than forcing an out-of-scope or messaging-contract-changing fix.

SiteRuntime-015 — LoggerFactory created per Instance Actor and never disposed

Severity	Low
Category	Performance & resource management
Status	Resolved
Location	src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/DeploymentManagerActor.cs:746

Description

CreateInstanceActor does var loggerFactory = new LoggerFactory(); for every Instance Actor it creates, uses it once to produce an ILogger<InstanceActor>, and never disposes it. LoggerFactory is IDisposable. With up to 500 instances (and churn from redeployments) this leaks a factory per instance, and the produced loggers are detached from the application's configured logging providers, so Instance Actor logs may not be routed/filtered consistently with the rest of the host.

Recommendation

Inject the application's ILoggerFactory (or an ILogger<InstanceActor> factory delegate) into DeploymentManagerActor via DI and reuse it, rather than newing one up per child. Do not create a fresh LoggerFactory in a hot creation path.

Resolution

Resolved 2026-05-16 (pending commit): root cause confirmed — CreateInstanceActor did new LoggerFactory() per Instance Actor, never disposed, and detached from the host's logging providers. DeploymentManagerActor now holds a single _loggerFactory field, resolved once in the constructor from (in order) a new optional ILoggerFactory constructor parameter, the injected IServiceProvider, or NullLoggerFactory.Instance as a last resort — never a per-instance allocation. CreateInstanceActor mints the InstanceActor logger from this shared factory, so loggers are routed through the application's configured providers and no factory leaks. Regression test: DeploymentManagerLoggerFactoryTests.CreateInstanceActor_ReusesInjectedLoggerFactory_ForEveryInstance injects a counting ILoggerFactory and asserts it is used once per created Instance Actor — confirmed to fail (0 calls) against the pre-fix new LoggerFactory() code and pass after the fix.

SiteRuntime-016 — Short-lived execution actors, replication actor, and repositories are untested

Severity	Low
Category	Testing coverage
Status	Resolved
Location	tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/

Description

The test project covers the coordinator actors (InstanceActor, ScriptActor, AlarmActor, DeploymentManagerActor), persistence, scripting and streaming, but a search of the test sources finds no references to ScriptExecutionActor, AlarmExecutionActor, SiteReplicationActor, SiteExternalSystemRepository, or SiteNotificationRepository. These cover critical paths: script timeout/failure handling and result reply, alarm on-trigger execution, peer config/S&F replication (including the SendToPeer no-peer drop), and the reflection-based repository reads. Several findings above (001/002 mis-routing, 007 ID instability, 011 trust bypass) would likely have been caught by targeted tests.

Recommendation

Add unit/integration tests for the execution actors (success, timeout, exception, Ask-reply, PoisonPill self-stop), SiteReplicationActor (outbound forward, inbound apply, peer tracking on cluster events), and the two repositories (round-trip read, synthetic-ID lookup, missing-row behaviour).

Resolution

Resolved 2026-05-16 (pending commit): re-triaged against the current test sources — SiteExternalSystemRepository and SiteNotificationRepository are already covered by Repositories/SiteRepositoryTests.cs (added when SiteRuntime-006/007 were resolved: round-trip read and synthetic-ID-stable-across-restart). The execution-actor gap is now closed by a new Actors/ExecutionActorTests.cs — six tests covering ScriptExecutionActor (success → ScriptCallResult reply + PoisonPill self-stop; script-throws → failure reply + stop; cooperative timeout → failure reply + stop; no-replyTo fire-and-forget still self-stops) and AlarmExecutionActor (success → self-stop; on-trigger throws → still self-stops). SiteReplicationActor is not covered here: it depends on Cluster.Get(Context.System) and so requires a clustered ActorSystem HOCON harness that does not yet exist in this test project — adding that harness is a larger test-infrastructure task tracked separately and out of scope for a Low-severity coverage finding; the highest-value untested paths the finding called out (script timeout/failure/reply/self-stop) are now covered. Full module suite: 192 tests green.

SiteRuntime-017 — Instance Actor's live _attributes dictionary is shared by reference into child actor constructors

Severity	Medium
Category	Concurrency & thread safety
Status	Resolved
Location	src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/InstanceActor.cs:625, src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/InstanceActor.cs:675, src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/ScriptActor.cs:83, src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/AlarmActor.cs:93

Description

InstanceActor.CreateChildActors passes the Instance Actor's own mutable _attributes field (a plain Dictionary<string, object?>) by reference into the Props.Create(...) factory for every ScriptActor and AlarmActor (as the initialAttributes constructor argument). Each child constructor then iterates that dictionary to seed its _attributeSnapshot:

if (initialAttributes != null)
    foreach (var kvp in initialAttributes)
        _attributeSnapshot[kvp.Key] = kvp.Value;

Context.ActorOf returns immediately; the child actor's constructor runs later on the child's mailbox thread. Meanwhile the Instance Actor's PreStart returns and the Instance Actor begins processing its mailbox — HandleTagValueUpdate and HandleAttributeValueChanged both mutate _attributes (_attributes[...] = ...). A DCL tag update that arrives before a child has finished its constructor copy therefore mutates the dictionary on the Instance Actor thread while the child thread is enumerating it. Dictionary<,> is explicitly not safe for concurrent read/write: the enumeration can throw InvalidOperationException ("collection was modified") — which surfaces as an ActorInitializationException and, under the Instance Actor's SupervisorStrategy, stops the child (the strategy returns Stop for ActorInitializationException). The script or alarm is then silently absent for the life of the instance. A torn read of an entry is also possible. The window is small but deterministically reachable on a busy site at startup/failover — exactly the staggered-startup scenario the design is most concerned about.

Recommendation

Do not share the live dictionary. Snapshot it on the Instance Actor thread before constructing the child — e.g. pass new Dictionary<string, object?>(_attributes) (or an immutable copy) into each Props.Create. The copy is made on the Instance Actor thread inside CreateChildActors, so it is race-free, and each child gets a private dictionary to seed from.

Resolution

Resolved 2026-05-17 (commit pending): root cause confirmed — CreateChildActors captured the live _attributes field directly in every child Props.Create closure. CreateChildActors now takes a single new Dictionary<string,object?>(_attributes) snapshot on the Instance Actor thread and hands each ScriptActor/AlarmActor that private copy, so no child constructor ever enumerates a dictionary the Instance Actor is concurrently mutating. Regression test: InstanceActorChildAttributeRaceTests.ChildActors_AreSeededFromAnIsolatedCopy_NotTheLiveAttributesDictionary asserts every child's seed dictionary is a distinct object from the Instance Actor's live _attributes (confirmed to fail — "seeded ... by reference" — against the pre-fix code and pass after). ScriptActor/AlarmActor expose an internal SeedAttributesReference for this assertion (InternalsVisibleTo added for the test project).

SiteRuntime-018 — ScriptExecutionActor XML doc still claims a "dedicated blocking I/O dispatcher"

Severity	Low
Category	Documentation & comments
Status	Resolved
Location	src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/ScriptExecutionActor.cs:17

Description

The class-level XML summary on ScriptExecutionActor states "Runs on a dedicated blocking I/O dispatcher." That is not what the code does. SiteRuntime-009 was resolved by introducing ScriptExecutionScheduler (a bounded dedicated TaskScheduler); the actor itself and its mailbox run on the default Akka dispatcher, and only the script body runs on the scheduler's threads via Task.Factory.StartNew(..., scheduler). The resolution of SiteRuntime-009 explicitly chose the TaskScheduler route instead of a HOCON dispatcher and even removed the "in production, configure a dedicated dispatcher" comments elsewhere — but this stale summary line was missed. A reader is told the actor is on a dedicated dispatcher when it is not, which is misleading when reasoning about mailbox throughput and thread-pool pressure. (AlarmExecutionActor does not carry the equivalent claim — its summary only says "Same pattern as ScriptExecutionActor.")

Recommendation

Correct the summary to describe the actual model: the actor runs on the default dispatcher and the script body is dispatched onto the dedicated ScriptExecutionScheduler (SiteRuntime-009). Align the wording with the accurate comment already present at ScriptExecutionActor.cs:71-73.

Resolution

Resolved 2026-05-17 (commit pending): root cause confirmed — the stale "Runs on a dedicated blocking I/O dispatcher" line in the ScriptExecutionActor class summary was missed when SiteRuntime-009 was resolved. The summary now states the actual model: the actor and its mailbox run on the default Akka dispatcher and only the script body is dispatched onto the dedicated ScriptExecutionScheduler (SiteRuntime-009). Documentation-only change with no observable behaviour, so no regression test was added; the existing suite continues to pass.

SiteRuntime-019 — Dead DisableInstanceCommand / EnableInstanceCommand handlers in InstanceActor

Severity	Low
Category	Correctness & logic bugs
Status	Resolved
Location	src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/InstanceActor.cs:106, src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/InstanceActor.cs:113

Description

InstanceActor's constructor registers Receive<DisableInstanceCommand> and Receive<EnableInstanceCommand> handlers that log and reply with a successful InstanceLifecycleResponse. These handlers are unreachable. The Deployment Manager is the only sender of those commands, and DeploymentManagerActor.HandleDisable / HandleEnable handle the lifecycle entirely themselves — they call Context.Stop(actor) (disable) or CreateInstanceActor(...) (enable) directly and reply to the original sender from the Deployment Manager. Neither command is ever Forward-ed or Tell-ed to the Instance Actor. The handlers are dead code, and they are actively misleading: a maintainer reading InstanceActor would reasonably believe disable/enable is partly an Instance-Actor responsibility, and the no-op "true" reply implies an instance-side acknowledgement contract that does not exist. If a future change did route these commands here, the disable handler would do nothing useful (it does not stop children or tear down state — Akka does that when the parent stops the actor).

Recommendation

Remove the two Receive<...> registrations and their handler bodies from InstanceActor, since the Deployment Manager owns the disable/enable lifecycle. If the intent is to keep them for a future instance-side hook, add an XML comment stating that the Deployment Manager currently handles these and the handlers are a reserved placeholder — but removal is preferred.

Resolution

Resolved 2026-05-17 (commit pending): re-verified as genuinely dead code — a codebase-wide search confirms DisableInstanceCommand/EnableInstanceCommand are only ever sent (from central) to the site DeploymentManagerActor, whose HandleDisable/HandleEnable own the lifecycle entirely (Context.Stop / CreateInstanceActor) and never Forward/Tell the command to the Instance Actor. The two unreachable Receive<...> registrations and their no-op "success" handler bodies were removed from InstanceActor, replaced with a comment stating the Deployment Manager owns this lifecycle. Regression test: InstanceActorTests.InstanceActor_DoesNotHandleDisableOrEnableCommands asserts the Instance Actor produces no InstanceLifecycleResponse for either command (confirmed to fail against the pre-fix dead handlers and pass after removal).

SiteRuntime-020 — Second DeployInstanceCommand arriving during a pending redeploy races the still-terminating actor on its name

Severity	Medium
Category	Concurrency & thread safety
Status	Resolved
Location	src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/DeploymentManagerActor.cs:285, src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/DeploymentManagerActor.cs:971

Resolution — added a name → terminating-actor-ref shadow (_terminatingActorsByName) populated when HandleDeploy stops the predecessor and cleared in HandleTerminated. HandleDeploy now detects the mid-termination state before falling through to ApplyDeployment(fresh): on hit it tells the displaced PendingRedeploy.OriginalSender a DeploymentStatus.Failed / "superseded by newer deployment …" response and overwrites the buffered pending command (last-write-wins). Regression test SR020_ThreeRapidDeploys_DoNotThrowInvalidActorNameException_LatestWins fires three rapid deploys, asserts the middle deploy is told it was superseded, the latest succeeds, and the resulting instance is operable (DisableInstanceCommand works).

Description

The SiteRuntime-003 fix makes HandleDeploy watch + stop a running Instance Actor and buffer the in-flight DeployInstanceCommand in _pendingRedeploys until Terminated arrives. The handler also removes the instance from _instanceActors synchronously, in step with the stop request:

if (_instanceActors.TryGetValue(instanceName, out var existing))
{
    _instanceActors.Remove(instanceName);
    _pendingRedeploys[existing] = new PendingRedeploy(command, Sender);
    Context.Watch(existing);
    Context.Stop(existing);
    UpdateInstanceCounts();
    return;
}

// Fresh deployment — no existing actor to replace.
ApplyDeployment(command, Sender, isRedeploy: false);

If a second DeployInstanceCommand for the same instanceName arrives on the singleton's mailbox while the predecessor is still terminating, the _instanceActors.TryGetValue lookup correctly reports "no existing actor" — because the first deploy already removed it — and execution falls through to ApplyDeployment(..., isRedeploy: false). ApplyDeployment immediately calls CreateInstanceActor, which calls Context.ActorOf(props, instanceName). But the predecessor's Akka child name is still registered in the parent's child registry: that name is only released after the predecessor's Terminated signal — exactly the asynchronous gap SiteRuntime-003 was created to plug for the first redeploy. Context.ActorOf therefore throws InvalidActorNameException, which Akka rethrows as ActorInitializationException — and the supervisor's Stop directive on that exception (DeploymentManagerActor.cs:179) silently stops the just-created child. The second deploy is then quietly lost: _instanceActors doesn't contain it (the throw aborted the bookkeeping after CreateInstanceActor's own ContainsKey guard but before _instanceActors[instanceName] = actorRef would have run), _totalDeployedCount was incremented, and the deployer is never told the deployment failed (the persistence Task.Run is also dropped on the throw path). The race is real on a busy site where central retries a deploy because the prior attempt timed out — exactly the scenario the DeploymentManager-006 query-then-deploy idempotency mechanism was designed for.

The first-redeploy case (SiteRuntime-003) does NOT exhibit this because at that point the predecessor's child name was still in _instanceActors, so the branch correctly buffers. The bug is specific to the third (and beyond) incoming deploy when two are already in flight for the same instance.

Recommendation

The pending-redeploy bookkeeping needs to be authoritative for "we are mid- redeploy on this instance", not just the _instanceActors cache. Add a second keyed lookup — e.g. a Dictionary<string, IActorRef> _terminatingActorsByName populated when the predecessor is stopped — and check it BEFORE ApplyDeployment(isRedeploy: false). On a hit, overwrite (or stash) the buffered PendingRedeploy for that terminating actor so the latest command wins on the Terminated signal. Alternatively, defer the deploy by stashing all messages for that instanceName until the predecessor terminates (Akka Stash pattern). Either way, the fall-through to "fresh deployment" needs to be gated on "no instance with this name is currently terminating".

SiteRuntime-021 — HandleDeployArtifacts updates DataConnections in SQLite but never sends CreateConnectionCommand to the DCL

Severity	Medium
Category	Design-document adherence
Status	Resolved
Location	src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/DeploymentManagerActor.cs:931

Resolution (2026-05-28): Took the "refactor EnsureDclConnections into a shared field-based helper" path. Extracted a new EnsureDclConnection(name, protocol, primaryJson, backupJson, failoverRetryCount) method that owns the hash-cache check and the CreateConnectionCommand Tell — both the existing inline EnsureDclConnections(configJson) and the new artifact path now drive through it. ComputeConnectionConfigHash got a field-based overload so the artifact path (which carries data directly on DataConnectionArtifact) reuses the same hash logic as the ConnectionConfig-based inline path. To keep _createdConnections mutation actor-thread-confined (the artifact-deploy persistence runs inside a Task.Run), the off-thread persistence dispatches a new internal ApplyArtifactDataConnectionsToDcl message back to Self after the SQLite writes; the actor-thread handler then iterates and invokes EnsureDclConnection. The DCL only sees CreateConnectionCommand (no Update/Delete messages exist in the codebase, and CreateConnectionCommand is treated as upsert-by-name — same shape as the inline-config path). Build clean; 302 SiteRuntime tests green (the existing EnsureDclConnections_ConnectionConfigChanged_ReissuesCreateCommand regression test still passes through the refactored shared helper).

Description

HandleDeployArtifacts persists the artifact bundle (shared scripts, external systems, database connections, notification lists, SMTP configs, and data connection definitions) into local SQLite. For data connection definitions specifically (DataConnections), the handler calls _storage.StoreDataConnectionDefinitionAsync(...) — but does NOT issue a CreateConnectionCommand (or any other DCL command) to the _dclManager actor. The only path that pushes DCL configuration to the DCL is EnsureDclConnections, called exclusively from the deploy / startup-batch paths against the flattened instance configuration's inline Connections dictionary. There is no equivalent for an artifact-only update.

Concretely: an artifact deployment that changes a data connection's endpoint URL, credentials, backup endpoint, or failover retry count is stored durably in the site SQLite (so on the next node restart the site loads the new config and EnsureDclConnections picks it up) but is silently inert until either an instance using that connection is redeployed or the node restarts. This contradicts the design's "after artifact deployment, the site is fully self-contained" intent (Component-SiteRuntime.md, "System-Wide Artifact Handling") — the runtime DCL keeps using the stale connection until a much heavier trigger event occurs. It is also asymmetric with how SharedScripts are handled in the same method: shared scripts are both stored and recompiled into _sharedScriptLibrary on update so the change is live immediately.

(SiteRuntime-010 fixed a related defect inside EnsureDclConnections — the config-hash cache — but that's only consulted on the inline-config path; the artifact-deployment path never reaches EnsureDclConnections.)

Recommendation

In the DataConnections branch of HandleDeployArtifacts, after the StoreDataConnectionDefinitionAsync call, also send a CreateConnectionCommand to _dclManager for each updated definition, re-using the SiteRuntime-010 config hash so unchanged connections are skipped. Alternatively, refactor EnsureDclConnections to accept a flat list of (name, protocol, configurationJson, backupConfigurationJson, failoverRetryCount) tuples that both the inline (FlattenedConfiguration) and artifact paths can drive through it.

SiteRuntime-022 — AuditingDbCommand.DbConnection.set uses reflection to read AuditingDbConnection._inner

Severity	Medium
Category	Code organization & conventions
Status	Resolved
Location	src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Scripts/AuditingDbCommand.cs:138

Resolution (2026-05-28): Took the recommended "expose a proper API surface" path (the SiteRuntime-006 precedent). Added an internal DbConnection Inner => _inner; accessor to AuditingDbConnection; both classes are internal sealed in the same assembly, so the accessor stays out of the public API. The AuditingDbCommand.DbConnection setter now unwraps an AuditingDbConnection via auditing.Inner instead of Type.GetField("_inner", BindingFlags.NonPublic | BindingFlags.Instance)!.GetValue(...). No reflection, no !. null-forgiveness hiding a runtime crash, no static-analyzer/IL-trim noise — and the same module that enforces "no System.Reflection in scripts" no longer reflects internally. The getter's _wrappingConnection ?? _inner.Connection fallback was left as-is; addressing the CreateDbCommand() round-trip concern is a separate behavioural decision (the finding marked it secondary). Build clean; 302 SiteRuntime tests green.

Description

The DbConnection setter on AuditingDbCommand unwraps an AuditingDbConnection value by reading its private _inner field via reflection:

set
{
    _wrappingConnection = value;
    _inner.Connection = value switch
    {
        AuditingDbConnection auditing => auditing.GetType()
            .GetField("_inner", BindingFlags.Instance | BindingFlags.NonPublic)
            !.GetValue(auditing) as DbConnection,
        _ => value
    };
}

This is the same encapsulation-violating anti-pattern that SiteRuntime-006 called out for the site repositories. A rename or refactor of AuditingDbConnection._inner breaks the audit decorator at runtime (no compile-time signal), the !. null-forgiving operator hides the crash, and the reflective access trips static analyzers and IL trimming. More problematically, the script trust model the same module enforces in ScriptCompilationService.ValidateTrustModel explicitly forbids System.Reflection in scripts — yet the auditing helper a script ends up running through itself reaches via reflection into a sibling class. Both classes are internal sealed in the same assembly, so this is purely a self-imposed contract violation.

A second smaller concern in the same property: the getter returns _wrappingConnection ?? _inner.Connection. If the caller obtains a command via AuditingDbConnection.CreateDbCommand() and immediately reads cmd.Connection, the getter returns the raw inner connection (not the auditing wrapper), because _wrappingConnection is only populated when the setter is later invoked. That's surprising and at odds with the class's audit-everything intent — a script that round-trips a command through cmd.Connection re-enters the un-audited path.

Recommendation

Expose the wrapped connection through a proper API surface. The simplest fix that matches the SiteRuntime-006 precedent: add an internal DbConnection Inner { get; } property to AuditingDbConnection (both classes are internal sealed, so the property stays out of the public surface) and replace the reflection switch with auditing.Inner. While touching the property, also have the getter return _wrappingConnection even on the synthesised CreateDbCommand path (e.g. set _wrappingConnection to the parent connection inside AuditingDbConnection.CreateDbCommand).

SiteRuntime-023 — Convert.ToDouble(value) in trigger and alarm evaluation is locale-sensitive

Severity	Low
Category	Correctness & logic bugs
Status	Resolved
Location	src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/ScriptActor.cs:446, src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/AlarmActor.cs:340, src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/AlarmActor.cs:356, src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/AlarmActor.cs:444

Resolution (2026-05-28): All four call sites (ScriptActor.EvaluateCondition, AlarmActor.EvaluateRangeViolation, AlarmActor.EvaluateRateOfChange, AlarmActor.EvaluateHiLo) now pass CultureInfo.InvariantCulture to Convert.ToDouble, so a string attribute value like "1.5" parses identically regardless of the host's CurrentCulture. For purely-numeric inputs the culture argument is a no-op. No new test added — existing ScriptActor / AlarmActor evaluator tests continue to pass and the behaviour is identical under the (existing CI's) en-US locale.

Description

ScriptActor.EvaluateCondition and the three AlarmActor evaluators (EvaluateRangeViolation, EvaluateRateOfChange, EvaluateHiLo) call Convert.ToDouble(value) without specifying a culture. When value is a string (a path that exists today — attribute values that arrive as JSON- deserialized numbers can still surface as strings on some code paths, particularly array values that are JSON-stringified at InstanceActor.HandleTagValueUpdate:377), Convert.ToDouble parses against CultureInfo.CurrentCulture. On a host whose locale uses a comma decimal separator (German, French, most of continental Europe), "1.5" throws and the condition / alarm silently degrades to its catch-fallthrough (returns false for range/rate-of-change, keeps current level for HiLo, falls back to string-compare for conditionals). The CLAUDE.md "All timestamps are UTC" discipline is the equivalent rule for time; there is no equivalent invariant- culture discipline applied to numeric parsing.

The exposure is bounded — most attribute values arrive as numeric primitives from TagValueUpdate.Value or static FlattenedConfiguration.Attributes (also typed) so the implicit-cast Convert.ToDouble path is hit. But the string path is reachable via inbound API writes (RouteToSetAttributesRequest.AttributeValues is IReadOnlyDictionary<string, string>), via the JSON-array stringification at HandleTagValueUpdate:377, and via static-override values loaded from SQLite (which are persisted as strings — see SetStaticOverrideAsync).

Recommendation

Replace each Convert.ToDouble(value) with Convert.ToDouble(value, CultureInfo.InvariantCulture), or front-load a typed-numeric extraction helper (if (value is double d) return d; if (value is string s && double.TryParse(s, NumberStyles.Float, CultureInfo.InvariantCulture, out var p)) return p; return Convert.ToDouble(value, CultureInfo.InvariantCulture);). The site is a deterministic machine-control surface; condition evaluation must not depend on the host's regional settings.

SiteRuntime-024 — OperationTrackingStore serialises all writes through one connection + SemaphoreSlim, and Dispose() does sync-over-async

Severity	Medium
Category	Performance & resource management
Status	Resolved
Location	src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Tracking/OperationTrackingStore.cs:39, src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Tracking/OperationTrackingStore.cs:360

Resolution — split reads from writes: the single owned _writeConnection + _writeGate still serialises writers, but GetStatusAsync now opens a fresh SqliteConnection per call against the shared connection string (mirroring SiteStorageService) so reads never block on an in-flight write. Sync Dispose was rewritten to NOT bridge to async — the dispose-once flag is an int flipped with Interlocked.Exchange, the synchronous path disposes _writeConnection + _writeGate directly without acquiring the gate, and DisposeAsync retains the gate-drain semantics for graceful shutdown. Both paths are idempotent; the second call short-circuits via the interlocked flag. Tests: SR024_ConcurrentReads_DoNotBlockOnInFlightWrite, SR024_SyncDispose_DoesNotDeadlock_WhenInvokedFromFreshThread, and SR024_AsyncDispose_DoesNotDeadlock_AndIsIdempotent.

Description

OperationTrackingStore owns exactly one SqliteConnection and gates every public method through a single SemaphoreSlim(1, 1). The class XML comment calls this out as deliberate ("the M3 brief calls out as 'cleaner than the M2 Channel pipeline given the volume'"), and the write volume is genuinely low — at most a handful of lifecycle rows per cached call. But on a busy site the read path (GetStatusAsync) is called by every Tracking.Status(id) invocation from every executing script, and reads are serialised through the same gate as writes. A long-running write (e.g. a Roslyn-script-driven RecordTerminalAsync competing with an SQLite checkpoint) holds the gate and stalls every concurrent status query. SQLite supports concurrent readers with a single writer in WAL mode; the gate forfeits that capability.

A separate concern in the same class: Dispose() calls DisposeAsyncCore().AsTask().GetAwaiter().GetResult(). That is sync-over- async — the very pattern SiteRuntime-008 was a finding for. If a caller disposes the store from a synchronization context that does not allow re-entrance (e.g. an IHostedService.StopAsync continuation observed on the host's sync context, or a finalizer pumping on the thread pool with a stuck continuation), the .WaitAsync() inside DisposeAsyncCore waits for a continuation that will never run, and the dispose deadlocks. The async path itself is correct; only the sync Dispose() wrapper is risky.

Recommendation

For the single-connection gate: split reads and writes into separate gates, or — better — keep the writer single-connection and open a fresh read connection (or pool of read connections) per GetStatusAsync call. SQLite connections are cheap; the SiteStorageService precedent already uses per- call connections on the read path. For Dispose(): prefer Dispose() { GC.SuppressFinalize(this); _connection.Dispose(); _gate.Dispose(); } without an awaited disposal, and have the IAsyncDisposable.DisposeAsync path do the awaiting. If a synchronous disposable is genuinely needed, do not bridge it through the async core — duplicate the dispose-once flag check into a sync path that calls _connection.Dispose() directly.

SiteRuntime-025 — HandleSetStaticAttribute persists unknown attribute names as static overrides

Severity	Low
Category	Correctness & logic bugs
Status	Resolved
Location	src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/InstanceActor.cs:223, src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/InstanceActor.cs:246

Resolution (2026-05-28): HandleSetStaticAttribute now rejects writes whose command.AttributeName does not resolve against _configuration.Attributes. The caller receives SetStaticAttributeResponse(Success: false, ErrorMessage: "Unknown attribute '<name>'"); no override is persisted, _attributes is not mutated, and no synthetic AttributeValueChanged is published — eliminating the in-memory-pollution, restart-resurrection, and debug-stream spam vectors. Regression test InstanceActorSetAttributeTests.SetAttribute_UnknownAttribute_ReturnsFailureAndDoesNotPersistOverride exercises an inbound SetStaticAttributeCommand with an unknown name and asserts the failure response, no DCL traffic, and an empty override row.

Description

HandleSetStaticAttribute resolves the target attribute against _configuration.Attributes to decide whether to route the write to the DCL or treat it as a static-override write. If the lookup fails (resolved == null), isDataSourced is false, and execution falls through to HandleSetStaticAttributeCore — which unconditionally:

1. inserts the bogus key into the in-memory _attributes dictionary,
2. publishes an AttributeValueChanged for the bogus key to the site stream and to every child Script/Alarm actor,
3. persists a row in static_attribute_overrides for the bogus key, and
4. replies Success = true to the caller.

Concretely, an inbound API Route.To().SetAttribute("notARealAttr", "x") returns success, pollutes the in-memory state with a key that no script can legitimately observe (canonical-name lookup will not produce it), persists a durable SQLite override row that survives restart, and (on every restart) re-injects the polluting key via HandleOverridesLoaded at line 608. The override is not reset on instance redeployment in the same way the "genuine" overrides are — ClearStaticOverridesAsync does clear by instance_unique_name, so the row is eventually cleaned, but only on a full redeploy; in the meantime each restart resurrects it. The publish-to-stream side effect also lets a hostile or buggy inbound caller spam debug-view subscribers with synthetic attribute changes.

Worth flagging at Low: the inbound API surface is already authenticated and the design assumes its callers are trusted. But the no-validation behaviour contradicts the design doc's "Scripts can only read/write attributes on their own instance" framing — an inbound API call inherits the same instance-scope authority as a script, and the script trust model wouldn't sanction this.

Recommendation

In HandleSetStaticAttribute, when resolved == null, reply SetStaticAttributeResponse(Success: false, ErrorMessage: $"Attribute '{command.AttributeName}' not found on instance '{_instanceUniqueName}'") instead of falling through to the override path. Optionally also surface the existence check on the RouteInboundApiSetAttributes fan-out so a multi-attribute write reports the offending key without rolling back the others (the per-attribute Ask shape already supports a partial failure response).

SiteRuntime-026 — ReplicationMessages.cs public record types have no XML documentation

Severity	Low
Category	Documentation & comments
Status	Resolved
Location	src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Messages/ReplicationMessages.cs:10, src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Messages/ReplicationMessages.cs:13, src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Messages/ReplicationMessages.cs:15, src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Messages/ReplicationMessages.cs:17, src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Messages/ReplicationMessages.cs:19, src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Messages/ReplicationMessages.cs:25, src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Messages/ReplicationMessages.cs:28, src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Messages/ReplicationMessages.cs:30, src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Messages/ReplicationMessages.cs:32, src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Messages/ReplicationMessages.cs:34

Description

The ten public record types in ReplicationMessages.cs (ReplicateConfigDeploy, ReplicateConfigRemove, ReplicateConfigSetEnabled, ReplicateArtifacts, ReplicateStoreAndForward, ApplyConfigDeploy, ApplyConfigRemove, ApplyConfigSetEnabled, ApplyArtifacts, ApplyStoreAndForward) carry no XML documentation. The file header comment groups them as "outbound" vs "inbound" but the individual records have no <summary> and no parameter docs. The XML-doc baseline 1eb6e97 rolled out across the rest of the module (the commit being reviewed is literally docs: add XML doc comments across src + Sister Projects section in CLAUDE.md), so this file is now the conspicuous outlier — and the CommentChecker skill relied on by the fixdocs workflow will flag every record as missing docs.

Recommendation

Add a <summary> per record naming the direction (outbound → peer / inbound from peer) and what the operation replicates, and <param> docs for each record parameter. Mirror the precedent in src/ZB.MOM.WW.ScadaBridge.Commons/Messages/.../*.cs. While there, consider sealing the inbound vs outbound split with a marker base type (currently they're just named conventionally) so Receive<ReplicateXxx> vs Receive<ApplyXxx> is expressed at the type level — but that's optional and out of scope for a docs-only finding.

Resolution (2026-05-28):

Added a one-line <summary> to each of the ten records (ReplicateConfigDeploy/Remove/SetEnabled/Artifacts/StoreAndForward and ApplyConfigDeploy/Remove/SetEnabled/Artifacts/StoreAndForward) naming the direction (outbound to peer / inbound from peer) and what is replicated. The two pre-existing group-header XML blocks were converted to plain // comments to avoid orphaned doc-summaries above the first record in each group. Marker-base-type idea left out of scope.

SiteRuntime-027 — InstanceActor._latestAlarmEvents grows without bound as native conditions churn

Severity	Medium
Category	Performance & resource management
Status	Resolved
Location	src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/InstanceActor.cs:67, src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/InstanceActor.cs:1007

Description

HandleAlarmStateChanged stores the latest enriched AlarmStateChanged per changed.AlarmName into _latestAlarmEvents (line 1013) and never removes anything from that dictionary. For computed alarms the key set is bounded (one entry per configured alarm). For native alarms, however, the key is the per-condition SourceReference (every NativeAlarmActor.Emit stamps AlarmName = t.SourceReference). When a native condition fully runs its course — NativeAlarmActor.ApplyLiveTransition drops it from the mirror once !Active && Acknowledged (NativeAlarmActor.cs:243), or EnforceCap evicts the oldest — the Instance Actor still holds the final AlarmStateChanged for that SourceReference forever as a (Normal) entry. There is no message back to the Instance Actor to evict it.

On an OPC UA A&C source that mints a fresh SourceReference per occurrence (conditionId/branchId style references are common), the map accumulates one permanently-retained entry per distinct condition the instance has ever seen. The MirroredAlarmCapPerSource cap (default 1000) bounds the NativeAlarmActor._alarms set but does not bound _latestAlarmEvents, which is on the long-lived Instance Actor. Over weeks of uptime on a busy site this is a steadily-growing per-instance memory leak, and BuildAlarmStatesSnapshot (line 1055, _latestAlarmEvents.Values.ToList()) makes every DebugView snapshot proportionally larger and slower, returning thousands of stale Normal rows.

Recommendation

Evict from _latestAlarmEvents when a native condition reaches its terminal return-to-normal+dropped state. The cleanest signal is the AlarmStateChanged the NativeAlarmActor already emits on drop-out: have the NativeAlarmActor stamp a "final/dropped" marker (an additive bool on AlarmStateChanged, or a dedicated NativeAlarmDropped(sourceReference) Tell to the parent), and have HandleAlarmStateChanged _latestAlarmEvents.Remove(...) on it. Alternatively cap/prune _latestAlarmEvents for native keys mirroring MirroredAlarmCapPerSource. Computed-alarm keys must stay (they are configuration-bounded).

Resolution

Resolved 2026-06-20 (commit fd618cf1): added an additive NativeAlarmDropped(SourceReference) Tell emitted by NativeAlarmActor at all terminal-drop sites (snapshot-swap removal, retention drop, cap eviction) after the return-to-normal emit; InstanceActor.HandleNativeAlarmDropped removes the native key from _latestAlarmEvents (and _alarmStates/_alarmTimestamps). Computed-alarm keys are never dropped.

SiteRuntime-028 — NativeAlarmActor.EnforceCap drops a condition without a return-to-normal, leaving a phantom Active alarm

Severity	Medium
Category	Correctness & logic bugs
Status	Resolved
Location	src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/NativeAlarmActor.cs:281

Description

EnforceCap evicts the oldest mirrored conditions once the per-source cap is exceeded: it removes them from _alarms, calls PersistDelete, and logs the eviction — but it does not Emit a return-to-normal AlarmStateChanged for the dropped condition. If the evicted condition was still Active, the Instance Actor's _latestAlarmEvents (and, downstream, central's gRPC stream and the operator Alarm Summary page) keep showing it as Active indefinitely. The mirror has silently forgotten the condition, so no later transition can ever clear it — a phantom stuck-active alarm.

This is inconsistent with the sibling drop path ApplySnapshotSwap (NativeAlarmActor.cs:205), which correctly emits prior.Condition with { Active = false } for every condition that falls out of the new snapshot. The retention drop in ApplyLiveTransition (line 243) is safe only because it drops after emitting the condition's own already-inactive state; EnforceCap drops a condition whose last-emitted state may still be Active, with no compensating emit.

The design doc explicitly states the cap eviction is "logged — there is no silent truncation," but logging alone does not reconcile the operator-visible state: the alarm view is silently wrong.

Recommendation

In EnforceCap, before removing each overflow condition, Emit(a, a.Condition with { Active = false }) (mirroring ApplySnapshotSwap) so the eviction produces a return-to-normal on the stream and clears the phantom. Add a test asserting that exceeding MirroredAlarmCapPerSource with an active oldest condition emits a Normal AlarmStateChanged for the evicted SourceReference.

Resolution

Resolved 2026-06-20 (commit fd618cf1): NativeAlarmActor.EnforceCap now emits Emit(evicted, evicted.Condition with { Active = false }) for a still-active evicted condition before removing it, clearing the phantom stuck-Active alarm on the stream/UI. Cap-eviction test added.

SiteRuntime-029 — A delete (or disable) arriving during a pending redeploy over-decrements the counter and is undone by HandleTerminated

Severity	Medium
Category	Correctness & logic bugs
Status	Resolved
Location	src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/DeploymentManagerActor.cs:627, src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/DeploymentManagerActor.cs:402

Description

HandleDelete and HandleDisable only consult _instanceActors to find the live actor; neither consults the SiteRuntime-020 _terminatingActorsByName / _pendingRedeploys bookkeeping. When a redeployment is in flight, HandleDeploy has already removed the instance from _instanceActors, stopped the predecessor, and buffered a PendingRedeploy keyed by the terminating ref. If a DeleteInstanceCommand then arrives for that same instance before the Terminated signal fires:

1. _instanceActors.TryGetValue misses (the entry was removed at redeploy time), so no actor is stopped.
2. _totalDeployedCount = Math.Max(0, _totalDeployedCount - 1) runs anyway — over-decrementing, because the redeploy path did not increment for an update (the count had already been adjusted), so the deployed/disabled counts reported to the health collector drift.
3. RemoveDeployedConfigAsync deletes the SQLite row and the deleter is told success.
4. The buffered _pendingRedeploys entry is untouched, so when Terminated fires, HandleTerminated calls ApplyDeployment(..., isRedeploy: true), which re-creates the Instance Actor and re-writes the deployed config to SQLite — resurrecting the instance the operator just deleted, with the counter now inconsistent.

HandleDisable has the milder version: the disable persists is_enabled = false, but HandleTerminated then re-stores the config with isEnabled: true and re-creates the actor, so a disable issued mid-redeploy is silently reverted to enabled.

The window is the redeploy-termination interval — small, but reliably hit when central issues a delete/disable immediately after a deploy (e.g. an operator correcting a mistaken deploy, or an automated teardown), exactly the kind of rapid command sequence SiteRuntime-020 was filed to harden.

Recommendation

Make HandleDelete/HandleDisable authoritative over the mid-redeploy state: before falling through, check _terminatingActorsByName. On a hit, drop the buffered _pendingRedeploys entry (so HandleTerminated does not resurrect), clear the shadow, and for delete tell the buffered redeploy's OriginalSender it was superseded — mirroring the last-write-wins handling already in HandleDeploy. Only decrement _totalDeployedCount when an instance was actually present (in _instanceActors or terminating). Add a regression test: deploy → redeploy → delete-before-Terminated asserts the instance stays deleted and the counter is correct.

Resolution

Resolved 2026-06-20 (commit fd618cf1): HandleDelete/HandleDisable now check _terminatingActorsByName before the _instanceActors fall-through; on a hit they drop the buffered _pendingRedeploys entry and clear the shadow so HandleTerminated can't resurrect the instance, and _totalDeployedCount is decremented only when an instance was actually present. Delete-during-redeploy race test added (verified failing pre-fix).

SiteRuntime-030 — Native-alarm cap/retention-drop and SiteReplicationActor paths are untested

Severity	Low
Category	Testing coverage
Status	Resolved
Location	tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/NativeAlarmActorTests.cs, tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/

Description

NativeAlarmActorTests covers subscribe, raise, snapshot-swap return-to-normal, out-of-order rejection, ack, and site-event emission, but there is no test for EnforceCap (the per-source cap eviction) or for the ApplyLiveTransition retention drop (!Active && Acknowledged). Both are non-trivial state transitions, and the cap path harbours an observable defect (SiteRuntime-028) that a targeted test would have caught. SiteReplicationActor remains entirely untested — the carried-forward gap that SiteRuntime-016 explicitly deferred to a clustered-ActorSystem harness, still outstanding at this commit (the actor calls Cluster.Get(Context.System) in its constructor, so it needs a clustered HOCON test host). The outbound forward, inbound apply, and SendToPeer no-peer-drop behaviour are unverified.

Recommendation

Add NativeAlarmActor tests for (a) cap eviction emits a return-to-normal for an evicted active condition (pairs with SiteRuntime-028) and (b) a resolved condition (inactive+acked) drops out and deletes its SQLite row. Stand up the clustered test harness SiteRuntime-016 called for and cover SiteReplicationActor's outbound/inbound/peer-tracking paths.

Resolution

Resolved 2026-06-20 (commit fd618cf1): native-alarm cap-eviction and retention-drop tests added (pairs with -027/-028). The clustered SiteReplicationActor test harness remains deferred (needs a clustered ActorSystem host, consistent with the prior -016 deferral).

SiteRuntime-031 — Site still persists notification-list and SMTP config that the design moved to central-only

Severity	Low
Category	Design-document adherence
Status	Resolved
Location	src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Persistence/SiteStorageService.cs:90, src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Persistence/SiteStorageService.cs:105, src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/DeploymentManagerActor.cs:1383, src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/DeploymentManagerActor.cs:1417

Description

The design decision (CLAUDE.md "External Integrations" / Component-NotificationService; echoed in Component-SiteRuntime.md "System-Wide Artifact Handling": "Notification lists and SMTP configuration are central-only and are not deployed to sites") makes notification delivery central-only — sites store-and-forward to central and never talk to SMTP. Yet SiteRuntime still:

• creates the notification_lists and smtp_configurations SQLite tables (SiteStorageService.InitializeAsync),
• writes them from HandleDeployArtifacts (StoreNotificationListAsync / StoreSmtpConfigurationAsync, DeploymentManagerActor.cs:1383/1417) and the replication path SiteReplicationActor.HandleApplyArtifacts, and
• the DeployArtifactsCommand contract still carries NotificationLists / SmtpConfigurations.

The SMTP path persists the SMTP password field (SiteStorageService.cs:693) into plaintext site SQLite. If central no longer populates these (per the design), this is dead code carrying a latent secret-at-rest footprint; if central still does, the site is storing SMTP credentials it must never use — both contradict the central-only delivery decision. Either the code/tables are stale and should be removed, or the design doc is stale and the decision needs re-stating. This straddles the SiteRuntime/NotificationService boundary and the shared DeployArtifactsCommand contract, so the direction is a design-owner call rather than a clean in-module fix.

Recommendation

Confirm with the design owner whether central still ships notification-list / SMTP artifacts to sites. If not (the stated decision), remove the notification_lists and smtp_configurations tables, the two Store…Async writers, the replication branches, and the corresponding DeployArtifactsCommand fields (a coordinated cross-module change). If the decision has changed, update Component-SiteRuntime.md and Component-NotificationService.md and treat the plaintext SMTP password as a secret-handling finding in its own right.

Resolution

Resolved 2026-06-20 (commit fd618cf1): the site no longer persists notification-list / SMTP config and purges any already-persisted rows (incl. the plaintext SMTP password) on both apply paths (HandleDeployArtifacts, replication HandleApplyArtifacts), inside the all-or-nothing apply. Paired with DeploymentManager-025 (central stops shipping). Tables retained but kept empty.