ScadaBridge/docs/requirements/Component-SiteRuntime.md

Component: Site Runtime

Purpose

The Site Runtime component manages the execution of deployed machine instances at site clusters. It encompasses the actor hierarchy that represents running instances, their scripts, and their alarms. It owns the site-side deployment lifecycle (receiving configs from central, compiling scripts, creating actors), script execution, alarm evaluation, and the site-wide Akka stream for attribute and alarm state changes.

This component replaces the previously separate Script Engine and Alarm Engine concepts, unifying them under a single actor hierarchy rooted at the Deployment Manager singleton.

Location

Site clusters only.

Responsibilities

• Run the Deployment Manager singleton (Akka.NET cluster singleton) on the active site node.
• On startup (or failover), read all deployed configurations from local SQLite and re-create the full actor hierarchy.
• Receive deployment commands from central: new/updated instance configurations, instance lifecycle commands (disable, enable, delete), and system-wide artifact updates.
• Compile C# scripts when deployments are received.
• Manage the Instance Actor hierarchy (Instance Actors, Script Actors, Alarm Actors).
• Execute scripts via Script Actors with support for concurrent execution.
• Evaluate alarm conditions via Alarm Actors and manage alarm state.
• Maintain the site-wide Akka stream for attribute value and alarm state changes.
• Execute shared scripts inline as compiled code libraries (no separate actors).
• Enforce script call recursion limits.

---

Actor Hierarchy

%%{init: {'theme':'base', 'themeVariables': {'textColor':'#111111','lineColor':'#555555','edgeLabelBackground':'#ffffff','fontSize':'15px'}}}%%
flowchart TD
    DMS["Deployment Manager Singleton<br/>(Cluster Singleton)"]
    IA1["Instance Actor<br/>('MachineA-001')"]
    IA2["Instance Actor<br/>('MachineA-002')"]
    IAMORE["… more Instance Actors"]

    SA1["Script Actor ('MonitorSpeed')<br/>— coordinator"]
    SA2["Script Actor ('CalculateOEE')<br/>— coordinator"]
    AA1["Alarm Actor ('OverTemp')<br/>— coordinator (computed)"]
    AA2["Alarm Actor ('LowPressure')<br/>— coordinator (computed)"]
    NAA1["Native Alarm Actor ('OpcUaServer1')<br/>— read-only mirror, peer to Alarm Actor"]

    SEA1["script run<br/>— launched task, not an actor"]
    SEA2["script run<br/>— launched task, not an actor"]
    AEA1["alarm on-trigger run<br/>— launched task, not an actor"]

    IA2CHILD["… (Script / Alarm Actors)"]

    DMS --> IA1
    DMS --> IA2
    DMS -.-> IAMORE

    IA1 --> SA1
    IA1 --> SA2
    IA1 --> AA1
    IA1 --> AA2
    IA1 --> NAA1

    SA1 -.-> SEA1
    SA2 -.-> SEA2
    AA1 -.-> AEA1

    IA2 -.-> IA2CHILD

    classDef start fill:#d5e8d4,stroke:#82b366,color:#111111;
    classDef proc fill:#dae8fc,stroke:#6c8ebf,color:#111111;
    classDef dec fill:#fff2cc,stroke:#d6b656,color:#111111;
    classDef warn fill:#ffe6cc,stroke:#d79b00,color:#111111;
    classDef bad fill:#f8cecc,stroke:#b85450,color:#111111;
    classDef alt fill:#e1d5e7,stroke:#9673a6,color:#111111;
    classDef muted fill:#f5f5f5,stroke:#999999,color:#666666;
    class DMS proc
    class IA1,IA2 start
    class SA1,SA2 dec
    class AA1,AA2 bad
    class NAA1 alt
    class SEA1,SEA2,AEA1 warn
    class IAMORE,IA2CHILD muted

---

Deployment Manager Singleton

Role

• Akka.NET cluster singleton — guaranteed to run on exactly one node in the site cluster (the active node).
• On failover, Akka.NET restarts the singleton on the new active node.

Startup Behavior

1. Read all deployed configurations from local SQLite.
2. Read all shared scripts from local storage.
3. Compile all scripts (instance scripts, alarm on-trigger scripts, shared scripts).
4. Create Instance Actors for all deployed, enabled instances as child actors. Instance Actors are created in staggered batches (e.g., 20 at a time with a short delay between batches) to prevent a reconnection storm — 500 Instance Actors all registering data subscriptions simultaneously would overwhelm OPC UA servers and network capacity.
5. Make compiled shared script code available to all Script Actors.

Startup config load is retried (S5): step 1's deployed-config read is best-effort against local SQLite, which can transiently fail (locked/busy). A failed load must not leave the site as a silent zero-instance node — it is re-attempted on a fixed interval until it succeeds, then the retry self-cancels.

Startup reconcile heals the RUNTIME, not just the config file (register row 67): SiteReconciliationActor runs on every site node (not a singleton) and performs one best-effort pass shortly after boot — it reports the node's local deployed inventory to central and fetches + stores any config central reports as a gap. That pass fires after step 1's deployed-config read has already run, so storing a config started nothing: Instance Actors are created only by the boot-time load above and by an explicit deploy/enable. An instance whose config was missing at boot (lost volume, first boot after a restore, rehomed data dir) therefore stayed dead until a second restart, whose load finally picked it up — silently, with nothing surfacing "config stored, actor not running". The pass now raises a fire-and-forget EnsureInstanceStartedCommand for every enabled instance it healed, routed through the deployment-manager singleton proxy so Instance Actors are only ever created under the singleton regardless of which node's pass found the gap. The handler is deliberately advisory: it no-ops when the actor is already running (both nodes legitimately raise it for the same instance), re-reads the durable row — which is the authority on both the config body and enabled/disabled state, so a disable landing after the fetch still wins and the instance stays stopped — defers to any in-flight deploy/redeploy for that instance, preserving the deploy gate's ordering guarantees and the all-or-nothing-per-instance rule (it never applies a config, only starts one already fully persisted), and treats a missing row as a quiet no-op (the normal case on an unreplicated site whose peer node healed its own file). A successful reconcile start is surfaced, not silent: a Warning log line plus an operational deployment site event, because reaching it means the node booted without a config for an instance that should have been running.

Instance reconcile sweep — the store and the runtime are kept in agreement, and disagreement is surfaced (register rows 78 + 72): the reconcile self-heal above covers configs the reconcile pass itself fetched. Two sibling divergences were not covered, and neither surfaced anything.

• Row 78 — a config that arrives by replication starts nothing. deployed_configurations is one of LocalDb's ten replicated tables, so on a replicated pair a config can appear in a node's file with no local actor involved at all — the CDC sync applier writes the row directly. Proven live 2026-08-15: a fresh site-a-a was seeded from its peer by snapshot replication within ~5 s of boot, i.e. after its boot-time load had read an empty table and invisibly to its own reconcile (central rightly reported no gap, because central's view of the site was correct). Enabled instances sat dead with their configs present until another restart.
• Row 72 — a refreshed config does not reach the running actor. For a stale ReconcileGapItem the pass writes the newer config via StoreDeployedConfigIfNewerAsync, a pure file write; the running Instance Actor keeps executing the previously compiled config until the next restart or redeploy.

Both are closed by one periodic sweep owned by the Deployment Manager singleton (ScadaBridge:SiteRuntime:InstanceSweepIntervalSeconds, default 60, validated >= 0; 0 disables it). Each tick re-reads the durable rows off-thread and, on the actor thread, for every enabled stored config: starts the Instance Actor if none is running (row 78), or — if one is running at a revision the store has since replaced — re-applies the stored config through the exact per-instance apply/restart path a central redeploy uses (row 72). A sweep is chosen over a LocalDb change hook deliberately: a hook would cover only the replication arrival path, only on replicated sites, and ILocalDb exposes no change-notification API at all, whereas the sweep is arrival-path-agnostic (replication seeding, post-boot snapshot resync, a hand-restored file, any future path). See docs/plans/2026-08-15-instance-sweep-self-heal-design.md.

The row-72 decision is to converge, not merely warn: the stored row is what central successfully deployed, so an actor serving the config central believes was replaced is the actual fault. Reusing HandleDeploy means the compile gate, warm-then-gate ordering, redeploy-supersede, the static-override / native-alarm reset and all-or-nothing-per-instance are the production ones — and a stop still does not cancel an in-flight script run, exactly as for a central redeploy.

Guards, all shared with the row-67 handler: a disabled instance is never started; the sweep defers to any deploy pipeline that owns the instance (compile warm in flight, predecessor still terminating, deployment awaiting its persistence/init join) so it can never interleave with a real redeploy; it is inert until boot-time Instance Actor creation has finished, so a tick between staggered batches does not "heal" instances the boot load is about to create; it makes one repair attempt per (instance, revision) pair, which is what stops a periodic self-heal becoming a restart loop for an actor that dies during init or a stored config the compile gate rejects (a rejection leaves the running instance untouched — the gate returns before anything is stopped); and reads never overlap. One guard is not in that list because it needed adding later: deferring to an in-flight deploy pipeline says nothing about a snapshot that landed after the pipeline finished, which is the stale-snapshot race — a tick whose read raced this node's own deploy/enable/disable/delete is discarded whole by the store-write generation check (see Stale-snapshot guard below). A sweep that finds nothing is completely silent. A repair emits the same Warning deployment site-event pattern as row 67, naming the trigger (deployment-manager sweep self-heal) so it is distinguishable from a startup-reconcile heal, and a stale-revision restart names both revisions; a rejected restart adds an Error event. The synthetic redeploy replies to the singleton itself rather than to NoSender, because dead letters are a monitored site health metric.

Starting an Instance Actor also creates its DCL connections — the ensure lives in the creation funnel, not at call sites (register row 85): a DCL connection actor is never created from the data_connection_definitions table — that stayed true when rows 89/90 gave the table a reader, because the overlay it feeds is an arbiter, not a creation source, and at the time of row 85 nothing read the table at all. A connection actor exists only as a side effect of starting an instance, built from the Connections map inside that instance's own flattened config via EnsureDclConnections — plus the separate system-wide artifact-deploy path. That ensure used to be each caller's responsibility, and only two of the four callers made it (the boot-time staggered batch and ApplyDeployment). The two that did not were StartStoredInstance — the path shared by the row-67 ensure-started handler and the row-78 sweep — and HandleEnableResult. Both therefore produced a running instance whose every tag subscribe was answered Unknown connection: <name> by the DCL manager, indefinitely, because nothing else on the node would ever create that connection actor. Proven live 2026-08-16: a replication-seeded node's instances were correctly started by the sweep and then failed every subscribe for 6+ minutes until a second pair restart let the boot load create the connections. It was never replication-specific — the boot-time load reads only enabled configs, so a node that boots with an instance disabled never creates its connections either, and enabling it afterwards hit the identical dead end on an ordinary unreplicated site. EnsureDclConnections now sits at the top of CreateInstanceActor, already the single Instance Actor creation funnel (it is where the running revision is stamped for the same reason), so all four paths get it and a fifth cannot forget. It is placed before the already-exists guard so the two paths that already called it keep their exact previous semantics, and it stays silent in steady state because EnsureDclConnection is hash-guarded on the last config pushed per connection name. Deliberately not done at the time: a sweep pass over data_connection_definitions — it would not have fixed row 85 (that table is empty on a site that has only ever received instance deploys), and re-hashing from it would fight the deploy path over the same per-name hash slot, recreating connection actors and dropping live subscriptions on a loop. See docs/plans/2026-08-15-instance-sweep-self-heal-design.md §7. That pass is now present, in a form that resolves the conflict rather than creating it — see DCL connection arbitration immediately below.

DCL connection arbitration — the definitions store wins wherever it has an opinion (register rows 89 + 90): data_connection_definitions now has a production reader. The rule is: a connection name with a row in that table is configured from that row; a name with no row is configured from the instance's flattened Connections entry, exactly as before. Rule 2 is not a fallback — it is the correct answer for a deploy-only site, where the table is legitimately empty while every connection that matters lives inline in the instance configs.

The register's premise needed two corrections, and they change what the fix is. First, the peer node's DCL is not "serving the old endpoint" — it is empty. The Deployment Manager is a site-{SiteId}-scoped singleton and the only sender of CreateConnectionCommand/RemoveConnectionCommand, sending to its own node's dcl-manager (which exists on every node but is populated only where the singleton runs). The real defect is that _createdConnections, the per-name hash cache, is in-memory and per singleton incarnation: a restart or failover rebuilds every connection from the Connections snapshot frozen into each instance's flattened config at that instance's deploy time, and a newer connection-only artifact deploy is silently discarded. That makes row 89 create-time, not message-delivery, and not replication-specific — the identical loss happens on an unreplicated single-node restart. Second, EnsureDclConnection's documented "a changed config re-issues the create so the DCL adopts it" was false: DataConnectionManagerActor.HandleCreateConnection is create-only (warns and returns for a name it already holds), so the artifact deploy's live push has never worked on the deploy-handling node either. That second correction was left as a residual by rows 89/90 and is closed by row 97 — see In-place reconfiguration below.

Mechanism. data_connection_definitions is read in the same off-thread task as deployed_configurations (ReadStoreSnapshotAsync), by both the boot load and every sweep tick, and applied to an actor-thread-confined arbitration overlay before the boot batch creates any Instance Actor and before the sweep's repair loop — so a connection is built from the winning config on its first push rather than created stale and corrected a tick later. The overlay is an arbiter, not a creation source: a definition row naming a connection no deployed instance references creates nothing. A CDC-delivered definition change therefore converges within one sweep tick on whichever node runs the singleton, with no new cross-node message. A definitions read that fails yields null and leaves the overlay untouched — deliberately distinct from an empty list, which is a real answer; clearing on a transient failure would reintroduce the ping-pong through the error path.

The ping-pong is killed by keying the hash on the winner. The conflict §7.4 identified is a property of where the hash is computed, not of the sweep: every ensure path now resolves the winning configuration before hashing or pushing, so for any name only one configuration can ever occupy the hash slot regardless of which path calls. The flip is not guarded against — it is unrepresentable. HandleApplyArtifactDataConnectionsToDcl writes its rows into the overlay in the same pass it pushes them, closing the same flip arriving through the artifact-deploy door.

Stale-snapshot guard — covers the WHOLE snapshot, both halves and both consumers (row 101). ReadStoreSnapshotAsync reads deployed_configurations and data_connection_definitions off the actor thread but is consumed only when its piped message lands, so a read dispatched before a mutation can land after the actor's own state has already moved past it. Every read therefore captures a single monotonic store-write generation on the actor thread at dispatch (_storeWriteGeneration), and a snapshot carrying a stale generation is discarded whole. One counter covers both tables on purpose: the snapshot is one consistent read and is discarded whole anyway, so a per-table counter would only buy the chance to apply half a snapshot the other half's mutation already invalidated.

Each half would do real damage. Definitions half: stale absences "prove" just-applied names disappeared — tombstoned, live connection actors stopped, and because the removal evicts the hash slot the overlay's push is gated on, nothing re-creates them until an instance redeploy; stale rows push superseded configs at live connections and flip-flop with the next fresh read. Configs half, row-72 leg: a snapshot carrying the revision a deploy just replaced makes the stale-revision repair run backward, re-applying the snapshot's own superseded config JSON through the redeploy path — and InstanceIsOwnedByADeployPipeline is no defence, because it only holds while a deploy is in flight and the stale snapshot can land after the pipeline released the instance. Configs half, row-78 leg: a snapshot read before an undeploy/delete or a disable committed still shows an enabled row while the actor is already stopped, so the start repair resurrects a just-stopped instance — and delete specifically clears _sweepAttemptedRevisions, so the once-per-revision claim is freshly available and does not stand in the way either. All three were reproduced by neutralising the guard and re-running the regression tests.

The generation is bumped twice per mutation — when the off-thread write is dispatched and again when its completion message lands on the actor thread. The pair is what leaves no window: an initiation-only bump would let a read dispatched between the decision and the commit compare equal and be applied even though it predates the write, and a completion-only bump would miss a read dispatched before the handler ran. With both, a read dispatched before the mutation is discarded, a read dispatched during it is discarded, and a read dispatched after the completion bump necessarily sees the committed row. Bump sites are every actor-thread path that mutates either table: deploy persist (dispatch + DeployPersistenceResult), enable (dispatch + EnableResult), disable and delete and the init-failure rollback (dispatch + a StoreWriteCompleted self-message that exists purely to bump after the commit), and the artifact apply (dispatch + ApplyArtifactDataConnectionsToDcl).

Writes that must NOT bump, and this is load-bearing: rows arriving by LocalDb CDC replication, by a snapshot resync, by an operator restoring the file, or by SiteReconciliationActor's startup fetch never pass through this actor — and discovering exactly those is the sweep's entire reason to exist (rows 72 + 78). They are not stale relative to actor state; they are the state the actor has yet to learn about.

The two consumers differ. A stale sweep result is dropped whole at Debug — no overlay apply, no start, no restart — because there is nothing to complete and the next tick is already a fresh read. A stale boot load cannot be dropped (a node whose startup read is discarded stays a silent zero-instance node), so it re-dispatches LoadDeployedConfigs(), capturing the current generation. Staleness requires a fresh mutation inside each attempt's window, so this converges in practice; as a livelock backstop the fifth consecutive superseded boot read is applied anyway with a Warning, on the grounds that a boot that never finishes is worse than one starting from a slightly superseded view that the sweep converges on its next tick. A superseded boot snapshot arriving after the boot has completed (the S5 retry timer racing a success) is dropped rather than re-read.

Surfaced originally as a test flake (a raced sweep removed PlcA where the test expected PlcGone). Deterministic reproductions: DclConnectionArbitrationTests.StaleSweepSnapshot_RacingAnArtifactApply_DoesNotRemoveJustAppliedConnections for the definitions half, and in InstanceSweepSelfHealTests — StaleSweepSnapshot_CarryingASupersededRevision_DoesNotRestartBackward, StaleSweepSnapshot_ShowingADeletedInstance_DoesNotResurrectIt, StaleBootSnapshot_IsReReadRatherThanApplied, each paired with a Fresh… twin pinning that the legitimate row-72 restart and row-78 start still happen.

Removal (row 90). Both removal paths share one helper and now record a tombstone for the incarnation: the artifact reconcile's RemovedConnectionNames, and the sweep noticing that a name it had adopted no longer has a row (a removal it can prove — the store owned it a tick ago and does not now). EnsureDclConnections skips a tombstoned name, so a removal survives the next instance start instead of being undone by it — a pre-existing bug, since the removal evicted the hash slot but left no memory of why. Tombstones are in-memory only by decision: a durable marker would be a second divergent authority, and flattened configs already resurrect a removed connection across a restart. A definition row reappearing clears the tombstone.

In-place reconfiguration — a changed config now reaches the LIVE connection (register row 97): resolving the winner decides what to push; it took a third command to make a push against an already-live connection mean anything. EnsureDclConnection has three outcomes, all keyed off the one per-name hash slot: unchanged → nothing at all (no message, no log — this is what keeps a healthy site silent across every sweep tick); new name → CreateConnectionCommand; changed → ReconfigureConnectionCommand, which the DCL manager forwards to the live connection actor. The actor swaps its adapter through the very machinery the primary→backup failover uses and re-issues every subscription it holds — no redeployment, no restart. The winner-resolution and tombstone logic above are untouched; only what happens after a change is detected changed. The log line is Reconfigured DCL connection {Connection} (protocol={Protocol}) in place at Information, replacing the Warning rows 89/90 added to make the gap honest. See Component-DataConnectionLayer.md → In-place reconfiguration for what the connection actor does, and why stop-and-recreate was rejected.

Residuals still open, recorded precisely in the decision memo: a removal is not durable across a restart; and a delete is undetectable on a node that never adopted the row. See docs/plans/2026-08-17-dcl-connection-arbitration-decision.md.

Per-instance compilation during staggered startup — CLOSED (WP3.1): this was a deferred optimization (P6) affecting failover time-to-recover; each Instance Actor Roslyn-compiled its own scripts inside its own start, serialising a site's whole recovery behind compilation. Each staggered startup batch now runs a pre-warm step: before the batch's Instance Actors are created, the Deployment Manager compiles that batch's distinct script bodies and trigger expressions on a background task and pipes BatchCompileWarmed back, so every PreStart compile is a cache hit. Instance Actor start still calls the compilation service — it remains the correctness backstop — but in every warmed path those calls are memoised lookups. Cost: one extra message per batch.

The Roslyn ScriptOptions are process-static, and must stay that way: the reference set backing every site-side compile is built once per process, never per compile. ScriptOptions.WithReferences(Assembly[]) resolves each assembly through MetadataReference.CreateFromFile, which does not cache — each call mints a fresh MetadataReference owning an AssemblyMetadata → PEReader → NativeHeapMemoryBlock, an unmanaged copy of the assembly metadata that nothing disposes. Building the options per compile leaks native memory permanently: no GC reclaims it, and it is invisible to gcdump and to the managed allocation counters, so the node's working set grows without the GC heap growing. This was a live defect (a Site node at 2,885 MB working set with 150 MB of live GC heap, ~2,469 MB of it on the default process heap across ~6,700 undisposed AssemblyMetadata instances). Note this is orthogonal to the compile cache above — the cache dedupes identical script bodies, so it bounds nothing when the bodies differ. (It no longer clears wholesale on overflow: WP3.1 replaced that latency cliff with approximate-LRU batch eviction, see Compile cache eviction below.) Pinned by a reference-equality regression test rather than by watching memory.

The compile cache is owned by the ScriptCompilationService singleton, not static (register row 100): SiteScriptCompileCache is an ordinary instance held by ScriptCompilationService, and "one Roslyn compile per script body per process" (N4) is exactly the statement that AddSiteRuntime registers that service as a DI singleton — the deploy gate (via DeployCompileValidator), every Instance Actor's PreStart, and SharedScriptLibrary all reach the one cache through the one service the Deployment Manager threads to them. The singleton lifetime is therefore load-bearing rather than incidental, and is pinned by a registration test (SiteRuntimeServiceRegistrationTests); downgrading it to Scoped/Transient would hand every consumer an empty cache and silently reintroduce WP3.1's recompile storms with no behavioural test failing. The class was previously static, which made its entry table and its Hits/Count counters one process-global blob every test class in the assembly wrote into; the Clear() method that existed only so tests could reset that blob is removed — it never had a production caller, and constructing a cache is the honest reset.

Compile cache eviction is approximate LRU, not a wholesale clear (WP3.1): SiteScriptCompileCache is bounded at 1024 entries — deliberately smaller than the TemplateEngine verdict cache's 4096, because entries pin compiled assemblies rather than verdict strings. It used to clear everything on overflow. Instance scripts and trigger expressions share this cache, so on a site large enough to cross the bound every overflow discarded up to 1023 live compiled scripts and the next deploy or Instance-Actor start paid a full recompile storm on actor threads. Overflow now evicts only the oldest ⅛ (128 entries) by last-access stamp: one 1024-element scan instead of 1023 future recompiles, and hot entries survive. Recency is an Interlocked access sequence rather than a clock — deterministic for tests and immune to clock steps; hits update it lock-free, and only the (rare) eviction sweep takes a lock, double-checked so concurrent inserts do not stampede it. The sweep snapshots through ConcurrentDictionary.ToArray() and not LINQ over the dictionary: LINQ's ToArray picks the ICollection.CopyTo fast path, which throws when a concurrent insert lands between its Count read and its copy, and inserts are deliberately lock-free.

Static options were only half the fix — the compile must also use the shared caching metadata resolver: the static ScriptOptions carry only the direct API-surface references (5 assemblies here), and each script.Compile() still binds their transitive closure (~105 assemblies on a site node). Every transitively-referenced assembly is resolved through the options' MetadataReferenceResolver, and Roslyn's default one has no cross-compilation cache — it calls MetadataReference.CreateFromFile afresh on every compile, minting the same undisposed AssemblyMetadata → PEReader → NativeHeapMemoryBlock triple per assembly per compiled script, pinned for the process lifetime by the compile cache. That per-compile re-resolution — not the per-compile options construction — was the dominant term (measured live: 21 site scripts holding 6,640 metadata objects, counts bit-identical before and after the static-options fix). ScriptCompilationService therefore also attaches the process-wide CachingScriptMetadataResolver.Instance (owned by the Script Analysis component, see Component-ScriptAnalysis.md) via ScriptOptions.WithMetadataResolver, so each distinct assembly is materialized once per process. The decorator returns exactly what the undecorated resolver returned — only object identity is de-duplicated — so compile results and the trust gate are unchanged.

Deployment Handling

• Receives flattened instance configurations from central via the Communication Layer.
• Stores the new configuration in local SQLite.
• Compiles all scripts in the configuration.
• Creates a new Instance Actor (for new instances) or updates an existing one (for redeployments).
• For redeployments: the existing Instance Actor and all its children are stopped, then a new Instance Actor is created with the updated configuration. Subscriptions are re-established.
• Reports deployment result (success/failure) back to central.

Dead-child bookkeeping + two-signal deploy join (S6). All Instance Actors are Watched; an unexpected Terminated evicts the actor ref from the singleton's map. Deviation from the naive design (2026-07-09, see master-tracker "Plan deviations during Wave 1 PLAN-03 execution"): deploy-success is NOT reported on persistence alone — the config store can commit before the Instance Actor's asynchronous PreStart throws. The design therefore uses a two-signal join: the Instance Actor sends a new InstanceActorInitialized readiness message at the end of a successful PreStart, and the Deployment Manager reports Success only once both persistence has committed and that readiness signal has arrived. An actor that dies during init never signals readiness → the Terminated fallback replies Failed and rolls the optimistic in-memory state back (the persisted-row rollback is deferred until the store commits so it cannot race the write).

System-Wide Artifact Handling

• Receives updated shared scripts, external system definitions, database connection definitions, and data connection definitions from central. (Notification lists and SMTP configuration are central-only and are not deployed to sites — see Component-NotificationService.md.)
• Stores all artifacts in local SQLite. After artifact deployment, the site is fully self-contained — all runtime configuration is read from local SQLite with no access to the central configuration database.
• Recompiles shared scripts and makes updated code available to all Script Actors.
• Reconciles deletions: each non-null artifact list is the complete set for its class, so after upserting, any stored row absent from the set is deleted (the artifact was deleted centrally — there is no per-artifact delete command). A removed shared script is also unregistered from the compiled script library, and a removed data connection's live DCL connection actor is stopped. A null list means "field not shipped" and reconciles nothing.

Deployed Secrets (2026-09-28)

• HandleDeployArtifacts additionally accepts the command's Secrets list (SecretArtifact, ciphertext only) and writes the full shipped set to the deployed_secrets table via SiteStorageService.UpsertDeployedSecretsAsync — upsert every shipped name, soft-remove (is_removed = 1) any stored name absent from the set. A null Secrets field (pre-feature central) touches nothing and answers with no SecretsApplied. A current central always ships a list — empty when the site references no secrets — so dropping a site's last secret: reference removes the secret from both nodes. Storage detail: Deployed Secrets (Site SQLite / LocalDb) below.
• After writing the table, the active node runs DeployedSecretsMirror.ReconcileAsync synchronously, before replying — so the deploy response's SecretsApplied count and any apply/verify error reflect this deploy's own outcome, not a value left over from the mirror's background schedule. SecretsApplied counts only the names this deploy shipped that applied and verified cleanly; a mirror failure for an unrelated, already-broken name (a different deploy's leftover, or a stale removal) is logged at Warning and excluded — it neither fails this deploy nor reduces its SecretsApplied.
• No DeployedSecretsMirror registered (secrets deployment not composed on this node) with a non-empty Secrets list fails the deploy with Secret deployment is not enabled on this site node; with an empty Secrets list it still succeeds (SecretsApplied = 0).
• The reply also carries SecretsRemoved: the names UpsertDeployedSecretsAsync soft-removed on this deploy (site-wide, whichever node's mirror later deletes them), or null when none. Central uses it for the removed list of its DeploySiteSecrets audit row, so a deploy that only retires secrets is audited centrally.
• Ordering against the DCL. The shipped secrets are written and mirrored before the deploy's data-connection definitions are pushed to the DCL (the definitions are still stored in SQLite at the same point, and the push still goes out if the secrets step fails), so a connection created or reconfigured by the same deploy resolves a newly shipped secret on its first connect. A rotation with unchanged connection config does not recreate the connection (the DCL push is hash-guarded) — the new value takes effect on that connection's next (re)connect. No forced reconnect is performed.
• See Deployed Secrets (Site SQLite / LocalDb) below for the mirror itself.

Instance Lifecycle Commands

• Disable: Stops the Instance Actor and its children. Retains the deployed configuration in SQLite so the instance can be re-enabled without redeployment.
• Enable: Creates a new Instance Actor from the stored configuration (same as startup).
• Delete: Stops the Instance Actor and its children, removes the deployed configuration from local SQLite. Does not clear store-and-forward messages.
• Ensure Started (EnsureInstanceStartedCommand, site-internal — never sent by central): advisory, idempotent request to make sure an already-persisted, enabled instance is actually running. Raised by the startup reconcile self-heal; see the note under Startup Behavior. Unlike the three commands above it has no reply and applies no configuration — it only starts an Instance Actor for a config that is already fully stored, and no-ops when the actor is running, the stored row says disabled, the row is absent, or a deploy for that instance is in flight.
• Instance reconcile sweep (internal timer, not a command): the periodic singleton-owned check that the running Instance Actors still agree with the durable config rows — starting an enabled instance that has none, and re-applying a stored config a running instance has fallen behind. See the note under Startup Behavior; the start half shares its implementation (and therefore its bookkeeping and site events) with Ensure Started, and the restart half goes through the ordinary redeploy path.

Debug Snapshot Routing

• Receives DebugSnapshotRequest from the Communication Layer and forwards to the Instance Actor by unique name (same lookup as SubscribeDebugViewRequest).
• Returns an error response if no Instance Actor exists for the requested unique name (instance not deployed or not enabled).

Server Certificate Trust (Per-Node Broadcast)

• The OPC UA trusted-peer certificate store is node-local — it lives in each node's PKI directory, so trusting a cert on only the active node would silently fail certificate validation after failover. To keep both nodes consistent, certificate trust is handled by a per-node actor and a broadcast.
• CertStoreActor is a per-node, non-singleton actor — it runs on every site node (unlike the Deployment Manager singleton). It owns its node's OPC UA PKI trusted-peer / rejected stores and handles three commands:
  • TrustServerCertCommand — write a base64/DER server certificate into the node's trusted-peer store (path-traversal-guarded thumbprints; the cert is decoded and persisted as a .der).
  • RemoveServerCertCommand — delete a certificate (by thumbprint) from the node's stores.
  • ListServerCertsCommand — enumerate the node's trusted-peer + rejected store contents (Subject / Issuer / Thumbprint / validity / status).
• The Deployment Manager singleton receives the central cert-trust verbs (relayed via the Communication Layer) and broadcasts TrustServerCertCommand / RemoveServerCertCommand to the CertStoreActor on both site nodes (via ActorSelection), so node-a and node-b PKI stores stay in sync. List reads from the local node. Trust is site-local — there is no central persistence of trusted certs (logged as a follow-up). The captured-but-untrusted server cert that seeds this flow comes from the DCL verify-endpoint probe (see Component-DataConnectionLayer.md), which never trusts on its own; the Central UI surfaces Trust / Remove (Admin-gated — see Component-CentralUI.md).
• Reconcile-on-join (UA1): a broadcast only reaches nodes that are Up at the time, so a node that was down when a cert was trusted would stay divergent forever. To close that gap, the Deployment Manager singleton subscribes to cluster MemberUp events; when a site node (re)joins, it reads its own trusted-peer store (ExportLocalTrustedCerts → CertStoreActor) and pushes each certificate to the joining node's CertStoreActor (additive union; each WriteCertToLocalStore is idempotent by thumbprint). Removal reconciliation and central persistence of trust decisions remain the documented follow-up — only additions converge automatically.

---

Instance Actor

Role

• Single source of truth for all runtime state of a deployed instance.
• Holds all attribute values (both static configuration values and live values from data connections).
• Holds current alarm states (active/normal), updated by child Alarm Actors.
• Publishes attribute value changes and alarm state changes to the site-wide Akka stream.

Trace arm (Flow Debugger, 2026-09-05)

• A SetStaticAttributeCommand / SetDataAttribute carrying Trace = true and a SourceExecutionId arms the attribute on the Instance Actor for TraceArmWindowSeconds (default 30, validated > 0). A static write publishes AttributeValueChanged with the writer's SourceExecutionId and Trace immediately; a data-sourced write is a real tag write, and the next device echo of that attribute inside the window consumes the arm and is published with the same ids — so the script run the echo triggers is parented to the writer's execution and runs traced. An echo outside the window, or a second echo, is an ordinary untraced change. Batch-write echoes are not armed (design §5.3); only the single-attribute write path arms.

Initialization

1. Load all attribute values from the flattened configuration (static defaults).

2. Set quality to uncertain for all attributes that have a data source reference. Static attributes (no data source reference) have quality good. The uncertain quality persists until the first value update arrives from the Data Connection Layer, distinguishing "not yet received" from "known good" or "connection lost."

3. Register data source references with the Data Connection Layer for subscriptions.

   Tag-subscribe retry (S4/UA6, hardened by Lever 1 of the MxGateway subscribe-storm design, 2026-08-28). The Instance Actor groups its data-sourced tag paths by connection and sends one SubscribeTagsRequest per connection, arming a per-connection watchdog on send so both a failed reply and a lost reply are retried. At most one request per connection is outstanding: a watchdog tick while the in-flight request is younger than TagSubscribeInFlightCeilingMs (45 s, sized above the gateway's 30 s command deadline) only re-arms — it never issues a duplicate — so a gateway that is slow to answer is not handed a fresh N-tag subscribe every interval on top of the one it is still processing (the amplifier behind the 2026-08-26 QA freeze). Past the ceiling the reply is presumed lost and a new request goes out; a late reply for the superseded id is ignored. The retry interval is exponential: floor TagSubscribeRetryIntervalMs (5 s), ×2 with ±20 % jitter per failed/lost attempt, capped at TagSubscribeRetryMaxIntervalMs (60 s); a Success reply or the connection returning to Good quality resets it to the floor — but the Good reset does not release an outstanding request (it still owns the connection until the ceiling), so it shortens the wait only when nothing is in flight; and once the backed-off interval exceeds the ceiling the guard is inert on ticks and the interval itself is the rate limit (one re-send per 60 s at the defaults). All three keys are startup-validated (SiteRuntimeOptionsValidator). Design: docs/plans/2026-08-28-mxgateway-subscribe-storm-prevention-design.md §3; plan: docs/plans/2026-08-28-tag-subscribe-inflight-guard-backoff.md.

4. Create child Script Actors (one per script defined on the instance).

5. Create child Alarm Actors (one per alarm defined on the instance).

Attribute Value Updates

• Receives tag value updates from the Data Connection Layer for attributes with data source references.
• Updates the in-memory attribute value.
• Notifies subscribed child Script Actors and Alarm Actors of the change.
• Publishes the change to the site-wide Akka stream.

Stream Message Format

• Attribute changes: [InstanceUniqueName].[AttributePath].[AttributeName], attribute value, attribute quality, attribute change timestamp.
• Alarm state changes: [InstanceUniqueName].[AlarmName], alarm state (active/normal), priority, timestamp.

GetAttribute / SetAttribute

• GetAttribute: Returns the current in-memory value for the requested attribute. Attribute read/write/wait helper continuations do not capture the calling scheduler (ConfigureAwait(false)), so synchronous Attributes[key] access can complete even when every script worker is blocked. AsyncLocal trace attribution still flows with execution context.
• SetAttribute (for attributes with data source reference): Sends a write request to the Data Connection Layer. The DCL writes to the physical device. If the write fails (connection down, device rejection, timeout), the error is returned synchronously to the calling script for handling. On success, the existing subscription picks up the confirmed value from the device and sends it back as a value update, which then updates the in-memory value. The in-memory value is not optimistically updated.
• SetAttribute (for static attributes): Updates the in-memory value and persists the override to local SQLite. On restart (or failover), the Instance Actor loads persisted overrides on top of the deployed configuration, preserving runtime-modified values. A redeployment of the instance resets all persisted overrides to the new deployed configuration values.

Debug View Support

• On request from central (via Communication Layer), the Instance Actor provides a snapshot of all current attribute values and alarm states.
• Subsequent changes are delivered via the SiteStreamManager → SiteStreamGrpcServer → gRPC stream to central. The Instance Actor publishes attribute value and alarm state changes to the SiteStreamManager; it does not forward events directly to the Communication Layer.
• The Instance Actor also handles one-shot DebugSnapshotRequest messages: it builds the same snapshot (attribute values and alarm states) and replies directly to the sender. Unlike SubscribeDebugViewRequest, no subscriber is registered and no stream is established. When the request sets AlarmsOnly (WP2.3 — the central live alarm cache's seed/reconcile fan-out, which discards attribute rows), only the alarm half is built and AttributeValues comes back empty; the flag defaults to false, so the Debug View path is unchanged.

Supervision Strategy

The Instance Actor supervises all child Script and Alarm Actors with explicit strategies:

Child Actor	Exception Type	Strategy	Rationale
Script Actor	Any exception	Resume	Script Actor is a coordinator — its state (trigger timers, last execution time) should survive run failures.
Alarm Actor	Any exception	Resume	Alarm Actor holds alarm state. Resume preserves state and continues evaluation on next value update.

Script Actor and Alarm Actor have no children of their own since WP3.1 removed the per-run execution actors, so neither declares a supervision strategy any more. A run's own exceptions were always contained inside the run's try/catch (they never reached supervision); a failure of the launch is caught by the coordinator, which answers the caller and releases the run slot rather than letting the throw escalate to the Instance Actor.

The Deployment Manager singleton supervises Instance Actors with a OneForOneStrategy — one Instance Actor's failure does not affect other instances.

When the Instance Actor is stopped (due to disable, delete, or redeployment), Akka.NET automatically stops all child actors.

---

Script Actor

Role

• Coordinator for a single script definition on an instance.
• Holds the compiled script code and trigger configuration.
• Manages trigger evaluation (interval timer, value change detection, conditional evaluation).
• Launches script runs directly for each invocation (no per-run child actor since WP3.1).

Trigger Management

• Interval: The Script Actor manages an internal timer. When the timer fires, it launches a run.
• Value Change: The Script Actor subscribes to attribute change notifications from its parent Instance Actor for the specific monitored attribute. When the attribute changes, it launches a run.
• Conditional: The Script Actor subscribes to attribute change notifications for the monitored attribute. On each update, it evaluates the condition (compares the attribute against a threshold). Firing depends on the fire mode (see below).
• Expression: The Script Actor evaluates a compiled boolean expression against an attribute snapshot on each attribute change. Firing depends on the fire mode (see below).
• Fire mode (Conditional + Expression):
  • OnTrue (default): Conditional fires on each matching attribute change; Expression fires once per false → true transition (edge-triggered). This is the original behavior — a trigger configuration with no mode field is treated as OnTrue.
  • WhileTrue: On the false → true edge the script fires once, then re-fires on a periodic timer while the condition stays true; on the true → false edge the timer stops. The re-fire cadence is the script's minimum time between runs; with none configured the trigger degrades to the single edge fire and logs a warning.
• Minimum time between runs: If configured, the Script Actor tracks the last execution time and skips trigger invocations that fire before the minimum interval has elapsed. For a WhileTrue trigger it doubles as the re-fire cadence.

Per-attribute change routing (P2). Rather than broadcasting every attribute change to all child Script/Alarm Actors, the Instance Actor routes each change only to the children that monitor that attribute (built from their trigger configs at spawn time); Expression-trigger children, which read an attribute snapshot, receive all changes. Each child still keeps its own trigger gate as defense-in-depth — the routing is an optimization, not the authority.

Concurrent Execution

• Each invocation launches a new run on the script-execution scheduler.
• Multiple runs can execute concurrently (e.g., a trigger fires while a previous Instance.CallScript invocation is still running), up to MaxConcurrentRunsPerScript — beyond which the newest is shed (see below).
• The Script Actor coordinates but does not block on run completion.

Script Run Launch (WP3.1 — replaces the Script Execution Actor)

• A run is launched directly by the Script Actor through ScriptRunLauncher. There is no per-run child actor.
  • The former ScriptExecutionActor / AlarmExecutionActor were already inert shells: neither declared a Receive handler (they executed from their constructor), neither had a PostStop, state, or stash, and neither's IActorRef was ever a message target — the whole lifecycle lived inside a detached task the actor never observed. What they cost was an actor cell, mailbox, and name registration per run, plus a per-spawn expression-tree Props.Create. Removing them changed no semantics; the run body moved verbatim into the shared launcher.
  • Everything the shells provided is preserved: exception and timeout containment, one DI scope per run disposed on every path, the site-event/health telemetry, the Ask reply, the completion notification (now the coordinator's own Self), and the audit ExecutionId / ParentExecutionId threading. A run still in flight when its Script Actor is stopped runs to completion and its completion message dead-letters, exactly as before — stopping does NOT cancel in-flight runs; redeploy/undeploy semantics are unchanged.
  • Per-run Started/Completed site events are sampled, not unconditional (WP3.2). The two Info rows an instance script run used to emit unconditionally were the dominant site_events writer under load (design memo: docs/plans/2026-08-15-site-events-policy-design.md). They are now off by default (ScadaBridge:SiteRuntime:PerRunScriptEvents, default false); a ScriptRunSummaryRecorder accumulates per-(instance, script) run counters from the same call sites and a site-only hosted flush service emits one aggregate "script" Info row per interval (ScriptRunSummaryIntervalSeconds, default 300s) with a top-50-scripts breakdown, capped so a pathological deployment cannot mint an oversized row — an idle interval emits nothing. ScadaBridge:SiteRuntime:PerRunScriptEventScripts restores the legacy per-run rows for named scripts ("Instance/Script" exact or "Instance/*" wildcard) for live debugging. All three keys are read from IOptionsMonitor<SiteRuntimeOptions> per run, so they are hot-togglable without a restart. Error-level rows (timeout, failure, stuck-watchdog, recursion-limit) are unaffected — always emitted, regardless of this policy. See Component-SiteEventLogging.md → Volume Policy for the full design.
  • One deliberate improvement: a failure of the launch itself (e.g. queueing onto a disposed scheduler) is caught by the Script Actor, which replies to the Ask caller and releases the run slot. The old per-run child's constructor throw was handled by a Stop supervision directive that sent no reply, leaving the caller to hang to its Ask timeout.
• The script body runs on the dedicated ScriptExecutionScheduler (a bounded set of dedicated threads), not the shared .NET thread pool, so blocking script I/O cannot starve the global pool or stall Akka dispatchers. The scheduler is process-wide by default (one pool per host), but each script/alarm actor takes it through an optional injection seam rather than reaching for the static directly: the Host injects nothing and gets the shared pool, while tests (or a future multi-site host) can hand an actor its own instance. The shared accessor also recreates a disposed pool rather than returning it, so a disposed scheduler can never silently poison later executions.
• Pool sizing is instance-scaled and grow-only (WP3.1). The pool was a fixed 8 threads regardless of load. It is now clamp(max(ScriptExecutionThreadCount, ceil(enabledInstances / 8)), 1, ScriptExecutionMaxThreadCount) — the existing ScriptExecutionThreadCount (default 8) becomes the floor, so configurations at or below 64 instances behave exactly as before, and the new ScriptExecutionMaxThreadCount (default 32) is the ceiling. Beyond the ceiling the per-script cap below is the real regulator. DeploymentManagerActor.UpdateInstanceCounts calls EnsureCapacity on every deploy / undeploy / enable / disable and once per staggered startup batch. Growth only: undeploying leaves idle threads, which cost nothing measurable and avoid drain/steal complexity.
• The deadline is armed at enqueue, not at dequeue (WP3.1). The run's timeout CancellationTokenSource is created on the actor thread before the body is queued, so queue wait consumes the script's own budget. A body that dequeues past its deadline skips execution entirely and takes the existing timeout path (site event, script-error counter, error reply, completion message): a saturated pool sheds stale work instead of running it late with a fresh full budget.
• Concurrent runs per script are capped (WP3.1). MaxConcurrentRunsPerScript (default 4) bounds runs in flight — queued or executing — for any one script or alarm on-trigger script. Over the cap the newest run is shed: the four already in flight are closest to their own deadlines and already charged against them, so nothing is ever reordered and no extra queue is needed (the scheduler's FIFO already is the queue). A shed increments ISiteHealthCollector.IncrementScriptRunShed (surfaced as ScriptRunShedCount), emits a script/Warning site event rate-limited to one per script per minute so a hot trigger cannot flood site_events, and — for an Ask-based CallScript — replies with an explicit error so a nested call or inbound-API route fails fast instead of hanging.
  • The cap gates only depth-0 launches (review fix). A trigger fire or a depth-0 CallScript/inbound-API route counts against MaxConcurrentRunsPerScript; a nested Instance.CallScript self-recursion (callDepth > 0) is exempt — it is already bounded by MaxScriptCallDepth instead, and the calling run is holding one of the four cap slots itself while it awaits the callee, so counting the nested launch too would spuriously shed legitimate self-recursion.

Traced runs (Flow Debugger, 2026-09-05)

• A run is traced when its ScriptCallRequest.Trace flag is set (a routed inbound call with TraceAllCalls or a Flow Debugger Live run), or when it was spawned by a value change whose AttributeValueChanged.Trace was set (see Trace arm under the Instance Actor). ScriptRunLauncher passes isTraced + a per-run FlowTraceCollector (cap 500 events, Truncated flag) into ScriptRuntimeContext, and on completion — success, timeout or failure — emits one ScriptTrace.ScriptRun audit row (occurred-at = run start, Extra.triggerKind = call or valuechange with the trigger attribute/value/source) through the site IAuditWriter.
• Every trace row is also collected in the run's collector and returned inline on ScriptCallResult.TraceEvents (with ScriptCallResult.ExecutionId), which DeploymentManagerActor maps onto RouteToCallResponse.ChildExecutionId / TraceEvents — Live gets its steps in the same reply; Replay reads the same rows from the Audit Log after the 5 s telemetry push.
• A value-change run spawned by a traced write inherits the writer's execution id as ParentExecutionId and runs traced itself, so a Flow Debugger value-change entry sees the run the write caused.
• Untraced runs emit nothing new — the flag defaults to false on every record and the collector is null.

Handling Instance.CallScript

• When an external caller (another script run, an alarm on-trigger run, or a routed call from the Inbound API) sends a CallScript message to the Script Actor, it launches a run to handle the call.
• The caller uses the Akka ask pattern and receives the return value when the execution completes.
• Trace flag inheritance: ScriptRuntimeContext.CallScript forwards the calling run's IsTraced on ScriptCallRequest.Trace, and CreateChildContextForSharedScript hands the same flag + collector to an inline shared-script context, so a nested run's ScriptRun / AttributeWrite / AttributeWait rows land in the same flow.

---

Alarm Actor

Role

• Coordinator for a single alarm definition on an instance.
• Evaluates alarm trigger conditions against attribute value updates.
• Manages alarm state (active/normal) in memory.
• Executes on-trigger scripts when the alarm activates.

Alarm Evaluation

• Subscribes to attribute change notifications from its parent Instance Actor for the attribute(s) referenced by its trigger definition.
• On each value update, evaluates the trigger condition:
  • Value Match: Incoming value equals the predefined target. Supports "!=X" prefix for not-equals semantics.
  • Range Violation: Value is outside the allowed min/max range.
  • Rate of Change: Value change rate exceeds the defined threshold over a configurable time window. Direction filter (rising / falling / either) restricts which side of the rate triggers.
  • HiLo: Multi-setpoint level alarm with up to four configurable setpoints (LoLo, Lo, Hi, HiHi). Any subset may be configured. Each setpoint may carry its own priority that overrides the alarm-level priority for that band.
• Expression trigger evaluation runs on the shared .NET thread pool behind a process-wide concurrency gate (TriggerEvalGate, sized by TriggerEvalMaxConcurrency, default max(2, ProcessorCount)) — not on the bounded script-execution pool. This is the WP3.1 fix for arch-review finding #4 (High): when evaluation shared that pool, N script bodies blocked in synchronous I/O stalled every Expression trigger on the node — scripts and alarms alike — for an unbounded time, and the evaluation's own 2 s timeout was constructed inside the queued body, so it did not start ticking until dequeue. An alarm that should have raised in milliseconds simply never raised, with neither a raise nor a timeout visible to the operator. Trigger expressions are non-blocking by construction (TriggerExpressionGlobals exposes only reads over an in-memory snapshot, and the script trust gate has already denied I/O, network, threading, and reflection), so the shared pool is where they belong; a second dedicated pool was considered and rejected as adding threads, gauges, and a second starvation surface for no isolation gain. The evaluation deadline (TriggerEvalTimeoutSeconds, default 2, previously hardcoded) is now armed at enqueue, so gate-wait time burns the same budget and a saturated gate yields a timely false rather than an unbounded stall. Per-actor coalescing (one evaluation in flight, one pending) is unchanged and caps waiters at one per Expression trigger, so the gate queue is bounded by trigger count.
• For binary trigger types (ValueMatch / RangeViolation / RateOfChange), when the condition is met and the alarm is currently in normal state, the alarm transitions to active:
  • Updates the alarm state on the parent Instance Actor (which publishes to the Akka stream).
  • If an on-trigger script is defined, launches its run via ScriptRunLauncher (see Alarm On-Trigger Run below) — no per-run child actor.
• When the condition clears and the alarm is in active state, the alarm transitions to normal.
• For HiLo triggers, the actor tracks the current AlarmLevel (None / Low / LowLow / High / HighHigh). Each level transition emits a fresh AlarmStateChanged with the new level and its priority; level escalations (e.g., High → HighHigh) and de-escalations (HighHigh → High) both produce events. The on-trigger script fires only on the Normal → non-None edge, not on escalations between alarm bands.
• No script execution on clear in any trigger type.

Alarm State

• Held in memory only — not persisted to SQLite. State comprises AlarmState (Active / Normal) and AlarmLevel (None for binary triggers; the active band for HiLo).
• On restart (or failover), alarm states are re-evaluated from incoming values. All alarms start in normal state with level None and transition based on incoming values.

Alarm On-Trigger Run

• Launched directly by the Alarm Actor when an on-trigger script needs to execute — the same ScriptRunLauncher path as an instance script run, with the firing alarm's name/level/priority/message exposed through the Alarm global. Bounded by the same MaxConcurrentRunsPerScript cap.
• Has access to the Instance Actor for GetAttribute/SetAttribute.
• Can call instance scripts via Instance.CallScript() — sends an ask message to the appropriate sibling Script Actor.
• Instance scripts cannot call alarm on-trigger scripts — the call direction is one-way.

Audit correlation of an on-trigger run (ParentExecutionId tag-cascade)

An alarm fired by a script- or inbound-API-initiated static attribute write is a genuine execution spawn, and the on-trigger run records the writing execution as its parent in the central Audit Log (#23). The originating id rides site-locally through three additive, nullable fields — no wire, proto or schema change:

Hop	Field
ScriptRuntimeContext.SetAttribute (this run's ExecutionId) / RouteToSetAttributesRequest.ParentExecutionId (the inbound request's)	→ SetStaticAttributeCommand.SourceExecutionId
Instance Actor static-write path (HandleSetStaticAttributeCore)	→ AttributeValueChanged.SourceExecutionId
Alarm Actor trigger evaluation → SpawnAlarmExecution	→ ScriptRunLauncher.LaunchAlarmScript → ScriptRuntimeContext.ParentExecutionId

All four computed trigger types participate. Expression triggers evaluate a whole attribute snapshot off the dispatcher, so the firing change is no longer in scope when the boolean returns: the writer of the newest value folded into the snapshot is captured with the snapshot and echoed back on the evaluation result, so a change arriving mid-flight cannot mis-attribute the raise.

The on-trigger run is correctly a tree root (ParentExecutionId NULL) when the firing value has no originating execution:

• Values from the Data Connection Layer — external device data has no spawning execution. This includes the confirmed value of a write to a data-sourced attribute: that write is forwarded to the device and the echo arrives on the subscription long after the writing execution ended, so only static attribute writes cascade.
• Deploy-time seeding and central Test Run / tooling writes.

AttributeValueChanged.SourceExecutionId is deliberately not projected onto the gRPC debug/stream wire shapes — the cascade is resolved inside the site node that owns the Instance Actor, and central reads the linkage from audit rows.

---

Native Alarm Actor

Role

• Read-only mirror of alarms raised natively by an external source — OPC UA Alarms & Conditions (A&C) servers and the MxAccess Gateway — surfaced into the Site Runtime alongside the alarms ScadaBridge computes itself.
• Created as a child of the Instance Actor and is a peer to the computed Alarm Actor (not a child of it). One NativeAlarmActor is spawned per resolved native alarm source binding on the instance.
• Mirrors source-of-truth condition state into the Instance Actor's view and onto the site-wide stream; it does not acknowledge, clear, or otherwise write back to the source. There is no ack-back path — the external source remains authoritative.

Construction

• Constructed with (ResolvedNativeAlarmSource source, string instanceName, IActorRef instanceActor, IActorRef dclManager, SiteStorageService storage, SiteRuntimeOptions options, ILogger logger, AlarmKind nativeKind = NativeOpcUa).
• nativeKind distinguishes the two native flavors and stamps the Kind on every emitted AlarmStateChanged. The Instance Actor selects it from the bound connection's protocol (see Instance Actor wiring below).

Lifecycle & Subscription

• PreStart: rehydrates any previously mirrored conditions for this source from the site SQLite native_alarm_state table, then subscribes to the source through the Data Connection Layer by sending a SubscribeAlarmsRequest to the DCL manager. The DCL routes the subscription to the bound connection's IAlarmSubscribableConnection implementation.
• Failed subscribe: schedules a retry timer at NativeAlarmRetryIntervalMs and re-attempts until the subscription is established. Rehydrated state remains visible in the meantime.
• NativeAlarmSourceUnavailable: the source connection has dropped. The actor retains its last-known mirrored conditions but marks them uncertain rather than purging them, so a transient disconnect does not flap every condition to normal. The set is reconciled against truth by the next reconnect snapshot.

Transition Handling (NativeAlarmTransitionUpdate)

• Owned subscription cleanup: Instance Actors send tag unsubscribes for each bound connection in PostStop; Native Alarm Actors send their alarm-source unsubscribe with their own actor identity. DCL DeathWatch provides cleanup if an owner terminates without a graceful message. Late cleanup from an old actor cannot remove its same-name replacement.
• OPC UA refresh scope: snapshot start, entries and completion preserve the exact source binding and refresh epoch. Native Alarm Actors ignore unrelated feeds and mismatched/duplicate completion epochs, and discard a partial refresh on source disconnect. MxGateway keeps its connection-wide snapshot framing.
• Snapshot / SnapshotComplete (reconnect reconciliation): Snapshot updates buffer into a staging set; SnapshotComplete performs an atomic swap of the mirrored set with the staged set. Any condition that was previously mirrored but is not present in the new snapshot emits a return-to-normal AlarmStateChanged and drops out. This is how the mirror self-corrects after an outage.
• Live transitions (Raise / Ack / Clear / Retrigger / StateChange): upsert the condition by SourceReference. Updates carrying a TransitionTime older than the currently held transition are ignored (out-of-order protection). Accepted transitions persist to SQLite and emit an enriched AlarmStateChanged upward to the Instance Actor.
• Retention: a mirrored condition is dropped once it is both inactive and acknowledged (!Active && Acknowledged) — the alarm has fully run its course at the source and no longer needs mirroring. The drop emits a final state change and deletes the SQLite row.
• Per-source cap: at most MirroredAlarmCapPerSource conditions are retained per source. When the cap is exceeded the oldest condition is dropped and the eviction is logged — there is no silent truncation. If the evicted condition is still active, a final return-to-normal AlarmStateChanged is emitted before it is dropped (mirroring the retention drop above), so a capped-out active condition is never left stuck-active downstream.

Persistence

• Mirrored condition state is persisted to the site SQLite native_alarm_state table on every accepted transition and removed on drop-out.
• Persistence is best-effort / fire-and-forget: a persistence failure is logged but never blocks the actor's mailbox and never aborts the upward AlarmStateChanged emit. The in-memory mirror remains authoritative for the running actor; SQLite exists to survive failover.

Supervision & Restart

• Supervised by the Instance Actor under the same OneForOneStrategy as the computed Alarm Actor — a native source fault is isolated to its own actor.
• On site restart or failover, the actor rehydrates its mirror from native_alarm_state in PreStart, then reconciles against the source via the reconnect snapshot. Native mirror state therefore survives failover (unlike computed alarm state, which is re-evaluated from values).
• Mirrored native state is cleared on redeploy/undeploy of the instance (mirroring the static-override reset): the stale rows for the instance are removed and the fresh actor re-subscribes from a clean slate.

---

Instance Actor — Native Alarm Wiring

The Instance Actor owns native-alarm setup alongside its computed Script and Alarm Actors:

• Spawning: for each entry in _configuration.NativeAlarmSources, the Instance Actor spawns a NativeAlarmActor. Spawning is skipped when there is no DCL manager (e.g., debug/test contexts with no data connections), since native alarms require a live source subscription.
• Kind derivation: the AlarmKind passed to each NativeAlarmActor is derived from the bound connection's protocol — Mx* protocols → NativeMxAccess, otherwise → NativeOpcUa.
• Latest-event retention: the Instance Actor retains the latest enriched AlarmStateChanged per alarm name in _latestAlarmEvents. The DebugView snapshot is built from this map so it carries the unified condition view plus native metadata for both computed and native alarms. Computed alarms that have not yet produced an event fall back to a Normal projection so the snapshot is complete.
• Idle native source binding placeholders: BuildAlarmStatesSnapshot() additionally emits one placeholder AlarmStateChanged for each configured native alarm source binding that currently has no live conditions in _latestAlarmEvents. The placeholder has IsConfiguredPlaceholder = true and carries the binding's canonical name in NativeSourceCanonicalName. The Instance Actor maintains a _nativeAlarmKinds map (sourceCanonicalName → AlarmKind) populated at spawn time to stamp the correct Kind on each placeholder. This ensures the Debug View Alarms tab shows every configured binding as a tree node even when quiet.
• NativeSourceCanonicalName on live events: every AlarmStateChanged emitted by a NativeAlarmActor stamps its source binding's canonical name in NativeSourceCanonicalName. The Debug View uses this field to place each live condition under the correct native-source binding node in the tree.
• Reset semantics: _latestAlarmEvents and the mirrored native state are cleared on redeploy/undeploy (same trigger as static-override reset) but rehydrate from SQLite on failover. In addition, a native condition's _latestAlarmEvents key is evicted per-condition whenever that condition leaves the mirror — snapshot-swap drop, retention drop, or cap eviction — driven by a NativeAlarmDropped signal from the NativeAlarmActor; this prevents unbounded _latestAlarmEvents growth for sources that mint a fresh SourceReference per occurrence.

---

Native Alarm State Persistence (Site SQLite)

SiteStorageService gains a native_alarm_state table backing the native mirror:

• Primary key: (instance_unique_name, source_canonical_name, source_reference) — one row per mirrored condition.
• Columns: condition_json (the serialized AlarmConditionState), metadata_json (display metadata — type/category/message/current+limit, UA4), and last_transition_at (the accepted TransitionTime).
• Operations: Upsert (on accepted transition), Delete (on condition drop-out), Get (PreStart rehydrate, scoped to instance + source), and ClearForInstance (redeploy/undeploy reset).
• Display metadata persistence (UA4): persisted rows include the condition's display metadata, so rehydrated conditions render fully (type / category / message / current + limit values) before the first fresh source snapshot arrives; a pre-UA4 row with no metadata rehydrates with empty display fields (unchanged behavior).
• Ack timestamp persistence: the condition's AckTime rides the same metadata_json blob, so an acknowledged alarm keeps its ack instant across restart/failover rather than reappearing as if it had never been acknowledged. It is deliberately not a new physical column: native_alarm_state is RegisterReplicated in SiteLocalDbSetup, and LocalDb builds that table's CDC triggers from the column list at registration time — an additive JSON property changes no schema, no triggers and no replication contract, and metadata_json is exactly the extension point UA4 introduced for this. A row written before the field existed deserializes it as null ("ack time unknown"), which is the correct fallback. See docs/plans/2026-06-30-mes-alarm-status-api.md §6.4.
• Coalesced writes (P4): per-transition upserts are coalesced (latest-per-source-reference) and flushed on a short single-shot timer rather than one SQLite write per transition, so an alarm storm collapses to one batched write; a pending upsert is dropped if the condition is deleted before the flush (no resurrection). Durability window is at most one flush interval on crash, reconciled by the next re-subscribe snapshot.
• This is a peer SQLite store to the existing deployed-configuration, store-and-forward, operation-tracking, and AuditLog stores. Unlike computed alarm state, native mirror state is intentionally persisted so it survives failover.

---

Deployed Secrets (Site SQLite / LocalDb)

Site secrets ride the artifact deploy (design docs/plans/2026-09-28-site-secrets-artifact-deploy-design.md; see that doc's Implementation notes (as built) for deviations from the original design). Three tables, two different replication postures:

• deployed_secrets — the 11th LocalDb-replicated table (SiteLocalDbSetup.ReplicatedTables, up from ten). Columns: name (PK), row_json (the shipped SecretArtifact — ciphertext, nonce, tag, wrapped DEK, KekId, revision; never plaintext), revision, updated_utc, is_removed (soft-remove flag, default 0), deployment_id, applied_at. No FKs (the standing FK guard — SiteLocalDbReplicatedTableFkGuardTests — applies to it like every other replicated table). UpsertDeployedSecretsAsync is full-set upsert-then-soft-remove, never a hard DELETE, so a removal is itself a replicated row change the standby node can react to.
• secret_mirror_status — the 12th LocalDb-replicated table (review finding M1). One row per site node: node_name (PK — INodeIdentityProvider.NodeName, i.e. node-a/node-b, falling back to the machine name when unconfigured), failures_json (a JSON array of {SecretName, Message} — the node's current mirror failures; names and messages only, never values or ciphertext), checked_at. No FKs. It is per-node data but replicated on purpose: each node writes only its own row, and replication is what lets the active node read the standby's row onto the site health report (only the active node sends one — see Mirror status on the health report below).
• mirrored_secrets — a node-local, unreplicated ledger (name PK, revision, mirrored_at) recording which names this specific node has applied into its own ISecretStore. Deliberately not registered with LocalDb: it is a per-node fact (which names this node mirrored), not site-wide state, and replicating it would make one node's mirroring progress masquerade as the other's.

DeployedSecretsMirror (SiteRuntime/Secrets/DeployedSecretsMirror.cs) is a per-node BackgroundService — registered on every site node, not a singleton — that reconciles deployed_secrets into the node's own local ISecretStore (site nodes never run a replication-decorated store; see Component-Host.md → Secrets):

• Schedule: runs once immediately at start, then every SiteRuntime:DeployedSecretsMirrorInterval (default 15s, validated > 0), and synchronously on demand right after an active-node artifact apply (see Deployed Secrets under Deployment Manager Singleton above). This 15s interval is the standby node's own convergence path — it never receives the active node's synchronous call, only the replicated deployed_secrets rows via LocalDb, so it picks them up on its own next tick. A pair that uses deployed secrets must therefore run LocalDb replication: on an unreplicated pair only the node that was active at deploy time holds the secrets, and after a failover the new active node has none until the next Deploy Artifacts (topology guide → Secrets: shared store, per-site deploy, and the KEK).
• Apply is confirm-then-ledger, not blind. ApplyReplicatedAsync is strict last-writer-wins on (UpdatedUtc, Revision) (a tie is a no-op) and DeleteAsync tombstones rather than hard-deletes — so the mirror always reads the row back after applying and only ledgers it in mirrored_secrets when the local ciphertext matches what was shipped. A tombstoned or clock-skewed local row that the ledger already owns is safely re-stamped (UpdatedUtc/Revision bumped past the local value) and re-applied; a live row with different ciphertext that the ledger does not already own is left completely untouched — not overwritten, not ledgered, not verified — and reported as an out-of-band conflict ("exists on this node with a newer local value; the deployed revision was not applied"), so an operator's genuinely local secret of the same name is never silently destroyed by a later central deploy or removal.
• Ownership, not tombstone-state, decides re-stamping. Once a name is ledgered in mirrored_secrets at any revision, this mirror treats that name as its own going forward — including re-stamping a mirror-owned name that drifted back to the deployed value on the next pass. This holds even when the name is already ledgered at the shipped revision: the mirror reads the local row back every pass, and one that is missing, tombstoned (an operator deleted it locally) or live with different ciphertext (an operator edited it) is healed through the same bumped-stamp re-apply, read-back and cache invalidation before the verify — a local delete is repaired, never misreported as a master-key mismatch. The drift is logged at Warning once per occurrence. Central is authoritative for every name it deploys.
• Unreadable rows are isolated. A deployed_secrets row whose row_json cannot be parsed is reported as a failure for that name (in Failed/FailedNames) instead of aborting the pass; an unreadable removed row is still honoured by name.
• Removal deletes from the local store only if the name was previously ledgered (present in mirrored_secrets) — an out-of-band local secret sharing a deployed name that was never mirrored is never touched by a central-side removal.
• Verify. Every pass trial-resolves every live row through ISecretResolver (even one not re-applied this pass, so a later KEK change is still caught) — a decrypt failure names the secret and its KekId and is reported as a distinct error (de-spammed: Error the first time a given (name, revision, KekId) fails, Debug on identical repeats, Error again after an intervening success).
• Isolation. One row failing to apply/verify does not block any other row in the same pass.
• Cache invalidation. A successful apply or delete invalidates the shared ISecretResolver cache before verifying, so the verify step (and any script/connection reading the secret) sees the fresh value immediately rather than waiting out the resolver's cache TTL.
• Never logs a secret value — log lines and the reported MirrorResult carry only names, revisions, and KekIds.
• Mirror status on the health report (review finding M1). A failure on the standby — above all a secret that will not decrypt because the standby's master key differs from central's — used to live only in that node's log, invisible until a failover broke its connections. After every pass the mirror records its node's current failure set in secret_mirror_status, but only when the set (sorted name+message pairs) differs from what this process last wrote, plus once on the first pass after start-up so a restart refreshes a stale row, plus at least every SiteRuntime:SecretMirrorStatusRefreshInterval (default 10 min) as a heartbeat — never on every 15 s tick, because every write is a replicated CDC change. The write is best-effort: a failure logs a Warning (repeats at Debug), never affects the pass result, and is retried next pass. SecretMirrorStatusReporter (SiteRuntime/Secrets/, every site node, 30 s, immediate first probe) reads every node's row into ISiteHealthCollector.SetSecretMirrorStatuses, so the active node's report carries both nodes' status as SiteHealthReport.SecretMirrorStatuses (see Component-HealthMonitoring.md). On an unreplicated pair only the active node's own row is visible — the same replication prerequisite deployed secrets already have. Stale rows are pruned: the row is keyed by node name, so a decommissioned or renamed node used to leave its row (and any Secret mirror failing badge) behind forever. The reporter now leaves off the report, and deletes, any row whose checked_at is older than SiteRuntime:SecretMirrorStatusStaleAfter (default 1 h, validated ≥ 2 × the refresh interval) — the delete is conditional on the checked_at it read, so a node that re-wrote its row in the meantime keeps it, and it replicates like any other write. A node down longer than that also drops off the list (its absence already shows in the cluster member count) and writes its row again on its first pass after it returns. Two guards keep a live node's row from being lost to the prune: the reporter prunes nothing during its first refresh interval after start-up (on a pair restarted together — an upgrade from a build without the heartbeat — every row is still old until each node's first write has replicated), and each mirror pass checks with a local read that its own row still holds what it last wrote, re-writing it otherwise (a peer's prune racing this node's write can replicate a delete over it; found on the rig).

---

Enriched AlarmStateChanged Message

The AlarmStateChanged message published by both Alarm Actors and Native Alarm Actors was extended additively (existing consumers keep working with computed defaults):

• Kind (AlarmKind): Computed for ScadaBridge-evaluated alarms; NativeOpcUa / NativeMxAccess for mirrored native alarms.
• Condition (AlarmConditionState): the unified condition view. Computed alarms supply a computed default; native alarms carry the mirrored source condition.
• Native metadata (populated for native alarms; defaulted/empty for computed): SourceReference, AlarmTypeName, Category, OperatorUser, OperatorComment, OriginalRaiseTime, CurrentValue, LimitValue.
• Computed-alarm projection: computed alarms are surfaced as auto-acknowledged with Severity = Priority, so a single enriched shape carries both computed and native alarms onto the stream and into the DebugView snapshot.
• NativeSourceCanonicalName (string?): the canonical name of the native alarm source binding a native condition belongs to (e.g. Motor1.NativeAlarms). null for computed alarms. Stamped by the NativeAlarmActor on every emitted event; also carried on idle-binding placeholder rows emitted by the Instance Actor.
• IsConfiguredPlaceholder (bool): true for a synthetic row emitted by the Instance Actor to represent a configured native source binding that currently has no live conditions. These rows are never persisted or forwarded as operational alarms — they exist solely so the Debug View can render every configured binding as a tree node. false on all other rows.
• AckTime (DateTimeOffset?): the instant the condition was acknowledged; null while unacknowledged, null on computed alarms (auto-acked but with no operator ack event), and cleared on re-raise. Stamped by the DCL adapter — the source's own ack instant where the protocol supplies one (OPC UA A&C), the DCL's observation time of the ack transition where it does not (MxAccess Gateway) — and carried verbatim by the NativeAlarmActor, which never invents or suppresses one. See Component-DataConnectionLayer.md → Ack Timestamp.

These fields are additive-only (init-only with defaults: null / false) and are mirrored additively on the gRPC AlarmStateUpdate proto as fields 22 (native_source_canonical_name, string), 23 (is_configured_placeholder, bool) and 24 (ack_time, google.protobuf.Timestamp, absent ⇒ null). StreamRelayActor and SiteStreamGrpcClient pack/unpack these fields without touching existing fields, preserving backward compatibility across version-skew boundaries.

The enriched message flows Instance Actor → site-wide Akka stream → SiteStreamManager → SiteStreamGrpcServer and is streamed to central as the gRPC AlarmStateUpdate event (see Component-Communication.md).

---

Shared Script Library

• Shared scripts are compiled at the site when received from central.
• Compiled code is stored in memory and made available to all Script Actors.
• When a script run calls Scripts.CallShared("scriptName", params), the shared script code executes inline in that run's context — it is a direct method invocation, not an actor message.
• This avoids serialization bottlenecks since there is no shared script actor to contend for.
• Shared scripts have access to the same runtime API as instance scripts (GetAttribute, SetAttribute, external systems, notifications, databases).
• Shared scripts always execute with Root scope (C5): Attributes[...] inside a shared script addresses root attributes even when the shared script is invoked from a composed-module script. Callers needing module-scoped access must pass the values in as parameters.

---

Script Lifecycle

All script types can be updated without restarting the cluster, but the mechanism differs per type.

Instance Scripts and Alarm On-Trigger Scripts

• Compiled at deployment time when the Deployment Manager receives a flattened configuration and creates the Instance Actor hierarchy.
• To update: modify the script in the template, then redeploy the instance (instance deploy --id <N>).
• Redeployment stops the existing Instance Actor and all its children, creates a new Instance Actor, and recompiles all scripts and alarms from the updated configuration.
• There is no way to hot-reload a single instance script without redeploying the entire instance.

Shared Scripts

• Compiled at the site when received from central via artifact deployment (deploy artifacts).
• The SharedScriptLibrary replaces its in-memory compiled code dictionary under a lock, making updated code immediately available to all Script Actors.
• To update: modify the shared script via the CLI or UI, then run deploy artifacts to push the change to sites.
• No instance redeployment is required — running instances pick up the new shared script code on the next Scripts.CallShared() invocation.

Inbound API Method Scripts

• See Component-InboundAPI.md for the compilation and hot-reload lifecycle of API method scripts.

Warning

: Editing scripts via direct SQL does not trigger recompilation. The in-memory compiled script will remain stale until the next node restart or redeployment. Always use the CLI, Management API, or Central UI to modify scripts.

---

Script Runtime API

Available to all script runs and alarm on-trigger runs:

Instance Attributes

• Instance.GetAttribute("name") — Read an attribute value from the parent Instance Actor.
• Instance.SetAttribute("name", value) — Write an attribute value. For data-connected attributes, writes to the DCL; for static attributes, updates in-memory and persists to local SQLite (survives restart/failover, reset on redeployment).

Other Scripts

• Instance.CallScript("scriptName", parameters) — Send an ask message to a sibling Script Actor. The target Script Actor launches a run, executes, and returns the result (or an explicit shed error if that script is already at its concurrent-run cap). The call includes the current recursion depth.
• Scripts.CallShared("scriptName", parameters) — Execute shared script code inline (direct method invocation). The call includes the current recursion depth.

External Systems

• ExternalSystem.Call("systemName", "methodName", params) — Synchronous HTTP call. Blocks until response or timeout. All failures return to script. Use when the script needs the result.
• ExternalSystem.CachedCall("systemName", "methodName", params) — Deferred delivery. Returns a TrackedOperationId tracking handle immediately rather than the response; the call is attempted immediately and, on transient failure, store-and-forwarded for retry. Use for outbound data pushes where deferred delivery is acceptable.
• The returned TrackedOperationId can be passed to Tracking.Status(id) (see Operation Tracking below) to observe delivery progress.

Notifications

• Notify.To("listName").Send("subject", "message") — Send a notification via a named notification list. Generates a TrackedOperationId (GUID) locally and returns it immediately; the notification is store-and-forwarded to the central cluster, which owns delivery. The script never contacts SMTP. (NotificationId is the notification-domain name for this same TrackedOperationId type.)
• Notify.Status("trackedOperationId") — A thin alias of Tracking.Status(id) retained for the notification domain. Returns a status record (status, retry count, last error, key timestamps). While the notification is still in the site store-and-forward buffer the site answers locally (status Forwarding); once forwarded the query round-trips to central.

Database Access

• Database.Connection("connectionName") — Obtain an audited ADO.NET connection for read/write. Commands and synchronous/asynchronous transactions retain the auditing connection on public readback; provider transactions are unwrapped only internally for command assignment. Each command execution emits one audit event, including commands reached through transaction.Connection or command.Connection. Commit, rollback, savepoints and disposal retain provider semantics.
• Database.CachedWrite("connectionName", "sql", parameters) — Submit a write operation for store-and-forward delivery. Returns a TrackedOperationId tracking handle immediately; pass it to Tracking.Status(id) to observe delivery progress.

Operation Tracking

• Tracking.Status("trackedOperationId") — Returns a status record (status, retry count, last error, key timestamps) for any tracked operation: a cached external system call, a cached database write, or a notification. For cached calls and writes the answer is always site-local and authoritative — the site owns the operation tracking table. (Notify.Status(...) is a thin alias scoped to the notification domain.)
• Cached-call telemetry is off the script thread (2026-09-06). A Database.CachedWrite / ExternalSystem.CachedCall emits three CachedCallTelemetry packets (submit, attempted, resolve) through CachedCallTelemetryForwarder, each an audit row plus a tracking row. Both halves used to be awaited to commit on the script thread; the audit half was the real cost — SqliteAuditWriter.WriteAsync returns the batch completion, and the writer coalesces for FlushIntervalMs (50 ms) before committing — so a cached call paid ~160 ms of a 176 ms run for bookkeeping (measured on the rig; now 17 ms). The forwarder now enqueues the audit row and observes its completion on a fault-logging continuation (the same pattern every other script-path audit emission already used), and OperationTrackingStore is a single-writer queue: RecordEnqueue/Attempt/TerminalAsync return once the row is queued, and one background pump applies each drained batch in one SQLite transaction. Preserved: FIFO order across the three lifecycle rows; read-your-writes — GetStatusAsync (behind Tracking.Status, and the telemetry actor's status join) and ReadChangedSinceAsync (central's PullSiteCalls) wait for every write queued before them; timestamps are stamped when recorded, not when applied; PurgeTerminalAsync returns after the purge lands; Both DisposeAsync and sync Dispose() drain the queue before closing the writer; host shutdown uses the async path. Accepted: a row queued but unapplied is lost on a hard crash (milliseconds), leaving the tracking row at its previous state until reconciled; the S&F buffer and audit rows are unaffected. The queue is bounded (8 192) with wait backpressure — it never drops. FlushAsync() is the explicit barrier for callers that go to the file directly. Trap for new audit emitters on a script path: never await IAuditWriter.WriteAsync there — it waits for the writer's flush, not the enqueue. Design + measurements: docs/plans/2026-09-06-tracking-store-async-writes-design.md.

Alarm State (read-only)

• Alarms.CurrentAsync() — Returns a snapshot of the instance's current alarm conditions as IReadOnlyList<ScriptAlarm> (Commons.Types.Scripts), covering computed alarms and mirrored native (OPC UA A&C / MxAccess Gateway) conditions alike. Served by a local Ask (GetAlarmSnapshotRequest/GetAlarmSnapshotResponse) to the script's own Instance Actor — the same mechanism attribute reads use, never a cross-cluster hop.
• Alarms.MesReportableAsync() — CurrentAsync() filtered to rows where ScriptAlarm.IsMesReportable is true (2026-09-11). Native alarms carry no per-alarm config row, so they never satisfy the filter — they still convey MES relevance via the 900–999 severity band (see docs/plans/2026-06-30-mes-alarm-status-api.md).
• Available to every script, unlike the Alarm global, which exists only inside an on-trigger handler and describes the single alarm that fired. This accessor is what lets an inbound-API-routed Call script answer "what is in alarm on this machine right now?"; before it, native mirrored conditions were reachable only from the Debug View.
• Read-only by design — native alarms are a read-only mirror of the source (no ack-back), so the accessor exposes no acknowledge/shelve operation.
• Not scope-prefixed, unlike Attributes: alarm identity is not a scope-relative attribute name (computed alarms are keyed by configured name, native conditions by a source-supplied reference), so every scope sees the same list rather than a silently truncated one.
• The snapshot is built by the same BuildAlarmStatesSnapshot() the Debug View uses, so it includes placeholder rows for configured-but-quiet native source bindings (IsConfiguredPlaceholder). Callers enumerating real alarms filter on Active && !IsConfiguredPlaceholder.
• ScriptAlarm is a flat projection of the retained AlarmStateChanged: Name, SourceReference, NativeSourceCanonicalName, Active, Acknowledged, Confirmed?, Shelved (all shelve sub-states collapse to one boolean), Suppressed, Severity (0–1000), Kind, Message, AlarmTypeName, Category, OperatorUser, OperatorComment, OriginalRaiseTime?, Timestamp, AckTime?, CurrentValue, LimitValue, IsConfiguredPlaceholder, IsMesReportable (trailing, 2026-09-11 — always false on native rows). The Condition is the authority for active/acked/severity, so computed and native alarms read identically from one filter expression.
• IsMesReportable is stamped in one place. InstanceActor.BuildAlarmStatesSnapshot() sets AlarmStateChanged.IsMesReportable from ResolvedAlarm.IsMesReportable on every row it produces — retained (fired/active) rows and the fallback/idle rows alike — so Alarms.CurrentAsync()/MesReportableAsync() and the Debug View snapshot always agree regardless of whether the alarm has fired. The flag is not carried on the gRPC stream or DebugAlarmStateDto (no proto change in this slice — deferred-work register row 122), so central-side surfaces (Debug View live stream, Alarm Summary) see false until that follow-up lands.
• The type lives in Commons so the runtime accessor and the Script Analysis compile-only surface (ScriptCompileSurface.CompileAlarmsAccessor) project to the same type — a script binding at the design-time gate binds identically at the site. Parity is pinned by CompileSurfaceParityTests.
• Design record: docs/plans/2026-06-30-mes-alarm-status-api.md §5.2.

File Access (Files, read-only, allow-listed — 2026-09-28)

• Files.ReadTextAsync(path) → string, Files.ReadLinesAsync(path) → IReadOnlyList<string> (\r\n/\n/\r; a trailing line break adds no empty last line), Files.ExistsAsync(path) → bool. Also reachable as Instance.Files. The use case that motivated it: a recipe/NC file whose path a Delmia/MES call hands to the machine, which a script parses and loads into attributes.
• The one sanctioned exception to the System.IO ban, and it is host-mediated. Scripts still cannot name any System.IO type — the trust policy is unchanged. The runtime FilesAccessor delegates to the site-singleton ScriptFileReader (SiteRuntime/Scripts/), which does every check on the host side.
• Off by default. ScadaBridge:SiteRuntime:ScriptFileAccess:AllowedRoots (list of absolute local/UNC directories, default empty) enables it; with no roots every call throws ScriptFileAccessException(Disabled), so an unconfigured site behaves exactly as before. ScriptFileAccess:MaxFileBytes (default 1 MiB, validated 1…64 MiB) caps one read. Both validated at boot (SiteRuntimeOptionsValidator — a relative root is rejected). Read per call through IOptionsMonitor.
• Guard, in order: roots configured → path non-empty, fully qualified, not a device path (\\?\, \\.\), no NTFS alternate data stream → canonical path (Path.GetFullPath, so .. is resolved first) strictly inside a root with a directory-boundary match (root C:\recipes does not admit C:\recipes2\x, and the root itself is not a file) → no symbolic link / junction / reparse point on the file or on any directory between the root and the file (the root itself may be a link — an administrator chose it) → size under the cap, re-checked while reading in case the file grows. Files are opened FileShare.ReadWrite | Delete so a file still being written by its producer can be read; text is BOM-detected, UTF-8 by default.
• Errors are one Commons type. Every refusal and I/O failure is ScriptFileAccessException (Commons.Types.Scripts) with a Failure of Disabled, InvalidPath, OutsideAllowedRoots, LinkNotAllowed, NotFound, TooLarge or IoError — scripts cannot name System.IO exception types, so this is what they catch. Commons.Types.Scripts is now one of the default script imports on every surface (runtime, deploy gate, Test Run editor), so the short name binds.
• Bounded by the run. Reads are cancelled by the script's own execution deadline as well as any token the script passes. Refusals (outside roots, links) log a Warning on the node; successful reads log at Debug. There is no audit row — file reads are not an audited channel (the AuditLog scope is outbound API/DB, notifications and inbound API).
• The node's service account still needs read permission on the path; the allow-list only ever narrows access. A time-of-check/time-of-use window exists between the link check and the open (a writer inside an allowed root could swap a file for a link in between) — acceptable because only the directories an administrator lists are reachable at all, and read-only.
• Site scripts only. Inbound API method scripts (central) have no Files global. The Central UI Test Run sandbox mirrors the member so scripts bind, but every call throws ScriptFileAccessException(Disabled) — central cannot reach a site's filesystem — so a script's own error handling runs as it would on an unconfigured site. The design-time ScriptCompileSurface.CompileFiles mirror is pinned by CompileSurfaceParityTests.
• Design record: docs/plans/2026-09-28-script-file-access-design.md.

Logging (Log)

• Log.Debug(message, exception?) / Log.Info / Log.Warn / Log.Error — leveled, per-run logging available to every script, alongside the Alarms accessor above (design docs/plans/2026-09-09-flow-debugger-step-logs-design.md D4/D5). Synchronous, void, and it never throws into the script: a failing sink is swallowed, because a debug observer must not fail a production run.
• Every line always reaches the node's ILogger (structured Instance / Script / ExecutionId / message), traced or not — an untraced run is an ordinary logging surface and pays nothing else. A ScriptTrace / ScriptLog audit row is emitted only when the run is traced (a Flow Debugger Live run, or a real call to a method with TraceAllCalls), so a timer or value-change script logging at high rate costs no audit volume. There is no site-event-log sink for Log.*.
• The implementation is ScriptLogger (SiteRuntime/Scripts/ScriptLogger.cs), created by ScriptRuntimeContext and exposed as ScriptGlobals.Log. One logger instance is one run's counter — CallShared shares the calling context, so an inline shared script shares the caller's counter, execution id and anchor row rather than starting its own.
• Runtime lines land on the step that produced them. While a run is traced, ScriptRuntimeContext wraps its own ILogger in a TraceCapturingLogger, which forwards everything to the inner logger unchanged and additionally copies Information and above into ScriptLog rows with logSource = "runtime" (Debug/Trace are deliberately not captured — internal chatter). IsEnabled returns true for Information and above regardless of the node's minimum level, because those entries are trace data rather than node log output. Attribution rides an ambient TraceStepScope (AsyncLocal<Guid?>): SetAttribute, WriteBatchAndWait, WaitInternal and the synchronous ExternalSystem.Call pre-mint their step row's EventId, open the scope around their body and pass the id to EmitTrace, so a warning the runtime logs inside a write shows on that write's node. Everything else falls back to the run's anchor ScriptRun row. Build ScriptLogger over the UNDECORATED logger — TraceCapturingLogger's constructor rejects an inner TraceCapturingLogger, since double wrapping would emit two rows per line.
• Bounded (D7). ScadaBridge:SiteRuntime:MaxLogLinesPerRun (default 200, validated > 0) caps the rows one run may emit; messages are clipped at 2 000 characters. Lines past the cap are counted and dropped, and a single closing warn / runtime row naming the dropped count is flushed at run end (it bypasses the counter — it is the one row that must always land).
• Exception detail. A failed run's ScriptRun row now carries errorDetail = ex.ToString() alongside the existing one-line errorMessage, so the Flow Debugger can show type, message, stack and inner exception. A timeout emits no detail — an OperationCanceledException stack is noise.
• Console.WriteLine is not the logging surface on a site. System is imported, so it compiles, and the Central UI Test Run sandbox does capture it; but on a site node Console.Out is process-global and shared by every concurrent script run, so per-run capture is unsound. Log.* is the leveled, per-run replacement, and the sandbox's mirror writes to the same console pane.

Parameter Access

• Parameters["key"] — Raw dictionary access (returns object?, requires manual casting).
• Parameters.Get<T>("key") — Typed access with descriptive error messages. Throws ScriptParameterException if parameter is missing, null, or cannot be converted to T.
• Parameters.Get<T?>("key") — Nullable typed access. Returns null if parameter is missing, null, or cannot be converted.
• Parameters.Get<T[]>("key") — Array access. Converts a list parameter to a typed array. Throws on first unconvertible element.
• Parameters.Get<List<T>>("key") — List access. Converts a list parameter to a typed List<T>.
• Supported types: bool, int, long, float, double, string, DateTime.

Recursion Limit

• Every script call (Instance.CallScript and Scripts.CallShared) increments a call depth counter.
• If the counter exceeds the maximum recursion depth (default: 10), the call fails with an error.
• The error is logged to the site event log.

---

Script Trust Model

Scripts execute in-process with constrained access. The following restrictions are enforced at compilation and runtime:

• Allowed: Access to the Script Runtime API (GetAttribute, SetAttribute, CallScript, CallShared, ExternalSystem, Notify, Database, Tracking, Alarms, Log), standard C# language features, basic .NET types (collections, string manipulation, math, date/time). System.Diagnostics.Stopwatch, Debug, and Activity are permitted.
• Forbidden: File system access (System.IO), process spawning (System.Diagnostics.Process), threading (System.Threading — except Tasks, CancellationToken, and CancellationTokenSource), reflection (System.Reflection), all raw network access (System.Net — must use ExternalSystem.Call), native interop (System.Runtime.InteropServices, Microsoft.Win32), assembly loading, unsafe code, dynamic, Activator.
• Execution timeout: Configurable per-script maximum execution time, measured from enqueue (WP3.1) — a run that spends its whole budget queued behind blocked scripts is shed at dequeue without executing, rather than being granted a fresh full budget for work whose triggering condition is already stale. Exceeding the timeout cancels the script cooperatively (S2/UA5): a script blocked in synchronous I/O or a tight CPU loop does not observe cancellation and continues to occupy its dedicated script-execution thread. A watchdog names the script once its thread has not returned within a grace period after the timeout — and, as of WP3.1, detaches and replaces that worker thread, so a wedged script costs the pool a thread only until the watchdog fires rather than for the rest of the process. The detached worker exits (instead of pulling more work) when its body finally returns, so capacity never silently doubles. Live detached workers are capped at the pool size — at the cap no replacement is started and an Error site event says so, because bounded starvation is preferable to unbounded thread growth when scripts wedge en masse. The scheduler's queue depth / busy-thread / oldest-busy-age / detached-thread gauges and the per-interval shed count are reported through Health Monitoring so a saturated, stuck, or shedding scheduler is visible.
• Memory: Scripts share the host process memory. No per-script memory limit, but the execution timeout prevents runaway allocations.

The forbidden-API policy is defined authoritatively in ScriptTrustPolicy (Script Analysis component, #25). ScriptCompilationService.ValidateTrustModel delegates to ScriptTrustValidator.FindViolations for the trust verdict; the Site Runtime also performs a CSharpScript.Compile against the real ScriptGlobals for execution. This is defence-in-depth static enforcement, not a true runtime sandbox.

Script Trust Boundary Auditing

Every script-trust-boundary call (External System Gateway, Database layer, Notify) emits an AuditEvent to IAuditWriter (site-local SQLite append). Hot path; never fails the calling action; failures logged via the SiteAuditWriteFailures health metric (see Component-HealthMonitoring.md). The central audit mirror and event schema live in Component-AuditLog.md.

Script Scoping Rules

• Scripts can only read/write attributes on their own instance (via the parent Instance Actor).
• Scripts can call other scripts on their own instance (via sibling Script Actors).
• Scripts can call shared scripts (inline execution).
• Scripts cannot access other instances' attributes or scripts.
• Alarm on-trigger scripts can call instance scripts; instance scripts cannot call alarm on-trigger scripts.

---

Tell vs. Ask Usage

Per Akka.NET best practices, internal actor communication uses Tell (fire-and-forget with reply-to) for the hot path:

• Tag value updates (DCL → Instance Actor): Tell. High-frequency, no response needed.
• Attribute change notifications (Instance Actor → Script/Alarm Actors): Tell. Fan-out notifications.
• Stream publishing (Instance Actor → Akka stream): Tell. Fire-and-forget.

Ask is reserved for system boundaries where a synchronous response is needed:

• Instance.CallScript(): Ask pattern from the running script (via ScriptRuntimeContext.CallScript) to sibling Script Actor. The caller needs the return value. Acceptable because script calls are infrequent relative to tag updates.
• Route.To().Call(): Ask from Inbound API to site Instance Actor via Communication Layer. External caller needs a response.
• Debug view snapshot: Ask from Communication Layer to Instance Actor for initial state.

Concurrency & Serialization

• The Instance Actor processes messages sequentially (standard Akka actor model). This means SetAttribute calls from concurrent script runs are serialized at the Instance Actor, preventing race conditions on attribute state.
• Script runs (launched via ScriptRunLauncher, no per-run child actor) may run concurrently, but all state mutations (attribute reads/writes, alarm state updates) are mediated through the parent Instance Actor's message queue.
• External side effects (external system calls, notifications, database writes) are not serialized — concurrent scripts may produce interleaved side effects. This is acceptable because each side effect is independent.

SiteStreamManager and gRPC Integration

• The SiteStreamManager implements the ISiteStreamSubscriber interface, allowing the Communication Layer's SiteStreamGrpcServer to subscribe to the stream for cross-cluster delivery via gRPC.
• When a gRPC SubscribeInstance call arrives, the SiteStreamGrpcServer creates a StreamRelayActor and subscribes it to SiteStreamManager for the requested instance. Events flow from SiteStreamManager → StreamRelayActor → Channel<SiteStreamEvent> → gRPC response stream to central.
• The SiteStreamManager filters events by instance unique name and forwards matching events to all registered subscribers (both local debug consumers and gRPC relay actors).

Site-Wide Stream Backpressure

• The site-wide Akka stream uses per-subscriber buffering with bounded buffers. Each subscriber (gRPC relay actors, future consumers) gets an independent buffer.
• If a subscriber falls behind (e.g., slow network on gRPC stream), its buffer fills and oldest events are dropped. This does not affect other subscribers or the publishing Instance Actors.
• Instance Actors publish to the stream with fire-and-forget semantics — publishing never blocks the actor.

Error Handling

Script Errors

• Unhandled exceptions and timeouts in a script run are logged locally to the site event log — unconditionally, regardless of the WP3.2 per-run Started/Completed sampling policy (see Script Run Launch above).
• The Script Actor (coordinator) is not affected — it remains active for future trigger events.
• Script failures are not reported to central (except as aggregated error rate metrics via Health Monitoring).

Alarm Evaluation Errors

• Errors during alarm condition evaluation are logged locally to the site event log.
• The Alarm Actor remains active and continues evaluating on subsequent value updates.
• Alarm evaluation error rates are reported to central via Health Monitoring.

Script Compilation Errors

• If script compilation fails when a deployment is received, the entire deployment for that instance is rejected. No partial state is applied.
• The failure is reported back to central as a failed deployment.
• Enforced by a site-side pre-compile validation gate (S3): before the Instance Actor is created or the config persisted, the Deployment Manager compiles all of the instance's scripts, alarm on-trigger scripts, and trigger expressions synchronously, as a pure prefix step of the deploy application on the singleton's actor thread — the verdict is never derived off-thread, because piping a verdict back to self would reorder concurrent deploys against delete/disable and break the redeploy-supersede mailbox-FIFO ordering. WP3.1 made it warm-then-gate: the same compile is first run off-thread purely to populate the process-wide compile cache (DeployCompileWarmed pipes back), so the authoritative synchronous gate is then all cache hits and no longer holds the singleton for a Roslyn compile. Ordering is preserved by a per-instance in-flight guard — while a warm is in flight for instance X, further mutating commands for X (deploy / delete / disable / enable) are queued and re-dispatched in arrival order once the deploy applies; a second deploy with nothing queued behind it supersedes the pending one last-write-wins and its displaced deployer is answered Failed-superseded rather than left to Ask-timeout. Commands for other instances flow freely, which is the point, and is safe because cross-instance ordering was never guaranteed to callers. Unchanged script bodies hit the compile cache, so the gate recompiles only genuinely new code. Any compile failure rejects the deployment with the collected compile errors. This makes the deployment result honest even if central's pre-deployment validation was bypassed or skew let a bad artifact through.
• Note: Pre-deployment validation at central should still catch compilation errors before they reach the site; the site gate is the authoritative backstop.

---

Dependencies

• Script Analysis (#25): ScriptCompilationService.ValidateTrustModel delegates the forbidden-API verdict to ScriptTrustValidator.FindViolations. The Site Runtime retains its own CSharpScript.Compile against the real ScriptGlobals for execution.
• Data Connection Layer: Provides tag value updates to Instance Actors. Receives write requests from Instance Actors. Also feeds Native Alarm Actors: connections implementing IAlarmSubscribableConnection (OPC UA A&C servers, MxAccess Gateway) deliver NativeAlarmTransitionUpdate events in response to a SubscribeAlarmsRequest, and signal NativeAlarmSourceUnavailable on connection loss. Subscription-handshake retry (S4/UA6): a tag-subscription handshake that either failed or whose response was lost is re-sent on a fixed interval per connection (default 5 s) — the retry is armed on every send and cancelled on the first success reply, closing both the failed-response and lost-response gaps; the NativeAlarmActor subscribe uses the same retry pattern.
• Store-and-Forward Engine: Handles reliable delivery for external system calls, cached database writes, and notifications submitted by scripts. For the notification category specifically, it forwards to the central cluster for delivery (not directly to SMTP). Owns the site-local operation tracking table that backs Tracking.Status(id).
• External System Gateway: Provides external system method invocations for scripts.
• Communication Layer: Receives deployments and lifecycle commands from central. Handles debug view requests. Reports deployment results.
• Site Event Logging: Records script executions, alarm events, deployment events, instance lifecycle events.
• Health Monitoring: Reports script error rates and alarm evaluation error rates.
• Local SQLite: Persists deployed configurations, system-wide artifacts (external system definitions, database connection definitions, data connection definitions). Sites also maintain peer SQLite stores for the Store-and-Forward buffer, the site event log, the operation tracking table, and the site-local AuditLog (see Component-AuditLog.md). The AuditLog file is purged on the same daily cadence as the others but respects the hard ForwardState invariant — rows still Pending forward are never purged, regardless of age.
• ZB.MOM.WW.Secrets.Abstractions (ISecretStore, ISecretResolver, ISecretCacheInvalidator): DeployedSecretsMirror applies deployed secrets into, and verifies against, the node's local secret store (composed by the Host — see Component-Host.md → Secrets). The Site Runtime depends only on the abstractions package, never the concrete store implementation.

Interactions

• Deployment Manager (central): Receives flattened configurations, system-wide artifact updates (including a site's referenced secrets, ciphertext only), and instance lifecycle commands.
• Data Connection Layer: Bidirectional — receives value updates, sends write-back commands.
• Communication Layer: Receives commands from central, sends deployment results, serves debug view data.
• Store-and-Forward Engine: Scripts route cached writes, notifications, and external system calls here.
• Health Monitoring: Periodically reports error rate metrics.

Medium architecture review lifecycle guarantees (2026-09-09)

• Alarm on-trigger runs resolve the same scoped integration services as ordinary scripts, including external systems, databases, notifications, operation tracking, auditing, and site/node provenance. The scope remains alive for the run and is disposed on every exit.
• Native alarm subscription retries retain the current correlation ID until its response deadline, reject stale replies, and back off to a bounded interval. Rehydration cannot overwrite a condition touched by a live event, including deletion, or a completed authoritative snapshot.
• Audited raw ADO connections expose no connection string and reject ConnectionString assignment and ChangeDatabase; their configured database name remains the audit target.
• Static override persistence runs in order off the actor dispatcher. Redeployment waits for both predecessor termination and successful final persistence acknowledgement before clearing old state; failed persistence fails the redeployment. At most four redeployment predecessor stops are in progress per deployment manager. Delete and disable retain the same-name stop barrier until the final flush and lifecycle persistence finish; an overlapping deploy/enable is rejected with a retryable busy diagnostic. Timer flushes keep coalescing while a previous SQLite write is pending.
• Synchronous operation-tracking disposal drains and joins its thread-pool writer before disposing SQLite resources, including when invoked under a synchronization context.