ScriptOptions.WithReferences(Assembly[]) resolves each assembly through
MetadataReference.CreateFromFile, which does not cache: every call mints a
fresh AssemblyMetadata -> PEReader -> NativeHeapMemoryBlock holding an
unmanaged copy of the assembly metadata that nothing disposes. Building the
options per compile therefore leaked native memory permanently — invisible to
the GC, to gcdump and to the managed allocation counters, so the working set
grew while the GC heap did not.
Diagnosed from a live dump of a wonder-app-vd03 Site node: 2,885 MB working
set 78 min after a cold start, only 150 MB live GC heap, ~2,469 MB on the
default process heap across ~6,700 undisposed AssemblyMetadata instances
against 473 DLLs on disk.
Three sites, all hoisted to static readonly:
- SiteRuntime ScriptCompilationService (the dumped one)
- InboundAPI InboundScriptExecutor — same defect on the central node; method
compiles recur on every re-registration and revision change
- CentralUI ScriptAnalysisService — CreateFromFile per sandbox run
ScriptAnalysis RoslynScriptCompiler also builds options per call but draws
from the static ScriptTrustPolicy.DefaultReferences, so it mints no metadata
and is left alone.
Guarded by reference-equality on the artifact rather than by watching memory:
a bytes-watching test would be flaky, and the leak is native so the managed
counters cannot see it at all. The test is proven to fail before the fix.
This does NOT close the AddTemplateScript OOM — that path was shown twice not
to compile scripts. It explains how a long-running node reaches a native
memory state where a large allocation fails with gigabytes free, which is a
lead worth re-testing, not a closure.
MES alarm-status API §5.2 (docs/plans/2026-06-30-mes-alarm-status-api.md,
Phase 1 tasks 2-4). Site `Call` scripts had NO way to read alarm condition
state: the `Alarm` global exists only inside an on-trigger handler and
describes the one alarm that fired, and native mirrored conditions were
reachable only from the Debug View. That gap blocked the CvdReactor
SimpleAlarmStatus/AlarmStatus scripts entirely -- they cannot be written
without it. `Alarms.CurrentAsync()` closes it.
The data was already local: the script runs inside its own Instance Actor's
context, so this is a LOCAL Ask -- the same mechanism attribute reads use, no
cross-cluster hop. A dedicated GetAlarmSnapshotRequest/Response is used rather
than reusing DebugSnapshotRequest, which would materialise every attribute
value on every alarm poll; both are served from the same
BuildAlarmStatesSnapshot(), so the script view and the operator's Debug View
can never disagree.
Deliberate shape decisions:
- NOT scope-prefixed, unlike Attributes. Alarm identity is not a
scope-relative attribute name (computed alarms are keyed by configured
name, native conditions by a source-supplied reference), so prefixing
would hand a composed script a silently truncated list.
- Read-only. Native alarms are a read-only mirror of the source (no
ack-back), so no acknowledge/shelve operation is exposed.
- Placeholder rows are NOT pre-filtered: a caller must be able to tell
"binding configured and quiet" from "binding unknown". The documented
filter is `Active && !IsConfiguredPlaceholder`.
- ScriptAlarm lives in Commons so the runtime accessor and the compile-only
surface project to the SAME type -- a script binding at the design-time
gate binds identically at the site. Condition is the authority for
active/acked/severity, so one filter expression works across computed and
native alarms.
Mirrored on BOTH design-time surfaces. ScriptCompileSurface is covered by the
reflection parity guard (AlarmsAccessor added to its mirror pairs). The Central
UI Test-Run SandboxScriptHost is the third, hand-maintained mirror that the
parity test cannot reach (Central UI does not reference Site Runtime); without
it the design page would false-flag CS1061 on scripts the deploy gate accepts.
It throws a labelled ScriptSandboxException at run time -- there is no central
route to per-instance alarm state, and returning an empty list would read as
"nothing is in alarm", which is worse than an error.
ScriptTrustPolicy needs NO change, and the reason is structural rather than
incidental: the trust boundary is a deny-list over API roots, not an allow-list
of context members. A test pins that no ForbiddenScopes entry prefixes the
Commons script-surface namespace, so a future deny-list entry cannot silently
make ScriptAlarm untouchable.
Tests: 6 accessor cases (Ask contract, full native projection incl. AckTime,
unacked, computed-alarm derivation, placeholder visibility, scope-independence),
2 InstanceActor snapshot cases incl. equality with the Debug View row set, the
full MES script shape compiling against ScriptCompileSurface, 2 trust cases,
and a sandbox diagnose-clean case reading every projected ScriptAlarm field.
M5.4 T4 threaded a `parentExecutionId` parameter through
AlarmActor.SpawnAlarmExecution → AlarmExecutionActor → ScriptRuntimeContext,
but every call site passed null — so alarm on-trigger runs were silently always
execution-tree roots, contradicting the "tag-cascade coverage is complete"
claim in CLAUDE.md and Component-AuditLog.md.
Source the id where a spawner genuinely exists: a static attribute write issued
by a site script (`Instance.SetAttribute`) or by an inbound API request
(`Route.To(...).SetAttributes(...)`, whose ParentExecutionId was already carried
to the site and then dropped). The id rides site-locally through three additive,
nullable fields — no wire, proto or central schema change:
ScriptRuntimeContext.SetAttribute / RouteToSetAttributesRequest.ParentExecutionId
→ SetStaticAttributeCommand.SourceExecutionId
→ AttributeValueChanged.SourceExecutionId (InstanceActor static-write path)
→ AlarmActor.SpawnAlarmExecution → AlarmExecutionActor → ScriptRuntimeContext
All four computed trigger types participate. Expression triggers evaluate off
the dispatcher, so the writer of the newest value folded into the snapshot is
captured *with* the snapshot and echoed home on ExpressionEvalResult /
ExpressionEvalFailed — a change arriving mid-flight cannot mis-attribute the
raise.
Deliberately still roots (documented, not deferred): alarms fired by Data
Connection Layer values (external device data has no spawning execution — this
includes the device echo of a script write to a *data-sourced* attribute, so
only static writes cascade), and ScriptActor value-change/conditional/
expression/timer trigger runs (a timer tick has no spawner; a WhileTrue/interval
run has no single identifiable write).
Tests: new SiteRuntime.Tests/Actors/AlarmCascadeParentExecutionTests pins all
three hops — SetAttribute stamps the run's ExecutionId, InstanceActor publishes
it on the change (and publishes null when absent), and ValueMatch/HiLo/
Expression alarms parent the on-trigger run to the writer while a DCL-originated
change leaves it a root.
Docs: CLAUDE.md and Component-AuditLog.md corrected from "complete" to the true
behaviour; Component-SiteRuntime.md gains an "Audit correlation of an on-trigger
run" section with the hop table and the by-design root cases.
Tasks 5 and 6 of the Phase 2 plan, committed together because their test
fallout is entangled — several fixtures construct both stores.
StoreAndForwardStorage and SiteStorageService now take ILocalDb. Connections
come from ILocalDb.CreateConnection(), which hands out an already-open,
pragma-configured connection carrying the zb_hlc_next() UDF the capture triggers
call; a raw connection would lack the UDF and every write to a replicated table
would fail closed. Deleted with the connection strings: S&F's
EnsureDatabaseDirectoryExists and its per-open busy_timeout pragma, and the site
service's BusyTimeoutFloorSeconds normalization — LocalDb owns all of it now.
DI: AddSiteRuntime's string overload is gone (nothing left to supply), so the
Host calls the no-arg form. ScadaBridge:Database:SiteDbPath and
StoreAndForwardOptions.SqliteDbPath survive only as the migrator's source
locations in Tasks 8/9.
Two things the plan did not anticipate, both worth reading:
1. FOUND A REAL LATENT DEFECT, from Phase 1, now fixed. The plan assumed
directory creation simply moved to LocalDb along with file ownership. It did
not: the LocalDb library never creates the parent directory, and
SqliteLocalDb opens the file eagerly in its constructor — so a missing
directory is a hard boot failure ("SQLite Error 14: unable to open database
file"), not a degraded start. The default site config points at the RELATIVE
path ./data/site-localdb.db, so any site node without a pre-existing data/
directory fails to boot. The docker rig escapes only because its volume mount
happens to create /app/data — a coincidence that would have hidden this until
a bare-metal or fresh deployment. This has been latent since Phase 1 made
LocalDb:Path required; deleting S&F's EnsureDatabaseDirectoryExists here
would have widened it. Re-established the guarantee at the layer that now
owns the path (SiteLocalDbDirectory.Ensure, called before AddZbLocalDb) and
pinned it with SiteLocalDbDirectoryTests. Non-vacuity is not assumed: two
tests written against the wrong assumption failed with exactly this
SQLite Error 14 before the fix existed.
2. Test fallout was ~7x the plan's estimate. The plan named "fixtures" in one
project; the constructor change actually reaches 40 files across 7 test
projects, and most used Mode=Memory;Cache=Shared — which LocalDb has no
equivalent for, so every one had to move to a real temp file. Rather than
copy the Phase 1 TestLocalDb fixture into 7 projects, added a shared
tests/ZB.MOM.WW.ScadaBridge.TestSupport library (not a test project) so the
WAL-sidecar cleanup and the "real, not stubbed" rationale live in one place.
Retargeted rather than deleted, in both directions: the S&F WAL test now asserts
against the LocalDb-backed store (WAL genuinely is LocalDb's job), while the
directory-creation test moved to Host.Tests (that guarantee is NOT LocalDb's).
SiteStorageServiceTests.Initialize_EnablesWalJournalMode got the same treatment.
DeploymentManagerMediumFindingsTests induced a persistence failure via an
unopenable path, which no longer reaches the assertion since the fixture now
throws first; it induces the same failure shape via an uninitialized store.
Verified: full solution build 0 warnings; SiteRuntime 532, Host 318,
AuditLog 355, ExternalSystemGateway 142, HealthMonitoring 97,
StoreAndForward 153 — 1597 passed, 0 failed.
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
Gitea #18: the process-wide ScriptExecutionScheduler was reached via a static
singleton (Shared) directly inside ScriptActor, ScriptExecutionActor, AlarmActor,
AlarmExecutionActor, and ScriptSchedulerStatsReporter, so it could not be
substituted — a shared mutable global for anything running more than one logical
site in a process, most visibly the test assembly. Add an optional scheduler
injection seam to all five: the Host injects nothing and gets the shared pool
(byte-for-byte unchanged), while a test (or a future multi-site host) hands an
actor its own instance. Spawning actors thread their scheduler down to the
execution children they create. Shared() now also recreates a disposed cached
instance rather than returning it (new IsDisposed guard), so a disposed scheduler
can never silently poison every later script/alarm execution.
Gitea #16: ScriptActorTests now runs on its OWN scheduler (disposed at class
teardown), so the faulted-task test can no longer strand a worker on the
process-wide pool and starve unrelated test classes (one prior run failed 22
tests). EvalGate's wait is bounded to 30 s (was unbounded — the actual leak
mechanism) and reset per test; the teardown releases one permit per worker
instead of Release(Entries), which under-released in exactly the failure case.
Full SiteRuntime suite: 6/6 runs green (524/524); was 3/6 failing on clean main.
Fixes: #18Fixes: #16
SiteScriptCompileCache.Hits is a process-global counter; other test classes in
the assembly compile scripts concurrently (not in this serialized collection),
so an exact post-Clear count is non-deterministic under a full-assembly parallel
run. Assert.Same(r1,r2) remains the definitive proof of one shared Roslyn
compile. Integration-surfaced; belongs with plan R2-03 T5/T6.
Replace value?.ToString() with AttributeValueCodec.Encode(value) in
AttributeAccessor indexer set and SetAsync, so a List<string>{"a","b"}
encodes to ["a","b"] instead of the garbage ToString representation.
Add using ZB.MOM.WW.ScadaBridge.Commons.Types. Tests verify the codec
contract (list→JSON array, scalar passthrough, null); full round-trip
through the accessor is not viable without a live Akka ActorSystem —
noted in-test with explanation.
Replace bare task-discard with ContinueWith(OnlyOnFaulted|ExecuteSynchronously) so a
faulted ISiteEventLogger is logged and swallowed rather than going to the unobserved-task
firehose. Replace the "ScriptRuntimeContext" class-name fallback with the meaningful
"InstanceScript:{instanceName}" identifier (matching the site-event-log source convention).
Update the method doc-comment to state the best-effort contract explicitly. Pin the new
fallback value in the shape-precision test.
Inject ISiteEventLogger into ScriptRuntimeContext (additive optional ctor
param, defaulted null, all existing callers source-compatible). Add a single
private EmitRecursionLimitEventAsync helper that fires-and-forgets a
"script"/Error site event; called at both recursion guard sites (CallScript
at ~:332 and ScriptCallHelper.CallShared at ~:499). ScriptExecutionActor
threads the already-resolved siteEventLogger singleton into the context;
AlarmExecutionActor leaves it null (no siteEventLogger wired there).
Existing _logger.LogError + throw behaviour unchanged.
Tests: RecursionLimitSiteEventTests — 5 tests covering both CallScript and
CallShared (ISiteEventLogger.LogEventAsync called once with category "script",
severity "Error"; null logger path does not throw).
CachedWrite buffered ALL write failures and retried forever, never returning a
synchronous failure to the script — permanent SQL errors (constraint/syntax/
permission) were treated as transient. Mirror the External-System API path:
attempt immediately, return Failed synchronously on permanent SQL errors (no
buffering), buffer only transient errors; the S&F retry path parks permanent
failures instead of retrying forever. New SqlErrorClassifier + PermanentDatabaseException.
Joseph Doherty