The compile cache was a `static` class, so its entry table and its Hits/Count
counters were one process-global blob every test class in the SiteRuntime
assembly wrote into. Row 91 last wave neutered the resulting flakes (per-test
GUID keys + one-sided counter bounds), but the shared mutable state itself
remained: any test calling Clear() evicted entries other parallel classes had
just inserted, and a Clear() from a class outside the serialized collection
would degrade the others to recompiles and re-open the row-91 flake class.
PREMISE CORRECTION: the register speculated production clears "per-instance or
per-deployment script entries" and offered ClearKeys(prefix) as the natural fit.
There are NO production callers of Clear() at all — grep of src/ finds only
GetOrAdd (ScriptCompilationService.CompileCore). Clear() existed solely so
tests could reset the global. A key-scoped API would therefore have been new
production surface built for nobody, so the instance-scoped seam is the fix and
Clear() is deleted outright rather than migrated.
- SiteScriptCompileCache: `internal static class` -> `internal sealed class`;
Cache/EvictionLock/_hits/_accessSequence become instance fields. LRU batch
eviction, the double-checked sweep and the ConcurrentDictionary.ToArray()
snapshot are byte-for-byte the same policy. Clear() removed.
- ScriptCompilationService owns one cache. Public ctor (the DI path) builds its
own; a new internal ctor takes one for tests. Production N4 sharing is
unchanged and now rests on the DI SINGLETON registration — the one service
instance is threaded to DeployCompileValidator and to every Instance Actor —
so that lifetime is load-bearing and is pinned by a new registration test.
- Tests: all three Clear() callers migrated off it. Each test now owns its cache,
which lets the assertions get STRONGER — the row-91 workarounds are retired:
Hits/Count are exact again (no >= baseline bounds), the per-test GUID key
token is gone, and ForeignEntrySlack (512 entries of allowance on the bound)
drops to the racing-thread count in the concurrency test. New test
SeparateCaches_DoNotShareEntriesOrCounters pins the scoping directly: filling
one cache past MaxEntries leaves another's entry, hit count and identity
intact. DeploymentWarmThenGateTests additionally asserts Count == 1 now that
its counter is private.
- The [Collection("SiteScriptCompileCache")] serialization from row 91 is
RELAXED (removed from all three classes) — they no longer touch any shared
state, so there is nothing left to serialize.
- Docs: Component-SiteRuntime.md gains the ownership/singleton note and drops
the Clear() reference; DeploymentManagerActor / DeployCompileValidator /
ServiceCollectionExtensions comments corrected to say where the cache lives.
Build: 0 warnings. SiteRuntime.Tests 643/643, three consecutive full-parallel
runs. AuditLog.Tests correlation classes (other ScriptCompilationService
consumers) green.
Implements WP3.2 stage (b) per docs/plans/2026-08-15-site-events-policy-design.md.
- Per-run instance-script Started/Completed Info site events are now off by
default (SiteRuntimeOptions.PerRunScriptEvents=false) instead of firing on
every run, closing the dominant site_events writer. Gated at the ScriptRunLauncher
call sites (moved there from ScriptExecutionActor by WP3.1). Error-level events
(timeout/failure/stuck-watchdog/recursion-limit) remain unconditional.
- ScriptRunSummaryRecorder accumulates per-(instance, script) run counters and a
new site-only ScriptRunSummaryFlushService emits one aggregate "script" Info
site event per ScriptRunSummaryIntervalSeconds (default 300s), top-50-script
breakdown with an "others" rollup, zero-activity intervals emit nothing.
- Per-script opt-in via PerRunScriptEventScripts ("Instance/Script" exact or
"Instance/*" wildcard), matched by the new pure ScriptRunEventPolicy. All three
options are read from IOptionsMonitor<SiteRuntimeOptions> per run, so the
policy is hot-togglable without a restart.
- Fixed the stale "event log is not replicated" comment at AkkaHostedService.cs
(~905): site_events IS registered in SiteLocalDbSetup.ReplicatedTables — the
singleton is what makes queries always hit the actively-written copy;
replication is what gives the singleton history to read after a failover
(memo Decision (b)). site_events replication itself is unchanged (still
registered) and already pinned by
tests/ZB.MOM.WW.ScadaBridge.Host.Tests/SiteLocalDbCdcRegistrationTests.cs.
- Updated Component-SiteEventLogging.md (Volume Policy section, corrected
Storage/replication rationale) and Component-SiteRuntime.md (Script Run
Launch + Error Handling sections).
WP2.6 (arch-review remediation, cross-cutting misc):
- SiteExternalSystemRepository: name/ID-indexed ExternalSystemDefinitionCache replaces
the fetch-all + reverse-map scan on every by-ID/method lookup; loaded once per
redeploy, invalidated by DeploymentManagerActor after HandleDeployArtifacts applies
external-system changes. Static JsonSerializerOptions for method-list parsing.
- Inbound API: short-TTL ApiMethodCache fronts the per-request ApiMethod repository
fetch; invalidated by name via the existing ScriptArtifactChangeSubscriber/
IScriptArtifactChangeBus pipeline, self-healing via TTL for changes the bus
doesn't cover (e.g. Management API edits).
- StoreAndForward: the cached-call audit-observer queue — the one unbounded channel
left in the system — is now bounded (ObserverQueueCapacity, default 10,000) with
DropOldest overflow and a dropped-notification counter.
- SiteStreamManager: alarm state changes now travel a dedicated publish
source/broadcast hub, isolated from the (far higher-volume) attribute path, so an
attribute storm can no longer evict a pending alarm transition; the alarm hand-off
queue is bounded with a drop counter surfaced on the site health report
(SiteStreamAlarmDropCount via the new SiteStreamAlarmDropReporter), and publishing
is skipped entirely at zero subscribers on either path.
- CLI ManagementHttpClient: explicit 30s HttpClient.Timeout on the shared
construction (was the 100s framework default), overridable via
SCADABRIDGE_HTTP_TIMEOUT_SECONDS.
Deviation: the failback-probe heartbeat item is NOT included — its only viable
surface (CentralChannelProvider.cs / heartbeat consumers) lives entirely in the
Communication project, explicitly off-limits to this work package this phase.
Tests: SiteRuntime.Tests (550), InboundAPI.Tests (278), StoreAndForward.Tests (133),
CLI.Tests (390), HealthMonitoring.Tests (97) — all green after full solution build.
Tasks 5 and 6 of the Phase 2 plan, committed together because their test
fallout is entangled — several fixtures construct both stores.
StoreAndForwardStorage and SiteStorageService now take ILocalDb. Connections
come from ILocalDb.CreateConnection(), which hands out an already-open,
pragma-configured connection carrying the zb_hlc_next() UDF the capture triggers
call; a raw connection would lack the UDF and every write to a replicated table
would fail closed. Deleted with the connection strings: S&F's
EnsureDatabaseDirectoryExists and its per-open busy_timeout pragma, and the site
service's BusyTimeoutFloorSeconds normalization — LocalDb owns all of it now.
DI: AddSiteRuntime's string overload is gone (nothing left to supply), so the
Host calls the no-arg form. ScadaBridge:Database:SiteDbPath and
StoreAndForwardOptions.SqliteDbPath survive only as the migrator's source
locations in Tasks 8/9.
Two things the plan did not anticipate, both worth reading:
1. FOUND A REAL LATENT DEFECT, from Phase 1, now fixed. The plan assumed
directory creation simply moved to LocalDb along with file ownership. It did
not: the LocalDb library never creates the parent directory, and
SqliteLocalDb opens the file eagerly in its constructor — so a missing
directory is a hard boot failure ("SQLite Error 14: unable to open database
file"), not a degraded start. The default site config points at the RELATIVE
path ./data/site-localdb.db, so any site node without a pre-existing data/
directory fails to boot. The docker rig escapes only because its volume mount
happens to create /app/data — a coincidence that would have hidden this until
a bare-metal or fresh deployment. This has been latent since Phase 1 made
LocalDb:Path required; deleting S&F's EnsureDatabaseDirectoryExists here
would have widened it. Re-established the guarantee at the layer that now
owns the path (SiteLocalDbDirectory.Ensure, called before AddZbLocalDb) and
pinned it with SiteLocalDbDirectoryTests. Non-vacuity is not assumed: two
tests written against the wrong assumption failed with exactly this
SQLite Error 14 before the fix existed.
2. Test fallout was ~7x the plan's estimate. The plan named "fixtures" in one
project; the constructor change actually reaches 40 files across 7 test
projects, and most used Mode=Memory;Cache=Shared — which LocalDb has no
equivalent for, so every one had to move to a real temp file. Rather than
copy the Phase 1 TestLocalDb fixture into 7 projects, added a shared
tests/ZB.MOM.WW.ScadaBridge.TestSupport library (not a test project) so the
WAL-sidecar cleanup and the "real, not stubbed" rationale live in one place.
Retargeted rather than deleted, in both directions: the S&F WAL test now asserts
against the LocalDb-backed store (WAL genuinely is LocalDb's job), while the
directory-creation test moved to Host.Tests (that guarantee is NOT LocalDb's).
SiteStorageServiceTests.Initialize_EnablesWalJournalMode got the same treatment.
DeploymentManagerMediumFindingsTests induced a persistence failure via an
unopenable path, which no longer reaches the assertion since the fixture now
throws first; it induces the same failure shape via an uninitialized store.
Verified: full solution build 0 warnings; SiteRuntime 532, Host 318,
AuditLog 355, ExternalSystemGateway 142, HealthMonitoring 97,
StoreAndForward 153 — 1597 passed, 0 failed.
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
Task 4 of the LocalDb Phase 1 adoption plan.
OperationTrackingStore took IOptions<OperationTrackingOptions> and opened its own
SqliteConnection from ConnectionString. It now takes ILocalDb and gets every
connection - writer and the two ad-hoc reader paths - from CreateConnection().
This is not cosmetic. OperationTracking is a RegisterReplicated table, so its
capture triggers call zb_hlc_next(). That UDF is registered per connection by
ILocalDb and by nothing else, so a raw SqliteConnection would fail closed on
every write. Connections from CreateConnection() also arrive already open -
calling Open()/OpenAsync() on one throws - hence the removed OpenAsync calls on
the reader paths.
OperationTrackingOptions.ConnectionString is now vestigial for this store; the
database location is LocalDb:Path. The options class stays (retention settings)
and the config key stays bound for the site config DB.
InitializeSchema is kept but is now always a no-op in the host: onReady runs
while ILocalDb is being constructed, strictly before this constructor can
receive it. It remains so a directly-constructed store (tests, tooling) is
self-sufficient.
Tests: the store fixture moves off mode=memory&cache=shared onto a real ILocalDb
over a temp file. There is no in-memory mode - LocalDbOptions.Path is a
filesystem path - and testing through a raw in-memory connection would no longer
resemble the host. Verifier connections stay raw on purpose: this fixture never
registers the table, so no trigger fires and the UDF is never reached.
Three DI fixtures needed LocalDb:Path added, because resolving
IOperationTrackingStore now forces ILocalDb construction:
SiteCompositionRootTests, SiteAuditWiringTests, and the AuditLog
CombinedTelemetryHarness.
Verified: build 0 warnings; Host 294/294, SiteRuntime 530/530, AuditLog 354/354.
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
VERIFY-THEN-FIX: the plan-authoring discovery is CONFIRMED. No code anywhere in
src/ registered IOperationTrackingStore in DI, yet AkkaHostedService and
AuditLog SCE both comment that AddSiteRuntime provides it. Every consumer
resolved it via GetService (null-tolerant) and silently ran degraded:
cached-drain scheduler never armed, PullSiteCalls reconciliation seam never
wired, Tracking.Status degraded to audit-only. This is a FUNCTIONAL FIX riding
the hygiene plan: AddSiteRuntime now registers the store (site-only), so
site-local cached-call tracking runs in its intended mode. OperationTrackingOptions
is now bound + eagerly validated in the Host site-options block.
Remove project bookkeeping citations from shipped code comments across the
solution: hyphenated task IDs (WP-14, StoreAndForward-025), milestone/task/
issue refs (M3, Task 4, Audit Log #23, #21), Bundle X task-bundle labels,
and C/D/K/S/T phase labels.
Comment text only — no code logic, string/log literals, or XML-doc structure
changed. Genuine descriptions are preserved (only the citation is stripped),
and technical lookalikes are retained (UTF-8, SHA-256, T00:00:00, M365,
UTC-5, pre-C4/pre-C5 schema versions). Flagged by the new CommentChecker
TaskReferenceInComment / TrackingReferenceInComment checks plus targeted
grep passes; full solution builds clean, append-only guard tests pass.
Resolve all 622 issues flagged by the enhanced CommentChecker: add missing
<returns> tags (incl. the standard phrasing on non-generic Task methods),
add missing <summary> tags, and replace misused/redundant <inheritdoc/> on
members that override or implement nothing with real documentation.
Documentation-only — no behavior change; solution builds clean.
Joseph Doherty