dohertj2/ScadaBridge
1
0
Fork 0
Code Issues 2 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

perf(sitelog): sampled per-run events; interval run summaries; site_events replication policy pinned

Browse Source 
Implements WP3.2 stage (b) per docs/plans/2026-08-15-site-events-policy-design.md.

- Per-run instance-script Started/Completed Info site events are now off by
  default (SiteRuntimeOptions.PerRunScriptEvents=false) instead of firing on
  every run, closing the dominant site_events writer. Gated at the ScriptRunLauncher
  call sites (moved there from ScriptExecutionActor by WP3.1). Error-level events
  (timeout/failure/stuck-watchdog/recursion-limit) remain unconditional.
- ScriptRunSummaryRecorder accumulates per-(instance, script) run counters and a
  new site-only ScriptRunSummaryFlushService emits one aggregate "script" Info
  site event per ScriptRunSummaryIntervalSeconds (default 300s), top-50-script
  breakdown with an "others" rollup, zero-activity intervals emit nothing.
- Per-script opt-in via PerRunScriptEventScripts ("Instance/Script" exact or
  "Instance/*" wildcard), matched by the new pure ScriptRunEventPolicy. All three
  options are read from IOptionsMonitor<SiteRuntimeOptions> per run, so the
  policy is hot-togglable without a restart.
- Fixed the stale "event log is not replicated" comment at AkkaHostedService.cs
  (~905): site_events IS registered in SiteLocalDbSetup.ReplicatedTables — the
  singleton is what makes queries always hit the actively-written copy;
  replication is what gives the singleton history to read after a failover
  (memo Decision (b)). site_events replication itself is unchanged (still
  registered) and already pinned by
  tests/ZB.MOM.WW.ScadaBridge.Host.Tests/SiteLocalDbCdcRegistrationTests.cs.
- Updated Component-SiteEventLogging.md (Volume Policy section, corrected
  Storage/replication rationale) and Component-SiteRuntime.md (Script Run
  Launch + Error Handling sections).
This commit is contained in:
Joseph Doherty
2026-08-14 22:56:08 -04:00
parent 799fd041ec
commit c254d0740e
15 changed files with 922 additions and 27 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/requirements/Component-SiteEventLogging.md
+54 -4
View File
@@ -19,7 +19,8 @@ Site clusters (event recording and storage). Central cluster (remote query acces
| Category | Events |
|----------|--------|
| Script Executions | Script started, completed, failed (with error details), recursion limit exceeded |
| Script Executions | Script started, completed *(sampled/opt-in — see Volume Policy below)*; failed, timed out, recursion limit exceeded (with error details; always logged) |
| Script Run Summary | Interval aggregate replacing the per-run rows by default — one Info row per flush interval summarizing every instance script run since the last flush (see Volume Policy) |
| Alarm Events | Alarm activated, alarm cleared (which alarm, which instance), alarm evaluation error |
| Deployment Events | Configuration received from central, scripts compiled, applied successfully, apply failed |
| Data Connection Status | Connected, disconnected, reconnected (per connection) |
@@ -27,6 +28,41 @@ Site clusters (event recording and storage). Central cluster (remote query acces
| Instance Lifecycle | Instance enabled, disabled, deleted |
| Notification | Site→central forward failure, long-buffered notification (still in the site buffer past a threshold) |
## Volume Policy — Script Run Events (WP3.2, 2026-08-15)
Per-run instance-script Started/Completed Info events were, before this policy, the dominant
`site_events` writer: two rows per run, translating to roughly a dozen physical row-writes
across a replicated site pair once CDC oplog/row-version rows and their eventual purge
tombstones are counted — for content nobody looks at unless something went wrong. The policy
below (design memo: `docs/plans/2026-08-15-site-events-policy-design.md`) keeps the signal
while removing the volume.
- **Off by default.** `ScadaBridge:SiteRuntime:PerRunScriptEvents` (default `false`) is the
global switch back to the legacy per-run Started/Completed rows for every script.
- **Interval aggregate instead.** `ScriptRunSummaryRecorder` accumulates per-`(instance,
script)` counters (started, completed, failed, timed out, total/max duration) from the same
call sites that used to emit the per-run rows. A hosted flush service emits **one** "script"
Info row per interval — `ScadaBridge:SiteRuntime:ScriptRunSummaryIntervalSeconds` (default
`300`, i.e. 5 minutes; `0` disables the flush) — with headline totals in the message (e.g.
"1,243 runs: 1,240 completed, 2 failed, 1 timed out across 17 scripts") and a per-script
breakdown in `Details`, capped at the **top 50 scripts by run activity** with an "others"
rollup entry so a pathological deployment cannot mint an oversized row. **An interval with
zero activity emits no row** — the standby, which never runs scripts, therefore never
produces a summary row naturally. Source is `"ScriptRunSummary"`; Instance ID is `NULL`
(the row spans every instance).
- **Per-script opt-in for debugging.** `ScadaBridge:SiteRuntime:PerRunScriptEventScripts`
(default empty) restores the legacy per-run rows for named scripts only, without paying the
volume cost for the rest of the deployment. Entries are `"{InstanceName}/{ScriptName}"` for
an exact match or `"{InstanceName}/*"` for every script on that instance. Effective only
while the global switch is off.
- **Hot-togglable.** Both keys are read from the live options snapshot at the top of every
run (`IOptionsMonitor<SiteRuntimeOptions>.CurrentValue`), so an operator can flip either one
— including narrowing an opt-in to a single misbehaving script — without a restart or a
redeploy.
- **Error-level events are unconditional.** Timeout, failure, stuck-watchdog, and
recursion-limit Error rows are unaffected by any of the above — they always fire, regardless
of `PerRunScriptEvents` or the opt-in list.
## Event Entry Schema
Each event entry contains:
@@ -40,9 +76,23 @@ Each event entry contains:
## Storage
- Events are stored in **local SQLite** on each site node.
- Each node maintains its own event log. Only the **active node** generates and stores events. Event logs are **not replicated** to the standby node. On failover, the new active node starts logging to its own SQLite database; historical events from the previous active node are no longer queryable via central until that node comes back online. This is acceptable because event logs are diagnostic, not transactional.
- **Retention**: 30 days. A **daily background job** runs on the active node and deletes all events older than 30 days. Hard delete — no archival.
- Events are stored in the **consolidated site LocalDb** SQLite file (`site_events` table;
see CLAUDE.md → Consolidated site database). Only the **active node** generates and stores
events (a standby runs no scripts, subsystems, or deployments to log).
- **`site_events` IS replicated** (`SiteLocalDbSetup.ReplicatedTables`) on a site pair with
replication configured — this reverses the component's pre-LocalDb-Phase-1 behavior, and the
reason is **failover history continuity, not either-node queries**: the query handler
(`EventLogHandlerActor`) is a **cluster singleton**, so a central query always routes to the
actively-written copy regardless of which node answers the gRPC call — replication is never
needed to make queries *work*. What replication buys is **content to read after a failover**:
when the active node dies, the singleton restarts on the survivor, and its query service reads
the survivor's *local* copy — which has history *only because it was replicated*. Without
replication, a failover would start the event log empty at precisely the moment an operator
is investigating an incident (WP3.2 design memo, Decision (b): `docs/plans/2026-08-15-site-events-policy-design.md`).
On a site pair running **without** replication configured (by deliberate choice, e.g. the
rig's site-b/site-c), the log stays node-local and a failover does start it fresh — the
documented trade of not configuring a peer, unchanged by this policy.
- **Retention**: 30 days. A **daily background job** runs on the active node and deletes all events older than 30 days. Hard delete — no archival. (Today, retention/cap deletes on a replicated node are captured by CDC like any other write — see the Volume Policy section above for why this is a small residual cost now that per-run rows are off by default.)
- **Storage cap**: A configurable maximum database size (default: 1 GB) is enforced. If the storage cap is reached before the 30-day retention window, the oldest events are purged first. This prevents disk exhaustion from alarm storms, script failure loops, or connection flapping.
## Central Access
docs/requirements/Component-SiteRuntime.md
+2 -1
View File
@@ -221,6 +221,7 @@ When the Instance Actor is stopped (due to disable, delete, or redeployment), Ak
- A run is launched **directly by the Script Actor** through `ScriptRunLauncher`. There is no per-run child actor.
- The former `ScriptExecutionActor` / `AlarmExecutionActor` were already inert shells: neither declared a `Receive` handler (they executed from their constructor), neither had a `PostStop`, state, or stash, and neither's `IActorRef` was ever a message target — the whole lifecycle lived inside a detached task the actor never observed. What they cost was an actor cell, mailbox, and name registration **per run**, plus a per-spawn expression-tree `Props.Create`. Removing them changed no semantics; the run body moved verbatim into the shared launcher.
- Everything the shells provided is preserved: exception and timeout containment, one DI scope per run disposed on every path, the site-event/health telemetry, the Ask reply, the completion notification (now the coordinator's own `Self`), and the audit `ExecutionId` / `ParentExecutionId` threading. A run still in flight when its Script Actor is stopped runs to completion and its completion message dead-letters, exactly as before — **stopping does NOT cancel in-flight runs**; redeploy/undeploy semantics are unchanged.
- **Per-run Started/Completed site events are sampled, not unconditional (WP3.2).** The two Info rows an instance script run used to emit unconditionally were the dominant `site_events` writer under load (design memo: `docs/plans/2026-08-15-site-events-policy-design.md`). They are now **off by default** (`ScadaBridge:SiteRuntime:PerRunScriptEvents`, default `false`); a `ScriptRunSummaryRecorder` accumulates per-`(instance, script)` run counters from the same call sites and a site-only hosted flush service emits **one aggregate "script" Info row per interval** (`ScriptRunSummaryIntervalSeconds`, default 300s) with a top-50-scripts breakdown, capped so a pathological deployment cannot mint an oversized row — an idle interval emits nothing. `ScadaBridge:SiteRuntime:PerRunScriptEventScripts` restores the legacy per-run rows for named scripts (`"Instance/Script"` exact or `"Instance/*"` wildcard) for live debugging. All three keys are read from `IOptionsMonitor<SiteRuntimeOptions>` per run, so they are hot-togglable without a restart. **Error-level rows (timeout, failure, stuck-watchdog, recursion-limit) are unaffected — always emitted, regardless of this policy.** See Component-SiteEventLogging.md → Volume Policy for the full design.
- One deliberate improvement: a failure of the *launch itself* (e.g. queueing onto a disposed scheduler) is caught by the Script Actor, which replies to the Ask caller and releases the run slot. The old per-run child's constructor throw was handled by a Stop supervision directive that sent no reply, leaving the caller to hang to its Ask timeout.
- The script body runs on the **dedicated `ScriptExecutionScheduler`** (a bounded set of dedicated threads), not the shared .NET thread pool, so blocking script I/O cannot starve the global pool or stall Akka dispatchers. The scheduler is **process-wide by default** (one pool per host), but each script/alarm actor takes it through an **optional injection seam** rather than reaching for the static directly: the Host injects nothing and gets the shared pool, while tests (or a future multi-site host) can hand an actor its own instance. The shared accessor also recreates a disposed pool rather than returning it, so a disposed scheduler can never silently poison later executions.
- **Pool sizing is instance-scaled and grow-only (WP3.1).** The pool was a fixed 8 threads regardless of load. It is now `clamp(max(ScriptExecutionThreadCount, ceil(enabledInstances / 8)), 1, ScriptExecutionMaxThreadCount)` — the existing `ScriptExecutionThreadCount` (default 8) becomes the **floor**, so configurations at or below 64 instances behave exactly as before, and the new `ScriptExecutionMaxThreadCount` (default 32) is the ceiling. Beyond the ceiling the per-script cap below is the real regulator. `DeploymentManagerActor.UpdateInstanceCounts` calls `EnsureCapacity` on every deploy / undeploy / enable / disable and once per staggered startup batch. Growth only: undeploying leaves idle threads, which cost nothing measurable and avoid drain/steal complexity.
@@ -531,7 +532,7 @@ Per Akka.NET best practices, internal actor communication uses **Tell** (fire-an
## Error Handling
### Script Errors
- Unhandled exceptions and timeouts in Script Execution Actors are **logged locally** to the site event log.
- Unhandled exceptions and timeouts in a script run are **logged locally** to the site event log — unconditionally, regardless of the WP3.2 per-run Started/Completed sampling policy (see Script Run Launch above).
- The Script Actor (coordinator) is **not affected** — it remains active for future trigger events.
- Script failures are **not reported to central** (except as aggregated error rate metrics via Health Monitoring).