dohertj2/ScadaBridge
1
0
Fork 0
Code Issues 2 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

perf(sitelog): sampled per-run events; interval run summaries; site_events replication policy pinned

Browse Source 
Implements WP3.2 stage (b) per docs/plans/2026-08-15-site-events-policy-design.md.

- Per-run instance-script Started/Completed Info site events are now off by
  default (SiteRuntimeOptions.PerRunScriptEvents=false) instead of firing on
  every run, closing the dominant site_events writer. Gated at the ScriptRunLauncher
  call sites (moved there from ScriptExecutionActor by WP3.1). Error-level events
  (timeout/failure/stuck-watchdog/recursion-limit) remain unconditional.
- ScriptRunSummaryRecorder accumulates per-(instance, script) run counters and a
  new site-only ScriptRunSummaryFlushService emits one aggregate "script" Info
  site event per ScriptRunSummaryIntervalSeconds (default 300s), top-50-script
  breakdown with an "others" rollup, zero-activity intervals emit nothing.
- Per-script opt-in via PerRunScriptEventScripts ("Instance/Script" exact or
  "Instance/*" wildcard), matched by the new pure ScriptRunEventPolicy. All three
  options are read from IOptionsMonitor<SiteRuntimeOptions> per run, so the
  policy is hot-togglable without a restart.
- Fixed the stale "event log is not replicated" comment at AkkaHostedService.cs
  (~905): site_events IS registered in SiteLocalDbSetup.ReplicatedTables — the
  singleton is what makes queries always hit the actively-written copy;
  replication is what gives the singleton history to read after a failover
  (memo Decision (b)). site_events replication itself is unchanged (still
  registered) and already pinned by
  tests/ZB.MOM.WW.ScadaBridge.Host.Tests/SiteLocalDbCdcRegistrationTests.cs.
- Updated Component-SiteEventLogging.md (Volume Policy section, corrected
  Storage/replication rationale) and Component-SiteRuntime.md (Script Run
  Launch + Error Handling sections).
This commit is contained in:
Joseph Doherty
2026-08-14 22:56:08 -04:00
parent 799fd041ec
commit c254d0740e
15 changed files with 922 additions and 27 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/requirements/Component-SiteEventLogging.md
+54 -4
View File
@@ -19,7 +19,8 @@ Site clusters (event recording and storage). Central cluster (remote query acces
| Category | Events |
|----------|--------|
| Script Executions | Script started, completed, failed (with error details), recursion limit exceeded |
| Script Executions | Script started, completed *(sampled/opt-in — see Volume Policy below)*; failed, timed out, recursion limit exceeded (with error details; always logged) |
| Script Run Summary | Interval aggregate replacing the per-run rows by default — one Info row per flush interval summarizing every instance script run since the last flush (see Volume Policy) |
| Alarm Events | Alarm activated, alarm cleared (which alarm, which instance), alarm evaluation error |
| Deployment Events | Configuration received from central, scripts compiled, applied successfully, apply failed |
| Data Connection Status | Connected, disconnected, reconnected (per connection) |
@@ -27,6 +28,41 @@ Site clusters (event recording and storage). Central cluster (remote query acces
| Instance Lifecycle | Instance enabled, disabled, deleted |
| Notification | Site→central forward failure, long-buffered notification (still in the site buffer past a threshold) |
## Volume Policy — Script Run Events (WP3.2, 2026-08-15)
Per-run instance-script Started/Completed Info events were, before this policy, the dominant
`site_events` writer: two rows per run, translating to roughly a dozen physical row-writes
across a replicated site pair once CDC oplog/row-version rows and their eventual purge
tombstones are counted — for content nobody looks at unless something went wrong. The policy
below (design memo: `docs/plans/2026-08-15-site-events-policy-design.md`) keeps the signal
while removing the volume.
- **Off by default.** `ScadaBridge:SiteRuntime:PerRunScriptEvents` (default `false`) is the
global switch back to the legacy per-run Started/Completed rows for every script.
- **Interval aggregate instead.** `ScriptRunSummaryRecorder` accumulates per-`(instance,
script)` counters (started, completed, failed, timed out, total/max duration) from the same
call sites that used to emit the per-run rows. A hosted flush service emits **one** "script"
Info row per interval — `ScadaBridge:SiteRuntime:ScriptRunSummaryIntervalSeconds` (default
`300`, i.e. 5 minutes; `0` disables the flush) — with headline totals in the message (e.g.
"1,243 runs: 1,240 completed, 2 failed, 1 timed out across 17 scripts") and a per-script
breakdown in `Details`, capped at the **top 50 scripts by run activity** with an "others"
rollup entry so a pathological deployment cannot mint an oversized row. **An interval with
zero activity emits no row** — the standby, which never runs scripts, therefore never
produces a summary row naturally. Source is `"ScriptRunSummary"`; Instance ID is `NULL`
(the row spans every instance).
- **Per-script opt-in for debugging.** `ScadaBridge:SiteRuntime:PerRunScriptEventScripts`
(default empty) restores the legacy per-run rows for named scripts only, without paying the
volume cost for the rest of the deployment. Entries are `"{InstanceName}/{ScriptName}"` for
an exact match or `"{InstanceName}/*"` for every script on that instance. Effective only
while the global switch is off.
- **Hot-togglable.** Both keys are read from the live options snapshot at the top of every
run (`IOptionsMonitor<SiteRuntimeOptions>.CurrentValue`), so an operator can flip either one
— including narrowing an opt-in to a single misbehaving script — without a restart or a
redeploy.
- **Error-level events are unconditional.** Timeout, failure, stuck-watchdog, and
recursion-limit Error rows are unaffected by any of the above — they always fire, regardless
of `PerRunScriptEvents` or the opt-in list.
## Event Entry Schema
Each event entry contains:
@@ -40,9 +76,23 @@ Each event entry contains:
## Storage
- Events are stored in **local SQLite** on each site node.
- Each node maintains its own event log. Only the **active node** generates and stores events. Event logs are **not replicated** to the standby node. On failover, the new active node starts logging to its own SQLite database; historical events from the previous active node are no longer queryable via central until that node comes back online. This is acceptable because event logs are diagnostic, not transactional.
- **Retention**: 30 days. A **daily background job** runs on the active node and deletes all events older than 30 days. Hard delete — no archival.
- Events are stored in the **consolidated site LocalDb** SQLite file (`site_events` table;
see CLAUDE.md → Consolidated site database). Only the **active node** generates and stores
events (a standby runs no scripts, subsystems, or deployments to log).
- **`site_events` IS replicated** (`SiteLocalDbSetup.ReplicatedTables`) on a site pair with
replication configured — this reverses the component's pre-LocalDb-Phase-1 behavior, and the
reason is **failover history continuity, not either-node queries**: the query handler
(`EventLogHandlerActor`) is a **cluster singleton**, so a central query always routes to the
actively-written copy regardless of which node answers the gRPC call — replication is never
needed to make queries *work*. What replication buys is **content to read after a failover**:
when the active node dies, the singleton restarts on the survivor, and its query service reads
the survivor's *local* copy — which has history *only because it was replicated*. Without
replication, a failover would start the event log empty at precisely the moment an operator
is investigating an incident (WP3.2 design memo, Decision (b): `docs/plans/2026-08-15-site-events-policy-design.md`).
On a site pair running **without** replication configured (by deliberate choice, e.g. the
rig's site-b/site-c), the log stays node-local and a failover does start it fresh — the
documented trade of not configuring a peer, unchanged by this policy.
- **Retention**: 30 days. A **daily background job** runs on the active node and deletes all events older than 30 days. Hard delete — no archival. (Today, retention/cap deletes on a replicated node are captured by CDC like any other write — see the Volume Policy section above for why this is a small residual cost now that per-run rows are off by default.)
- **Storage cap**: A configurable maximum database size (default: 1 GB) is enforced. If the storage cap is reached before the 30-day retention window, the oldest events are purged first. This prevents disk exhaustion from alarm storms, script failure loops, or connection flapping.
## Central Access