perf(sitelog): sampled per-run events; interval run summaries; site_events replication policy pinned

Implements WP3.2 stage (b) per docs/plans/2026-08-15-site-events-policy-design.md.

- Per-run instance-script Started/Completed Info site events are now off by
  default (SiteRuntimeOptions.PerRunScriptEvents=false) instead of firing on
  every run, closing the dominant site_events writer. Gated at the ScriptRunLauncher
  call sites (moved there from ScriptExecutionActor by WP3.1). Error-level events
  (timeout/failure/stuck-watchdog/recursion-limit) remain unconditional.
- ScriptRunSummaryRecorder accumulates per-(instance, script) run counters and a
  new site-only ScriptRunSummaryFlushService emits one aggregate "script" Info
  site event per ScriptRunSummaryIntervalSeconds (default 300s), top-50-script
  breakdown with an "others" rollup, zero-activity intervals emit nothing.
- Per-script opt-in via PerRunScriptEventScripts ("Instance/Script" exact or
  "Instance/*" wildcard), matched by the new pure ScriptRunEventPolicy. All three
  options are read from IOptionsMonitor<SiteRuntimeOptions> per run, so the
  policy is hot-togglable without a restart.
- Fixed the stale "event log is not replicated" comment at AkkaHostedService.cs
  (~905): site_events IS registered in SiteLocalDbSetup.ReplicatedTables — the
  singleton is what makes queries always hit the actively-written copy;
  replication is what gives the singleton history to read after a failover
  (memo Decision (b)). site_events replication itself is unchanged (still
  registered) and already pinned by
  tests/ZB.MOM.WW.ScadaBridge.Host.Tests/SiteLocalDbCdcRegistrationTests.cs.
- Updated Component-SiteEventLogging.md (Volume Policy section, corrected
  Storage/replication rationale) and Component-SiteRuntime.md (Script Run
  Launch + Error Handling sections).
This commit is contained in:
Joseph Doherty
2026-08-14 22:56:08 -04:00
parent 799fd041ec
commit c254d0740e
15 changed files with 922 additions and 27 deletions
@@ -141,4 +141,44 @@ public class SiteRuntimeOptions
/// <see cref="StreamBufferSize"/>).
/// </summary>
public int AlarmPublishQueueCapacity { get; set; } = 2000;
/// <summary>
/// WP3.2 (site_events volume policy — docs/plans/2026-08-15-site-events-policy-design.md):
/// global switch restoring the legacy per-run Started/Completed Info site events for
/// EVERY instance script run. Off by default: those two rows per run were the dominant
/// <c>site_events</c> writer (~12 physical row-writes across a replicated pair per run,
/// counting CDC oplog + row-version rows and their eventual purge tombstones), for
/// content nobody looks at unless something went wrong. With this off, an interval
/// aggregate (<see cref="Scripts.ScriptRunSummaryRecorder"/>) replaces the per-run rows
/// and script Error events (timeout/failure/stuck-watchdog) are unaffected — they are
/// unconditional regardless of this switch. Read via
/// <see cref="Microsoft.Extensions.Options.IOptionsMonitor{TOptions}"/> at the top of
/// every run, so it is hot-togglable (no restart, no redeploy). Default: false.
/// </summary>
public bool PerRunScriptEvents { get; set; }
/// <summary>
/// WP3.2: per-script opt-in restoring the legacy per-run Started/Completed Info site
/// events for NAMED scripts only, for live debugging without flipping the global
/// <see cref="PerRunScriptEvents"/> switch (and therefore without paying its volume
/// cost for every other script). Effective only while the global switch is off.
/// Entries are <c>"{InstanceName}/{ScriptName}"</c> for an exact match, or
/// <c>"{InstanceName}/*"</c> to opt in every script on that instance. Matched by
/// <see cref="Scripts.ScriptRunEventPolicy.ShouldEmitPerRun"/>, read from
/// <see cref="Microsoft.Extensions.Options.IOptionsMonitor{TOptions}"/> per run — hot
/// -togglable like <see cref="PerRunScriptEvents"/>. Default: empty (no opt-ins).
/// </summary>
public List<string> PerRunScriptEventScripts { get; set; } = new();
/// <summary>
/// WP3.2: flush interval (seconds) for <see cref="Scripts.ScriptRunSummaryRecorder"/>'s
/// interval aggregate — one "script" Info site event per interval summarizing every
/// instance script run since the last flush (headline totals + a top-50-scripts
/// breakdown, capped so a pathological deployment cannot mint an oversized row). An
/// interval with zero activity emits no row. <c>0</c> disables the summary flush
/// entirely (the recorder still accumulates counters; they are just never emitted or
/// reset). Default: 300 (5 minutes) — at most 288 summary rows/day, versus 2 rows per
/// script run under the legacy per-run behaviour.
/// </summary>
public int ScriptRunSummaryIntervalSeconds { get; set; } = 300;
}