perf(sitelog): sampled per-run events; interval run summaries; site_events replication policy pinned

Implements WP3.2 stage (b) per docs/plans/2026-08-15-site-events-policy-design.md. - Per-run instance-script Started/Completed Info site events are now off by default (SiteRuntimeOptions.PerRunScriptEvents=false) instead of firing on every run, closing the dominant site_events writer. Gated at the ScriptRunLauncher call sites (moved there from ScriptExecutionActor by WP3.1). Error-level events (timeout/failure/stuck-watchdog/recursion-limit) remain unconditional. - ScriptRunSummaryRecorder accumulates per-(instance, script) run counters and a new site-only ScriptRunSummaryFlushService emits one aggregate "script" Info site event per ScriptRunSummaryIntervalSeconds (default 300s), top-50-script breakdown with an "others" rollup, zero-activity intervals emit nothing. - Per-script opt-in via PerRunScriptEventScripts ("Instance/Script" exact or "Instance/*" wildcard), matched by the new pure ScriptRunEventPolicy. All three options are read from IOptionsMonitor<SiteRuntimeOptions> per run, so the policy is hot-togglable without a restart. - Fixed the stale "event log is not replicated" comment at AkkaHostedService.cs (~905): site_events IS registered in SiteLocalDbSetup.ReplicatedTables — the singleton is what makes queries always hit the actively-written copy; replication is what gives the singleton history to read after a failover (memo Decision (b)). site_events replication itself is unchanged (still registered) and already pinned by tests/ZB.MOM.WW.ScadaBridge.Host.Tests/SiteLocalDbCdcRegistrationTests.cs. - Updated Component-SiteEventLogging.md (Volume Policy section, corrected Storage/replication rationale) and Component-SiteRuntime.md (Script Run Launch + Error Handling sections).
2026-08-14 22:56:08 -04:00
parent 799fd041ec
commit c254d0740e
15 changed files with 922 additions and 27 deletions
@@ -0,0 +1,84 @@
+using Microsoft.Extensions.Hosting;
+using Microsoft.Extensions.Logging;
+using Microsoft.Extensions.Options;
+using ZB.MOM.WW.ScadaBridge.SiteEventLogging;
+
+namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Scripts;
+
+/// <summary>
+/// WP3.2 (site_events volume policy — docs/plans/2026-08-15-site-events-policy-design.md):
+/// site-only hosted service that periodically flushes <see cref="ScriptRunSummaryRecorder"/>
+/// into one aggregate "script" Info site event. Registered by
+/// <c>ServiceCollectionExtensions.AddSiteRuntime</c>, which only the site composition root
+/// calls (<c>SiteServiceRegistration.Configure</c>) — central never runs scripts, so central
+/// never registers this service, which is what makes it "Site nodes only" per the design memo.
+///
+/// <para>The interval is read from <see cref="IOptionsMonitor{TOptions}"/> on every tick (not
+/// captured once at construction) so <see cref="SiteRuntimeOptions.ScriptRunSummaryIntervalSeconds"/>
+/// is hot-reloadable — an operator can shorten or lengthen it, or disable it (<c>0</c>), live.
+/// Mirrors <c>ScriptSchedulerStatsReporter</c>'s shape: fixed cadence, exceptions logged and
+/// swallowed so the loop survives every flush failure.</para>
+/// </summary>
+public sealed class ScriptRunSummaryFlushService : BackgroundService
+{
+    /// <summary>Poll cadence used while summaries are disabled (<c>ScriptRunSummaryIntervalSeconds == 0</c>) — coarse enough to amortise re-checking whether the interval was re-enabled live.</summary>
+    private static readonly TimeSpan DisabledPollInterval = TimeSpan.FromSeconds(30);
+
+    private readonly ScriptRunSummaryRecorder _recorder;
+    private readonly ISiteEventLogger _siteEventLogger;
+    private readonly IOptionsMonitor<SiteRuntimeOptions> _optionsMonitor;
+    private readonly ILogger<ScriptRunSummaryFlushService> _logger;
+
+    /// <summary>Initializes a new instance of <see cref="ScriptRunSummaryFlushService"/>.</summary>
+    /// <param name="recorder">The recorder whose accumulated counters this service flushes.</param>
+    /// <param name="siteEventLogger">The site event logger the flushed summary row is written to.</param>
+    /// <param name="optionsMonitor">Supplies the hot-reloadable flush interval.</param>
+    /// <param name="logger">Logger instance.</param>
+    public ScriptRunSummaryFlushService(
+        ScriptRunSummaryRecorder recorder,
+        ISiteEventLogger siteEventLogger,
+        IOptionsMonitor<SiteRuntimeOptions> optionsMonitor,
+        ILogger<ScriptRunSummaryFlushService> logger)
+    {
+        _recorder = recorder ?? throw new ArgumentNullException(nameof(recorder));
+        _siteEventLogger = siteEventLogger ?? throw new ArgumentNullException(nameof(siteEventLogger));
+        _optionsMonitor = optionsMonitor ?? throw new ArgumentNullException(nameof(optionsMonitor));
+        _logger = logger ?? throw new ArgumentNullException(nameof(logger));
+    }
+
+    /// <inheritdoc />
+    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
+    {
+        while (!stoppingToken.IsCancellationRequested)
+        {
+            var intervalSeconds = _optionsMonitor.CurrentValue.ScriptRunSummaryIntervalSeconds;
+            var wait = intervalSeconds > 0 ? TimeSpan.FromSeconds(intervalSeconds) : DisabledPollInterval;
+
+            try
+            {
+                await Task.Delay(wait, stoppingToken).ConfigureAwait(false);
+            }
+            catch (OperationCanceledException)
+            {
+                break;
+            }
+
+            if (intervalSeconds <= 0)
+            {
+                // Summaries disabled for this tick's window: counters keep accumulating in
+                // the recorder (harmless — the next enabled flush just reports a longer
+                // window), we simply don't emit or reset yet.
+                continue;
+            }
+
+            try
+            {
+                await _recorder.FlushAsync(_siteEventLogger).ConfigureAwait(false);
+            }
+            catch (Exception ex)
+            {
+                _logger.LogWarning(ex, "ScriptRunSummaryFlushService flush failed; next tick will retry.");
+            }
+        }
+    }
+}