fix(alarms): engine cache prune + bounded dispose, source Warning, cached dead-letter depth, script-log rate limit (E3)

- R05:1/12 ScriptedAlarmEngine: _valueCache is cleared in LoadAsync (it is
  re-seeded for the referenced paths) so retired tags no longer accrete across
  deploys; OnUpstreamChange no longer writes the cache after Dispose.
- R05:3 ScriptedAlarmEngine.Dispose drains in-flight work for at most 5 s
  (DisposeDrainTimeout), then cancels it through a shutdown token every
  DrainDirtyAsync/ShelvingCheckAsync now runs on, logs a Warning and gives
  one second of grace — a stalled IAlarmStateStore.SaveAsync can no longer
  hang host shutdown. A save that completes inside the bound is still drained
  (positive control test).
- R05:6 ScriptedAlarmSource logs a throwing OnAlarmEvent subscriber at
  Warning (naming the alarm) instead of swallowing it.
- R05:7 LocalDbStoreAndForwardSink keeps a cached dead-letter counter in step
  with every dead-letter/revive/purge path and resyncs it with the queued
  counter on the same cadence; GetStatus no longer runs a COUNT(*) scan.
- R05:9 ScriptLogRateLimiter (20 per source per minute, fixed window, one
  summary entry naming the dropped count when the window rolls) behind
  ScriptLogTopicSink (key = script|vtag|alarm) and ScriptLogCompanionSink
  (key = script name).

Refs #734

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
This commit is contained in:
Joseph Doherty
2026-10-22 10:17:30 -04:00
parent 2e9160317f
commit 6cbf763719
10 changed files with 683 additions and 32 deletions
@@ -1,4 +1,5 @@
using System.Collections.Concurrent;
using Serilog;
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
namespace ZB.MOM.WW.OtOpcUa.Core.ScriptedAlarms;
@@ -23,19 +24,27 @@ namespace ZB.MOM.WW.OtOpcUa.Core.ScriptedAlarms;
/// adapter doesn't maintain per-subscription state beyond the filter set — it
/// checks each emission against every live subscription.
/// </para>
/// <para>
/// A throwing <see cref="OnAlarmEvent"/> subscriber is logged at Warning (naming the alarm) and
/// does not crash the adapter — the same posture as the engine's own <c>FireEvent</c>. It used to
/// be swallowed silently: a lost alarm event with no trace.
/// </para>
/// </remarks>
public sealed class ScriptedAlarmSource : IAlarmSource, IDisposable
{
private readonly ScriptedAlarmEngine _engine;
private readonly ILogger _logger;
private readonly ConcurrentDictionary<string, Subscription> _subscriptions
= new(StringComparer.Ordinal);
private bool _disposed;
/// <summary>Initializes a new instance of the <see cref="ScriptedAlarmSource"/> class.</summary>
/// <param name="engine">The scripted alarm engine to expose.</param>
public ScriptedAlarmSource(ScriptedAlarmEngine engine)
/// <param name="logger">Where a throwing subscriber is reported; null ⇒ Serilog's global logger.</param>
public ScriptedAlarmSource(ScriptedAlarmEngine engine, ILogger? logger = null)
{
_engine = engine ?? throw new ArgumentNullException(nameof(engine));
_logger = logger ?? Log.Logger;
_engine.OnEvent += OnEngineEvent;
}
@@ -98,7 +107,13 @@ public sealed class ScriptedAlarmSource : IAlarmSource, IDisposable
Severity: evt.Severity,
SourceTimestampUtc: evt.TimestampUtc);
try { OnAlarmEvent?.Invoke(this, payload); }
catch { /* subscriber exceptions don't crash the adapter */ }
catch (Exception ex)
{
// Subscriber exceptions don't crash the adapter — but they are not silent either: a lost
// alarm event must leave a trace the operator can find.
_logger.Warning(ex, "ScriptedAlarmSource OnAlarmEvent subscriber threw for {AlarmId} ({Emission})",
evt.AlarmId, evt.Emission);
}
}
}