feat(sitestream): central transient per-site live alarm cache w/ seed-then-stream + failover (plan #10 T4)

Adds the active-central-node, in-memory, reference-counted per-site live alarm
cache backing the operator Alarm Summary page. No persisted central alarm store
([PERM]) — no EF entity/table/migration/DbSet.

- ISiteAlarmLiveCache (new): singleton seam — Subscribe(siteId, onChanged) ->
  IDisposable (ref-counted, linger stop on last-out), GetCurrentAlarms, IsLive.
- SiteAlarmAggregatorActor (new): one per site. Seed-then-stream ordering copied
  from DebugStreamBridgeActor — open the site-wide alarm-only gRPC stream first,
  buffer live deltas while the snapshot fan-out runs, flush with per-key dedup
  (AlarmKey = InstanceUniqueName|AlarmName|SourceReference), then live pass-through
  into an in-memory dict. Placeholder rows seeded from the snapshot and never
  expected on the stream; a real-alarm delta (distinct key) never wipes them.
  NodeA<->NodeB reconnect (retry budget + stability window), reconnect re-seeds,
  periodic reconcile (authoritative clear-and-rebuild) corrects instance-set drift,
  and a budget-exhausted stream self-heals on the next reconcile tick.
- SiteAlarmLiveCacheService (new): DI singleton facade — viewer reference-counting,
  linger-delayed last-out stop (version + TryRemove(ref) race guards), the
  snapshot fan-out seed (RequestDebugSnapshotAsync per Enabled instance, capped),
  bounded start-retry self-heal on transient start failure, and the immutable
  published-snapshot store the page reads. Cache mutated only on the actor thread;
  viewer callbacks invoked outside the lock.
- CommunicationOptions: LiveAlarmCacheLinger (30s), LiveAlarmCacheReconcileInterval
  (60s), LiveAlarmCacheSeedConcurrency (8), LiveAlarmCacheMaxSubscribersPerSite
  (200). Task 6 formalizes eager validation + telemetry.
- DI registration + AkkaHostedService SetActorSystem wiring on the active central node.
- Tests: 14 actor (seed/stream ordering, dedup, native-alarm parity, placeholder
  coherence, reconnect re-seed, reconcile replace, self-heal, stop) + 6 service
  (shared start, linger stop, resubscribe cancels stop, idempotent dispose, unknown
  site, transient-start self-heal). Code-reviewer pass: no persistence, no Critical;
  both Important findings addressed.

Claude-Session: https://claude.ai/code/session_01MtdgwpEeCUn6cUA5f1LMPj
This commit is contained in:
Joseph Doherty
2026-07-10 12:18:02 -04:00
parent 6d5ecc92a5
commit 696a4ffea2
8 changed files with 1739 additions and 0 deletions
@@ -0,0 +1,62 @@
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Streaming;
namespace ZB.MOM.WW.ScadaBridge.Communication;
/// <summary>
/// Central, transient, in-memory per-site live alarm cache (plan #10, Task 4).
/// A DI singleton shared by every Alarm Summary Blazor circuit on the active central
/// node. For each site with at least one active viewer it runs ONE shared per-site
/// aggregator that seeds from the existing debug-snapshot fan-out and stays warm on a
/// single site-wide, alarm-only gRPC stream (<c>SubscribeSite</c>), so the page
/// reflects alarm transitions in near-real-time instead of re-polling every 15s.
/// <para>
/// <b>Hard constraint (locked <c>[PERM]</c>):</b> there is NO persisted central alarm
/// store. This cache is purely in-memory and lives only on the active central node —
/// no EF entity/table/migration backs it. On a NodeA↔NodeB failover the new active
/// node re-seeds from scratch.
/// </para>
/// <para>
/// <b>Reference-counted lifecycle:</b> the first <see cref="Subscribe"/> for a site
/// starts its aggregator (open stream + snapshot seed); the last subscriber leaving
/// stops it after a short linger (see
/// <see cref="CommunicationOptions.LiveAlarmCacheLinger"/>) to avoid re-seed thrash.
/// One gRPC stream per <em>site</em>, not per browser circuit.
/// </para>
/// </summary>
public interface ISiteAlarmLiveCache
{
/// <summary>
/// Registers a viewer for a site's live alarm feed and returns an
/// <see cref="IDisposable"/> that unregisters it. The first subscriber for a site
/// starts the shared aggregator; the last subscriber's <see cref="IDisposable.Dispose"/>
/// schedules a linger-delayed stop. <paramref name="onChanged"/> is raised (on the
/// aggregator's thread) after each applied change — the handler must not block and
/// should marshal any UI work onto its own dispatcher (mirrors
/// <c>IDeploymentStatusNotifier</c>). Disposing the returned handle is idempotent.
/// </summary>
/// <param name="siteId">The numeric site id whose alarms the viewer wants.</param>
/// <param name="onChanged">Callback invoked whenever the site's cached alarm set changes.</param>
/// <returns>A disposable that unregisters this viewer (idempotent).</returns>
IDisposable Subscribe(int siteId, Action onChanged);
/// <summary>
/// Returns the current in-memory alarm snapshot for a site — exactly what the page
/// should render. Empty when the site has no aggregator or has not yet completed its
/// first seed. Safe to call from Blazor render threads: it returns an immutable
/// point-in-time list (the aggregator publishes fresh immutable lists; readers never
/// see a partially-mutated collection).
/// </summary>
/// <param name="siteId">The numeric site id.</param>
/// <returns>The current cached alarms (possibly empty), never <c>null</c>.</returns>
IReadOnlyList<AlarmStateChanged> GetCurrentAlarms(int siteId);
/// <summary>
/// Whether the site's aggregator has completed its initial seed and is serving live
/// state. The page can use this to decide whether to keep its 15s poll fallback
/// (Task 5): <c>false</c> means "not live yet — keep polling"; <c>true</c> means the
/// cache is authoritative.
/// </summary>
/// <param name="siteId">The numeric site id.</param>
/// <returns><c>true</c> once the site's aggregator has seeded and published at least once.</returns>
bool IsLive(int siteId);
}