feat(sitestream): central transient per-site live alarm cache w/ seed-then-stream + failover (plan #10 T4)

Adds the active-central-node, in-memory, reference-counted per-site live alarm
cache backing the operator Alarm Summary page. No persisted central alarm store
([PERM]) — no EF entity/table/migration/DbSet.

- ISiteAlarmLiveCache (new): singleton seam — Subscribe(siteId, onChanged) ->
  IDisposable (ref-counted, linger stop on last-out), GetCurrentAlarms, IsLive.
- SiteAlarmAggregatorActor (new): one per site. Seed-then-stream ordering copied
  from DebugStreamBridgeActor — open the site-wide alarm-only gRPC stream first,
  buffer live deltas while the snapshot fan-out runs, flush with per-key dedup
  (AlarmKey = InstanceUniqueName|AlarmName|SourceReference), then live pass-through
  into an in-memory dict. Placeholder rows seeded from the snapshot and never
  expected on the stream; a real-alarm delta (distinct key) never wipes them.
  NodeA<->NodeB reconnect (retry budget + stability window), reconnect re-seeds,
  periodic reconcile (authoritative clear-and-rebuild) corrects instance-set drift,
  and a budget-exhausted stream self-heals on the next reconcile tick.
- SiteAlarmLiveCacheService (new): DI singleton facade — viewer reference-counting,
  linger-delayed last-out stop (version + TryRemove(ref) race guards), the
  snapshot fan-out seed (RequestDebugSnapshotAsync per Enabled instance, capped),
  bounded start-retry self-heal on transient start failure, and the immutable
  published-snapshot store the page reads. Cache mutated only on the actor thread;
  viewer callbacks invoked outside the lock.
- CommunicationOptions: LiveAlarmCacheLinger (30s), LiveAlarmCacheReconcileInterval
  (60s), LiveAlarmCacheSeedConcurrency (8), LiveAlarmCacheMaxSubscribersPerSite
  (200). Task 6 formalizes eager validation + telemetry.
- DI registration + AkkaHostedService SetActorSystem wiring on the active central node.
- Tests: 14 actor (seed/stream ordering, dedup, native-alarm parity, placeholder
  coherence, reconnect re-seed, reconcile replace, self-heal, stop) + 6 service
  (shared start, linger stop, resubscribe cancels stop, idempotent dispose, unknown
  site, transient-start self-heal). Code-reviewer pass: no persistence, no Critical;
  both Important findings addressed.

Claude-Session: https://claude.ai/code/session_01MtdgwpEeCUn6cUA5f1LMPj
This commit is contained in:
Joseph Doherty
2026-07-10 12:18:02 -04:00
parent 6d5ecc92a5
commit 696a4ffea2
8 changed files with 1739 additions and 0 deletions
@@ -424,6 +424,11 @@ akka {{
var debugStreamService = _serviceProvider.GetService<DebugStreamService>();
debugStreamService?.SetActorSystem(_actorSystem!);
// Wire up the aggregated live alarm cache with the ActorSystem so it can host
// per-site aggregator actors on the active central node (plan #10, Task 4).
var siteAlarmLiveCache = _serviceProvider.GetService<ZB.MOM.WW.ScadaBridge.Communication.SiteAlarmLiveCacheService>();
siteAlarmLiveCache?.SetActorSystem(_actorSystem!);
// Management Service — accessible via ClusterClient
var mgmtLogger = _serviceProvider.GetRequiredService<ILoggerFactory>()