feat(sitestream): central transient per-site live alarm cache w/ seed-then-stream + failover (plan #10 T4)

Adds the active-central-node, in-memory, reference-counted per-site live alarm
cache backing the operator Alarm Summary page. No persisted central alarm store
([PERM]) — no EF entity/table/migration/DbSet.

- ISiteAlarmLiveCache (new): singleton seam — Subscribe(siteId, onChanged) ->
  IDisposable (ref-counted, linger stop on last-out), GetCurrentAlarms, IsLive.
- SiteAlarmAggregatorActor (new): one per site. Seed-then-stream ordering copied
  from DebugStreamBridgeActor — open the site-wide alarm-only gRPC stream first,
  buffer live deltas while the snapshot fan-out runs, flush with per-key dedup
  (AlarmKey = InstanceUniqueName|AlarmName|SourceReference), then live pass-through
  into an in-memory dict. Placeholder rows seeded from the snapshot and never
  expected on the stream; a real-alarm delta (distinct key) never wipes them.
  NodeA<->NodeB reconnect (retry budget + stability window), reconnect re-seeds,
  periodic reconcile (authoritative clear-and-rebuild) corrects instance-set drift,
  and a budget-exhausted stream self-heals on the next reconcile tick.
- SiteAlarmLiveCacheService (new): DI singleton facade — viewer reference-counting,
  linger-delayed last-out stop (version + TryRemove(ref) race guards), the
  snapshot fan-out seed (RequestDebugSnapshotAsync per Enabled instance, capped),
  bounded start-retry self-heal on transient start failure, and the immutable
  published-snapshot store the page reads. Cache mutated only on the actor thread;
  viewer callbacks invoked outside the lock.
- CommunicationOptions: LiveAlarmCacheLinger (30s), LiveAlarmCacheReconcileInterval
  (60s), LiveAlarmCacheSeedConcurrency (8), LiveAlarmCacheMaxSubscribersPerSite
  (200). Task 6 formalizes eager validation + telemetry.
- DI registration + AkkaHostedService SetActorSystem wiring on the active central node.
- Tests: 14 actor (seed/stream ordering, dedup, native-alarm parity, placeholder
  coherence, reconnect re-seed, reconcile replace, self-heal, stop) + 6 service
  (shared start, linger stop, resubscribe cancels stop, idempotent dispose, unknown
  site, transient-start self-heal). Code-reviewer pass: no persistence, no Critical;
  both Important findings addressed.

Claude-Session: https://claude.ai/code/session_01MtdgwpEeCUn6cUA5f1LMPj
This commit is contained in:
Joseph Doherty
2026-07-10 12:18:02 -04:00
parent 6d5ecc92a5
commit 696a4ffea2
8 changed files with 1739 additions and 0 deletions
@@ -88,4 +88,38 @@ public class CommunicationOptions
/// that are deployed once and never re-deployed. Default 1 hour ≫ <see cref="PendingDeploymentTtl"/>.
/// </summary>
public TimeSpan PendingDeploymentPurgeInterval { get; set; } = TimeSpan.FromHours(1);
// ── Aggregated live alarm cache (plan #10, Task 4) ───────────────────────────
// Introduced by Task 4 with sane defaults; Task 6 formalizes eager validation
// (positive interval/linger, positive concurrency/subscriber cap) and telemetry.
/// <summary>
/// How long the shared per-site live alarm aggregator keeps its gRPC stream +
/// in-memory cache warm after its LAST Alarm Summary viewer leaves, before it is
/// torn down. A short linger avoids re-seed thrash when an operator navigates away
/// and straight back. Default 30s.
/// </summary>
public TimeSpan LiveAlarmCacheLinger { get; set; } = TimeSpan.FromSeconds(30);
/// <summary>
/// Cadence of the per-site aggregator's periodic reconcile snapshot fan-out. The
/// backstop that corrects instance-set drift (enable/disable/deploy/delete) and any
/// alarm delta missed by the live stream, since the aggregated stream never carries
/// an explicit "removed" event. Default 60s.
/// </summary>
public TimeSpan LiveAlarmCacheReconcileInterval { get; set; } = TimeSpan.FromSeconds(60);
/// <summary>
/// Max concurrent per-instance snapshot fetches during a live-cache seed/reconcile
/// fan-out — the aggregated analogue of the Alarm Summary poll's
/// <c>MaxConcurrentFetches</c>. Default 8.
/// </summary>
public int LiveAlarmCacheSeedConcurrency { get; set; } = 8;
/// <summary>
/// Upper bound on concurrent Alarm Summary viewers sharing one site's aggregator.
/// A defensive cap only — one gRPC stream per site is shared regardless of viewer
/// count; this just bounds the subscriber list. Default 200.
/// </summary>
public int LiveAlarmCacheMaxSubscribersPerSite { get; set; } = 200;
}