feat(sitestream): central transient per-site live alarm cache w/ seed-then-stream + failover (plan #10 T4)
Adds the active-central-node, in-memory, reference-counted per-site live alarm cache backing the operator Alarm Summary page. No persisted central alarm store ([PERM]) — no EF entity/table/migration/DbSet. - ISiteAlarmLiveCache (new): singleton seam — Subscribe(siteId, onChanged) -> IDisposable (ref-counted, linger stop on last-out), GetCurrentAlarms, IsLive. - SiteAlarmAggregatorActor (new): one per site. Seed-then-stream ordering copied from DebugStreamBridgeActor — open the site-wide alarm-only gRPC stream first, buffer live deltas while the snapshot fan-out runs, flush with per-key dedup (AlarmKey = InstanceUniqueName|AlarmName|SourceReference), then live pass-through into an in-memory dict. Placeholder rows seeded from the snapshot and never expected on the stream; a real-alarm delta (distinct key) never wipes them. NodeA<->NodeB reconnect (retry budget + stability window), reconnect re-seeds, periodic reconcile (authoritative clear-and-rebuild) corrects instance-set drift, and a budget-exhausted stream self-heals on the next reconcile tick. - SiteAlarmLiveCacheService (new): DI singleton facade — viewer reference-counting, linger-delayed last-out stop (version + TryRemove(ref) race guards), the snapshot fan-out seed (RequestDebugSnapshotAsync per Enabled instance, capped), bounded start-retry self-heal on transient start failure, and the immutable published-snapshot store the page reads. Cache mutated only on the actor thread; viewer callbacks invoked outside the lock. - CommunicationOptions: LiveAlarmCacheLinger (30s), LiveAlarmCacheReconcileInterval (60s), LiveAlarmCacheSeedConcurrency (8), LiveAlarmCacheMaxSubscribersPerSite (200). Task 6 formalizes eager validation + telemetry. - DI registration + AkkaHostedService SetActorSystem wiring on the active central node. - Tests: 14 actor (seed/stream ordering, dedup, native-alarm parity, placeholder coherence, reconnect re-seed, reconcile replace, self-heal, stop) + 6 service (shared start, linger stop, resubscribe cancels stop, idempotent dispose, unknown site, transient-start self-heal). Code-reviewer pass: no persistence, no Critical; both Important findings addressed. Claude-Session: https://claude.ai/code/session_01MtdgwpEeCUn6cUA5f1LMPj
This commit is contained in:
Joseph Doherty
@@ -0,0 +1,519 @@
|
||||
using Akka.Actor;
|
||||
using Akka.Event;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Streaming;
|
||||
using ZB.MOM.WW.ScadaBridge.Communication.Grpc;
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.Communication.Actors;
|
||||
|
||||
/// <summary>
|
||||
/// One-per-site aggregator on the active central node backing the operator Alarm
|
||||
/// Summary live cache (plan #10, Task 4). Holds a transient in-memory
|
||||
/// <c>Dictionary<AlarmKey, AlarmStateChanged></c> of the whole site's current
|
||||
/// alarm state — NO persistence (locked <c>[PERM]</c>: no central alarm store). Created
|
||||
/// and torn down by <see cref="SiteAlarmLiveCacheService"/> under viewer reference-count.
|
||||
/// <para>
|
||||
/// <b>Seed-then-stream ordering</b> (copied from <see cref="DebugStreamBridgeActor"/>):
|
||||
/// the site-wide, alarm-only gRPC stream (<c>SubscribeSite</c>) is opened FIRST in
|
||||
/// <see cref="PreStart"/> so live deltas start flowing during the seed's snapshot-build
|
||||
/// + transit window; deltas arriving while a seed/reconcile fan-out is in flight are
|
||||
/// <em>buffered in arrival order</em>. When the fan-out completes the cache is rebuilt
|
||||
/// authoritatively from the fresh snapshot, then the buffer is flushed with per-key
|
||||
/// dedup against the seed, then the actor passes live deltas straight into the cache.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>Placeholder reconciliation:</b> the snapshot fan-out carries
|
||||
/// <see cref="AlarmStateChanged.IsConfiguredPlaceholder"/> rows (a configured native
|
||||
/// source binding with no active conditions); the live stream drops them server-side.
|
||||
/// Placeholders are seeded and never expected on the live stream — a live delta for a
|
||||
/// real alarm has a distinct <c>AlarmKey</c> so it can never wipe a placeholder row and
|
||||
/// vice-versa. Placeholder-vs-real coherence is refreshed by the periodic reconcile.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>Failover + drift:</b> a gRPC error flips NodeA↔NodeB with the same retry budget +
|
||||
/// stability window as <see cref="DebugStreamBridgeActor"/>, and each reconnect triggers
|
||||
/// a RE-SEED (never silently serve stale). A periodic reconcile snapshot
|
||||
/// (<see cref="_reconcileInterval"/>, default 60s) corrects instance-set drift and any
|
||||
/// missed delta.
|
||||
/// </para>
|
||||
/// All state is mutated only on the actor thread: gRPC callbacks and fan-out results are
|
||||
/// marshalled back via <c>Self.Tell</c>, so the cache needs no internal lock. The
|
||||
/// published snapshot handed to the service is a fresh immutable list (reference swap),
|
||||
/// so Blazor render threads never observe a partially-mutated cache.
|
||||
/// </summary>
|
||||
public sealed class SiteAlarmAggregatorActor : ReceiveActor, IWithTimers
|
||||
{
|
||||
private readonly ILoggingAdapter _log = Context.GetLogger();
|
||||
private readonly string _siteIdentifier;
|
||||
private readonly string _correlationId;
|
||||
private readonly Func<CancellationToken, Task<IReadOnlyList<AlarmStateChanged>>> _seedFn;
|
||||
private readonly Action<IReadOnlyList<AlarmStateChanged>> _publish;
|
||||
private readonly SiteStreamGrpcClientFactory _grpcFactory;
|
||||
private readonly string _grpcNodeAAddress;
|
||||
private readonly string _grpcNodeBAddress;
|
||||
private readonly TimeSpan _reconcileInterval;
|
||||
|
||||
private const int MaxRetries = 3;
|
||||
private const string ReconnectTimerKey = "alarm-grpc-reconnect";
|
||||
private const string StabilityTimerKey = "alarm-grpc-stability";
|
||||
private const string ReconcileTimerKey = "alarm-reconcile";
|
||||
|
||||
/// <summary>Delay between gRPC reconnection attempts. Settable for tests.</summary>
|
||||
internal static TimeSpan ReconnectDelay { get; set; } = TimeSpan.FromSeconds(5);
|
||||
|
||||
/// <summary>
|
||||
/// How long a freshly-opened gRPC stream must stay up before its retry budget is
|
||||
/// considered recovered (mirrors <see cref="DebugStreamBridgeActor.StabilityWindow"/>).
|
||||
/// Settable for tests.
|
||||
/// </summary>
|
||||
internal static TimeSpan StabilityWindow { get; set; } = TimeSpan.FromSeconds(60);
|
||||
|
||||
private int _retryCount;
|
||||
private bool _useNodeA = true;
|
||||
private bool _stopped;
|
||||
|
||||
/// <summary>
|
||||
/// True once the live gRPC stream has been given up (retry budget exhausted). Reconcile
|
||||
/// snapshots keep serving in the meantime; the next reconcile tick self-heals the stream
|
||||
/// by resetting the retry budget and reopening it, so a sustained site outage does not
|
||||
/// permanently drop the live feed. Actor-thread only.
|
||||
/// </summary>
|
||||
private bool _streamDown;
|
||||
private CancellationTokenSource? _grpcCts;
|
||||
private CancellationTokenSource? _lifetimeCts;
|
||||
|
||||
/// <summary>Current whole-site alarm state, keyed by <see cref="AlarmKey"/>. Actor-thread only.</summary>
|
||||
private readonly Dictionary<string, AlarmStateChanged> _cache = new();
|
||||
|
||||
/// <summary>True once the first seed has completed and been published. Actor-thread only.</summary>
|
||||
private bool _seeded;
|
||||
|
||||
/// <summary>True while a seed/reconcile snapshot fan-out is in flight (deltas buffer). Actor-thread only.</summary>
|
||||
private bool _fanoutInFlight;
|
||||
|
||||
/// <summary>Ordered buffer of live deltas that arrived while a fan-out was in flight. Actor-thread only.</summary>
|
||||
private readonly List<AlarmStateChanged> _buffer = new();
|
||||
|
||||
private const int BufferWarnThreshold = 10_000;
|
||||
private bool _bufferWarned;
|
||||
|
||||
/// <inheritdoc />
|
||||
public ITimerScheduler Timers { get; set; } = null!;
|
||||
|
||||
/// <summary>
|
||||
/// Creates a per-site alarm aggregator.
|
||||
/// </summary>
|
||||
/// <param name="siteIdentifier">Site identifier (for logging / gRPC client keying).</param>
|
||||
/// <param name="correlationId">Correlation id for the site-wide gRPC subscription.</param>
|
||||
/// <param name="seedFn">
|
||||
/// Snapshot fan-out that returns the whole site's current alarm rows (including
|
||||
/// placeholders), best-effort and tolerant of per-instance failure. Re-run on every
|
||||
/// seed and reconcile, so it re-enumerates the site's Enabled instances each call.
|
||||
/// </param>
|
||||
/// <param name="publish">
|
||||
/// Publishes a fresh immutable snapshot of the cache to the owning service, which
|
||||
/// stores it and raises viewer <c>onChanged</c> callbacks. Invoked on the actor thread.
|
||||
/// </param>
|
||||
/// <param name="grpcFactory">Factory caching one gRPC client per (site, endpoint).</param>
|
||||
/// <param name="grpcNodeAAddress">gRPC address of the site's node A.</param>
|
||||
/// <param name="grpcNodeBAddress">gRPC address of the site's node B.</param>
|
||||
/// <param name="reconcileInterval">Periodic reconcile snapshot cadence.</param>
|
||||
public SiteAlarmAggregatorActor(
|
||||
string siteIdentifier,
|
||||
string correlationId,
|
||||
Func<CancellationToken, Task<IReadOnlyList<AlarmStateChanged>>> seedFn,
|
||||
Action<IReadOnlyList<AlarmStateChanged>> publish,
|
||||
SiteStreamGrpcClientFactory grpcFactory,
|
||||
string grpcNodeAAddress,
|
||||
string grpcNodeBAddress,
|
||||
TimeSpan reconcileInterval)
|
||||
{
|
||||
_siteIdentifier = siteIdentifier;
|
||||
_correlationId = correlationId;
|
||||
_seedFn = seedFn;
|
||||
_publish = publish;
|
||||
_grpcFactory = grpcFactory;
|
||||
_grpcNodeAAddress = grpcNodeAAddress;
|
||||
_grpcNodeBAddress = grpcNodeBAddress;
|
||||
_reconcileInterval = reconcileInterval;
|
||||
|
||||
// Live delta from the site-wide alarm stream (marshalled in via Self.Tell).
|
||||
// A received delta must NOT reset the retry budget (a flapping stream that
|
||||
// delivers one delta between failures would otherwise never trip MaxRetries).
|
||||
Receive<AlarmStateChanged>(HandleLiveDelta);
|
||||
|
||||
// A seed/reconcile fan-out completed.
|
||||
Receive<SeedCompleted>(OnSeedCompleted);
|
||||
|
||||
// A seed/reconcile fan-out threw as a whole (individual per-instance faults are
|
||||
// swallowed inside seedFn and degrade to fewer rows, not a whole-fan-out failure).
|
||||
Receive<SeedFailed>(OnSeedFailed);
|
||||
|
||||
// Periodic reconcile tick (and the re-seed kicked after a reconnect).
|
||||
Receive<RunReconcile>(_ => OnReconcileTick());
|
||||
|
||||
// Stream stayed up for StabilityWindow — recover the retry budget.
|
||||
Receive<GrpcAlarmStreamStable>(_ =>
|
||||
{
|
||||
if (_stopped) return;
|
||||
_retryCount = 0;
|
||||
_log.Debug("Site-alarm gRPC stream for {0} stable; retry count reset", _siteIdentifier);
|
||||
});
|
||||
|
||||
// gRPC stream error — flip node + reconnect + re-seed.
|
||||
Receive<GrpcAlarmStreamError>(msg =>
|
||||
{
|
||||
_log.Warning("Site-alarm gRPC stream error for {0}: {1}", _siteIdentifier, msg.Exception.Message);
|
||||
HandleGrpcError();
|
||||
});
|
||||
|
||||
Receive<ReconnectAlarmStream>(_ => OpenGrpcStream());
|
||||
|
||||
// Owning service asks us to stop (last viewer left + linger elapsed).
|
||||
Receive<StopSiteAlarmAggregator>(_ =>
|
||||
{
|
||||
_log.Info("Stopping site-alarm aggregator for {0}", _siteIdentifier);
|
||||
CleanupGrpc();
|
||||
_stopped = true;
|
||||
Context.Stop(Self);
|
||||
});
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
protected override void PreStart()
|
||||
{
|
||||
_log.Info("Starting site-alarm aggregator for site {0}", _siteIdentifier);
|
||||
_lifetimeCts = new CancellationTokenSource();
|
||||
|
||||
// Stream-first: open the site-wide alarm stream BEFORE the first seed so deltas
|
||||
// in the seed window are captured (buffered) rather than lost.
|
||||
OpenGrpcStream();
|
||||
|
||||
// Kick the initial seed fan-out.
|
||||
StartFanout(isInitial: true);
|
||||
|
||||
// Periodic reconcile backstop.
|
||||
Timers.StartPeriodicTimer(ReconcileTimerKey, new RunReconcile(), _reconcileInterval, _reconcileInterval);
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
protected override void PostStop()
|
||||
{
|
||||
_grpcCts?.Cancel();
|
||||
_grpcCts?.Dispose();
|
||||
_grpcCts = null;
|
||||
_lifetimeCts?.Cancel();
|
||||
_lifetimeCts?.Dispose();
|
||||
_lifetimeCts = null;
|
||||
base.PostStop();
|
||||
}
|
||||
|
||||
// ── Reconcile tick ──────────────────────────────────────────────────────────
|
||||
|
||||
/// <summary>
|
||||
/// Periodic reconcile: always re-run the snapshot fan-out (corrects drift + missed
|
||||
/// deltas), and if the live stream was previously given up, self-heal it by resetting
|
||||
/// the retry budget and reopening — so a sustained outage never permanently kills the
|
||||
/// live feed.
|
||||
/// </summary>
|
||||
private void OnReconcileTick()
|
||||
{
|
||||
if (_stopped) return;
|
||||
StartFanout(isInitial: false);
|
||||
if (_streamDown)
|
||||
{
|
||||
_log.Info("Site-alarm gRPC stream for {0} was down; reopening on reconcile tick", _siteIdentifier);
|
||||
_retryCount = 0;
|
||||
OpenGrpcStream();
|
||||
}
|
||||
}
|
||||
|
||||
// ── Seed / reconcile fan-out ────────────────────────────────────────────────
|
||||
|
||||
/// <summary>
|
||||
/// Kicks a snapshot fan-out as a background task, marshalling the result back via
|
||||
/// <c>Self.Tell</c>. While in flight, live deltas buffer. A reconcile that arrives
|
||||
/// while a fan-out is already running is skipped (no stacking).
|
||||
/// </summary>
|
||||
private void StartFanout(bool isInitial)
|
||||
{
|
||||
if (_stopped) return;
|
||||
if (_fanoutInFlight)
|
||||
{
|
||||
// Already seeding/reconciling — don't stack a second fan-out.
|
||||
return;
|
||||
}
|
||||
|
||||
_fanoutInFlight = true;
|
||||
var self = Self;
|
||||
var ct = _lifetimeCts?.Token ?? CancellationToken.None;
|
||||
|
||||
Task.Run(async () =>
|
||||
{
|
||||
try
|
||||
{
|
||||
var alarms = await _seedFn(ct);
|
||||
self.Tell(new SeedCompleted(alarms, isInitial));
|
||||
}
|
||||
catch (OperationCanceledException) when (ct.IsCancellationRequested)
|
||||
{
|
||||
// Actor stopping — drop silently.
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
self.Tell(new SeedFailed(ex, isInitial));
|
||||
}
|
||||
}, ct);
|
||||
}
|
||||
|
||||
private void OnSeedCompleted(SeedCompleted msg)
|
||||
{
|
||||
if (_stopped) return;
|
||||
|
||||
// Rebuild the cache authoritatively from the fresh snapshot (this is what makes
|
||||
// reconcile able to DROP rows for instances/alarms that disappeared — a merge
|
||||
// could never remove a stale row since the live stream sends no "removed" event).
|
||||
_cache.Clear();
|
||||
foreach (var alarm in msg.Alarms)
|
||||
{
|
||||
var key = AlarmKey(alarm);
|
||||
if (!_cache.TryGetValue(key, out var existing) || alarm.Timestamp >= existing.Timestamp)
|
||||
_cache[key] = alarm;
|
||||
}
|
||||
|
||||
// Flush the deltas buffered during the fan-out, deduped against the seed: a
|
||||
// buffered delta whose key is in the fresh snapshot with an equal-or-newer
|
||||
// timestamp is already reflected → drop; a strictly-newer (or new-key) delta is
|
||||
// applied. Inclusive-on-snapshot boundary matches DebugStreamBridgeActor.
|
||||
FlushBuffer();
|
||||
|
||||
_fanoutInFlight = false;
|
||||
_seeded = true;
|
||||
|
||||
_log.Debug("Site-alarm {0} {1} complete: {2} alarm row(s)",
|
||||
_siteIdentifier, msg.IsInitial ? "seed" : "reconcile", _cache.Count);
|
||||
|
||||
Publish();
|
||||
}
|
||||
|
||||
private void OnSeedFailed(SeedFailed msg)
|
||||
{
|
||||
if (_stopped) return;
|
||||
|
||||
_log.Warning(msg.Exception,
|
||||
"Site-alarm {0} {1} fan-out failed; keeping current cache and relying on the next reconcile",
|
||||
_siteIdentifier, msg.IsInitial ? "seed" : "reconcile");
|
||||
|
||||
// Don't lose deltas captured during the failed window — apply them pass-through
|
||||
// into the (possibly stale/empty) cache. The next reconcile re-seeds authoritatively.
|
||||
_fanoutInFlight = false;
|
||||
FlushBuffer(dedupAgainstSeed: false);
|
||||
|
||||
// Only publish if we already had a seed (so IsLive doesn't flip true on a
|
||||
// failed initial seed — the page keeps its poll fallback until we truly seed).
|
||||
if (_seeded)
|
||||
Publish();
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Flushes the pre-fan-out buffer in arrival order. When <paramref name="dedupAgainstSeed"/>
|
||||
/// is true (normal completion) a buffered delta already reflected in the just-installed
|
||||
/// snapshot (same key, timestamp <= cache entry) is dropped; otherwise every buffered
|
||||
/// delta is applied pass-through.
|
||||
/// </summary>
|
||||
private void FlushBuffer(bool dedupAgainstSeed = true)
|
||||
{
|
||||
if (_buffer.Count == 0) return;
|
||||
|
||||
foreach (var delta in _buffer)
|
||||
{
|
||||
if (dedupAgainstSeed)
|
||||
ApplyDelta(delta, requireStrictlyNewer: true);
|
||||
else
|
||||
ApplyDelta(delta, requireStrictlyNewer: false);
|
||||
}
|
||||
_buffer.Clear();
|
||||
}
|
||||
|
||||
// ── Live delta handling ─────────────────────────────────────────────────────
|
||||
|
||||
private void HandleLiveDelta(AlarmStateChanged delta)
|
||||
{
|
||||
if (_stopped) return;
|
||||
|
||||
if (_fanoutInFlight)
|
||||
{
|
||||
_buffer.Add(delta);
|
||||
if (!_bufferWarned && _buffer.Count > BufferWarnThreshold)
|
||||
{
|
||||
_bufferWarned = true;
|
||||
_log.Warning(
|
||||
"Site-alarm pre-seed buffer for {0} exceeded {1} deltas while a fan-out was in flight " +
|
||||
"(deltas retained, not dropped).",
|
||||
_siteIdentifier, BufferWarnThreshold);
|
||||
}
|
||||
return;
|
||||
}
|
||||
|
||||
// Pass-through: apply and publish only if the cache actually changed.
|
||||
if (ApplyDelta(delta, requireStrictlyNewer: false))
|
||||
Publish();
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Applies one alarm delta to the cache keyed by <see cref="AlarmKey"/>. Returns
|
||||
/// <c>true</c> if the cache changed. A stale (older) delta for an existing key is
|
||||
/// ignored. Never carries a placeholder (the live stream drops those), so a live
|
||||
/// delta for a real alarm can only add/replace its own key — it never touches a
|
||||
/// placeholder row under a different key.
|
||||
/// </summary>
|
||||
private bool ApplyDelta(AlarmStateChanged delta, bool requireStrictlyNewer)
|
||||
{
|
||||
var key = AlarmKey(delta);
|
||||
if (_cache.TryGetValue(key, out var existing))
|
||||
{
|
||||
var newer = requireStrictlyNewer
|
||||
? delta.Timestamp > existing.Timestamp
|
||||
: delta.Timestamp >= existing.Timestamp;
|
||||
if (!newer) return false;
|
||||
}
|
||||
_cache[key] = delta;
|
||||
return true;
|
||||
}
|
||||
|
||||
private void Publish()
|
||||
{
|
||||
// Fresh immutable list — reference swap; readers never see a partial mutation.
|
||||
var snapshot = _cache.Values.ToList();
|
||||
try
|
||||
{
|
||||
_publish(snapshot);
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
_log.Warning(ex, "Site-alarm publish callback threw for {0}; ignoring", _siteIdentifier);
|
||||
}
|
||||
}
|
||||
|
||||
// ── gRPC stream lifecycle (mirrors DebugStreamBridgeActor) ──────────────────
|
||||
|
||||
private void OpenGrpcStream()
|
||||
{
|
||||
if (_stopped) return;
|
||||
|
||||
_streamDown = false;
|
||||
var endpoint = _useNodeA ? _grpcNodeAAddress : _grpcNodeBAddress;
|
||||
_log.Info("Opening site-alarm gRPC stream for {0} to {1}", _siteIdentifier, endpoint);
|
||||
|
||||
_grpcCts?.Cancel();
|
||||
_grpcCts?.Dispose();
|
||||
_grpcCts = new CancellationTokenSource();
|
||||
|
||||
Timers.StartSingleTimer(StabilityTimerKey, new GrpcAlarmStreamStable(), StabilityWindow);
|
||||
|
||||
var client = _grpcFactory.GetOrCreate(_siteIdentifier, endpoint);
|
||||
var self = Self;
|
||||
var ct = _grpcCts.Token;
|
||||
|
||||
Task.Run(async () =>
|
||||
{
|
||||
await client.SubscribeSiteAsync(
|
||||
_correlationId,
|
||||
alarm => self.Tell(alarm),
|
||||
ex => self.Tell(new GrpcAlarmStreamError(ex)),
|
||||
ct);
|
||||
}, ct);
|
||||
}
|
||||
|
||||
private void HandleGrpcError()
|
||||
{
|
||||
if (_stopped) return;
|
||||
|
||||
// Stream failed before the stability window — retry budget NOT recovered.
|
||||
Timers.Cancel(StabilityTimerKey);
|
||||
|
||||
_retryCount++;
|
||||
|
||||
if (_retryCount > MaxRetries)
|
||||
{
|
||||
// Give up the stream, but do NOT stop the aggregator: the periodic reconcile
|
||||
// still refreshes the cache from ClusterClient snapshots, so the page keeps a
|
||||
// (slower) live-ish view rather than going dark. A later reconcile-triggered
|
||||
// reconnect is not attempted here; the stream is simply left down.
|
||||
_log.Error("Site-alarm gRPC stream for {0} exceeded max retries ({1}); leaving stream down, " +
|
||||
"reconcile snapshots continue and the next reconcile tick will retry the stream",
|
||||
_siteIdentifier, MaxRetries);
|
||||
_streamDown = true;
|
||||
CleanupGrpc();
|
||||
return;
|
||||
}
|
||||
|
||||
// Unsubscribe the failed stream on the previous endpoint (TryGet, never
|
||||
// GetOrCreate) so the site releases its relay actor instead of leaving a zombie.
|
||||
var previousEndpoint = _useNodeA ? _grpcNodeAAddress : _grpcNodeBAddress;
|
||||
_grpcFactory.TryGet(_siteIdentifier, previousEndpoint)?.Unsubscribe(_correlationId);
|
||||
|
||||
// Flip to the other node.
|
||||
_useNodeA = !_useNodeA;
|
||||
|
||||
// A failover flip must RE-SEED (never silently serve stale) — kick a reconcile
|
||||
// fan-out alongside the reconnect. Buffering during the fan-out keeps the new
|
||||
// stream's deltas coherent with the fresh snapshot.
|
||||
StartFanout(isInitial: false);
|
||||
|
||||
if (_retryCount == 1)
|
||||
Self.Tell(new ReconnectAlarmStream());
|
||||
else
|
||||
Timers.StartSingleTimer(ReconnectTimerKey, new ReconnectAlarmStream(), ReconnectDelay);
|
||||
}
|
||||
|
||||
private void CleanupGrpc()
|
||||
{
|
||||
_grpcCts?.Cancel();
|
||||
_grpcCts?.Dispose();
|
||||
_grpcCts = null;
|
||||
|
||||
var endpoint = _useNodeA ? _grpcNodeAAddress : _grpcNodeBAddress;
|
||||
_grpcFactory.TryGet(_siteIdentifier, endpoint)?.Unsubscribe(_correlationId);
|
||||
}
|
||||
|
||||
// ── Dedup key (copied from DebugStreamBridgeActor.AlarmKey with attribution) ──
|
||||
|
||||
/// <summary>
|
||||
/// NUL delimiter so distinct identities never collide on a shared boundary. Cannot
|
||||
/// appear in an instance/alarm name. Mirrors <see cref="DebugStreamBridgeActor"/>.
|
||||
/// </summary>
|
||||
private const char KeyDelimiter = '\u0000';
|
||||
|
||||
/// <summary>
|
||||
/// Per-alarm dedup identity = (InstanceUniqueName, AlarmName, SourceReference) —
|
||||
/// identical to <c>DebugStreamBridgeActor.AlarmKey</c> so native per-condition alarms
|
||||
/// sharing an AlarmName but differing by source reference are not conflated. Each
|
||||
/// nullable component is guarded to prevent silent null/empty key collisions.
|
||||
/// </summary>
|
||||
private static string AlarmKey(AlarmStateChanged a) =>
|
||||
string.Concat(
|
||||
a.InstanceUniqueName ?? string.Empty, KeyDelimiter,
|
||||
a.AlarmName ?? string.Empty, KeyDelimiter,
|
||||
a.SourceReference ?? string.Empty);
|
||||
}
|
||||
|
||||
/// <summary>Message asking a <see cref="SiteAlarmAggregatorActor"/> to stop (last viewer left).</summary>
|
||||
public sealed record StopSiteAlarmAggregator;
|
||||
|
||||
/// <summary>Internal: a seed/reconcile snapshot fan-out completed with the whole-site alarm rows.</summary>
|
||||
internal sealed record SeedCompleted(IReadOnlyList<AlarmStateChanged> Alarms, bool IsInitial);
|
||||
|
||||
/// <summary>Internal: a seed/reconcile snapshot fan-out threw as a whole.</summary>
|
||||
internal sealed record SeedFailed(Exception Exception, bool IsInitial);
|
||||
|
||||
/// <summary>Internal: periodic reconcile tick (and the re-seed kicked after a reconnect).</summary>
|
||||
internal sealed record RunReconcile;
|
||||
|
||||
/// <summary>Internal: site-alarm gRPC stream error occurred.</summary>
|
||||
internal sealed record GrpcAlarmStreamError(Exception Exception);
|
||||
|
||||
/// <summary>Internal: reconnect the site-alarm gRPC stream (flip node).</summary>
|
||||
internal sealed record ReconnectAlarmStream;
|
||||
|
||||
/// <summary>Internal: the current site-alarm gRPC stream has stayed up long enough to recover the retry budget.</summary>
|
||||
internal sealed record GrpcAlarmStreamStable;
|
||||
Reference in New Issue
Block a user