dohertj2/ScadaBridge
1
0
Fork 0
Code Issues 2 Pull Requests Actions Packages Projects Releases Wiki Activity

fix(comms): review findings — consumer-based debug orphan net, foreign-cancel triad, honest onConnected, served-row-exact retirement, full-rate reconcile

Browse Source 
F1 (HIGH) DebugStreamBridgeActor: the 5-minute orphan net measured the MAILBOX
(SetReceiveTimeout), and once stream events were correctly marked
INotInfluenceReceiveTimeout nothing recurring reset it — the snapshot lands once
and GrpcStreamStable once — so every healthy session self-terminated at ~6 min
with a false "Site disconnected". Replaced with a periodic self-tick
(ConsumerLivenessCheckInterval, 30s) over a consumer-last-seen stamp renewed only
by DebugStreamConsumerAlive, which DebugStreamService Tells on a shared timer to
every session still in its registry (holding a session there IS "a consumer is
attached" — both the Blazor view and the SignalR hub release it on
dispose/disconnect, and it works headless). Reverting the wrapper was rejected: it
would restore the quiet-instance orphan bug.

F2 (MED) SiteStreamGrpcClient: the RpcException(Cancelled) filter now requires
cts.IsCancellationRequested. A peer-originated / channel-dispose Cancelled fired
none of onError/onCompleted/onConnected, leaving SiteAlarmAggregatorActor with
_streamDown=false forever (IsLive stuck true, reconcile reopen guard never fired).

F3 (MED) SiteStreamGrpcClient: a header TIMEOUT is no longer reported as
connected — that shape is exactly what an unreachable site produces, and it
cleared _streamDown, consumed _seedOnConnect and launched a full snapshot fan-out
at a dead site. AwaitHeadersAsync returns bool; the first received event is the
fallback connected signal, fired at most once from headers OR first event.

F4 (LOW-MED) SqliteAuditWriter.MarkReconciledUpToAsync: the blanket below-cursor
UPDATE retired late-stamped inserts that were never served (then age-purged —
silent loss). The flip is now bounded by insertion order: a Pending row retires
only if its rowid is at or below the high-water mark of rows this instance has
served from ReadPendingSinceAsync (clamped on purge, since SQLite reuses rowids);
Forwarded rows are exempt (central ACKed them over the push path). At-least-once
is unchanged.

F5 (LOW) Documented the liveness dependency (a served row never covered by a later
cursor stays Pending forever; PurgeExpiredAsync never purges Pending) in
ISiteAuditQueue + Component-AuditLog.md, and added a cheap site-health signal:
SiteAuditBacklogReporter logs a rate-limited warning when the existing
oldest-pending metric exceeds 24h.

F6 (MED) SiteAlarmAggregatorActor: _fanoutSinceLastTick was armed by the
reconcile's OWN fan-out, so steady state ran fan-out→skip→fan-out→skip — one
reconcile per 2x interval (120s), halving the not-reporting refresh and the alarm
reconcile backstop. The skip is now armed only by connect/failover-driven seeds
(initial, _seedOnConnect, and a re-seed queued behind one).

Tests: Communication.Tests 691 passed (+13), AuditLog.Tests 382 passed (+5).
This commit is contained in:
Joseph Doherty
2026-08-14 23:52:25 -04:00
parent b1de9dfdd4
commit fd5e023d08
16 changed files with 1192 additions and 84 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/ZB.MOM.WW.ScadaBridge.Communication/Actors/SiteAlarmAggregatorActor.cs
+70 -21
View File
@@ -144,15 +144,38 @@ public sealed class SiteAlarmAggregatorActor : ReceiveActor, IWithTimers
private bool _seedOnConnect;
/// <summary>
/// Set whenever a fan-out finishes (success or failure); consumed by the next reconcile
/// tick, which skips its own fan-out when it finds the flag set. That makes the
/// connect-driven seed and the periodic backstop mutually exclusive — the pair used to
/// Set when a CONNECT- OR FAILOVER-DRIVEN fan-out finishes (the initial seed, a re-seed
/// consumed by a successful (re)connect, or a re-seed queued behind one); consumed by the
/// next reconcile tick, which skips its own fan-out when it finds the flag set. That makes
/// the connect-driven seed and the periodic backstop mutually exclusive — the pair used to
/// run BOTH, which is the "full unconditional snapshot every 60s" waste — while bounding
/// staleness at two intervals (a tick can be skipped at most once in a row).
/// <para>
/// A TICK-driven fan-out deliberately does NOT set it. It used to: the tick's own fan-out
/// completion armed the skip, so steady state ran fan-out → skip → fan-out → skip, i.e.
/// one reconcile per TWO intervals — halving the not-reporting refresh rate and the alarm
/// reconcile backstop for no reason (nothing was duplicated to suppress).
/// </para>
/// Actor-thread only.
/// </summary>
private bool _fanoutSinceLastTick;
/// <summary>
/// Whether the in-flight fan-out is connect/failover-driven and therefore makes the next
/// reconcile tick redundant (see <see cref="_fanoutSinceLastTick"/>). Stable for the
/// fan-out's lifetime: <see cref="StartFanout"/> never overwrites it while one is in
/// flight (a colliding request queues instead). Actor-thread only.
/// </summary>
private bool _fanoutSuppressesNextTick;
/// <summary>
/// Same flag for a fan-out that was queued behind an in-flight one
/// (<see cref="_reseedQueued"/>) — OR-ed across colliding requests so a queued
/// connect/failover re-seed keeps its suppression even if a tick collided too.
/// Actor-thread only.
/// </summary>
private bool _queuedFanoutSuppressesNextTick;
/// <summary>
/// True between issuing a stream open and its first outcome (connected / error /
/// completed), so a reconcile tick never stacks a second open on an in-flight one.
@@ -269,7 +292,9 @@ public sealed class SiteAlarmAggregatorActor : ReceiveActor, IWithTimers
Receive<RetrySeed>(_ =>
{
if (_stopped) return;
StartFanout(isInitial: false);
// A backoff retry of a failed fan-out is not a connect/failover seed: it must
// not stand the next reconcile tick down.
StartFanout(isInitial: false, suppressesNextTick: false);
});
// The site-wide stream is confirmed established (response headers received). This —
@@ -356,7 +381,7 @@ public sealed class SiteAlarmAggregatorActor : ReceiveActor, IWithTimers
// Kick the initial seed fan-out. The connect callback for this first stream must
// NOT seed again on top of it, so the flag starts cleared.
_seedOnConnect = false;
StartFanout(isInitial: true);
StartFanout(isInitial: true, suppressesNextTick: true);
// Periodic reconcile backstop. Single-shot and re-armed with fresh jitter each
// tick (a periodic timer would lock every site to the same phase forever).
@@ -398,9 +423,11 @@ public sealed class SiteAlarmAggregatorActor : ReceiveActor, IWithTimers
/// <summary>
/// Periodic reconcile backstop: re-run the snapshot fan-out (corrects instance-set drift
/// + any missed delta) UNLESS one already completed inside this window — a reconnect-driven
/// seed makes the tick redundant, and running both was the "full unconditional snapshot
/// every 60s" waste (WP2.3). If the live stream was previously given up, self-heal it by
/// + any missed delta) UNLESS a CONNECT- OR FAILOVER-DRIVEN seed already completed inside
/// this window — that seed makes the tick redundant, and running both was the "full
/// unconditional snapshot every 60s" waste (WP2.3). A tick's OWN fan-out never arms that
/// skip: it used to, which made steady state alternate fan-out/skip and halved the
/// effective reconcile rate. If the live stream was previously given up, self-heal it by
/// resetting the retry budget and reopening, so a sustained outage never permanently kills
/// the live feed.
/// </summary>
@@ -441,7 +468,11 @@ public sealed class SiteAlarmAggregatorActor : ReceiveActor, IWithTimers
}
if (!reopening)
StartFanout(isInitial: false);
{
// Tick-driven: deliberately does NOT set the skip flag, so the NEXT tick fans
// out too. Setting it here was the "one reconcile per two intervals" bug.
StartFanout(isInitial: false, suppressesNextTick: false);
}
}
// ── Seed / reconcile fan-out ────────────────────────────────────────────────
@@ -451,7 +482,7 @@ public sealed class SiteAlarmAggregatorActor : ReceiveActor, IWithTimers
/// <c>Self.Tell</c>. While in flight, live deltas buffer. A reconcile that arrives
/// while a fan-out is already running is skipped (no stacking).
/// </summary>
private void StartFanout(bool isInitial)
private void StartFanout(bool isInitial, bool suppressesNextTick)
{
if (_stopped) return;
if (_fanoutInFlight)
@@ -460,11 +491,16 @@ public sealed class SiteAlarmAggregatorActor : ReceiveActor, IWithTimers
// one — its snapshot read-time predates the stream death, so skipping it would
// serve stale up to the next 60s reconcile (N7.1). An initial-seed collision
// never queues (there is only ever one).
if (!isInitial) _reseedQueued = true;
if (!isInitial)
{
_reseedQueued = true;
_queuedFanoutSuppressesNextTick |= suppressesNextTick;
}
return;
}
_fanoutInFlight = true;
_fanoutSuppressesNextTick = suppressesNextTick;
var self = Self;
var ct = _lifetimeCts?.Token ?? CancellationToken.None;
@@ -514,7 +550,9 @@ public sealed class SiteAlarmAggregatorActor : ReceiveActor, IWithTimers
var flushChanged = FlushBuffer();
_fanoutInFlight = false;
_fanoutSinceLastTick = true;
// Only a connect/failover-driven seed makes the next tick redundant — a tick's own
// fan-out must never suppress the tick after it (that halved the reconcile rate).
if (_fanoutSuppressesNextTick) _fanoutSinceLastTick = true;
_consecutiveSeedFailures = 0;
Timers.Cancel(SeedRetryTimerKey);
var firstSeed = !_seeded;
@@ -531,11 +569,21 @@ public sealed class SiteAlarmAggregatorActor : ReceiveActor, IWithTimers
Publish();
// A failover re-seed requested while this fan-out was in flight runs now (N7.1).
if (_reseedQueued)
{
_reseedQueued = false;
StartFanout(isInitial: false);
}
RunQueuedFanoutIfAny();
}
/// <summary>
/// Runs a fan-out that collided with an in-flight one, carrying the suppression flag it
/// was queued with so a queued connect/failover re-seed still stands the next tick down
/// (and a queued TICK fan-out still does not).
/// </summary>
private void RunQueuedFanoutIfAny()
{
if (!_reseedQueued) return;
_reseedQueued = false;
var suppresses = _queuedFanoutSuppressesNextTick;
_queuedFanoutSuppressesNextTick = false;
StartFanout(isInitial: false, suppressesNextTick: suppresses);
}
/// <summary>
@@ -570,7 +618,9 @@ public sealed class SiteAlarmAggregatorActor : ReceiveActor, IWithTimers
// Don't lose deltas captured during the failed window — apply them pass-through
// into the (possibly stale/empty) cache. The next reconcile re-seeds authoritatively.
_fanoutInFlight = false;
_fanoutSinceLastTick = true;
// Same rule as the success path: only a connect/failover-driven fan-out stands the
// next reconcile tick down.
if (_fanoutSuppressesNextTick) _fanoutSinceLastTick = true;
var flushChanged = FlushBuffer(dedupAgainstSeed: false);
// Only publish if we already had a seed (so IsLive doesn't flip true on a
@@ -585,8 +635,7 @@ public sealed class SiteAlarmAggregatorActor : ReceiveActor, IWithTimers
// A failover re-seed requested while this fan-out was in flight runs now (N7.1).
if (_reseedQueued)
{
_reseedQueued = false;
StartFanout(isInitial: false);
RunQueuedFanoutIfAny();
return;
}
@@ -794,7 +843,7 @@ public sealed class SiteAlarmAggregatorActor : ReceiveActor, IWithTimers
{
_seedOnConnect = false;
_log.Info("Site-alarm gRPC stream for {0} (re)connected; running one re-seed", _siteIdentifier);
StartFanout(isInitial: false);
StartFanout(isInitial: false, suppressesNextTick: true);
}
// Liveness recovered — republish so viewers stop falling back to polling even if the