dohertj2/ScadaBridge
1
0
Fork 0
Code Issues 2 Pull Requests Actions Packages Projects Releases Wiki Activity

fix(comms): review findings — consumer-based debug orphan net, foreign-cancel triad, honest onConnected, served-row-exact retirement, full-rate reconcile

Browse Source 
F1 (HIGH) DebugStreamBridgeActor: the 5-minute orphan net measured the MAILBOX
(SetReceiveTimeout), and once stream events were correctly marked
INotInfluenceReceiveTimeout nothing recurring reset it — the snapshot lands once
and GrpcStreamStable once — so every healthy session self-terminated at ~6 min
with a false "Site disconnected". Replaced with a periodic self-tick
(ConsumerLivenessCheckInterval, 30s) over a consumer-last-seen stamp renewed only
by DebugStreamConsumerAlive, which DebugStreamService Tells on a shared timer to
every session still in its registry (holding a session there IS "a consumer is
attached" — both the Blazor view and the SignalR hub release it on
dispose/disconnect, and it works headless). Reverting the wrapper was rejected: it
would restore the quiet-instance orphan bug.

F2 (MED) SiteStreamGrpcClient: the RpcException(Cancelled) filter now requires
cts.IsCancellationRequested. A peer-originated / channel-dispose Cancelled fired
none of onError/onCompleted/onConnected, leaving SiteAlarmAggregatorActor with
_streamDown=false forever (IsLive stuck true, reconcile reopen guard never fired).

F3 (MED) SiteStreamGrpcClient: a header TIMEOUT is no longer reported as
connected — that shape is exactly what an unreachable site produces, and it
cleared _streamDown, consumed _seedOnConnect and launched a full snapshot fan-out
at a dead site. AwaitHeadersAsync returns bool; the first received event is the
fallback connected signal, fired at most once from headers OR first event.

F4 (LOW-MED) SqliteAuditWriter.MarkReconciledUpToAsync: the blanket below-cursor
UPDATE retired late-stamped inserts that were never served (then age-purged —
silent loss). The flip is now bounded by insertion order: a Pending row retires
only if its rowid is at or below the high-water mark of rows this instance has
served from ReadPendingSinceAsync (clamped on purge, since SQLite reuses rowids);
Forwarded rows are exempt (central ACKed them over the push path). At-least-once
is unchanged.

F5 (LOW) Documented the liveness dependency (a served row never covered by a later
cursor stays Pending forever; PurgeExpiredAsync never purges Pending) in
ISiteAuditQueue + Component-AuditLog.md, and added a cheap site-health signal:
SiteAuditBacklogReporter logs a rate-limited warning when the existing
oldest-pending metric exceeds 24h.

F6 (MED) SiteAlarmAggregatorActor: _fanoutSinceLastTick was armed by the
reconcile's OWN fan-out, so steady state ran fan-out→skip→fan-out→skip — one
reconcile per 2x interval (120s), halving the not-reporting refresh and the alarm
reconcile backstop. The skip is now armed only by connect/failover-driven seeds
(initial, _seedOnConnect, and a re-seed queued behind one).

Tests: Communication.Tests 691 passed (+13), AuditLog.Tests 382 passed (+5).
This commit is contained in:
Joseph Doherty
2026-08-14 23:52:25 -04:00
parent b1de9dfdd4
commit fd5e023d08
16 changed files with 1192 additions and 84 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/ZB.MOM.WW.ScadaBridge.Communication/Actors/DebugStreamBridgeActor.cs
+95 -24
View File
@@ -47,6 +47,7 @@ public class DebugStreamBridgeActor : ReceiveActor, IWithTimers
private const string ReconnectTimerKey = "grpc-reconnect";
private const string StabilityTimerKey = "grpc-stability";
private const string SnapshotTimerKey = "debug-snapshot-deadline";
private const string ConsumerLivenessTimerKey = "debug-consumer-liveness";
/// <summary>Delay between gRPC reconnection attempts.</summary>
internal static TimeSpan ReconnectDelay { get; set; } = TimeSpan.FromSeconds(5);
@@ -69,6 +70,33 @@ public class DebugStreamBridgeActor : ReceiveActor, IWithTimers
/// </summary>
internal static TimeSpan StabilityWindow { get; set; } = TimeSpan.FromSeconds(60);
/// <summary>
/// Orphan window: how long the session may go without ANY sign of life from its CONSUMER
/// before it self-terminates. Renewed by <see cref="DebugStreamConsumerAlive"/>, which
/// <c>DebugStreamService</c> Tells on a timer for every session still attached to a
/// consumer (Blazor debug view or the SignalR hub) — so it measures the consumer, never
/// the stream. Settable for tests.
/// </summary>
internal static TimeSpan ConsumerIdleTimeout { get; set; } = TimeSpan.FromMinutes(5);
/// <summary>
/// How often the actor checks the consumer-last-seen stamp against
/// <see cref="ConsumerIdleTimeout"/>. A self-tick rather than <c>SetReceiveTimeout</c>:
/// the receive timeout measures the MAILBOX, which conflates site chatter with consumer
/// liveness — and once stream events were correctly excluded from it (via
/// <see cref="LiveDebugStreamEvent"/>) nothing recurring reset it at all, so every healthy
/// session self-terminated one window after its snapshot with a false "Site disconnected".
/// Settable for tests.
/// </summary>
internal static TimeSpan ConsumerLivenessCheckInterval { get; set; } = TimeSpan.FromSeconds(30);
/// <summary>
/// When the consumer was last known to be attached (UTC). Seeded in <see cref="PreStart"/>
/// so a session gets a full window to receive its first keepalive, then refreshed by every
/// <see cref="DebugStreamConsumerAlive"/>. Actor-thread only.
/// </summary>
private DateTime _consumerLastSeenUtc = DateTime.UtcNow;
private int _retryCount;
private bool _useNodeA = true;
private bool _stopped;
@@ -188,7 +216,7 @@ public class DebugStreamBridgeActor : ReceiveActor, IWithTimers
// non-deployed instance — cancel it (and any buffered gap events are
// discarded with the actor). No pass-through.
// _stopped is set AFTER CleanupGrpc() to match the ordering in the
// DebugStreamTerminated and ReceiveTimeout handlers (cosmetic consistency).
// DebugStreamTerminated and consumer-liveness handlers (cosmetic consistency).
CleanupGrpc();
_stopped = true;
_preSnapshotBuffer.Clear();
@@ -217,8 +245,8 @@ public class DebugStreamBridgeActor : ReceiveActor, IWithTimers
// Hard snapshot deadline (WP2.3). Nothing else ends a session stuck in the
// buffering phase: the site's reply was lost, so no gRPC error fires, the stream
// keeps delivering events, and (with the wrapper above) they no longer even reset
// the orphan timeout. Fail the session so the consumer is told and can reopen.
// keeps delivering events, and stream traffic does not renew the orphan net (which
// measures the consumer). Fail the session so the consumer is told and can reopen.
Receive<DebugSnapshotDeadline>(_ =>
{
if (_stopped || _snapshotDelivered) return;
@@ -234,21 +262,19 @@ public class DebugStreamBridgeActor : ReceiveActor, IWithTimers
Context.Stop(Self);
});
// Domain events arriving via Self.Tell from the gRPC callback, wrapped so they do
// NOT influence the receive timeout (WP2.3): the orphan safety net exists to end a
// session whose CONSUMER is gone, and a busy site's event flood used to keep that
// net permanently reset — an abandoned session on a chatty instance never timed out.
// Receiving an event must not reset _retryCount either: a flapping stream that
// delivers a single event between failures would otherwise never trip MaxRetries.
// The retry budget is recovered only by GrpcStreamStable (a stream that has stayed
// up for StabilityWindow). Before the snapshot has been delivered, BUFFER (in arrival
// order) rather than deliver — these may be gap-window events; after the snapshot has
// been flushed, pass through directly (phase-dependent behavior).
// Domain events arriving via Self.Tell from the gRPC callback. Stream traffic never
// proves the CONSUMER is still there, so it deliberately does not touch the orphan
// net (which now measures the consumer keepalive, not the mailbox). Receiving an
// event must not reset _retryCount either: a flapping stream that delivers a single
// event between failures would otherwise never trip MaxRetries. The retry budget is
// recovered only by GrpcStreamStable (a stream that has stayed up for
// StabilityWindow). Before the snapshot has been delivered, BUFFER (in arrival order)
// rather than deliver — these may be gap-window events; after the snapshot has been
// flushed, pass through directly (phase-dependent behavior).
Receive<LiveDebugStreamEvent>(wrapped => HandleStreamEvent(wrapped.Event));
// Unwrapped forms are still accepted (a direct Tell from a test or a future
// in-process producer); those DO influence the receive timeout, which is correct —
// they are not the high-volume stream path.
// in-process producer) and take the identical path.
Receive<AttributeValueChanged>(changed => HandleStreamEvent(changed));
Receive<AlarmStateChanged>(changed => HandleStreamEvent(changed));
@@ -315,11 +341,32 @@ public class DebugStreamBridgeActor : ReceiveActor, IWithTimers
Context.Stop(Self);
});
// Orphan safety net — if nobody stops us within 5 minutes, self-terminate
Context.SetReceiveTimeout(TimeSpan.FromMinutes(5));
Receive<ReceiveTimeout>(_ =>
// Consumer keepalive: DebugStreamService Tells this on a timer for every session it
// still holds (i.e. still attached to a Blazor debug view / SignalR connection). It
// is the ONLY thing that renews the orphan window — deliberately, so neither a chatty
// site nor a silent one can influence it.
Receive<DebugStreamConsumerAlive>(_ =>
{
_log.Warning("Debug stream for {0} timed out (orphaned session), stopping", _instanceUniqueName);
if (_stopped) return;
_consumerLastSeenUtc = DateTime.UtcNow;
});
// Orphan safety net, CONSUMER-measured (WP2.3 follow-up). A periodic self-tick
// compares the consumer-last-seen stamp against ConsumerIdleTimeout; the previous
// SetReceiveTimeout(5 min) measured the mailbox instead, and once stream events were
// (correctly) marked INotInfluenceReceiveTimeout nothing recurring reset it — the
// snapshot arrives once and GrpcStreamStable once, so EVERY healthy session died at
// ~6 minutes and the consumer was told "Site disconnected".
Receive<ConsumerLivenessTick>(_ =>
{
if (_stopped) return;
var idle = DateTime.UtcNow - _consumerLastSeenUtc;
if (idle < ConsumerIdleTimeout) return;
_log.Warning(
"Debug stream for {0} has had no consumer activity for {1:F0}s (orphaned session), stopping",
_instanceUniqueName, idle.TotalSeconds);
Timers.Cancel(ConsumerLivenessTimerKey);
CleanupGrpc();
SendUnsubscribe();
_stopped = true;
@@ -507,6 +554,15 @@ public class DebugStreamBridgeActor : ReceiveActor, IWithTimers
// Arm the hard snapshot deadline alongside the request.
if (SnapshotTimeout > TimeSpan.Zero)
Timers.StartSingleTimer(SnapshotTimerKey, new DebugSnapshotDeadline(), SnapshotTimeout);
// Arm the consumer-liveness net. The stamp starts now, so the session always gets a
// full ConsumerIdleTimeout to see its first keepalive from DebugStreamService.
_consumerLastSeenUtc = DateTime.UtcNow;
if (ConsumerIdleTimeout > TimeSpan.Zero && ConsumerLivenessCheckInterval > TimeSpan.Zero)
{
Timers.StartPeriodicTimer(
ConsumerLivenessTimerKey, new ConsumerLivenessTick(), ConsumerLivenessCheckInterval);
}
}
/// <inheritdoc />
@@ -546,7 +602,7 @@ public class DebugStreamBridgeActor : ReceiveActor, IWithTimers
await client.SubscribeAsync(
_correlationId,
_instanceUniqueName,
// Wrapped: stream traffic must not reset the orphan receive timeout.
// Wrapped so the stream path is explicit (it never renews the orphan net).
evt => self.Tell(new LiveDebugStreamEvent(evt)),
ex => self.Tell(new GrpcStreamError(ex, generation)),
() => self.Tell(new GrpcStreamCompleted(generation)),
@@ -660,11 +716,26 @@ public record StopDebugStream;
/// <summary>
/// Envelope for a live gRPC stream event (<c>AttributeValueChanged</c>/
/// <c>AlarmStateChanged</c>). Implements <see cref="INotInfluenceReceiveTimeout"/> so a busy
/// site's event flood cannot keep resetting the orphan-session receive timeout — the timeout
/// measures consumer/session liveness, not site chatter (WP2.3).
/// <c>AlarmStateChanged</c>). Kept as a distinct envelope so the high-volume stream path is
/// explicit at the call site; the orphan net no longer keys off the mailbox at all (it
/// measures the consumer keepalive), so a busy site's event flood can neither hold a dead
/// session open nor — as briefly happened — be the only thing keeping a healthy one alive.
/// </summary>
internal record LiveDebugStreamEvent(object Event) : INotInfluenceReceiveTimeout;
internal record LiveDebugStreamEvent(object Event);
/// <summary>
/// Consumer keepalive: <c>DebugStreamService</c> Tells one of these to every bridge actor
/// whose session is still attached to a consumer, on
/// <see cref="DebugStreamBridgeActor.ConsumerLivenessCheckInterval"/>-scale cadence. Renewing
/// the consumer-last-seen stamp is its ONLY effect.
/// </summary>
public record DebugStreamConsumerAlive;
/// <summary>
/// Internal self-tick that checks the consumer-last-seen stamp against
/// <see cref="DebugStreamBridgeActor.ConsumerIdleTimeout"/>.
/// </summary>
internal record ConsumerLivenessTick;
/// <summary>
/// Internal message: the hard deadline for the initial <c>DebugViewSnapshot</c> expired.
src/ZB.MOM.WW.ScadaBridge.Communication/Actors/SiteAlarmAggregatorActor.cs
+70 -21
View File
@@ -144,15 +144,38 @@ public sealed class SiteAlarmAggregatorActor : ReceiveActor, IWithTimers
private bool _seedOnConnect;
/// <summary>
/// Set whenever a fan-out finishes (success or failure); consumed by the next reconcile
/// tick, which skips its own fan-out when it finds the flag set. That makes the
/// connect-driven seed and the periodic backstop mutually exclusive — the pair used to
/// Set when a CONNECT- OR FAILOVER-DRIVEN fan-out finishes (the initial seed, a re-seed
/// consumed by a successful (re)connect, or a re-seed queued behind one); consumed by the
/// next reconcile tick, which skips its own fan-out when it finds the flag set. That makes
/// the connect-driven seed and the periodic backstop mutually exclusive — the pair used to
/// run BOTH, which is the "full unconditional snapshot every 60s" waste — while bounding
/// staleness at two intervals (a tick can be skipped at most once in a row).
/// <para>
/// A TICK-driven fan-out deliberately does NOT set it. It used to: the tick's own fan-out
/// completion armed the skip, so steady state ran fan-out → skip → fan-out → skip, i.e.
/// one reconcile per TWO intervals — halving the not-reporting refresh rate and the alarm
/// reconcile backstop for no reason (nothing was duplicated to suppress).
/// </para>
/// Actor-thread only.
/// </summary>
private bool _fanoutSinceLastTick;
/// <summary>
/// Whether the in-flight fan-out is connect/failover-driven and therefore makes the next
/// reconcile tick redundant (see <see cref="_fanoutSinceLastTick"/>). Stable for the
/// fan-out's lifetime: <see cref="StartFanout"/> never overwrites it while one is in
/// flight (a colliding request queues instead). Actor-thread only.
/// </summary>
private bool _fanoutSuppressesNextTick;
/// <summary>
/// Same flag for a fan-out that was queued behind an in-flight one
/// (<see cref="_reseedQueued"/>) — OR-ed across colliding requests so a queued
/// connect/failover re-seed keeps its suppression even if a tick collided too.
/// Actor-thread only.
/// </summary>
private bool _queuedFanoutSuppressesNextTick;
/// <summary>
/// True between issuing a stream open and its first outcome (connected / error /
/// completed), so a reconcile tick never stacks a second open on an in-flight one.
@@ -269,7 +292,9 @@ public sealed class SiteAlarmAggregatorActor : ReceiveActor, IWithTimers
Receive<RetrySeed>(_ =>
{
if (_stopped) return;
StartFanout(isInitial: false);
// A backoff retry of a failed fan-out is not a connect/failover seed: it must
// not stand the next reconcile tick down.
StartFanout(isInitial: false, suppressesNextTick: false);
});
// The site-wide stream is confirmed established (response headers received). This —
@@ -356,7 +381,7 @@ public sealed class SiteAlarmAggregatorActor : ReceiveActor, IWithTimers
// Kick the initial seed fan-out. The connect callback for this first stream must
// NOT seed again on top of it, so the flag starts cleared.
_seedOnConnect = false;
StartFanout(isInitial: true);
StartFanout(isInitial: true, suppressesNextTick: true);
// Periodic reconcile backstop. Single-shot and re-armed with fresh jitter each
// tick (a periodic timer would lock every site to the same phase forever).
@@ -398,9 +423,11 @@ public sealed class SiteAlarmAggregatorActor : ReceiveActor, IWithTimers
/// <summary>
/// Periodic reconcile backstop: re-run the snapshot fan-out (corrects instance-set drift
/// + any missed delta) UNLESS one already completed inside this window — a reconnect-driven
/// seed makes the tick redundant, and running both was the "full unconditional snapshot
/// every 60s" waste (WP2.3). If the live stream was previously given up, self-heal it by
/// + any missed delta) UNLESS a CONNECT- OR FAILOVER-DRIVEN seed already completed inside
/// this window — that seed makes the tick redundant, and running both was the "full
/// unconditional snapshot every 60s" waste (WP2.3). A tick's OWN fan-out never arms that
/// skip: it used to, which made steady state alternate fan-out/skip and halved the
/// effective reconcile rate. If the live stream was previously given up, self-heal it by
/// resetting the retry budget and reopening, so a sustained outage never permanently kills
/// the live feed.
/// </summary>
@@ -441,7 +468,11 @@ public sealed class SiteAlarmAggregatorActor : ReceiveActor, IWithTimers
}
if (!reopening)
StartFanout(isInitial: false);
{
// Tick-driven: deliberately does NOT set the skip flag, so the NEXT tick fans
// out too. Setting it here was the "one reconcile per two intervals" bug.
StartFanout(isInitial: false, suppressesNextTick: false);
}
}
// ── Seed / reconcile fan-out ────────────────────────────────────────────────
@@ -451,7 +482,7 @@ public sealed class SiteAlarmAggregatorActor : ReceiveActor, IWithTimers
/// <c>Self.Tell</c>. While in flight, live deltas buffer. A reconcile that arrives
/// while a fan-out is already running is skipped (no stacking).
/// </summary>
private void StartFanout(bool isInitial)
private void StartFanout(bool isInitial, bool suppressesNextTick)
{
if (_stopped) return;
if (_fanoutInFlight)
@@ -460,11 +491,16 @@ public sealed class SiteAlarmAggregatorActor : ReceiveActor, IWithTimers
// one — its snapshot read-time predates the stream death, so skipping it would
// serve stale up to the next 60s reconcile (N7.1). An initial-seed collision
// never queues (there is only ever one).
if (!isInitial) _reseedQueued = true;
if (!isInitial)
{
_reseedQueued = true;
_queuedFanoutSuppressesNextTick |= suppressesNextTick;
}
return;
}
_fanoutInFlight = true;
_fanoutSuppressesNextTick = suppressesNextTick;
var self = Self;
var ct = _lifetimeCts?.Token ?? CancellationToken.None;
@@ -514,7 +550,9 @@ public sealed class SiteAlarmAggregatorActor : ReceiveActor, IWithTimers
var flushChanged = FlushBuffer();
_fanoutInFlight = false;
_fanoutSinceLastTick = true;
// Only a connect/failover-driven seed makes the next tick redundant — a tick's own
// fan-out must never suppress the tick after it (that halved the reconcile rate).
if (_fanoutSuppressesNextTick) _fanoutSinceLastTick = true;
_consecutiveSeedFailures = 0;
Timers.Cancel(SeedRetryTimerKey);
var firstSeed = !_seeded;
@@ -531,11 +569,21 @@ public sealed class SiteAlarmAggregatorActor : ReceiveActor, IWithTimers
Publish();
// A failover re-seed requested while this fan-out was in flight runs now (N7.1).
if (_reseedQueued)
{
_reseedQueued = false;
StartFanout(isInitial: false);
}
RunQueuedFanoutIfAny();
}
/// <summary>
/// Runs a fan-out that collided with an in-flight one, carrying the suppression flag it
/// was queued with so a queued connect/failover re-seed still stands the next tick down
/// (and a queued TICK fan-out still does not).
/// </summary>
private void RunQueuedFanoutIfAny()
{
if (!_reseedQueued) return;
_reseedQueued = false;
var suppresses = _queuedFanoutSuppressesNextTick;
_queuedFanoutSuppressesNextTick = false;
StartFanout(isInitial: false, suppressesNextTick: suppresses);
}
/// <summary>
@@ -570,7 +618,9 @@ public sealed class SiteAlarmAggregatorActor : ReceiveActor, IWithTimers
// Don't lose deltas captured during the failed window — apply them pass-through
// into the (possibly stale/empty) cache. The next reconcile re-seeds authoritatively.
_fanoutInFlight = false;
_fanoutSinceLastTick = true;
// Same rule as the success path: only a connect/failover-driven fan-out stands the
// next reconcile tick down.
if (_fanoutSuppressesNextTick) _fanoutSinceLastTick = true;
var flushChanged = FlushBuffer(dedupAgainstSeed: false);
// Only publish if we already had a seed (so IsLive doesn't flip true on a
@@ -585,8 +635,7 @@ public sealed class SiteAlarmAggregatorActor : ReceiveActor, IWithTimers
// A failover re-seed requested while this fan-out was in flight runs now (N7.1).
if (_reseedQueued)
{
_reseedQueued = false;
StartFanout(isInitial: false);
RunQueuedFanoutIfAny();
return;
}
@@ -794,7 +843,7 @@ public sealed class SiteAlarmAggregatorActor : ReceiveActor, IWithTimers
{
_seedOnConnect = false;
_log.Info("Site-alarm gRPC stream for {0} (re)connected; running one re-seed", _siteIdentifier);
StartFanout(isInitial: false);
StartFanout(isInitial: false, suppressesNextTick: true);
}
// Liveness recovered — republish so viewers stop falling back to polling even if the