dohertj2/ScadaBridge
1
0
Fork 0
Code Issues 2 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

fix(comms): review findings — consumer-based debug orphan net, foreign-cancel triad, honest onConnected, served-row-exact retirement, full-rate reconcile

Browse Source 
F1 (HIGH) DebugStreamBridgeActor: the 5-minute orphan net measured the MAILBOX
(SetReceiveTimeout), and once stream events were correctly marked
INotInfluenceReceiveTimeout nothing recurring reset it — the snapshot lands once
and GrpcStreamStable once — so every healthy session self-terminated at ~6 min
with a false "Site disconnected". Replaced with a periodic self-tick
(ConsumerLivenessCheckInterval, 30s) over a consumer-last-seen stamp renewed only
by DebugStreamConsumerAlive, which DebugStreamService Tells on a shared timer to
every session still in its registry (holding a session there IS "a consumer is
attached" — both the Blazor view and the SignalR hub release it on
dispose/disconnect, and it works headless). Reverting the wrapper was rejected: it
would restore the quiet-instance orphan bug.

F2 (MED) SiteStreamGrpcClient: the RpcException(Cancelled) filter now requires
cts.IsCancellationRequested. A peer-originated / channel-dispose Cancelled fired
none of onError/onCompleted/onConnected, leaving SiteAlarmAggregatorActor with
_streamDown=false forever (IsLive stuck true, reconcile reopen guard never fired).

F3 (MED) SiteStreamGrpcClient: a header TIMEOUT is no longer reported as
connected — that shape is exactly what an unreachable site produces, and it
cleared _streamDown, consumed _seedOnConnect and launched a full snapshot fan-out
at a dead site. AwaitHeadersAsync returns bool; the first received event is the
fallback connected signal, fired at most once from headers OR first event.

F4 (LOW-MED) SqliteAuditWriter.MarkReconciledUpToAsync: the blanket below-cursor
UPDATE retired late-stamped inserts that were never served (then age-purged —
silent loss). The flip is now bounded by insertion order: a Pending row retires
only if its rowid is at or below the high-water mark of rows this instance has
served from ReadPendingSinceAsync (clamped on purge, since SQLite reuses rowids);
Forwarded rows are exempt (central ACKed them over the push path). At-least-once
is unchanged.

F5 (LOW) Documented the liveness dependency (a served row never covered by a later
cursor stays Pending forever; PurgeExpiredAsync never purges Pending) in
ISiteAuditQueue + Component-AuditLog.md, and added a cheap site-health signal:
SiteAuditBacklogReporter logs a rate-limited warning when the existing
oldest-pending metric exceeds 24h.

F6 (MED) SiteAlarmAggregatorActor: _fanoutSinceLastTick was armed by the
reconcile's OWN fan-out, so steady state ran fan-out→skip→fan-out→skip — one
reconcile per 2x interval (120s), halving the not-reporting refresh and the alarm
reconcile backstop. The skip is now armed only by connect/failover-driven seeds
(initial, _seedOnConnect, and a re-seed queued behind one).

Tests: Communication.Tests 691 passed (+13), AuditLog.Tests 382 passed (+5).
This commit is contained in:
Joseph Doherty
2026-08-14 23:52:25 -04:00
parent b1de9dfdd4
commit fd5e023d08
16 changed files with 1192 additions and 84 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/ZB.MOM.WW.ScadaBridge.Communication/Actors/DebugStreamBridgeActor.cs
+95 -24
View File
@@ -47,6 +47,7 @@ public class DebugStreamBridgeActor : ReceiveActor, IWithTimers
private const string ReconnectTimerKey = "grpc-reconnect";
private const string StabilityTimerKey = "grpc-stability";
private const string SnapshotTimerKey = "debug-snapshot-deadline";
private const string ConsumerLivenessTimerKey = "debug-consumer-liveness";
/// <summary>Delay between gRPC reconnection attempts.</summary>
internal static TimeSpan ReconnectDelay { get; set; } = TimeSpan.FromSeconds(5);
@@ -69,6 +70,33 @@ public class DebugStreamBridgeActor : ReceiveActor, IWithTimers
/// </summary>
internal static TimeSpan StabilityWindow { get; set; } = TimeSpan.FromSeconds(60);
/// <summary>
/// Orphan window: how long the session may go without ANY sign of life from its CONSUMER
/// before it self-terminates. Renewed by <see cref="DebugStreamConsumerAlive"/>, which
/// <c>DebugStreamService</c> Tells on a timer for every session still attached to a
/// consumer (Blazor debug view or the SignalR hub) — so it measures the consumer, never
/// the stream. Settable for tests.
/// </summary>
internal static TimeSpan ConsumerIdleTimeout { get; set; } = TimeSpan.FromMinutes(5);
/// <summary>
/// How often the actor checks the consumer-last-seen stamp against
/// <see cref="ConsumerIdleTimeout"/>. A self-tick rather than <c>SetReceiveTimeout</c>:
/// the receive timeout measures the MAILBOX, which conflates site chatter with consumer
/// liveness — and once stream events were correctly excluded from it (via
/// <see cref="LiveDebugStreamEvent"/>) nothing recurring reset it at all, so every healthy
/// session self-terminated one window after its snapshot with a false "Site disconnected".
/// Settable for tests.
/// </summary>
internal static TimeSpan ConsumerLivenessCheckInterval { get; set; } = TimeSpan.FromSeconds(30);
/// <summary>
/// When the consumer was last known to be attached (UTC). Seeded in <see cref="PreStart"/>
/// so a session gets a full window to receive its first keepalive, then refreshed by every
/// <see cref="DebugStreamConsumerAlive"/>. Actor-thread only.
/// </summary>
private DateTime _consumerLastSeenUtc = DateTime.UtcNow;
private int _retryCount;
private bool _useNodeA = true;
private bool _stopped;
@@ -188,7 +216,7 @@ public class DebugStreamBridgeActor : ReceiveActor, IWithTimers
// non-deployed instance — cancel it (and any buffered gap events are
// discarded with the actor). No pass-through.
// _stopped is set AFTER CleanupGrpc() to match the ordering in the
// DebugStreamTerminated and ReceiveTimeout handlers (cosmetic consistency).
// DebugStreamTerminated and consumer-liveness handlers (cosmetic consistency).
CleanupGrpc();
_stopped = true;
_preSnapshotBuffer.Clear();
@@ -217,8 +245,8 @@ public class DebugStreamBridgeActor : ReceiveActor, IWithTimers
// Hard snapshot deadline (WP2.3). Nothing else ends a session stuck in the
// buffering phase: the site's reply was lost, so no gRPC error fires, the stream
// keeps delivering events, and (with the wrapper above) they no longer even reset
// the orphan timeout. Fail the session so the consumer is told and can reopen.
// keeps delivering events, and stream traffic does not renew the orphan net (which
// measures the consumer). Fail the session so the consumer is told and can reopen.
Receive<DebugSnapshotDeadline>(_ =>
{
if (_stopped || _snapshotDelivered) return;
@@ -234,21 +262,19 @@ public class DebugStreamBridgeActor : ReceiveActor, IWithTimers
Context.Stop(Self);
});
// Domain events arriving via Self.Tell from the gRPC callback, wrapped so they do
// NOT influence the receive timeout (WP2.3): the orphan safety net exists to end a
// session whose CONSUMER is gone, and a busy site's event flood used to keep that
// net permanently reset — an abandoned session on a chatty instance never timed out.
// Receiving an event must not reset _retryCount either: a flapping stream that
// delivers a single event between failures would otherwise never trip MaxRetries.
// The retry budget is recovered only by GrpcStreamStable (a stream that has stayed
// up for StabilityWindow). Before the snapshot has been delivered, BUFFER (in arrival
// order) rather than deliver — these may be gap-window events; after the snapshot has
// been flushed, pass through directly (phase-dependent behavior).
// Domain events arriving via Self.Tell from the gRPC callback. Stream traffic never
// proves the CONSUMER is still there, so it deliberately does not touch the orphan
// net (which now measures the consumer keepalive, not the mailbox). Receiving an
// event must not reset _retryCount either: a flapping stream that delivers a single
// event between failures would otherwise never trip MaxRetries. The retry budget is
// recovered only by GrpcStreamStable (a stream that has stayed up for
// StabilityWindow). Before the snapshot has been delivered, BUFFER (in arrival order)
// rather than deliver — these may be gap-window events; after the snapshot has been
// flushed, pass through directly (phase-dependent behavior).
Receive<LiveDebugStreamEvent>(wrapped => HandleStreamEvent(wrapped.Event));
// Unwrapped forms are still accepted (a direct Tell from a test or a future
// in-process producer); those DO influence the receive timeout, which is correct —
// they are not the high-volume stream path.
// in-process producer) and take the identical path.
Receive<AttributeValueChanged>(changed => HandleStreamEvent(changed));
Receive<AlarmStateChanged>(changed => HandleStreamEvent(changed));
@@ -315,11 +341,32 @@ public class DebugStreamBridgeActor : ReceiveActor, IWithTimers
Context.Stop(Self);
});
// Orphan safety net — if nobody stops us within 5 minutes, self-terminate
Context.SetReceiveTimeout(TimeSpan.FromMinutes(5));
Receive<ReceiveTimeout>(_ =>
// Consumer keepalive: DebugStreamService Tells this on a timer for every session it
// still holds (i.e. still attached to a Blazor debug view / SignalR connection). It
// is the ONLY thing that renews the orphan window — deliberately, so neither a chatty
// site nor a silent one can influence it.
Receive<DebugStreamConsumerAlive>(_ =>
{
_log.Warning("Debug stream for {0} timed out (orphaned session), stopping", _instanceUniqueName);
if (_stopped) return;
_consumerLastSeenUtc = DateTime.UtcNow;
});
// Orphan safety net, CONSUMER-measured (WP2.3 follow-up). A periodic self-tick
// compares the consumer-last-seen stamp against ConsumerIdleTimeout; the previous
// SetReceiveTimeout(5 min) measured the mailbox instead, and once stream events were
// (correctly) marked INotInfluenceReceiveTimeout nothing recurring reset it — the
// snapshot arrives once and GrpcStreamStable once, so EVERY healthy session died at
// ~6 minutes and the consumer was told "Site disconnected".
Receive<ConsumerLivenessTick>(_ =>
{
if (_stopped) return;
var idle = DateTime.UtcNow - _consumerLastSeenUtc;
if (idle < ConsumerIdleTimeout) return;
_log.Warning(
"Debug stream for {0} has had no consumer activity for {1:F0}s (orphaned session), stopping",
_instanceUniqueName, idle.TotalSeconds);
Timers.Cancel(ConsumerLivenessTimerKey);
CleanupGrpc();
SendUnsubscribe();
_stopped = true;
@@ -507,6 +554,15 @@ public class DebugStreamBridgeActor : ReceiveActor, IWithTimers
// Arm the hard snapshot deadline alongside the request.
if (SnapshotTimeout > TimeSpan.Zero)
Timers.StartSingleTimer(SnapshotTimerKey, new DebugSnapshotDeadline(), SnapshotTimeout);
// Arm the consumer-liveness net. The stamp starts now, so the session always gets a
// full ConsumerIdleTimeout to see its first keepalive from DebugStreamService.
_consumerLastSeenUtc = DateTime.UtcNow;
if (ConsumerIdleTimeout > TimeSpan.Zero && ConsumerLivenessCheckInterval > TimeSpan.Zero)
{
Timers.StartPeriodicTimer(
ConsumerLivenessTimerKey, new ConsumerLivenessTick(), ConsumerLivenessCheckInterval);
}
}
/// <inheritdoc />
@@ -546,7 +602,7 @@ public class DebugStreamBridgeActor : ReceiveActor, IWithTimers
await client.SubscribeAsync(
_correlationId,
_instanceUniqueName,
// Wrapped: stream traffic must not reset the orphan receive timeout.
// Wrapped so the stream path is explicit (it never renews the orphan net).
evt => self.Tell(new LiveDebugStreamEvent(evt)),
ex => self.Tell(new GrpcStreamError(ex, generation)),
() => self.Tell(new GrpcStreamCompleted(generation)),
@@ -660,11 +716,26 @@ public record StopDebugStream;
/// <summary>
/// Envelope for a live gRPC stream event (<c>AttributeValueChanged</c>/
/// <c>AlarmStateChanged</c>). Implements <see cref="INotInfluenceReceiveTimeout"/> so a busy
/// site's event flood cannot keep resetting the orphan-session receive timeout — the timeout
/// measures consumer/session liveness, not site chatter (WP2.3).
/// <c>AlarmStateChanged</c>). Kept as a distinct envelope so the high-volume stream path is
/// explicit at the call site; the orphan net no longer keys off the mailbox at all (it
/// measures the consumer keepalive), so a busy site's event flood can neither hold a dead
/// session open nor — as briefly happened — be the only thing keeping a healthy one alive.
/// </summary>
internal record LiveDebugStreamEvent(object Event) : INotInfluenceReceiveTimeout;
internal record LiveDebugStreamEvent(object Event);
/// <summary>
/// Consumer keepalive: <c>DebugStreamService</c> Tells one of these to every bridge actor
/// whose session is still attached to a consumer, on
/// <see cref="DebugStreamBridgeActor.ConsumerLivenessCheckInterval"/>-scale cadence. Renewing
/// the consumer-last-seen stamp is its ONLY effect.
/// </summary>
public record DebugStreamConsumerAlive;
/// <summary>
/// Internal self-tick that checks the consumer-last-seen stamp against
/// <see cref="DebugStreamBridgeActor.ConsumerIdleTimeout"/>.
/// </summary>
internal record ConsumerLivenessTick;
/// <summary>
/// Internal message: the hard deadline for the initial <c>DebugViewSnapshot</c> expired.