dohertj2/ScadaBridge
1
0
Fork 0
Code Issues 2 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

fix(comms): review findings — consumer-based debug orphan net, foreign-cancel triad, honest onConnected, served-row-exact retirement, full-rate reconcile

Browse Source 
F1 (HIGH) DebugStreamBridgeActor: the 5-minute orphan net measured the MAILBOX
(SetReceiveTimeout), and once stream events were correctly marked
INotInfluenceReceiveTimeout nothing recurring reset it — the snapshot lands once
and GrpcStreamStable once — so every healthy session self-terminated at ~6 min
with a false "Site disconnected". Replaced with a periodic self-tick
(ConsumerLivenessCheckInterval, 30s) over a consumer-last-seen stamp renewed only
by DebugStreamConsumerAlive, which DebugStreamService Tells on a shared timer to
every session still in its registry (holding a session there IS "a consumer is
attached" — both the Blazor view and the SignalR hub release it on
dispose/disconnect, and it works headless). Reverting the wrapper was rejected: it
would restore the quiet-instance orphan bug.

F2 (MED) SiteStreamGrpcClient: the RpcException(Cancelled) filter now requires
cts.IsCancellationRequested. A peer-originated / channel-dispose Cancelled fired
none of onError/onCompleted/onConnected, leaving SiteAlarmAggregatorActor with
_streamDown=false forever (IsLive stuck true, reconcile reopen guard never fired).

F3 (MED) SiteStreamGrpcClient: a header TIMEOUT is no longer reported as
connected — that shape is exactly what an unreachable site produces, and it
cleared _streamDown, consumed _seedOnConnect and launched a full snapshot fan-out
at a dead site. AwaitHeadersAsync returns bool; the first received event is the
fallback connected signal, fired at most once from headers OR first event.

F4 (LOW-MED) SqliteAuditWriter.MarkReconciledUpToAsync: the blanket below-cursor
UPDATE retired late-stamped inserts that were never served (then age-purged —
silent loss). The flip is now bounded by insertion order: a Pending row retires
only if its rowid is at or below the high-water mark of rows this instance has
served from ReadPendingSinceAsync (clamped on purge, since SQLite reuses rowids);
Forwarded rows are exempt (central ACKed them over the push path). At-least-once
is unchanged.

F5 (LOW) Documented the liveness dependency (a served row never covered by a later
cursor stays Pending forever; PurgeExpiredAsync never purges Pending) in
ISiteAuditQueue + Component-AuditLog.md, and added a cheap site-health signal:
SiteAuditBacklogReporter logs a rate-limited warning when the existing
oldest-pending metric exceeds 24h.

F6 (MED) SiteAlarmAggregatorActor: _fanoutSinceLastTick was armed by the
reconcile's OWN fan-out, so steady state ran fan-out→skip→fan-out→skip — one
reconcile per 2x interval (120s), halving the not-reporting refresh and the alarm
reconcile backstop. The skip is now armed only by connect/failover-driven seeds
(initial, _seedOnConnect, and a re-seed queued behind one).

Tests: Communication.Tests 691 passed (+13), AuditLog.Tests 382 passed (+5).
This commit is contained in:
Joseph Doherty
2026-08-14 23:52:25 -04:00
parent b1de9dfdd4
commit fd5e023d08
16 changed files with 1192 additions and 84 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/Grpc/DebugStreamBridgeActorTests.cs
+139 -9
View File
@@ -11,6 +11,13 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Tests.Grpc;
/// <summary>
/// Tests for DebugStreamBridgeActor with gRPC streaming integration.
/// </summary>
/// <remarks>
/// Shares the <c>DebugStreamStatics</c> xUnit collection with <c>DebugStreamServiceTests</c>:
/// both tune the actor's static test seams (<c>SnapshotTimeout</c>, <c>ConsumerIdleTimeout</c>,
/// …), and xUnit parallelizes distinct classes by default, which would let one class's
/// try/finally restore clobber the other's window mid-test.
/// </remarks>
[Collection("DebugStreamStatics")]
public class DebugStreamBridgeActorTests : TestKit
{
private const string SiteId = "site-alpha";
@@ -1034,14 +1041,11 @@ public class DebugStreamBridgeActorTests : TestKit
}
[Fact]
public void StreamEvents_AreWrapped_SoTheyDoNotResetTheOrphanReceiveTimeout()
public void StreamEvents_StillReachTheConsumer_ThroughTheWrappedCallbackPath()
{
// Structural pin for the fix: the gRPC callback wraps every event in an envelope
// marked INotInfluenceReceiveTimeout, so a busy site can no longer keep an
// abandoned session alive indefinitely by feeding it events.
Assert.True(typeof(Akka.Actor.INotInfluenceReceiveTimeout)
.IsAssignableFrom(typeof(LiveDebugStreamEvent)));
// The gRPC callback wraps every event in LiveDebugStreamEvent; the wrapper must be
// transparent to delivery (it exists only to keep the high-volume path explicit —
// the orphan net keys off the consumer keepalive, not the mailbox).
var ctx = CreateBridgeActor();
ctx.CommProbe.ExpectMsg<SiteEnvelope>();
AwaitCondition(() => ctx.MockGrpcClient.SubscribeCalls.Count == 1, TimeSpan.FromSeconds(3));
@@ -1052,8 +1056,6 @@ public class DebugStreamBridgeActorTests : TestKit
new List<AlarmStateChanged>(),
DateTimeOffset.UtcNow));
// An event delivered through the real gRPC callback path still reaches the
// consumer — the wrapper is transparent to delivery.
var evt = new AttributeValueChanged(
InstanceName, "Modules.IO", "Temperature", 42.5, "Good", DateTimeOffset.UtcNow);
ctx.MockGrpcClient.SubscribeCalls[0].OnEvent(evt);
@@ -1063,6 +1065,134 @@ public class DebugStreamBridgeActorTests : TestKit
lock (ctx.ReceivedEvents) { return ctx.ReceivedEvents.OfType<AttributeValueChanged>().Any(); }
}, TimeSpan.FromSeconds(3));
}
// ----- Orphan net: measures the CONSUMER, not the mailbox ----- //
[Fact]
public void HealthySession_StreamingEvents_WithConsumerKeepalives_SurvivesWellPastTheOrphanWindow()
{
// THE regression this closes: with the orphan net armed off the mailbox
// (SetReceiveTimeout) and stream events correctly excluded from it, nothing recurring
// reset it — the snapshot lands once, GrpcStreamStable once — so a perfectly healthy
// session self-terminated one window later and the consumer was told
// "Site disconnected". Here the session streams events and receives the keepalive
// DebugStreamService sends while it is attached; it must live through MANY windows.
DebugStreamBridgeActor.ConsumerIdleTimeout = TimeSpan.FromMilliseconds(400);
DebugStreamBridgeActor.ConsumerLivenessCheckInterval = TimeSpan.FromMilliseconds(50);
try
{
var ctx = CreateBridgeActor();
ctx.CommProbe.ExpectMsg<SiteEnvelope>();
AwaitCondition(() => ctx.MockGrpcClient.SubscribeCalls.Count == 1, TimeSpan.FromSeconds(3));
Watch(ctx.BridgeActor);
ctx.BridgeActor.Tell(new DebugViewSnapshot(
InstanceName,
new List<AttributeValueChanged>(),
new List<AlarmStateChanged>(),
DateTimeOffset.UtcNow));
// ~5 orphan windows of pure stream traffic + consumer keepalives, and no other
// mailbox activity whatsoever (no reconnects, no snapshots, no stop).
var deadline = DateTime.UtcNow.AddSeconds(2);
var delivered = 0;
while (DateTime.UtcNow < deadline)
{
ctx.MockGrpcClient.SubscribeCalls[0].OnEvent(new AttributeValueChanged(
InstanceName, "Modules.IO", "Temperature", 20.0 + delivered, "Good",
DateTimeOffset.UtcNow));
delivered++;
// What DebugStreamService's shared timer does for an attached session.
ctx.BridgeActor.Tell(new DebugStreamConsumerAlive());
Thread.Sleep(100);
}
ExpectNoMsg(TimeSpan.FromMilliseconds(100));
Assert.False(ctx.TerminatedFlag[0]);
// Still serving: the events all arrived and the actor is alive.
AwaitCondition(() =>
{
lock (ctx.ReceivedEvents) { return ctx.ReceivedEvents.OfType<AttributeValueChanged>().Count() == delivered; }
}, TimeSpan.FromSeconds(3));
Assert.False(ctx.BridgeActor.IsNobody());
}
finally
{
DebugStreamBridgeActor.ConsumerIdleTimeout = TimeSpan.FromMinutes(5);
DebugStreamBridgeActor.ConsumerLivenessCheckInterval = TimeSpan.FromSeconds(30);
}
}
[Fact]
public void OrphanedSession_ConsumerGone_StillTerminates_EvenWhileEventsKeepArriving()
{
// The other half of the contract: site chatter must NOT hold an abandoned session
// open. No keepalive arrives (the consumer is gone), so the session terminates,
// unsubscribes from the site and reports termination — while events stream in.
DebugStreamBridgeActor.ConsumerIdleTimeout = TimeSpan.FromMilliseconds(400);
DebugStreamBridgeActor.ConsumerLivenessCheckInterval = TimeSpan.FromMilliseconds(50);
try
{
var ctx = CreateBridgeActor();
ctx.CommProbe.ExpectMsg<SiteEnvelope>();
AwaitCondition(() => ctx.MockGrpcClient.SubscribeCalls.Count == 1, TimeSpan.FromSeconds(3));
Watch(ctx.BridgeActor);
ctx.BridgeActor.Tell(new DebugViewSnapshot(
InstanceName,
new List<AttributeValueChanged>(),
new List<AlarmStateChanged>(),
DateTimeOffset.UtcNow));
// Keep the site chatty for longer than the orphan window — with no keepalive.
var deadline = DateTime.UtcNow.AddSeconds(1);
while (DateTime.UtcNow < deadline)
{
ctx.MockGrpcClient.SubscribeCalls[0].OnEvent(new AttributeValueChanged(
InstanceName, "Modules.IO", "Temperature", 42.5, "Good", DateTimeOffset.UtcNow));
Thread.Sleep(50);
}
ExpectTerminated(ctx.BridgeActor, TimeSpan.FromSeconds(3));
Assert.True(ctx.TerminatedFlag[0]);
Assert.Contains("corr-1", ctx.MockGrpcClient.UnsubscribedCorrelationIds);
}
finally
{
DebugStreamBridgeActor.ConsumerIdleTimeout = TimeSpan.FromMinutes(5);
DebugStreamBridgeActor.ConsumerLivenessCheckInterval = TimeSpan.FromSeconds(30);
}
}
[Fact]
public void ConsumerKeepalive_RenewsTheWindow_AfterANearMiss()
{
// A single late keepalive is enough to save a session — the stamp is refreshed, not
// a one-shot grace period.
DebugStreamBridgeActor.ConsumerIdleTimeout = TimeSpan.FromMilliseconds(500);
DebugStreamBridgeActor.ConsumerLivenessCheckInterval = TimeSpan.FromMilliseconds(50);
try
{
var ctx = CreateBridgeActor();
ctx.CommProbe.ExpectMsg<SiteEnvelope>();
Watch(ctx.BridgeActor);
Thread.Sleep(350); // most of the window burnt
ctx.BridgeActor.Tell(new DebugStreamConsumerAlive()); // …then the consumer checks in
Thread.Sleep(350); // past the ORIGINAL deadline
Assert.False(ctx.TerminatedFlag[0]);
// …and once the keepalives stop, it does terminate.
ExpectTerminated(ctx.BridgeActor, TimeSpan.FromSeconds(3));
}
finally
{
DebugStreamBridgeActor.ConsumerIdleTimeout = TimeSpan.FromMinutes(5);
DebugStreamBridgeActor.ConsumerLivenessCheckInterval = TimeSpan.FromSeconds(30);
}
}
}
/// <summary>