fix(comms): review findings — consumer-based debug orphan net, foreign-cancel triad, honest onConnected, served-row-exact retirement, full-rate reconcile

F1 (HIGH) DebugStreamBridgeActor: the 5-minute orphan net measured the MAILBOX (SetReceiveTimeout), and once stream events were correctly marked INotInfluenceReceiveTimeout nothing recurring reset it — the snapshot lands once and GrpcStreamStable once — so every healthy session self-terminated at ~6 min with a false "Site disconnected". Replaced with a periodic self-tick (ConsumerLivenessCheckInterval, 30s) over a consumer-last-seen stamp renewed only by DebugStreamConsumerAlive, which DebugStreamService Tells on a shared timer to every session still in its registry (holding a session there IS "a consumer is attached" — both the Blazor view and the SignalR hub release it on dispose/disconnect, and it works headless). Reverting the wrapper was rejected: it would restore the quiet-instance orphan bug. F2 (MED) SiteStreamGrpcClient: the RpcException(Cancelled) filter now requires cts.IsCancellationRequested. A peer-originated / channel-dispose Cancelled fired none of onError/onCompleted/onConnected, leaving SiteAlarmAggregatorActor with _streamDown=false forever (IsLive stuck true, reconcile reopen guard never fired). F3 (MED) SiteStreamGrpcClient: a header TIMEOUT is no longer reported as connected — that shape is exactly what an unreachable site produces, and it cleared _streamDown, consumed _seedOnConnect and launched a full snapshot fan-out at a dead site. AwaitHeadersAsync returns bool; the first received event is the fallback connected signal, fired at most once from headers OR first event. F4 (LOW-MED) SqliteAuditWriter.MarkReconciledUpToAsync: the blanket below-cursor UPDATE retired late-stamped inserts that were never served (then age-purged — silent loss). The flip is now bounded by insertion order: a Pending row retires only if its rowid is at or below the high-water mark of rows this instance has served from ReadPendingSinceAsync (clamped on purge, since SQLite reuses rowids); Forwarded rows are exempt (central ACKed them over the push path). At-least-once is unchanged. F5 (LOW) Documented the liveness dependency (a served row never covered by a later cursor stays Pending forever; PurgeExpiredAsync never purges Pending) in ISiteAuditQueue + Component-AuditLog.md, and added a cheap site-health signal: SiteAuditBacklogReporter logs a rate-limited warning when the existing oldest-pending metric exceeds 24h. F6 (MED) SiteAlarmAggregatorActor: _fanoutSinceLastTick was armed by the reconcile's OWN fan-out, so steady state ran fan-out→skip→fan-out→skip — one reconcile per 2x interval (120s), halving the not-reporting refresh and the alarm reconcile backstop. The skip is now armed only by connect/failover-driven seeds (initial, _seedOnConnect, and a re-seed queued behind one). Tests: Communication.Tests 691 passed (+13), AuditLog.Tests 382 passed (+5).
2026-08-14 23:52:25 -04:00
parent b1de9dfdd4
commit fd5e023d08
16 changed files with 1192 additions and 84 deletions
@@ -243,6 +243,45 @@ public class SiteStreamPullAuditEventsTests : TestKit
        Assert.True(response.MoreAvailable);
    }

+    [Fact]
+    public async Task PullAuditEvents_RetiresBeforeItServes_SoThisBatchIsNeverSelfRetired()
+    {
+        // Ordering is load-bearing for the "only served rows retire" invariant. The queue
+        // bounds the cursor flip by insertion order — the high-water mark of rows it has
+        // SERVED — so the retire step must run BEFORE the read: retiring first can only ever
+        // reach rows served by an EARLIER pull, never the ones this call is about to serve
+        // (which would defeat at-least-once), and never a row inserted after them (the
+        // late-stamped insert that used to be silently retired and then age-purged).
+        var queue = Substitute.For<ISiteAuditQueue>();
+        queue.ReadPendingSinceAsync(
+                Arg.Any<DateTime>(), Arg.Any<int>(), Arg.Any<string?>(), Arg.Any<CancellationToken>())
+            .Returns((IReadOnlyList<AuditEvent>)new[] { NewEvent() });
+
+        var server = CreateServer();
+        server.SetSiteAuditQueue(queue);
+
+        var cursorTime = DateTime.SpecifyKind(new DateTime(2026, 5, 20, 10, 0, 0), DateTimeKind.Utc);
+        await server.PullAuditEvents(
+            new PullAuditEventsRequest
+            {
+                SinceUtc = Timestamp.FromDateTime(cursorTime),
+                BatchSize = 100,
+            },
+            NewContext());
+
+        Received.InOrder(() =>
+        {
+            queue.MarkReconciledUpToAsync(cursorTime, null, Arg.Any<CancellationToken>());
+            queue.ReadPendingSinceAsync(cursorTime, 100, null, Arg.Any<CancellationToken>());
+        });
+
+        // Serving still flips nothing by itself — the next cursor is the only receipt.
+        await queue.DidNotReceive().MarkReconciledAsync(
+            Arg.Any<IReadOnlyList<Guid>>(), Arg.Any<CancellationToken>());
+        await queue.DidNotReceive().MarkForwardedAsync(
+            Arg.Any<IReadOnlyList<Guid>>(), Arg.Any<CancellationToken>());
+    }
+
    [Fact]
    public async Task PullAuditEvents_MarkReconciledUpToThrows_ResponseStillReturned()
    {