fix(comms): review findings — consumer-based debug orphan net, foreign-cancel triad, honest onConnected, served-row-exact retirement, full-rate reconcile

F1 (HIGH) DebugStreamBridgeActor: the 5-minute orphan net measured the MAILBOX (SetReceiveTimeout), and once stream events were correctly marked INotInfluenceReceiveTimeout nothing recurring reset it — the snapshot lands once and GrpcStreamStable once — so every healthy session self-terminated at ~6 min with a false "Site disconnected". Replaced with a periodic self-tick (ConsumerLivenessCheckInterval, 30s) over a consumer-last-seen stamp renewed only by DebugStreamConsumerAlive, which DebugStreamService Tells on a shared timer to every session still in its registry (holding a session there IS "a consumer is attached" — both the Blazor view and the SignalR hub release it on dispose/disconnect, and it works headless). Reverting the wrapper was rejected: it would restore the quiet-instance orphan bug. F2 (MED) SiteStreamGrpcClient: the RpcException(Cancelled) filter now requires cts.IsCancellationRequested. A peer-originated / channel-dispose Cancelled fired none of onError/onCompleted/onConnected, leaving SiteAlarmAggregatorActor with _streamDown=false forever (IsLive stuck true, reconcile reopen guard never fired). F3 (MED) SiteStreamGrpcClient: a header TIMEOUT is no longer reported as connected — that shape is exactly what an unreachable site produces, and it cleared _streamDown, consumed _seedOnConnect and launched a full snapshot fan-out at a dead site. AwaitHeadersAsync returns bool; the first received event is the fallback connected signal, fired at most once from headers OR first event. F4 (LOW-MED) SqliteAuditWriter.MarkReconciledUpToAsync: the blanket below-cursor UPDATE retired late-stamped inserts that were never served (then age-purged — silent loss). The flip is now bounded by insertion order: a Pending row retires only if its rowid is at or below the high-water mark of rows this instance has served from ReadPendingSinceAsync (clamped on purge, since SQLite reuses rowids); Forwarded rows are exempt (central ACKed them over the push path). At-least-once is unchanged. F5 (LOW) Documented the liveness dependency (a served row never covered by a later cursor stays Pending forever; PurgeExpiredAsync never purges Pending) in ISiteAuditQueue + Component-AuditLog.md, and added a cheap site-health signal: SiteAuditBacklogReporter logs a rate-limited warning when the existing oldest-pending metric exceeds 24h. F6 (MED) SiteAlarmAggregatorActor: _fanoutSinceLastTick was armed by the reconcile's OWN fan-out, so steady state ran fan-out→skip→fan-out→skip — one reconcile per 2x interval (120s), halving the not-reporting refresh and the alarm reconcile backstop. The skip is now armed only by connect/failover-driven seeds (initial, _seedOnConnect, and a re-seed queued behind one). Tests: Communication.Tests 691 passed (+13), AuditLog.Tests 382 passed (+5).
2026-08-14 23:52:25 -04:00
parent b1de9dfdd4
commit fd5e023d08
16 changed files with 1192 additions and 84 deletions
@@ -1,4 +1,5 @@
 using System.Diagnostics;
+using Microsoft.Extensions.Logging;
 using Microsoft.Extensions.Logging.Abstractions;
 using Microsoft.Extensions.Options;
 using NSubstitute;
@@ -115,4 +116,67 @@ public class SiteAuditBacklogReporterCadenceTests

        Assert.Equal(TimeSpan.FromSeconds(3), reporter.RefreshInterval);
    }
+
+    // ----- Stale-Pending signal (review F5) ----- //
+
+    [Fact]
+    public void StalePendingBacklog_IsWarned_ThenRateLimited()
+    {
+        // Pending rows are exempt from the retention purge by design, so a standing Pending
+        // backlog is the one site-store condition that never self-heals on age — it clears
+        // only when central acknowledges the rows. It is worth a log line, not just a number
+        // on the health report, and the warning must not spam every 30 s poll.
+        var logger = new CapturingLogger();
+        var reporter = new SiteAuditBacklogReporter(
+            Substitute.For<ISiteAuditQueue>(),
+            Substitute.For<ISiteHealthCollector>(),
+            logger,
+            TimeSpan.FromHours(1),
+            null);
+
+        var stale = DateTime.UtcNow - SiteAuditBacklogReporter.StalePendingThreshold - TimeSpan.FromHours(1);
+        reporter.WarnIfPendingIsStale(stale, pendingCount: 4321);
+        reporter.WarnIfPendingIsStale(stale, pendingCount: 4321); // same poll cycle-ish
+
+        var warning = Assert.Single(logger.Entries, e => e.Level == LogLevel.Warning);
+        Assert.Contains("4321", warning.Message);
+        Assert.Contains("pending", warning.Message, StringComparison.OrdinalIgnoreCase);
+    }
+
+    [Fact]
+    public void FreshOrEmptyPendingBacklog_IsNotWarned()
+    {
+        var logger = new CapturingLogger();
+        var reporter = new SiteAuditBacklogReporter(
+            Substitute.For<ISiteAuditQueue>(),
+            Substitute.For<ISiteHealthCollector>(),
+            logger,
+            TimeSpan.FromHours(1),
+            null);
+
+        reporter.WarnIfPendingIsStale(null, pendingCount: 0);                    // nothing pending
+        reporter.WarnIfPendingIsStale(DateTime.UtcNow.AddMinutes(-5), 12);       // a normal drain lag
+
+        Assert.DoesNotContain(logger.Entries, e => e.Level == LogLevel.Warning);
+    }
+
+    /// <summary>Captures log entries so the stale-pending signal can be asserted.</summary>
+    private sealed class CapturingLogger : ILogger<SiteAuditBacklogReporter>
+    {
+        public List<(LogLevel Level, Exception? Exception, string Message)> Entries { get; } = new();
+
+        public IDisposable? BeginScope<TState>(TState state) where TState : notnull => null;
+
+        public bool IsEnabled(LogLevel logLevel) => true;
+
+        public void Log<TState>(
+            LogLevel logLevel,
+            EventId eventId,
+            TState state,
+            Exception? exception,
+            Func<TState, Exception?, string> formatter)
+        {
+            Entries.Add((logLevel, exception, formatter(state, exception)));
+        }
+    }
 }