fix(comms): review findings — consumer-based debug orphan net, foreign-cancel triad, honest onConnected, served-row-exact retirement, full-rate reconcile
F1 (HIGH) DebugStreamBridgeActor: the 5-minute orphan net measured the MAILBOX (SetReceiveTimeout), and once stream events were correctly marked INotInfluenceReceiveTimeout nothing recurring reset it — the snapshot lands once and GrpcStreamStable once — so every healthy session self-terminated at ~6 min with a false "Site disconnected". Replaced with a periodic self-tick (ConsumerLivenessCheckInterval, 30s) over a consumer-last-seen stamp renewed only by DebugStreamConsumerAlive, which DebugStreamService Tells on a shared timer to every session still in its registry (holding a session there IS "a consumer is attached" — both the Blazor view and the SignalR hub release it on dispose/disconnect, and it works headless). Reverting the wrapper was rejected: it would restore the quiet-instance orphan bug. F2 (MED) SiteStreamGrpcClient: the RpcException(Cancelled) filter now requires cts.IsCancellationRequested. A peer-originated / channel-dispose Cancelled fired none of onError/onCompleted/onConnected, leaving SiteAlarmAggregatorActor with _streamDown=false forever (IsLive stuck true, reconcile reopen guard never fired). F3 (MED) SiteStreamGrpcClient: a header TIMEOUT is no longer reported as connected — that shape is exactly what an unreachable site produces, and it cleared _streamDown, consumed _seedOnConnect and launched a full snapshot fan-out at a dead site. AwaitHeadersAsync returns bool; the first received event is the fallback connected signal, fired at most once from headers OR first event. F4 (LOW-MED) SqliteAuditWriter.MarkReconciledUpToAsync: the blanket below-cursor UPDATE retired late-stamped inserts that were never served (then age-purged — silent loss). The flip is now bounded by insertion order: a Pending row retires only if its rowid is at or below the high-water mark of rows this instance has served from ReadPendingSinceAsync (clamped on purge, since SQLite reuses rowids); Forwarded rows are exempt (central ACKed them over the push path). At-least-once is unchanged. F5 (LOW) Documented the liveness dependency (a served row never covered by a later cursor stays Pending forever; PurgeExpiredAsync never purges Pending) in ISiteAuditQueue + Component-AuditLog.md, and added a cheap site-health signal: SiteAuditBacklogReporter logs a rate-limited warning when the existing oldest-pending metric exceeds 24h. F6 (MED) SiteAlarmAggregatorActor: _fanoutSinceLastTick was armed by the reconcile's OWN fan-out, so steady state ran fan-out→skip→fan-out→skip — one reconcile per 2x interval (120s), halving the not-reporting refresh and the alarm reconcile backstop. The skip is now armed only by connect/failover-driven seeds (initial, _seedOnConnect, and a re-seed queued behind one). Tests: Communication.Tests 691 passed (+13), AuditLog.Tests 382 passed (+5).
This commit is contained in:
@@ -143,6 +143,29 @@ public interface ISiteAuditQueue
|
||||
/// <paramref name="sinceUtc"/> are proven received; rows at the boundary instant are left
|
||||
/// alone. Idempotent; already-Reconciled rows are untouched.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>Only-served-rows-retire.</b> The cursor is a timestamp, and
|
||||
/// <see cref="AuditEvent.OccurredAtUtc"/> is stamped by the caller, so a row can be
|
||||
/// INSERTED after a batch was served yet carry a timestamp below the (by then advanced)
|
||||
/// cursor. Implementations MUST NOT retire such a row: it was never served, the keyset
|
||||
/// read has moved past it, and retiring it would make it purgeable — silent audit loss.
|
||||
/// The SQLite implementation bounds the flip by insertion order (<c>rowid</c> high-water
|
||||
/// mark of rows it has served), exempting rows already
|
||||
/// <see cref="ZB.MOM.WW.ScadaBridge.Commons.Types.Enums.AuditForwardState.Forwarded"/>
|
||||
/// because central ACKED those through the telemetry push path.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>Liveness dependency (documented, accepted).</b> A row that WAS served but is never
|
||||
/// covered by a later cursor — central reconciliation stopped for good, or the site node
|
||||
/// restarted and its bound reset — stays
|
||||
/// <see cref="ZB.MOM.WW.ScadaBridge.Commons.Types.Enums.AuditForwardState.Pending"/>
|
||||
/// indefinitely, and <see cref="PurgeExpiredAsync"/> never purges Pending. That is the
|
||||
/// retention invariant working as designed (an unacknowledged row is not droppable), but
|
||||
/// it means the site store's floor is bounded by reconciliation actually running. The
|
||||
/// backlog is observable: <see cref="GetBacklogStatsAsync"/> reports the pending count and
|
||||
/// oldest-pending instant on every site health report, and the site reporter logs a
|
||||
/// warning once the oldest pending row exceeds its stale threshold.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
/// <param name="sinceUtc">The cursor timestamp central has consumed up to (UTC).</param>
|
||||
/// <param name="afterId">The last consumed <see cref="AuditEvent.EventId"/> at that instant, or null.</param>
|
||||
|
||||
Reference in New Issue
Block a user