fix(comms): review findings — consumer-based debug orphan net, foreign-cancel triad, honest onConnected, served-row-exact retirement, full-rate reconcile
F1 (HIGH) DebugStreamBridgeActor: the 5-minute orphan net measured the MAILBOX (SetReceiveTimeout), and once stream events were correctly marked INotInfluenceReceiveTimeout nothing recurring reset it — the snapshot lands once and GrpcStreamStable once — so every healthy session self-terminated at ~6 min with a false "Site disconnected". Replaced with a periodic self-tick (ConsumerLivenessCheckInterval, 30s) over a consumer-last-seen stamp renewed only by DebugStreamConsumerAlive, which DebugStreamService Tells on a shared timer to every session still in its registry (holding a session there IS "a consumer is attached" — both the Blazor view and the SignalR hub release it on dispose/disconnect, and it works headless). Reverting the wrapper was rejected: it would restore the quiet-instance orphan bug. F2 (MED) SiteStreamGrpcClient: the RpcException(Cancelled) filter now requires cts.IsCancellationRequested. A peer-originated / channel-dispose Cancelled fired none of onError/onCompleted/onConnected, leaving SiteAlarmAggregatorActor with _streamDown=false forever (IsLive stuck true, reconcile reopen guard never fired). F3 (MED) SiteStreamGrpcClient: a header TIMEOUT is no longer reported as connected — that shape is exactly what an unreachable site produces, and it cleared _streamDown, consumed _seedOnConnect and launched a full snapshot fan-out at a dead site. AwaitHeadersAsync returns bool; the first received event is the fallback connected signal, fired at most once from headers OR first event. F4 (LOW-MED) SqliteAuditWriter.MarkReconciledUpToAsync: the blanket below-cursor UPDATE retired late-stamped inserts that were never served (then age-purged — silent loss). The flip is now bounded by insertion order: a Pending row retires only if its rowid is at or below the high-water mark of rows this instance has served from ReadPendingSinceAsync (clamped on purge, since SQLite reuses rowids); Forwarded rows are exempt (central ACKed them over the push path). At-least-once is unchanged. F5 (LOW) Documented the liveness dependency (a served row never covered by a later cursor stays Pending forever; PurgeExpiredAsync never purges Pending) in ISiteAuditQueue + Component-AuditLog.md, and added a cheap site-health signal: SiteAuditBacklogReporter logs a rate-limited warning when the existing oldest-pending metric exceeds 24h. F6 (MED) SiteAlarmAggregatorActor: _fanoutSinceLastTick was armed by the reconcile's OWN fan-out, so steady state ran fan-out→skip→fan-out→skip — one reconcile per 2x interval (120s), halving the not-reporting refresh and the alarm reconcile backstop. The skip is now armed only by connect/failover-driven seeds (initial, _seedOnConnect, and a re-seed queued behind one). Tests: Communication.Tests 691 passed (+13), AuditLog.Tests 382 passed (+5).
This commit is contained in:
@@ -261,7 +261,15 @@ room is a compliance violation, not a self-healing behavior. To bound that
|
||||
growth in practice, the site emits a `SiteAuditBacklog` health metric (pending
|
||||
row count, oldest pending age, bytes on disk); crossing operator-configured
|
||||
thresholds surfaces a warning on the relevant site tile in the Health
|
||||
dashboard, mirroring the Store-and-Forward Engine's backlog metric.
|
||||
dashboard, mirroring the Store-and-Forward Engine's backlog metric, and
|
||||
`SiteAuditBacklogReporter` additionally logs a rate-limited warning once the
|
||||
oldest pending row passes 24 h.
|
||||
|
||||
The same invariant carries a liveness dependency worth stating explicitly: the
|
||||
`Pending` floor clears only when central acknowledges the rows — a telemetry ack
|
||||
or a reconciliation cursor that covers them — never on age. See *Reconciliation
|
||||
pull* below for the served-row retirement rule that decides which rows can be
|
||||
acknowledged by a cursor at all.
|
||||
|
||||
Central is the durable home. Site SQLite is a write-buffer with a forwarding
|
||||
guarantee.
|
||||
@@ -367,6 +375,38 @@ it — a lagging drain is meant to surface as the stalled signal. The id tiebrea
|
||||
what makes that safe against a same-instant burst larger than one batch: the cursor
|
||||
advances on every tick even when the timestamp cannot.
|
||||
|
||||
**Only rows that were actually served may retire.** `OccurredAtUtc` is stamped by the
|
||||
caller, so a row can be *inserted* after a batch was served yet carry a timestamp
|
||||
*below* central's (by then advanced) cursor — a back-dated stamp, a clock nudge, a
|
||||
write flushed late. A cursor flip that keyed on the timestamp alone retired exactly
|
||||
those rows: never served, never servable again (the keyset read has moved past them),
|
||||
and, being `Reconciled`, purged on age. That is silent audit loss, and it is a failure
|
||||
mode the pre-WP2.3 explicit id-set flip could not produce, so `MarkReconciledUpToAsync`
|
||||
carries a second, insertion-order bound:
|
||||
|
||||
> A `Pending` row retires only if its insertion order is at or below the high-water
|
||||
> mark of rows this site node has actually served from `ReadPendingSinceAsync`
|
||||
> (SQLite `rowid`). A `Forwarded` row is exempt — central ACKED it over the telemetry
|
||||
> push path, which is proof independent of the pull.
|
||||
|
||||
The bound is per-process, so after a site-node restart the first pull retires only
|
||||
`Forwarded` rows and the pull after it resumes normal retirement — conservative in the
|
||||
safe direction (retirement is delayed, never a row lost). The gRPC handler's ordering
|
||||
(retire, *then* read) is what keeps the bound from ever vouching for the batch it is
|
||||
about to serve.
|
||||
|
||||
**Liveness note (accepted, documented).** A row that was served but never covered by a
|
||||
later cursor — central reconciliation stopped for good, or the bound reset over a
|
||||
restart before the next cursor arrived — stays `Pending` indefinitely, and the site
|
||||
retention purge never purges `Pending`. That is the hard `ForwardState` invariant
|
||||
working as intended (an unacknowledged row is not droppable), but it means the site
|
||||
store's floor depends on reconciliation actually running, not merely on the retention
|
||||
window elapsing. It is observable rather than silent: `GetBacklogStatsAsync` reports
|
||||
the pending count and oldest-pending instant on every site health report
|
||||
(`SiteAuditBacklog`), and `SiteAuditBacklogReporter` logs a rate-limited warning once
|
||||
the oldest pending row exceeds `StalePendingThreshold` (24 h) naming the drain and the
|
||||
reconciliation pull as the things to check.
|
||||
|
||||
### Central direct-write (central-originated events)
|
||||
|
||||
Events originating at central never touch site SQLite. Inbound API writes one
|
||||
|
||||
Reference in New Issue
Block a user