fix(comms): review findings — consumer-based debug orphan net, foreign-cancel triad, honest onConnected, served-row-exact retirement, full-rate reconcile

F1 (HIGH) DebugStreamBridgeActor: the 5-minute orphan net measured the MAILBOX
(SetReceiveTimeout), and once stream events were correctly marked
INotInfluenceReceiveTimeout nothing recurring reset it — the snapshot lands once
and GrpcStreamStable once — so every healthy session self-terminated at ~6 min
with a false "Site disconnected". Replaced with a periodic self-tick
(ConsumerLivenessCheckInterval, 30s) over a consumer-last-seen stamp renewed only
by DebugStreamConsumerAlive, which DebugStreamService Tells on a shared timer to
every session still in its registry (holding a session there IS "a consumer is
attached" — both the Blazor view and the SignalR hub release it on
dispose/disconnect, and it works headless). Reverting the wrapper was rejected: it
would restore the quiet-instance orphan bug.

F2 (MED) SiteStreamGrpcClient: the RpcException(Cancelled) filter now requires
cts.IsCancellationRequested. A peer-originated / channel-dispose Cancelled fired
none of onError/onCompleted/onConnected, leaving SiteAlarmAggregatorActor with
_streamDown=false forever (IsLive stuck true, reconcile reopen guard never fired).

F3 (MED) SiteStreamGrpcClient: a header TIMEOUT is no longer reported as
connected — that shape is exactly what an unreachable site produces, and it
cleared _streamDown, consumed _seedOnConnect and launched a full snapshot fan-out
at a dead site. AwaitHeadersAsync returns bool; the first received event is the
fallback connected signal, fired at most once from headers OR first event.

F4 (LOW-MED) SqliteAuditWriter.MarkReconciledUpToAsync: the blanket below-cursor
UPDATE retired late-stamped inserts that were never served (then age-purged —
silent loss). The flip is now bounded by insertion order: a Pending row retires
only if its rowid is at or below the high-water mark of rows this instance has
served from ReadPendingSinceAsync (clamped on purge, since SQLite reuses rowids);
Forwarded rows are exempt (central ACKed them over the push path). At-least-once
is unchanged.

F5 (LOW) Documented the liveness dependency (a served row never covered by a later
cursor stays Pending forever; PurgeExpiredAsync never purges Pending) in
ISiteAuditQueue + Component-AuditLog.md, and added a cheap site-health signal:
SiteAuditBacklogReporter logs a rate-limited warning when the existing
oldest-pending metric exceeds 24h.

F6 (MED) SiteAlarmAggregatorActor: _fanoutSinceLastTick was armed by the
reconcile's OWN fan-out, so steady state ran fan-out→skip→fan-out→skip — one
reconcile per 2x interval (120s), halving the not-reporting refresh and the alarm
reconcile backstop. The skip is now armed only by connect/failover-driven seeds
(initial, _seedOnConnect, and a re-seed queued behind one).

Tests: Communication.Tests 691 passed (+13), AuditLog.Tests 382 passed (+5).
This commit is contained in:
Joseph Doherty
2026-08-14 23:52:25 -04:00
parent b1de9dfdd4
commit fd5e023d08
16 changed files with 1192 additions and 84 deletions
+41 -1
View File
@@ -261,7 +261,15 @@ room is a compliance violation, not a self-healing behavior. To bound that
growth in practice, the site emits a `SiteAuditBacklog` health metric (pending
row count, oldest pending age, bytes on disk); crossing operator-configured
thresholds surfaces a warning on the relevant site tile in the Health
dashboard, mirroring the Store-and-Forward Engine's backlog metric.
dashboard, mirroring the Store-and-Forward Engine's backlog metric, and
`SiteAuditBacklogReporter` additionally logs a rate-limited warning once the
oldest pending row passes 24 h.
The same invariant carries a liveness dependency worth stating explicitly: the
`Pending` floor clears only when central acknowledges the rows — a telemetry ack
or a reconciliation cursor that covers them — never on age. See *Reconciliation
pull* below for the served-row retirement rule that decides which rows can be
acknowledged by a cursor at all.
Central is the durable home. Site SQLite is a write-buffer with a forwarding
guarantee.
@@ -367,6 +375,38 @@ it — a lagging drain is meant to surface as the stalled signal. The id tiebrea
what makes that safe against a same-instant burst larger than one batch: the cursor
advances on every tick even when the timestamp cannot.
**Only rows that were actually served may retire.** `OccurredAtUtc` is stamped by the
caller, so a row can be *inserted* after a batch was served yet carry a timestamp
*below* central's (by then advanced) cursor — a back-dated stamp, a clock nudge, a
write flushed late. A cursor flip that keyed on the timestamp alone retired exactly
those rows: never served, never servable again (the keyset read has moved past them),
and, being `Reconciled`, purged on age. That is silent audit loss, and it is a failure
mode the pre-WP2.3 explicit id-set flip could not produce, so `MarkReconciledUpToAsync`
carries a second, insertion-order bound:
> A `Pending` row retires only if its insertion order is at or below the high-water
> mark of rows this site node has actually served from `ReadPendingSinceAsync`
> (SQLite `rowid`). A `Forwarded` row is exempt — central ACKED it over the telemetry
> push path, which is proof independent of the pull.
The bound is per-process, so after a site-node restart the first pull retires only
`Forwarded` rows and the pull after it resumes normal retirement — conservative in the
safe direction (retirement is delayed, never a row lost). The gRPC handler's ordering
(retire, *then* read) is what keeps the bound from ever vouching for the batch it is
about to serve.
**Liveness note (accepted, documented).** A row that was served but never covered by a
later cursor — central reconciliation stopped for good, or the bound reset over a
restart before the next cursor arrived — stays `Pending` indefinitely, and the site
retention purge never purges `Pending`. That is the hard `ForwardState` invariant
working as intended (an unacknowledged row is not droppable), but it means the site
store's floor depends on reconciliation actually running, not merely on the retention
window elapsing. It is observable rather than silent: `GetBacklogStatsAsync` reports
the pending count and oldest-pending instant on every site health report
(`SiteAuditBacklog`), and `SiteAuditBacklogReporter` logs a rate-limited warning once
the oldest pending row exceeds `StalePendingThreshold` (24 h) naming the drain and the
reconciliation pull as the things to check.
### Central direct-write (central-originated events)
Events originating at central never touch site SQLite. Inbound API writes one