Merge branch 'worktree-agent-a465fb3cd6ec48cd3' into arch-review-remediation

This commit is contained in:
Joseph Doherty
2026-08-14 23:53:00 -04:00
16 changed files with 1192 additions and 84 deletions
+41 -1
View File
@@ -263,7 +263,15 @@ room is a compliance violation, not a self-healing behavior. To bound that
growth in practice, the site emits a `SiteAuditBacklog` health metric (pending
row count, oldest pending age, bytes on disk); crossing operator-configured
thresholds surfaces a warning on the relevant site tile in the Health
dashboard, mirroring the Store-and-Forward Engine's backlog metric.
dashboard, mirroring the Store-and-Forward Engine's backlog metric, and
`SiteAuditBacklogReporter` additionally logs a rate-limited warning once the
oldest pending row passes 24 h.
The same invariant carries a liveness dependency worth stating explicitly: the
`Pending` floor clears only when central acknowledges the rows — a telemetry ack
or a reconciliation cursor that covers them — never on age. See *Reconciliation
pull* below for the served-row retirement rule that decides which rows can be
acknowledged by a cursor at all.
Central is the durable home. Site SQLite is a write-buffer with a forwarding
guarantee.
@@ -387,6 +395,38 @@ it — a lagging drain is meant to surface as the stalled signal. The id tiebrea
what makes that safe against a same-instant burst larger than one batch: the cursor
advances on every tick even when the timestamp cannot.
**Only rows that were actually served may retire.** `OccurredAtUtc` is stamped by the
caller, so a row can be *inserted* after a batch was served yet carry a timestamp
*below* central's (by then advanced) cursor — a back-dated stamp, a clock nudge, a
write flushed late. A cursor flip that keyed on the timestamp alone retired exactly
those rows: never served, never servable again (the keyset read has moved past them),
and, being `Reconciled`, purged on age. That is silent audit loss, and it is a failure
mode the pre-WP2.3 explicit id-set flip could not produce, so `MarkReconciledUpToAsync`
carries a second, insertion-order bound:
> A `Pending` row retires only if its insertion order is at or below the high-water
> mark of rows this site node has actually served from `ReadPendingSinceAsync`
> (SQLite `rowid`). A `Forwarded` row is exempt — central ACKED it over the telemetry
> push path, which is proof independent of the pull.
The bound is per-process, so after a site-node restart the first pull retires only
`Forwarded` rows and the pull after it resumes normal retirement — conservative in the
safe direction (retirement is delayed, never a row lost). The gRPC handler's ordering
(retire, *then* read) is what keeps the bound from ever vouching for the batch it is
about to serve.
**Liveness note (accepted, documented).** A row that was served but never covered by a
later cursor — central reconciliation stopped for good, or the bound reset over a
restart before the next cursor arrived — stays `Pending` indefinitely, and the site
retention purge never purges `Pending`. That is the hard `ForwardState` invariant
working as intended (an unacknowledged row is not droppable), but it means the site
store's floor depends on reconciliation actually running, not merely on the retention
window elapsing. It is observable rather than silent: `GetBacklogStatsAsync` reports
the pending count and oldest-pending instant on every site health report
(`SiteAuditBacklog`), and `SiteAuditBacklogReporter` logs a rate-limited warning once
the oldest pending row exceeds `StalePendingThreshold` (24 h) naming the drain and the
reconciliation pull as the things to check.
### Central direct-write (central-originated events)
Events originating at central never touch site SQLite. Inbound API writes one