fix(site-call-audit): reconciliation pull serves only un-acked tracking rows (node-local ack ledger), batched page upsert, prune removed sites' cursors (review 2026-10-21 06/PERF-2 + lane H follow-up)

This commit is contained in:
Joseph Doherty
2026-10-22 09:35:30 -04:00
parent e69be1654f
commit 88273693f9
12 changed files with 789 additions and 10 deletions
+34 -1
View File
@@ -135,8 +135,41 @@ Pinned)` on the EventStream (transition-only, mirroring
health-observable condition rather than a silent log line, and the latch clears
with `Pinned=false` once a later tick makes progress.
**The pull serves only what central has not acknowledged.** The site's
`ReadChangedSinceAsync` skips every tracking row whose *current* version
(`UpdatedAtUtc`) central already accepted through the cached-telemetry push, so in
steady state a reconciliation pull carries only genuinely lost telemetry rather than
a second copy of every lifecycle step — and a central failover, which resets the
in-memory cursors to `DateTime.MinValue`, no longer replays the whole tracking
retention window of every site. When central acks an `IngestCachedTelemetry`
packet (it commits the packet's `SiteCalls` upsert with its audit row), the site's
`SiteAuditTelemetryActor` records `(TrackedOperationId, snapshot UpdatedAtUtc)` in
the **node-local** `OperationTrackingCentralAck` table — deliberately a separate,
unregistered table, never a column on the LocalDb-replicated `OperationTracking`
table (same posture as `mirrored_secrets`): only the node that pushed saw central
accept it. The filter matches the version exactly, so any later change (even one
stamped by a clock that stepped back) is served again; acks are monotonic per row
and pruned with the tracking rows they describe. **Failover:** a site node that
has just become active holds no ack rows, so its pull serves every row once
(bounded by central's cursor); central's idempotent monotonic upsert absorbs the
duplicates and the ledger rebuilds from that node's own pushes. A lost ack only
costs one extra pull of that row.
**Each pulled page is one batched upsert.** `ISiteCallAuditRepository.UpsertManyAsync`
runs the same two per-row statements as `UpsertAsync` (insert-if-absent, then the
ownership-checked monotonic update) for up to 128 rows per command — a 500-row page
is 4 round trips instead of 500. A chunk that faults (an insert race, or an
ownership violation) is re-applied row by row through `UpsertAsync`, which is inert
for rows already applied; the rest of the page is still applied and the first
row failure is rethrown, so the site's cursor stays put exactly as before.
**Removed sites are pruned.** Each tick drops the cursor and pinned latch of any
site the enumerator no longer returns (a pinned one publishes its `Pinned=false`
transition first), so the dictionaries do not grow with site churn; a site that
reappears restarts from `DateTime.MinValue`.
**The drain runs off the mailbox.** A reconciliation pass is unbounded work —
every site, up to a page ceiling of network pulls each, one upsert per row — so
every site, up to a page ceiling of network pulls each, one batched upsert per page — so
it runs as a background task with a `PipeTo`-delivered completion message and a
single-flight guard, not as an actor message handler. A handler occupies the
actor for its whole duration, so a post-outage catch-up used to park telemetry