A cached-telemetry audit row whose OperationTracking snapshot could not be
resolved was skipped and left Pending. Nothing ever removed it, so the next
drain re-read it, failed identically, and logged again — forever.
The severity was worse than the original write-up said. The queue is read
oldest-first with a fixed BatchSize (default 256), so once a batch's worth of
permanently-unresolvable rows collected at the head, every drain re-read
exactly those rows and NEVER REACHED the newer rows behind them. That is a
permanent stall of the cached-telemetry path, not the "log flood, no data loss"
the issue was first filed as. Measured on the rig at ~2,800 warnings/minute,
surviving both a process restart and a container restart.
Fix: after a grace period (CachedTrackingGraceSeconds, default 300 s) the
operational half is abandoned and the row marked Forwarded. A row with no
CorrelationId is abandoned immediately — it can never resolve.
Marking Forwarded does not drop audit data. That state means "no longer owed by
the drain, still eligible for reconciliation", which is exactly this situation:
ReadPendingSinceAsync covers Forwarded as well as Pending and central dedups on
EventId, so the reconciliation pull still delivers the audit half. What is lost
is the operational (SiteCalls) half, which is unrecoverable anyway once the
tracking row is gone.
Three deliberate boundaries:
- Inside the grace window the row is still retried — a missing snapshot is
normally a brief write race, and abandoning on the first failed lookup would
discard the operational half of every cached call that lost that race.
- A tracking-store THROW never abandons, however old the row: a throw is a
store fault (locked, corrupt, mid-restore), not a verdict about the row.
- Logging is per drain pass, not per row. Deferred rows dropped to Debug —
being inside the grace window is normal operation, not a warning.
CachedDrain_OrphanRow_NoTrackingSnapshot_IsSkipped_DoesNotCrash was RETARGETED
rather than deleted: it pinned the defect ("skipped and stays Pending"), so its
orphan assertion is inverted and the half that still holds is kept verbatim.
Three tests added, including the starvation regression. Both "must NOT abandon"
tests carry a positive control on the read count, so they cannot pass by virtue
of a drain that never ran.
Verified non-vacuous: with abandonment disabled the two abandonment tests go
red and the two guards stay green. Build 0 warnings; AuditLog 358, Host 330,
SiteRuntime 512, StoreAndForward 130, Communication 312, Commons 684,
Integration 94 — all pass.
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
An audit row whose tracking snapshot cannot be resolved is skipped and left
Pending, on the reasoning that central reconciliation will pick it up. Nothing
removes it from the local drain queue, so the next tick re-reads it, fails
identically, and logs again — forever. Measured on the rig at ~2,800
warnings/minute, sustained across both a process restart and a container
restart, until the audit database itself was discarded.
The code comment already names the cause ("the tracking store was reset"), so
this is an anticipated input, not an exotic one. Two production triggers need
no operator error: the tracking retention window elapsing before the drain
catches up (long central outage, large backlog), or restoring one store
independently of the other. The two stores are easy to desync because they live
in different places — audit in auditlog.db, which on the docker rig is INSIDE
the container, and tracking in the bind-mounted LocalDb database.
Skipping the row is right; leaving it Pending with no other state change is
not. There is no attempt counter, no backoff, no terminal state, and one
Warning per row per pass. Suggested fixes ranked by effort, the cheapest being
to rate-limit the warning the way MaintenanceBackgroundService already does for
oplog caps.
No data loss — the audit rows are intact and still reach central by the
reconciliation path. Found while cleaning the rig after the Phase 2 live gate.
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
Joseph Doherty