An audit row whose tracking snapshot cannot be resolved is skipped and left
Pending, on the reasoning that central reconciliation will pick it up. Nothing
removes it from the local drain queue, so the next tick re-reads it, fails
identically, and logs again — forever. Measured on the rig at ~2,800
warnings/minute, sustained across both a process restart and a container
restart, until the audit database itself was discarded.
The code comment already names the cause ("the tracking store was reset"), so
this is an anticipated input, not an exotic one. Two production triggers need
no operator error: the tracking retention window elapsing before the drain
catches up (long central outage, large backlog), or restoring one store
independently of the other. The two stores are easy to desync because they live
in different places — audit in auditlog.db, which on the docker rig is INSIDE
the container, and tracking in the bind-mounted LocalDb database.
Skipping the row is right; leaving it Pending with no other state change is
not. There is no attempt counter, no backoff, no terminal state, and one
Warning per row per pass. Suggested fixes ranked by effort, the cheapest being
to rate-limit the warning the way MaintenanceBackgroundService already does for
oplog caps.
No data loss — the audit rows are intact and still reach central by the
reconciliation path. Found while cleaning the rig after the Phase 2 live gate.
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
Joseph Doherty