# Cached-telemetry drain hot-loops forever on a row whose tracking snapshot is gone **Date:** 2026-07-20 · **Status:** RESOLVED (2026-07-20) · **Severity:** High — revised up from Medium on investigation (permanent stall of the cached-telemetry path, not just a log flood) · **Area:** AuditLog / Site Telemetry ## Resolution (2026-07-20) Fixed by letting an unresolvable row LEAVE the queue. `SiteAuditTelemetryActor.OnCachedDrainAsync` now abandons the operational half of a cached row whose tracking snapshot is still unresolvable after a grace period (`SiteAuditTelemetryOptions.CachedTrackingGraceSeconds`, default **300 s**) and marks it `Forwarded`. A row with no `CorrelationId` at all is abandoned immediately — it can never resolve. **Marking `Forwarded` does not drop the audit data.** That state's role in this machine is "no longer owed by the drain, still eligible for reconciliation", which is exactly the situation: `ISiteAuditQueue.ReadPendingSinceAsync` covers `Forwarded` rows as well as `Pending` ones and central dedups on `EventId`, so the reconciliation pull still delivers them. What is genuinely lost is the operational (`SiteCalls`) half — unrecoverable regardless, since the tracking row it would have been built from no longer exists. Three further behaviours, each deliberate: - **Inside the grace window the row is still retried.** A missing snapshot is normally a brief write race (the audit row lands microseconds before the tracking row); abandoning on the first failed lookup would discard the operational half of every cached call that lost that race. - **A tracking-store THROW never abandons.** A throw is a store fault (locked, corrupt, mid-restore), not a verdict about the row. Those rows stay `Pending` however old they are. - **Logging is per drain pass, not per row.** One summary line each for abandoned, lookup-failed (with the first exception attached) and deferred rows. The deferred case dropped to Debug — being inside the grace window is normal operation, not a warning. ### Severity was revised UP during the fix The original write-up called this a log flood with "no data loss", which understated it. The queue is read **oldest-first with a fixed `BatchSize` (default 256)**, so once a batch's worth of permanently-unresolvable rows collects at the head, every drain re-reads exactly those rows, fails identically, and **never reaches the newer rows behind them**. The cached-telemetry path stalls permanently. The audit half still reached central by reconciliation, so "no data loss" held — but "Medium" did not. **Fixed in:** `SiteAuditTelemetryActor.cs` (`AbandonUnresolvableAsync` + the rewritten skip branches), `SiteAuditTelemetryOptions.cs` (`CachedTrackingGraceSeconds`). **Tests:** `SiteAuditTelemetryActorTests` — `CachedDrain_OrphanRow_PastGrace_IsAbandoned_...`, `..._InsideGrace_IsRetried_NotAbandoned`, `..._FullBatchOfUnresolvableRows_DoesNotStarveTheQueue`, `..._TrackingStoreThrow_DoesNotAbandonTheRow`. The pre-existing `CachedDrain_OrphanRow_NoTrackingSnapshot_IsSkipped_DoesNotCrash` was **retargeted, not deleted** — it had pinned the defect ("skipped and stays Pending"), so its orphan assertion is inverted while the half that still holds is kept verbatim. Verified non-vacuous: with abandonment disabled the two abandonment tests go red; the two must-NOT-abandon guards stay green in both directions. The diagnosis below is retained as the record of how it was found. ## Summary `SiteAuditTelemetryActor`'s cached-telemetry drain reads Pending audit rows, looks up each row's tracking snapshot by `CorrelationId`, and pushes the combined packet to central. When the lookup returns `null` the row is **skipped and deliberately left Pending** (`SiteAuditTelemetryActor.cs:307`), on the reasoning that "central reconciliation will pick it up". Nothing ever removes such a row from the local drain queue. The next tick re-reads it, fails the same lookup, logs the same warning, and leaves it Pending again — **forever**. With a batch of unresolvable rows the actor spins at its non-idle rate and emits one warning per row per pass. Measured on the docker rig: **~2 800 warnings/minute, sustained**, surviving both a process restart and a container restart, until the audit database itself was discarded. ``` [09:57:41 WRN] [Site/scadabridge-site-a-a] Cached-telemetry drain: no tracking snapshot for a5392796-291f-4f5b-9fbf-5817c1ec76c7 (TrackedOperationId 59bd4bf8-…); skipping. ``` ## Why the rows became unresolvable Two independent stores must agree: - the **audit** rows live in `auditlog.db` (site-local, and on the docker rig **inside the container at `/app/auditlog.db`**, not on the bind-mounted data volume); - the **tracking** rows live in `OperationTracking`, which LocalDb Phase 1 moved into the consolidated `LocalDb:Path` database (bind-mounted). Anything that resets one without the other strands every audit row that referenced it. The code comment already anticipates the cause — *"possible if the audit row is older than the tracking retention window, or the tracking store was reset"* — so this is a known-and-accepted input, not an exotic one. **Two realistic production triggers, neither requiring operator error:** 1. **Tracking retention expiry.** If the tracking retention window elapses before the audit drain catches up — a long central outage, a large backlog — the snapshots are pruned out from under still-Pending audit rows and every one of them becomes a permanent hot-loop entry. 2. **Restoring or resetting one store independently of the other**, e.g. rebuilding a node's LocalDb file from its peer while its container-local `auditlog.db` survives. It was hit here by (2): the two site-a LocalDb databases were dropped during a rig cleanup while `auditlog.db` — being inside the container — survived. ## Why the current handling is not enough Skipping the row is correct; **leaving it Pending with no other state change is not**. The row is now in a state it can never leave: - no attempt counter, so an unresolvable row is indistinguishable from a transiently-failing one; - no backoff, so the actor runs at full non-idle rate against a queue that can never shrink; - no terminal state, so it is retried for the life of the database; - one Warning per row per pass, which buries every other log line on the node. The "central reconciliation will pick it up" comment is about the **audit half** reaching central by another path. That may well be true — but it does not release the row from the local drain queue, which is what actually loops. ## Suggested fix Give an unresolvable row somewhere to go. Roughly, in increasing order of effort: 1. **Bound the retries.** Add an attempt count; past a threshold mark the row terminal (`TrackingUnavailable`) and stop re-reading it. Emit a single summary Warning with the count rather than one per row per pass. 2. **Rate-limit the warning** to one per drain episode regardless of row count — the same pattern `MaintenanceBackgroundService` already uses for the oplog caps-exceeded warning (`_snapshotFlagWarned`). 3. **Push the audit half alone** when the tracking snapshot is missing, rather than skipping the row entirely, so the row can be marked emitted and leave the queue. Needs a decision on whether central accepts a packet with no tracking half. (2) alone would remove the operational damage; (1) or (3) is needed to stop the wasted I/O. ## Reproduction 1. Run a site node until it has cached-call audit rows with tracking correlations. 2. Stop the node; delete its consolidated LocalDb database (which holds `OperationTracking`); leave `auditlog.db` in place. 3. Start the node. The drain warning repeats indefinitely; the rate does not decay. ## Notes - **No data loss.** The audit rows are intact and still reach central by the reconciliation path; what is broken is the local drain's ability to ever finish. - Discovered while cleaning the rig after the LocalDb Phase 2 live gate (`docs/plans/2026-07-19-localdb-phase2-live-gate.md`), which is also where the related "deleting an instance orphans its buffered messages" observation is recorded.