ScadaBridge/docs/known-issues/2026-07-20-cached-telemetry-drain-hot-loop.md

# Cached-telemetry drain hot-loops forever on a row whose tracking snapshot is gone

**Date:** 2026-07-20 · **Status:** RESOLVED (2026-07-20) · **Severity:** High — revised up from Medium on investigation (permanent stall of the cached-telemetry path, not just a log flood)
· **Area:** AuditLog / Site Telemetry

## Resolution (2026-07-20)

Fixed by letting an unresolvable row LEAVE the queue.
`SiteAuditTelemetryActor.OnCachedDrainAsync` now abandons the operational half of a cached row
whose tracking snapshot is still unresolvable after a grace period
(`SiteAuditTelemetryOptions.CachedTrackingGraceSeconds`, default **300 s**) and marks it
`Forwarded`. A row with no `CorrelationId` at all is abandoned immediately — it can never resolve.

**Marking `Forwarded` does not drop the audit data.** That state's role in this machine is "no
longer owed by the drain, still eligible for reconciliation", which is exactly the situation:
`ISiteAuditQueue.ReadPendingSinceAsync` covers `Forwarded` rows as well as `Pending` ones and
central dedups on `EventId`, so the reconciliation pull still delivers them. What is genuinely lost
is the operational (`SiteCalls`) half — unrecoverable regardless, since the tracking row it would
have been built from no longer exists.

Three further behaviours, each deliberate:

- **Inside the grace window the row is still retried.** A missing snapshot is normally a brief
  write race (the audit row lands microseconds before the tracking row); abandoning on the first
  failed lookup would discard the operational half of every cached call that lost that race.
- **A tracking-store THROW never abandons.** A throw is a store fault (locked, corrupt,
  mid-restore), not a verdict about the row. Those rows stay `Pending` however old they are.
- **Logging is per drain pass, not per row.** One summary line each for abandoned, lookup-failed
  (with the first exception attached) and deferred rows. The deferred case dropped to Debug — being
  inside the grace window is normal operation, not a warning.

### Severity was revised UP during the fix

The original write-up called this a log flood with "no data loss", which understated it. The queue
is read **oldest-first with a fixed `BatchSize` (default 256)**, so once a batch's worth of
permanently-unresolvable rows collects at the head, every drain re-reads exactly those rows, fails
identically, and **never reaches the newer rows behind them**. The cached-telemetry path stalls
permanently. The audit half still reached central by reconciliation, so "no data loss" held — but
"Medium" did not.

**Fixed in:** `SiteAuditTelemetryActor.cs` (`AbandonUnresolvableAsync` + the rewritten skip
branches), `SiteAuditTelemetryOptions.cs` (`CachedTrackingGraceSeconds`).
**Tests:** `SiteAuditTelemetryActorTests` — `CachedDrain_OrphanRow_PastGrace_IsAbandoned_...`,
`..._InsideGrace_IsRetried_NotAbandoned`, `..._FullBatchOfUnresolvableRows_DoesNotStarveTheQueue`,
`..._TrackingStoreThrow_DoesNotAbandonTheRow`. The pre-existing
`CachedDrain_OrphanRow_NoTrackingSnapshot_IsSkipped_DoesNotCrash` was **retargeted, not deleted** —
it had pinned the defect ("skipped and stays Pending"), so its orphan assertion is inverted while
the half that still holds is kept verbatim. Verified non-vacuous: with abandonment disabled the two
abandonment tests go red; the two must-NOT-abandon guards stay green in both directions.

The diagnosis below is retained as the record of how it was found.

## Summary

`SiteAuditTelemetryActor`'s cached-telemetry drain reads Pending audit rows, looks up each row's
tracking snapshot by `CorrelationId`, and pushes the combined packet to central. When the lookup
returns `null` the row is **skipped and deliberately left Pending**
(`SiteAuditTelemetryActor.cs:307`), on the reasoning that "central reconciliation will pick it up".

Nothing ever removes such a row from the local drain queue. The next tick re-reads it, fails the
same lookup, logs the same warning, and leaves it Pending again — **forever**. With a batch of
unresolvable rows the actor spins at its non-idle rate and emits one warning per row per pass.

Measured on the docker rig: **~2 800 warnings/minute, sustained**, surviving both a process restart
and a container restart, until the audit database itself was discarded.

```
[09:57:41 WRN] [Site/scadabridge-site-a-a] Cached-telemetry drain: no tracking snapshot for
  a5392796-291f-4f5b-9fbf-5817c1ec76c7 (TrackedOperationId 59bd4bf8-…); skipping.
```

## Why the rows became unresolvable

Two independent stores must agree:

- the **audit** rows live in `auditlog.db` (site-local, and on the docker rig **inside the
  container at `/app/auditlog.db`**, not on the bind-mounted data volume);
- the **tracking** rows live in `OperationTracking`, which LocalDb Phase 1 moved into the
  consolidated `LocalDb:Path` database (bind-mounted).

Anything that resets one without the other strands every audit row that referenced it. The code
comment already anticipates the cause — *"possible if the audit row is older than the tracking
retention window, or the tracking store was reset"* — so this is a known-and-accepted input, not an
exotic one.

**Two realistic production triggers, neither requiring operator error:**

1. **Tracking retention expiry.** If the tracking retention window elapses before the audit drain
   catches up — a long central outage, a large backlog — the snapshots are pruned out from under
   still-Pending audit rows and every one of them becomes a permanent hot-loop entry.
2. **Restoring or resetting one store independently of the other**, e.g. rebuilding a node's
   LocalDb file from its peer while its container-local `auditlog.db` survives.

It was hit here by (2): the two site-a LocalDb databases were dropped during a rig cleanup while
`auditlog.db` — being inside the container — survived.

## Why the current handling is not enough

Skipping the row is correct; **leaving it Pending with no other state change is not**. The row is
now in a state it can never leave:

- no attempt counter, so an unresolvable row is indistinguishable from a transiently-failing one;
- no backoff, so the actor runs at full non-idle rate against a queue that can never shrink;
- no terminal state, so it is retried for the life of the database;
- one Warning per row per pass, which buries every other log line on the node.

The "central reconciliation will pick it up" comment is about the **audit half** reaching central by
another path. That may well be true — but it does not release the row from the local drain queue,
which is what actually loops.

## Suggested fix

Give an unresolvable row somewhere to go. Roughly, in increasing order of effort:

1. **Bound the retries.** Add an attempt count; past a threshold mark the row terminal
   (`TrackingUnavailable`) and stop re-reading it. Emit a single summary Warning with the count
   rather than one per row per pass.
2. **Rate-limit the warning** to one per drain episode regardless of row count — the same pattern
   `MaintenanceBackgroundService` already uses for the oplog caps-exceeded warning (`_snapshotFlagWarned`).
3. **Push the audit half alone** when the tracking snapshot is missing, rather than skipping the row
   entirely, so the row can be marked emitted and leave the queue. Needs a decision on whether
   central accepts a packet with no tracking half.

(2) alone would remove the operational damage; (1) or (3) is needed to stop the wasted I/O.

## Reproduction

1. Run a site node until it has cached-call audit rows with tracking correlations.
2. Stop the node; delete its consolidated LocalDb database (which holds `OperationTracking`);
   leave `auditlog.db` in place.
3. Start the node. The drain warning repeats indefinitely; the rate does not decay.

## Notes

- **No data loss.** The audit rows are intact and still reach central by the reconciliation path;
  what is broken is the local drain's ability to ever finish.
- Discovered while cleaning the rig after the LocalDb Phase 2 live gate
  (`docs/plans/2026-07-19-localdb-phase2-live-gate.md`), which is also where the related
  "deleting an instance orphans its buffered messages" observation is recorded.