ScadaBridge/docs/known-issues/2026-07-20-cached-telemetry-drain-hot-loop.md

Cached-telemetry drain hot-loops forever on a row whose tracking snapshot is gone

Date: 2026-07-20 · Status: RESOLVED (2026-07-20) · Severity: High — revised up from Medium on investigation (permanent stall of the cached-telemetry path, not just a log flood) · Area: AuditLog / Site Telemetry

Resolution (2026-07-20)

Fixed by letting an unresolvable row LEAVE the queue. SiteAuditTelemetryActor.OnCachedDrainAsync now abandons the operational half of a cached row whose tracking snapshot is still unresolvable after a grace period (SiteAuditTelemetryOptions.CachedTrackingGraceSeconds, default 300 s) and marks it Forwarded. A row with no CorrelationId at all is abandoned immediately — it can never resolve.

Marking Forwarded does not drop the audit data. That state's role in this machine is "no longer owed by the drain, still eligible for reconciliation", which is exactly the situation: ISiteAuditQueue.ReadPendingSinceAsync covers Forwarded rows as well as Pending ones and central dedups on EventId, so the reconciliation pull still delivers them. What is genuinely lost is the operational (SiteCalls) half — unrecoverable regardless, since the tracking row it would have been built from no longer exists.

Three further behaviours, each deliberate:

• Inside the grace window the row is still retried. A missing snapshot is normally a brief write race (the audit row lands microseconds before the tracking row); abandoning on the first failed lookup would discard the operational half of every cached call that lost that race.
• A tracking-store THROW never abandons. A throw is a store fault (locked, corrupt, mid-restore), not a verdict about the row. Those rows stay Pending however old they are.
• Logging is per drain pass, not per row. One summary line each for abandoned, lookup-failed (with the first exception attached) and deferred rows. The deferred case dropped to Debug — being inside the grace window is normal operation, not a warning.

Severity was revised UP during the fix

The original write-up called this a log flood with "no data loss", which understated it. The queue is read oldest-first with a fixed BatchSize (default 256), so once a batch's worth of permanently-unresolvable rows collects at the head, every drain re-reads exactly those rows, fails identically, and never reaches the newer rows behind them. The cached-telemetry path stalls permanently. The audit half still reached central by reconciliation, so "no data loss" held — but "Medium" did not.

Fixed in: SiteAuditTelemetryActor.cs (AbandonUnresolvableAsync + the rewritten skip branches), SiteAuditTelemetryOptions.cs (CachedTrackingGraceSeconds). Tests: SiteAuditTelemetryActorTests — CachedDrain_OrphanRow_PastGrace_IsAbandoned_..., ..._InsideGrace_IsRetried_NotAbandoned, ..._FullBatchOfUnresolvableRows_DoesNotStarveTheQueue, ..._TrackingStoreThrow_DoesNotAbandonTheRow. The pre-existing CachedDrain_OrphanRow_NoTrackingSnapshot_IsSkipped_DoesNotCrash was retargeted, not deleted — it had pinned the defect ("skipped and stays Pending"), so its orphan assertion is inverted while the half that still holds is kept verbatim. Verified non-vacuous: with abandonment disabled the two abandonment tests go red; the two must-NOT-abandon guards stay green in both directions.

The diagnosis below is retained as the record of how it was found.

Summary

SiteAuditTelemetryActor's cached-telemetry drain reads Pending audit rows, looks up each row's tracking snapshot by CorrelationId, and pushes the combined packet to central. When the lookup returns null the row is skipped and deliberately left Pending (SiteAuditTelemetryActor.cs:307), on the reasoning that "central reconciliation will pick it up".

Nothing ever removes such a row from the local drain queue. The next tick re-reads it, fails the same lookup, logs the same warning, and leaves it Pending again — forever. With a batch of unresolvable rows the actor spins at its non-idle rate and emits one warning per row per pass.

Measured on the docker rig: ~2 800 warnings/minute, sustained, surviving both a process restart and a container restart, until the audit database itself was discarded.

[09:57:41 WRN] [Site/scadabridge-site-a-a] Cached-telemetry drain: no tracking snapshot for
  a5392796-291f-4f5b-9fbf-5817c1ec76c7 (TrackedOperationId 59bd4bf8-…); skipping.

Why the rows became unresolvable

Two independent stores must agree:

• the audit rows live in auditlog.db (site-local, and on the docker rig inside the container at /app/auditlog.db, not on the bind-mounted data volume);
• the tracking rows live in OperationTracking, which LocalDb Phase 1 moved into the consolidated LocalDb:Path database (bind-mounted).

Anything that resets one without the other strands every audit row that referenced it. The code comment already anticipates the cause — "possible if the audit row is older than the tracking retention window, or the tracking store was reset" — so this is a known-and-accepted input, not an exotic one.

Two realistic production triggers, neither requiring operator error:

1. Tracking retention expiry. If the tracking retention window elapses before the audit drain catches up — a long central outage, a large backlog — the snapshots are pruned out from under still-Pending audit rows and every one of them becomes a permanent hot-loop entry.
2. Restoring or resetting one store independently of the other, e.g. rebuilding a node's LocalDb file from its peer while its container-local auditlog.db survives.

It was hit here by (2): the two site-a LocalDb databases were dropped during a rig cleanup while auditlog.db — being inside the container — survived.

Why the current handling is not enough

Skipping the row is correct; leaving it Pending with no other state change is not. The row is now in a state it can never leave:

• no attempt counter, so an unresolvable row is indistinguishable from a transiently-failing one;
• no backoff, so the actor runs at full non-idle rate against a queue that can never shrink;
• no terminal state, so it is retried for the life of the database;
• one Warning per row per pass, which buries every other log line on the node.

The "central reconciliation will pick it up" comment is about the audit half reaching central by another path. That may well be true — but it does not release the row from the local drain queue, which is what actually loops.

Suggested fix

Give an unresolvable row somewhere to go. Roughly, in increasing order of effort:

1. Bound the retries. Add an attempt count; past a threshold mark the row terminal (TrackingUnavailable) and stop re-reading it. Emit a single summary Warning with the count rather than one per row per pass.
2. Rate-limit the warning to one per drain episode regardless of row count — the same pattern MaintenanceBackgroundService already uses for the oplog caps-exceeded warning (_snapshotFlagWarned).
3. Push the audit half alone when the tracking snapshot is missing, rather than skipping the row entirely, so the row can be marked emitted and leave the queue. Needs a decision on whether central accepts a packet with no tracking half.

(2) alone would remove the operational damage; (1) or (3) is needed to stop the wasted I/O.

Reproduction

1. Run a site node until it has cached-call audit rows with tracking correlations.
2. Stop the node; delete its consolidated LocalDb database (which holds OperationTracking); leave auditlog.db in place.
3. Start the node. The drain warning repeats indefinitely; the rate does not decay.

Notes

• No data loss. The audit rows are intact and still reach central by the reconciliation path; what is broken is the local drain's ability to ever finish.
• Discovered while cleaning the rig after the LocalDb Phase 2 live gate (docs/plans/2026-07-19-localdb-phase2-live-gate.md), which is also where the related "deleting an instance orphans its buffered messages" observation is recorded.