Merge row 34: site_events purge burst measured (2.73M-row worst case = 2.7x MaxOplogRows) and paced — shared 250k/tick budget, 5-min catch-up, ceiling-hit counter

This commit is contained in:
Joseph Doherty
2026-08-16 04:17:16 -04:00
10 changed files with 823 additions and 31 deletions
@@ -94,12 +94,35 @@ Each event entry contains:
documented trade of not configuring a peer, unchanged by this policy.
- **Retention**: 30 days. A **daily background job** runs on the active node and deletes all events older than 30 days. Hard delete — no archival. Deletes are sliced into bounded 1000-row batches per DELETE statement rather than one unbounded statement, so a large expired backlog does not hold the write lock (and every concurrent recorder flush) for the duration of the purge.
- **Storage cap**: A configurable maximum database size (default: 1 GB) is enforced. If the storage cap is reached before the 30-day retention window, the oldest events are purged first. This prevents disk exhaustion from alarm storms, script failure loops, or connection flapping.
- **Per-tick delete ceiling** (`MaxPurgedRowsPerTick`, default **250 000**; `0` = unlimited): one
purge tick deletes at most this many rows, **shared** by the retention purge and the storage-cap
trim — two independently-capped phases would let a tick emit twice the intended burst. A tick that
stops at the ceiling schedules the next tick after `PurgeCatchUpInterval` (default **5 minutes**)
instead of `PurgeInterval`, so retention still converges: a worst-case backlog drains over
~11 accelerated ticks (≈55 minutes) rather than 11 days at the daily cadence, with every
individual burst bounded. The ceiling exists because the purge is CDC-captured (next bullet) and
an unbounded tick could exceed LocalDb's oplog budget; measured evidence and the chosen defaults
are in `docs/plans/2026-08-15-site-events-purge-burst-measurement.md`. A ceiling-limited tick logs
a Warning (rate-limited to one line per 15 minutes) and increments
`scadabridge.site.event_log.purge_ceiling_hits` — the counter is the unthrottled series that
distinguishes a backlog converging from a site whose event rate outruns one paced tick.
Consequence of the accelerated retry: the purge loop is a delay loop, not a `PeriodicTimer`, so
the cadence is anchored to the **end** of the previous purge rather than to process start.
- **Purge deletes are CDC-captured on a replicated site — the resulting backlog spike is expected.**
Retention and cap deletes are ordinary row changes on a replicated table; there is deliberately no
purge-exemption path (LocalDb Phase 2 — CDC does all three jobs, and a table-wide exemption is the
same mechanism that made `ReplaceAllAsync` unsafe). One oplog row is queued per deleted event, so
the site health report's `LocalDbOplogBacklog` (Prometheus `localdb_oplog_depth`) jumps by the size
of the batch at purge time and drains as the peer acks it. **Healthy signature:**
of the batch at purge time and drains as the peer acks it. **The height of that jump is bounded by
`MaxPurgedRowsPerTick`, not by how much history expired** — measured (register row 34): a full 1 GB
event log holds ~2.73 million rows at 393 bytes/row, and purging it unbounded queued 2 731 947
oplog rows, **2.7x LocalDb's `MaxOplogRows` default of 1 000 000**, which would make the
maintenance tick prune the oplog and flag a full ten-table snapshot resync. How much of that is
*resident* depends on the peer — the oplog drains concurrently, and a loopback peer held the same
burst to a 391 000 peak — so the overrun is conditional on peer throughput and unconditional when
the peer is down. Bounding the tick is what makes the outcome independent of the peer. Pacing the
tick, not exempting the purge from capture, is the fix — the exemption remains ruled out.
**Healthy signature:**
`LocalDbReplicationConnected` stays true across the spike and the backlog returns to ~0; a backlog
that keeps climbing, or climbs while `LocalDbReplicationConnected` is false, is a genuine
replication fault and not the purge. The purge logs an Information line naming the row count and