Merge row 34: site_events purge burst measured (2.73M-row worst case = 2.7x MaxOplogRows) and paced — shared 250k/tick budget, 5-min catch-up, ceiling-hit counter
This commit is contained in:
@@ -94,12 +94,35 @@ Each event entry contains:
|
||||
documented trade of not configuring a peer, unchanged by this policy.
|
||||
- **Retention**: 30 days. A **daily background job** runs on the active node and deletes all events older than 30 days. Hard delete — no archival. Deletes are sliced into bounded 1000-row batches per DELETE statement rather than one unbounded statement, so a large expired backlog does not hold the write lock (and every concurrent recorder flush) for the duration of the purge.
|
||||
- **Storage cap**: A configurable maximum database size (default: 1 GB) is enforced. If the storage cap is reached before the 30-day retention window, the oldest events are purged first. This prevents disk exhaustion from alarm storms, script failure loops, or connection flapping.
|
||||
- **Per-tick delete ceiling** (`MaxPurgedRowsPerTick`, default **250 000**; `0` = unlimited): one
|
||||
purge tick deletes at most this many rows, **shared** by the retention purge and the storage-cap
|
||||
trim — two independently-capped phases would let a tick emit twice the intended burst. A tick that
|
||||
stops at the ceiling schedules the next tick after `PurgeCatchUpInterval` (default **5 minutes**)
|
||||
instead of `PurgeInterval`, so retention still converges: a worst-case backlog drains over
|
||||
~11 accelerated ticks (≈55 minutes) rather than 11 days at the daily cadence, with every
|
||||
individual burst bounded. The ceiling exists because the purge is CDC-captured (next bullet) and
|
||||
an unbounded tick could exceed LocalDb's oplog budget; measured evidence and the chosen defaults
|
||||
are in `docs/plans/2026-08-15-site-events-purge-burst-measurement.md`. A ceiling-limited tick logs
|
||||
a Warning (rate-limited to one line per 15 minutes) and increments
|
||||
`scadabridge.site.event_log.purge_ceiling_hits` — the counter is the unthrottled series that
|
||||
distinguishes a backlog converging from a site whose event rate outruns one paced tick.
|
||||
Consequence of the accelerated retry: the purge loop is a delay loop, not a `PeriodicTimer`, so
|
||||
the cadence is anchored to the **end** of the previous purge rather than to process start.
|
||||
- **Purge deletes are CDC-captured on a replicated site — the resulting backlog spike is expected.**
|
||||
Retention and cap deletes are ordinary row changes on a replicated table; there is deliberately no
|
||||
purge-exemption path (LocalDb Phase 2 — CDC does all three jobs, and a table-wide exemption is the
|
||||
same mechanism that made `ReplaceAllAsync` unsafe). One oplog row is queued per deleted event, so
|
||||
the site health report's `LocalDbOplogBacklog` (Prometheus `localdb_oplog_depth`) jumps by the size
|
||||
of the batch at purge time and drains as the peer acks it. **Healthy signature:**
|
||||
of the batch at purge time and drains as the peer acks it. **The height of that jump is bounded by
|
||||
`MaxPurgedRowsPerTick`, not by how much history expired** — measured (register row 34): a full 1 GB
|
||||
event log holds ~2.73 million rows at 393 bytes/row, and purging it unbounded queued 2 731 947
|
||||
oplog rows, **2.7x LocalDb's `MaxOplogRows` default of 1 000 000**, which would make the
|
||||
maintenance tick prune the oplog and flag a full ten-table snapshot resync. How much of that is
|
||||
*resident* depends on the peer — the oplog drains concurrently, and a loopback peer held the same
|
||||
burst to a 391 000 peak — so the overrun is conditional on peer throughput and unconditional when
|
||||
the peer is down. Bounding the tick is what makes the outcome independent of the peer. Pacing the
|
||||
tick, not exempting the purge from capture, is the fix — the exemption remains ruled out.
|
||||
**Healthy signature:**
|
||||
`LocalDbReplicationConnected` stays true across the spike and the backlog returns to ~0; a backlog
|
||||
that keeps climbing, or climbs while `LocalDbReplicationConnected` is false, is a genuine
|
||||
replication fault and not the purge. The purge logs an Information line naming the row count and
|
||||
|
||||
Reference in New Issue
Block a user