perf(siteeventlog): pace the site_events purge burst — per-tick delete ceiling + accelerated catch-up (row 34)
Measured the unbounded purge burst and found it can force a full ten-table
snapshot resync, then bounded it. No purge-exemption path was added — the
deletes are still CDC-captured like any other write, which stays ruled out
by design; what changed is how many a single tick performs.
MEASURED (harness: real EventLogPurgeService over a real LocalDb file with
site_events genuinely RegisterReplicated, plus a two-node loopback gRPC
replication pair; docker rig untouched):
393 bytes/row incl. indexes => a 1 GB event log holds ~2,731,947 rows.
Expiring all of them in ONE tick: 285 s, 2,731,947 oplog rows,
284 MB oplog payload, 2.73 M row-version tombstones (7-day retention),
i.e. 2.7x LocalDb's MaxOplogRows default of 1,000,000. Crossing that makes
the 60 s maintenance tick prune the oplog and set needs_snapshot — a FULL
snapshot of all ten replicated tables, bought by a routine retention purge.
The resident backlog is produced-minus-drained, so a healthy peer blunts it:
on loopback the same burst peaked at 391,424 and cleared in 6.0 min. That is
the problem, not the reassurance — the old safety margin was a function of
peer throughput and vanished with the peer down. A bounded tick is safe at
any peer speed.
Drain is not a bottleneck: 7.6k-19.8k tombstones/s (20k / 500k / 2.73M runs).
SHIPPED:
- SiteEventLogOptions.MaxPurgedRowsPerTick (default 250,000; 0 = unlimited
escape hatch, negatives rejected). One SHARED budget across the retention
purge and the cap trim — two independently-capped phases would let a tick
emit twice the intended burst. 4x margin under MaxOplogRows, which is also
shared with every other replicated table's ordinary traffic.
- SiteEventLogOptions.PurgeCatchUpInterval (default 5 min, validated > 0 and
<= PurgeInterval). Without it pacing would make retention converge in 11
DAYS at the daily cadence; with it the worst case converges in ~11 ticks
(~55 min), every burst bounded. 5 min rather than 1 so ticks cannot outrun
the peer's drain and re-accumulate the backlog.
- A ceiling is claimed only when work genuinely remains (bounded EXISTS probe),
so an expired set landing exactly on the budget does not arm a no-op retry.
- Visibility: rate-limited (15 min) Warning naming the ceiling and the
accelerated interval, plus an unthrottled counter
scadabridge.site.event_log.purge_ceiling_hits — its SHAPE distinguishes a
backlog converging from a site whose event rate outruns one paced tick.
- The purge loop is a delay loop rather than a PeriodicTimer, so its cadence
now anchors to the END of the previous purge. Documented.
ROW-33 INTERPLAY VERIFIED: pacing lowers each spike and lengthens total purge
duration, but a full 250k tick drains in ~23 s (~4 min even at 10x a real
link's penalty) against the 15-minute sustained dwell and a 5-minute gap to
the next tick. The UNPACED 2.73M burst had only 2.5x of margin (6.0 min), so
pacing makes the badge assumption robust rather than merely preserving it.
Docs travel together: Component-SiteEventLogging.md, topology-guide.md
("Reading the replication backlog" + the row-33 badge note), CLAUDE.md's
purge-burst sentence, and a dated measurement/decision note under docs/plans/.
Fail-before: the new behavioural tests do not compile against the pre-fix
service (RunPurge was void); with PurgeBudget neutralised to always-unlimited,
6 of the new tests fail and all 19 pre-existing ones pass.
Tests: SiteEventLogging 93/93, Commons 715/715, HealthMonitoring 115/115,
Host 505/505. Solution build clean.
This commit is contained in:
@@ -28,6 +28,43 @@ public class SiteEventLogOptions
|
||||
/// <summary>Interval between purge runs; default 24 hours.</summary>
|
||||
public TimeSpan PurgeInterval { get; set; } = TimeSpan.FromHours(24);
|
||||
/// <summary>
|
||||
/// Ceiling on how many <c>site_events</c> rows ONE purge tick may delete, across the
|
||||
/// retention purge and the storage-cap trim combined. Default 250 000; <c>0</c> means
|
||||
/// unlimited (the pre-pacing behaviour, kept as an escape hatch).
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// <b>Why a ceiling exists at all.</b> <c>site_events</c> is CDC-captured on a
|
||||
/// replication-enabled node and there is deliberately no purge-exemption path, so a purge
|
||||
/// queues one oplog row per deleted event. Measured worst case (register row 34,
|
||||
/// <c>docs/plans/2026-08-15-site-events-purge-burst-measurement.md</c>): a 1 GB event log
|
||||
/// holds ~2.73 million rows at 393 bytes/row, and expiring all of them in one tick queued
|
||||
/// 2 731 947 oplog rows — <b>2.7x over LocalDb's <c>MaxOplogRows</c> default of
|
||||
/// 1 000 000</b>. Crossing that ceiling makes LocalDb's maintenance tick prune the oplog
|
||||
/// and set <c>needs_snapshot</c>, which costs a FULL snapshot resync of all ten replicated
|
||||
/// tables. The ceiling keeps a single tick's burst safely inside that budget (250 000 is
|
||||
/// a 4x margin) while <see cref="PurgeCatchUpInterval"/> keeps retention converging.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// The remainder is not dropped: a tick that hits the ceiling schedules the next tick after
|
||||
/// <see cref="PurgeCatchUpInterval"/> instead of <see cref="PurgeInterval"/>, so a backlog
|
||||
/// drains over minutes rather than days while every individual burst stays bounded.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
public int MaxPurgedRowsPerTick { get; set; } = 250_000;
|
||||
/// <summary>
|
||||
/// Delay before the next purge tick when the previous tick stopped at
|
||||
/// <see cref="MaxPurgedRowsPerTick"/>; default 5 minutes. Ignored when the ceiling was not
|
||||
/// reached — an ordinary tick waits <see cref="PurgeInterval"/>.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// Sized so the peer drains one tick's burst before the next one lands: 250 000 tombstones
|
||||
/// measured at ~10 800 rows/s over a loopback pair (≈23 s), leaving an order of magnitude of
|
||||
/// headroom for a real inter-node link. A worst-case 2.73 M-row backlog therefore converges
|
||||
/// in ~11 ticks ≈ 55 minutes instead of 11 days at the daily interval.
|
||||
/// </remarks>
|
||||
public TimeSpan PurgeCatchUpInterval { get; set; } = TimeSpan.FromMinutes(5);
|
||||
/// <summary>
|
||||
/// Bound on the background write queue. Default 10 000 events.
|
||||
/// Overflow uses <c>BoundedChannelFullMode.DropOldest</c> — callers never block; the
|
||||
/// dropped event's <c>Task</c> is faulted and <c>FailedWriteCount</c> is incremented
|
||||
|
||||
Reference in New Issue
Block a user