perf(siteeventlog): pace the site_events purge burst — per-tick delete ceiling + accelerated catch-up (row 34)
Measured the unbounded purge burst and found it can force a full ten-table
snapshot resync, then bounded it. No purge-exemption path was added — the
deletes are still CDC-captured like any other write, which stays ruled out
by design; what changed is how many a single tick performs.
MEASURED (harness: real EventLogPurgeService over a real LocalDb file with
site_events genuinely RegisterReplicated, plus a two-node loopback gRPC
replication pair; docker rig untouched):
393 bytes/row incl. indexes => a 1 GB event log holds ~2,731,947 rows.
Expiring all of them in ONE tick: 285 s, 2,731,947 oplog rows,
284 MB oplog payload, 2.73 M row-version tombstones (7-day retention),
i.e. 2.7x LocalDb's MaxOplogRows default of 1,000,000. Crossing that makes
the 60 s maintenance tick prune the oplog and set needs_snapshot — a FULL
snapshot of all ten replicated tables, bought by a routine retention purge.
The resident backlog is produced-minus-drained, so a healthy peer blunts it:
on loopback the same burst peaked at 391,424 and cleared in 6.0 min. That is
the problem, not the reassurance — the old safety margin was a function of
peer throughput and vanished with the peer down. A bounded tick is safe at
any peer speed.
Drain is not a bottleneck: 7.6k-19.8k tombstones/s (20k / 500k / 2.73M runs).
SHIPPED:
- SiteEventLogOptions.MaxPurgedRowsPerTick (default 250,000; 0 = unlimited
escape hatch, negatives rejected). One SHARED budget across the retention
purge and the cap trim — two independently-capped phases would let a tick
emit twice the intended burst. 4x margin under MaxOplogRows, which is also
shared with every other replicated table's ordinary traffic.
- SiteEventLogOptions.PurgeCatchUpInterval (default 5 min, validated > 0 and
<= PurgeInterval). Without it pacing would make retention converge in 11
DAYS at the daily cadence; with it the worst case converges in ~11 ticks
(~55 min), every burst bounded. 5 min rather than 1 so ticks cannot outrun
the peer's drain and re-accumulate the backlog.
- A ceiling is claimed only when work genuinely remains (bounded EXISTS probe),
so an expired set landing exactly on the budget does not arm a no-op retry.
- Visibility: rate-limited (15 min) Warning naming the ceiling and the
accelerated interval, plus an unthrottled counter
scadabridge.site.event_log.purge_ceiling_hits — its SHAPE distinguishes a
backlog converging from a site whose event rate outruns one paced tick.
- The purge loop is a delay loop rather than a PeriodicTimer, so its cadence
now anchors to the END of the previous purge. Documented.
ROW-33 INTERPLAY VERIFIED: pacing lowers each spike and lengthens total purge
duration, but a full 250k tick drains in ~23 s (~4 min even at 10x a real
link's penalty) against the 15-minute sustained dwell and a 5-minute gap to
the next tick. The UNPACED 2.73M burst had only 2.5x of margin (6.0 min), so
pacing makes the badge assumption robust rather than merely preserving it.
Docs travel together: Component-SiteEventLogging.md, topology-guide.md
("Reading the replication backlog" + the row-33 badge note), CLAUDE.md's
purge-burst sentence, and a dated measurement/decision note under docs/plans/.
Fail-before: the new behavioural tests do not compile against the pre-fix
service (RunPurge was void); with PurgeBudget neutralised to always-unlimited,
6 of the new tests fail and all 19 pre-existing ones pass.
Tests: SiteEventLogging 93/93, Commons 715/715, HealthMonitoring 115/115,
Host 505/505. Solution build clean.
This commit is contained in:
@@ -111,6 +111,21 @@ public static class ScadaBridgeTelemetry
|
||||
Meter.CreateCounter<long>("scadabridge.central.transport.import_sessions.rejected", unit: "1",
|
||||
description: "Bundle-import sessions rejected because the concurrent-import-session cap was exhausted.");
|
||||
|
||||
/// <summary>
|
||||
/// Register row 34: incremented once per site-event-log purge tick that stopped at
|
||||
/// <c>SiteEventLogOptions.MaxPurgedRowsPerTick</c> with expired rows still to delete.
|
||||
///
|
||||
/// A single hit is routine (a backlog is being worked through on accelerated catch-up
|
||||
/// ticks) — what the series is for is the SHAPE. A short burst of hits that stops is a
|
||||
/// backlog converging; hits that never stop mean the event log is being written faster
|
||||
/// than one paced tick can purge, i.e. the ceiling or the catch-up interval is now
|
||||
/// mis-sized for this site's event rate. The paired log line is rate-limited and so
|
||||
/// cannot answer that question on its own.
|
||||
/// </summary>
|
||||
private static readonly Counter<long> _siteEventLogPurgeCeilingHits =
|
||||
Meter.CreateCounter<long>("scadabridge.site.event_log.purge_ceiling_hits", unit: "1",
|
||||
description: "Site event log purge ticks that stopped at the per-tick delete ceiling with work remaining.");
|
||||
|
||||
// ---------------- Histograms ----------------
|
||||
|
||||
/// <summary>
|
||||
@@ -272,6 +287,13 @@ public static class ScadaBridgeTelemetry
|
||||
/// </summary>
|
||||
public static void RecordImportSessionRejected() => _transportImportSessionsRejected.Add(1);
|
||||
|
||||
/// <summary>
|
||||
/// Records one site-event-log purge tick that stopped at its per-tick delete ceiling
|
||||
/// (register row 34). Call ONLY when rows genuinely remain to be purged — a tick that
|
||||
/// finished the work and happened to land exactly on the ceiling has nothing to report.
|
||||
/// </summary>
|
||||
public static void RecordSiteEventLogPurgeCeilingHit() => _siteEventLogPurgeCeilingHits.Add(1);
|
||||
|
||||
/// <summary>
|
||||
/// Records that a central→site gRPC subscriber stream opened (increments
|
||||
/// <c>scadabridge.site.stream.subscribers.active</c>). Renamed from
|
||||
|
||||
Reference in New Issue
Block a user