perf(siteeventlog): pace the site_events purge burst — per-tick delete ceiling + accelerated catch-up (row 34)
Measured the unbounded purge burst and found it can force a full ten-table
snapshot resync, then bounded it. No purge-exemption path was added — the
deletes are still CDC-captured like any other write, which stays ruled out
by design; what changed is how many a single tick performs.
MEASURED (harness: real EventLogPurgeService over a real LocalDb file with
site_events genuinely RegisterReplicated, plus a two-node loopback gRPC
replication pair; docker rig untouched):
393 bytes/row incl. indexes => a 1 GB event log holds ~2,731,947 rows.
Expiring all of them in ONE tick: 285 s, 2,731,947 oplog rows,
284 MB oplog payload, 2.73 M row-version tombstones (7-day retention),
i.e. 2.7x LocalDb's MaxOplogRows default of 1,000,000. Crossing that makes
the 60 s maintenance tick prune the oplog and set needs_snapshot — a FULL
snapshot of all ten replicated tables, bought by a routine retention purge.
The resident backlog is produced-minus-drained, so a healthy peer blunts it:
on loopback the same burst peaked at 391,424 and cleared in 6.0 min. That is
the problem, not the reassurance — the old safety margin was a function of
peer throughput and vanished with the peer down. A bounded tick is safe at
any peer speed.
Drain is not a bottleneck: 7.6k-19.8k tombstones/s (20k / 500k / 2.73M runs).
SHIPPED:
- SiteEventLogOptions.MaxPurgedRowsPerTick (default 250,000; 0 = unlimited
escape hatch, negatives rejected). One SHARED budget across the retention
purge and the cap trim — two independently-capped phases would let a tick
emit twice the intended burst. 4x margin under MaxOplogRows, which is also
shared with every other replicated table's ordinary traffic.
- SiteEventLogOptions.PurgeCatchUpInterval (default 5 min, validated > 0 and
<= PurgeInterval). Without it pacing would make retention converge in 11
DAYS at the daily cadence; with it the worst case converges in ~11 ticks
(~55 min), every burst bounded. 5 min rather than 1 so ticks cannot outrun
the peer's drain and re-accumulate the backlog.
- A ceiling is claimed only when work genuinely remains (bounded EXISTS probe),
so an expired set landing exactly on the budget does not arm a no-op retry.
- Visibility: rate-limited (15 min) Warning naming the ceiling and the
accelerated interval, plus an unthrottled counter
scadabridge.site.event_log.purge_ceiling_hits — its SHAPE distinguishes a
backlog converging from a site whose event rate outruns one paced tick.
- The purge loop is a delay loop rather than a PeriodicTimer, so its cadence
now anchors to the END of the previous purge. Documented.
ROW-33 INTERPLAY VERIFIED: pacing lowers each spike and lengthens total purge
duration, but a full 250k tick drains in ~23 s (~4 min even at 10x a real
link's penalty) against the 15-minute sustained dwell and a 5-minute gap to
the next tick. The UNPACED 2.73M burst had only 2.5x of margin (6.0 min), so
pacing makes the badge assumption robust rather than merely preserving it.
Docs travel together: Component-SiteEventLogging.md, topology-guide.md
("Reading the replication backlog" + the row-33 badge note), CLAUDE.md's
purge-burst sentence, and a dated measurement/decision note under docs/plans/.
Fail-before: the new behavioural tests do not compile against the pre-fix
service (RunPurge was void); with PurgeBudget neutralised to always-unlimited,
6 of the new tests fail and all 19 pre-existing ones pass.
Tests: SiteEventLogging 93/93, Commons 715/715, HealthMonitoring 115/115,
Host 505/505. Solution build clean.
This commit is contained in:
@@ -223,19 +223,35 @@ one of the ten replicated tables, and its retention DELETE is captured by CDC ex
|
||||
ordinary write — by design, since LocalDb Phase 2 there is deliberately no purge-exemption path
|
||||
(the same property that makes a mass DELETE dangerous, which is why `ReplaceAllAsync` was deleted
|
||||
rather than reinstated). One oplog row is therefore queued per deleted event, and the backlog jumps
|
||||
by the size of the day's expired batch.
|
||||
by the size of the batch the tick deleted.
|
||||
|
||||
**The spike is bounded per tick** (register row 34, measured:
|
||||
`docs/plans/2026-08-15-site-events-purge-burst-measurement.md`). One purge tick deletes at most
|
||||
`ScadaBridge:SiteEventLog:MaxPurgedRowsPerTick` rows — default **250 000**, across the retention
|
||||
purge and the cap trim combined — so the height of the spike is a property of the *configuration*,
|
||||
not of how much history had expired. That matters because the burst used to be unbounded: a full
|
||||
1 GB event log holds ~2.73 million rows, and expiring all of them at once queued 2 731 947 oplog
|
||||
rows, **2.7x LocalDb's `MaxOplogRows` (default 1 000 000)** — enough for the maintenance tick to
|
||||
prune the oplog and flag a *full ten-table snapshot resync*. A healthy peer blunts that (the oplog
|
||||
drains while the purge runs — measured peak was 391 000, not 2.73 M), which is precisely the problem:
|
||||
the old safety margin was a function of how fast the peer happened to be, and vanished when the peer
|
||||
was down. Set `MaxPurgedRowsPerTick` to `0` to restore the old unbounded behaviour.
|
||||
|
||||
- **When.** Every `ScadaBridge:SiteEventLog:PurgeInterval` (default **24 h**), plus once at
|
||||
startup. The timer is anchored to the **active node's process start**, not to a wall-clock hour,
|
||||
so the burst lands at a different time of day after each failover or restart — do not expect it
|
||||
at a fixed hour. The storage-cap trim (default 1 GB) can produce the same shape off-schedule, and
|
||||
is usually the larger of the two.
|
||||
startup — **except** after a tick that hit the ceiling, which schedules the next tick after
|
||||
`ScadaBridge:SiteEventLog:PurgeCatchUpInterval` (default **5 minutes**) instead. So a large
|
||||
backlog appears as a *run* of bounded spikes minutes apart rather than one huge one; a
|
||||
worst-case 2.73 M-row backlog converges in ~11 ticks ≈ 55 minutes. The cadence is anchored to the
|
||||
**end of the previous purge**, not to a wall-clock hour, so the burst lands at a different time of
|
||||
day after each failover or restart — do not expect it at a fixed hour. The storage-cap trim
|
||||
(default 1 GB) can produce the same shape off-schedule, and draws from the same per-tick budget.
|
||||
- **Where it shows.** Only on a node with replication configured — the rig's site-a. site-b/site-c
|
||||
have no capture triggers at all and report no backlog for a purge.
|
||||
- **What healthy looks like.** `LocalDbReplicationConnected` stays **true** across the spike, and
|
||||
the backlog drains back to ~0 as the peer acks the batch — within seconds to a couple of minutes
|
||||
depending on batch size (delta messages are bounded by `LocalDb:Replication:MaxBatchBytes`,
|
||||
default 2 MB, and secondarily by `MaxBatchSize`). No dead letters, no schema-mismatch errors.
|
||||
the backlog drains back to ~0 as the peer acks the batch. Measured on a loopback pair at
|
||||
~10 800 tombstones/second, so a full 250 000-row tick drains in well under a minute (delta
|
||||
messages are bounded by `LocalDb:Replication:MaxBatchBytes`, default 2 MB, and secondarily by
|
||||
`MaxBatchSize`). No dead letters, no schema-mismatch errors.
|
||||
- **What is actually wrong.** `LocalDbReplicationConnected` **false** while the backlog climbs, a
|
||||
backlog that keeps rising across successive readings rather than draining, or a backlog that
|
||||
never returns near zero between bursts. Those point at the sync stream — an ApiKey mismatch
|
||||
@@ -248,13 +264,30 @@ logs an Information line next to the purge count:
|
||||
```
|
||||
Purged 41230 events older than 30 days
|
||||
Purged 41230 site_events rows on a replication-enabled node — a transient LocalDb oplog backlog
|
||||
is expected while the deletes replicate to the peer. It drains on its own;
|
||||
of up to that many rows is expected while the deletes replicate to the peer. It drains on its own;
|
||||
LocalDbReplicationConnected staying true with the backlog returning to ~0 is the healthy signature.
|
||||
A tick deletes at most 250000 rows, so the spike is bounded by that number rather than by how much
|
||||
history had expired.
|
||||
```
|
||||
|
||||
An unreplicated node logs only the first line. If a backlog spike has no such line near it in the
|
||||
active node's log, the purge is *not* the explanation and the spike is worth investigating.
|
||||
|
||||
**A tick that hit the ceiling adds a Warning**, rate-limited to one line per 15 minutes so a
|
||||
converging backlog does not flood the log:
|
||||
|
||||
```
|
||||
Event log purge stopped at its per-tick ceiling of 250000 rows (250000 deleted); expired events
|
||||
remain and the next purge runs in 00:05:00 instead of 1.00:00:00.
|
||||
```
|
||||
|
||||
Every hit — throttled line or not — increments the counter
|
||||
`scadabridge_site_event_log_purge_ceiling_hits`. **Read the counter's shape, not a single hit:** a
|
||||
short run of hits that stops is a backlog converging (normal, e.g. the first purge after enabling
|
||||
replication on a long-running site); hits that never stop mean the site writes events faster than
|
||||
one paced tick can purge, i.e. `MaxPurgedRowsPerTick` or `PurgeCatchUpInterval` is mis-sized for
|
||||
that site's event rate.
|
||||
|
||||
**Two dashboard surfaces make this reviewable rather than a log-archaeology exercise** (register
|
||||
row 33):
|
||||
|
||||
@@ -263,7 +296,13 @@ row 33):
|
||||
above `ScadaBridge:HealthMonitoring:LocalDbBacklogThreshold` (default **1000**) continuously for
|
||||
`ScadaBridge:HealthMonitoring:LocalDbBacklogSustainedFor` (default **15 minutes**). The daily
|
||||
purge burst described above drains in seconds to a couple of minutes, so it never reaches the
|
||||
dwell and never badges the site — which is the entire reason the dwell exists. A bare threshold
|
||||
dwell and never badges the site — which is the entire reason the dwell exists. **The per-tick
|
||||
ceiling reinforces that rather than threatening it:** pacing lowers the spike (250 000 rows
|
||||
maximum instead of an unbounded 2.7 M) at the cost of more ticks, and a 250 000-row tick was
|
||||
measured draining in well under a minute — even allowing 10x for a real inter-node link that is
|
||||
~4 minutes, against a 15-minute dwell and a 5-minute gap to the next tick. (The *unpaced* 2.73 M
|
||||
burst drained in 6.0 minutes on loopback: inside the dwell, but with only 2.5x of margin, so a
|
||||
materially slower link would have badged.) A bare threshold
|
||||
would fire once a day on every replicated site and be ignored within a week. The dwell is
|
||||
validated to be at least one `ReportInterval`, since a shorter window could be satisfied by a
|
||||
single sample and would collapse back into a bare threshold.
|
||||
|
||||
Reference in New Issue
Block a user