perf(siteeventlog): pace the site_events purge burst — per-tick delete ceiling + accelerated catch-up (row 34)
Measured the unbounded purge burst and found it can force a full ten-table
snapshot resync, then bounded it. No purge-exemption path was added — the
deletes are still CDC-captured like any other write, which stays ruled out
by design; what changed is how many a single tick performs.
MEASURED (harness: real EventLogPurgeService over a real LocalDb file with
site_events genuinely RegisterReplicated, plus a two-node loopback gRPC
replication pair; docker rig untouched):
393 bytes/row incl. indexes => a 1 GB event log holds ~2,731,947 rows.
Expiring all of them in ONE tick: 285 s, 2,731,947 oplog rows,
284 MB oplog payload, 2.73 M row-version tombstones (7-day retention),
i.e. 2.7x LocalDb's MaxOplogRows default of 1,000,000. Crossing that makes
the 60 s maintenance tick prune the oplog and set needs_snapshot — a FULL
snapshot of all ten replicated tables, bought by a routine retention purge.
The resident backlog is produced-minus-drained, so a healthy peer blunts it:
on loopback the same burst peaked at 391,424 and cleared in 6.0 min. That is
the problem, not the reassurance — the old safety margin was a function of
peer throughput and vanished with the peer down. A bounded tick is safe at
any peer speed.
Drain is not a bottleneck: 7.6k-19.8k tombstones/s (20k / 500k / 2.73M runs).
SHIPPED:
- SiteEventLogOptions.MaxPurgedRowsPerTick (default 250,000; 0 = unlimited
escape hatch, negatives rejected). One SHARED budget across the retention
purge and the cap trim — two independently-capped phases would let a tick
emit twice the intended burst. 4x margin under MaxOplogRows, which is also
shared with every other replicated table's ordinary traffic.
- SiteEventLogOptions.PurgeCatchUpInterval (default 5 min, validated > 0 and
<= PurgeInterval). Without it pacing would make retention converge in 11
DAYS at the daily cadence; with it the worst case converges in ~11 ticks
(~55 min), every burst bounded. 5 min rather than 1 so ticks cannot outrun
the peer's drain and re-accumulate the backlog.
- A ceiling is claimed only when work genuinely remains (bounded EXISTS probe),
so an expired set landing exactly on the budget does not arm a no-op retry.
- Visibility: rate-limited (15 min) Warning naming the ceiling and the
accelerated interval, plus an unthrottled counter
scadabridge.site.event_log.purge_ceiling_hits — its SHAPE distinguishes a
backlog converging from a site whose event rate outruns one paced tick.
- The purge loop is a delay loop rather than a PeriodicTimer, so its cadence
now anchors to the END of the previous purge. Documented.
ROW-33 INTERPLAY VERIFIED: pacing lowers each spike and lengthens total purge
duration, but a full 250k tick drains in ~23 s (~4 min even at 10x a real
link's penalty) against the 15-minute sustained dwell and a 5-minute gap to
the next tick. The UNPACED 2.73M burst had only 2.5x of margin (6.0 min), so
pacing makes the badge assumption robust rather than merely preserving it.
Docs travel together: Component-SiteEventLogging.md, topology-guide.md
("Reading the replication backlog" + the row-33 badge note), CLAUDE.md's
purge-burst sentence, and a dated measurement/decision note under docs/plans/.
Fail-before: the new behavioural tests do not compile against the pre-fix
service (RunPurge was void); with PurgeBudget neutralised to always-unlimited,
6 of the new tests fail and all 19 pre-existing ones pass.
Tests: SiteEventLogging 93/93, Commons 715/715, HealthMonitoring 115/115,
Host 505/505. Solution build clean.
This commit is contained in:
Joseph Doherty
@@ -0,0 +1,214 @@
|
||||
# `site_events` purge burst — measurement + pacing decision (register row 34)
|
||||
|
||||
**Date:** 2026-08-15
|
||||
**Row:** deferred-work register #34 — *"`site_events` purge burst is unbounded in the oplog and unmeasured at scale."*
|
||||
**Outcome:** measured; **pacing implemented** (a per-tick delete ceiling with an accelerated
|
||||
catch-up tick). One separate defect found on the storage-cap path — see *New issue* at the end.
|
||||
|
||||
---
|
||||
|
||||
## 1. What the row asked
|
||||
|
||||
`site_events` lives in the consolidated LocalDb file and is CDC-replicated on a replication-enabled
|
||||
site. The daily retention purge (30 days) and the storage-cap trim (1 GB) slice their deletes at
|
||||
1000 rows per DELETE statement, but the **whole** expired set went in one tick, so the oplog gained
|
||||
one row per deleted event with no pacing. The wire path is bounded (`MaxBatchBytes` 2 MB; the rig
|
||||
also pins `MaxBatchSize: 64` to bound the DB read page) but the oplog **write** was not, and nothing
|
||||
had been measured.
|
||||
|
||||
The design commitment that had to survive, and did: **there is no purge-exemption path.** Purge
|
||||
deletes are captured like any other write — the same property that made `ReplaceAllAsync` unsafe to
|
||||
reintroduce. Nothing below adds an exemption or an oplog bypass; the fix bounds how much one tick
|
||||
deletes, not what gets captured.
|
||||
|
||||
## 2. How it was measured
|
||||
|
||||
A throwaway console harness (`dotnet` console app, scratchpad — not committed) running the **real**
|
||||
`EventLogPurgeService` over a **real** `ZB.MOM.WW.LocalDb` file with `site_events` genuinely
|
||||
`RegisterReplicated`, plus a two-node loopback replication pair for the drain leg:
|
||||
|
||||
| Phase | What it does |
|
||||
| --- | --- |
|
||||
| `calibrate` | Seeds realistically-shaped rows, measures `page_count`/`freelist_count`/`page_size` delta ⇒ bytes per row ⇒ rows at the 1 GB cap. |
|
||||
| `burst` | Seeds N expired rows, registers CDC, parks the oplog at "everything acked", runs one real `RunPurge()`. Records purge wall time, oplog rows, oplog payload bytes, row-version tombstones, file size before/after. |
|
||||
| `cap` | Same but with rows *inside* retention and a small `MaxStorageMb`, so only the storage-cap trim runs. Run twice — replicated and not — to isolate the CDC contribution. |
|
||||
| `drain` | Two full replication stacks (initiator + passive `MapZbLocalDbSync`) over Kestrel h2c on loopback, both nodes registered for `site_events`. Seeds through the normal write path so the pair converges naturally, then purges on node A and polls the O(1) backlog expression until it returns to 0 and node B's table is empty. |
|
||||
|
||||
Row content mirrors the post-WP3.2 event stream: mostly short Info rows, 1-in-10 carrying a details
|
||||
payload, instance ids on 2-in-3. `docker/` was not touched.
|
||||
|
||||
Machine: macOS (darwin 25.5.0), .NET 10, Release build, server GC, local SSD. Loopback drain numbers
|
||||
are therefore an **upper bound on throughput** — a real inter-node link is slower, and the margins
|
||||
below are stated with that in mind.
|
||||
|
||||
## 3. Numbers
|
||||
|
||||
### 3.1 How big the worst case actually is
|
||||
|
||||
```
|
||||
calibrate: 200,000 rows
|
||||
bytes per row (incl. the four indexes) 393
|
||||
rows at the 1 GB storage cap 2,731,947
|
||||
```
|
||||
|
||||
The 1 GB cap is the real bound on a single retention burst: the worst realistic case is a log at the
|
||||
cap in which everything has expired (a site restarted after a long outage, a clock correction, or
|
||||
simply the first purge after `site_events` retention was ever satisfied).
|
||||
|
||||
### 3.2 The unpaced retention burst
|
||||
|
||||
| Seeded expired rows | Purge wall time | Oplog rows queued | Oplog payload | Row-version tombstones | File before → after |
|
||||
| ---: | ---: | ---: | ---: | ---: | --- |
|
||||
| 500,000 | 36.2 s | 500,000 | 52.0 MB | 500,000 | 254 MB → 129 MB |
|
||||
| **2,731,947** (1 GB cap) | **285.1 s** | **2,731,947** | **284.1 MB** | **2,731,947** | 1.39 GB → 707 MB |
|
||||
|
||||
**The headline finding:** the worst-case burst *queues* **2.7x LocalDb's `MaxOplogRows` default of
|
||||
1,000,000**. Crossing that ceiling is not cosmetic — `OplogStore.EnforceCapsAsync`, which the
|
||||
replication maintenance service runs every 60 s, prunes the oldest oplog rows **and sets
|
||||
`needs_snapshot = 1`**, the database-wide flag. The next handshake therefore serves a **full
|
||||
snapshot of all ten registered tables**. A routine retention purge could, on its own, force a full
|
||||
resync of the site's entire replicated state.
|
||||
|
||||
**How much of that 2.73 M is actually resident depends on the peer** — the oplog is drained
|
||||
concurrently, so the *peak backlog* is the burst minus what the peer acks while the purge runs. §3.3
|
||||
measures it: on a loopback pair the peak was 391 k, not 2.73 M, because the peer kept up. That does
|
||||
not make the risk theoretical, it makes it **conditional on peer speed**, which is exactly the
|
||||
property a design should not depend on:
|
||||
|
||||
- **peer down / disconnected** — nothing drains, the full 2.73 M is resident. Over the cap outright.
|
||||
- **peer slower than ~63 % of loopback** — the production deficit outruns the drain and the peak
|
||||
crosses 1 M. Arithmetic from §3.3: the purge produced 8,548 rows/s and the loopback peer consumed
|
||||
7,585 rows/s; halve the peer's rate and the peak lands ≈1.5 M.
|
||||
- **peer at loopback speed** — peak 391 k, under the cap, but still 391x the row-33 badge threshold.
|
||||
|
||||
Pacing removes the dependency entirely: a 250 k tick cannot exceed the cap at *any* peer speed,
|
||||
including a peer that is down.
|
||||
|
||||
Secondary costs at worst case: 285 s of purge (the write lock is taken and released 2,732 times, so
|
||||
recorder flushes interleave — but they contend for nearly five minutes), 284 MB of oplog payload,
|
||||
and 2.73 M row-version tombstones retained for `TombstoneRetention` (7 days). The file was still
|
||||
707 MB *after* deleting a 1 GB event log, because the replication bookkeeping the delete generated
|
||||
is itself in that file.
|
||||
|
||||
### 3.3 Drain
|
||||
|
||||
Two-node loopback pair, `MaxBatchSize: 64` (the rig's pin), `MaxBatchBytes` at its 2 MB default.
|
||||
Both nodes registered for `site_events`; rows are seeded through the normal write path so the pair
|
||||
converges naturally before the purge runs.
|
||||
|
||||
| Rows | Insert leg (seed + replicate) | Purge | Backlog → 0 | **Peak backlog** | Tombstones/s | Inside the 15-min dwell? |
|
||||
| ---: | ---: | ---: | ---: | ---: | ---: | --- |
|
||||
| 20,000 | 1.3 s (15.0 k/s) | 0.5 s | 1.0 s | 17,424 | 19,807 | yes |
|
||||
| 500,000 | 65.7 s (7.6 k/s) | 34.4 s | 46.3 s | 150,912 | 10,795 | yes |
|
||||
| **2,731,947** (1 GB cap) | 996.9 s (2.7 k/s) | 319.6 s | **360.2 s** | **391,424** | 7,585 | yes (6.0 min) |
|
||||
|
||||
**Drain is not the bottleneck, and a healthy peer blunts the burst.** At every size the pump
|
||||
consumed tombstones almost as fast as the purge produced them, so the resident backlog stayed far
|
||||
below the burst size — 391 k peak on a 2.73 M burst. The tombstone leg is roughly 4x cheaper per row
|
||||
than the insert leg, as expected: a delete oplog entry carries a null `row_json`.
|
||||
|
||||
**That is exactly why the unpaced burst was still unsafe**, though. The peak is
|
||||
`produced − drained`, so it is a function of the *peer's* throughput, and the margin evaporates as
|
||||
soon as the peer is slower than loopback or absent (see §3.2). A design whose oplog-cap safety
|
||||
depends on the peer keeping up is one bad link away from a full ten-table resync.
|
||||
|
||||
**Row-33 interplay (the sustained badge).** The badge needs the backlog above 1000 continuously for
|
||||
15 minutes.
|
||||
|
||||
- **Paced (shipped default).** A full 250 000-row tick drains in ~23 s at the measured rate; even
|
||||
assuming a real link is **10x slower** than loopback it clears in ~4 minutes — comfortably inside
|
||||
the dwell, with the next tick 5 minutes away. Pacing lowers each spike and lengthens *total*
|
||||
purge duration, but every individual spike still drains between ticks, so the badge assumption
|
||||
holds with a large margin. **Verified, not assumed:** the 500 k measurement is 2x a paced tick and
|
||||
drained in 46 s.
|
||||
- **Unpaced (previous behaviour).** 6.0 minutes on loopback — inside the dwell, but only 2.5x of
|
||||
margin, at a peak of 391 k (391x the badge threshold). At a 3x-slower link the same burst badges.
|
||||
So pacing does not merely preserve the row-33 assumption; it is what makes it robust.
|
||||
|
||||
### 3.4 The storage-cap trim
|
||||
|
||||
`MaxStorageMb = 64`, 400,000 rows seeded inside the retention window (so only the cap path runs):
|
||||
|
||||
| Replicated? | Events deleted | Remaining | Measured size before → after | Reached the cap? |
|
||||
| --- | ---: | ---: | --- | --- |
|
||||
| no | 248,000 | 152,000 | 158.0 MB → 66.98 MB | **yes** (cap 67.11 MB) |
|
||||
| yes | 174,000 | 226,000 | 203.6 MB → 167.7 MB | **no — stopped 2.5x over cap** |
|
||||
|
||||
See *New issue* below.
|
||||
|
||||
## 4. Decision — pace it
|
||||
|
||||
Pacing is warranted, on the strength of §3.2: an unbounded tick can exceed `MaxOplogRows` and buy a
|
||||
full ten-table snapshot resync. Drain *speed* (§3.3) was not the reason — that is healthy — but the
|
||||
fact that the safety margin **depends on** drain speed is: with the peer down, the whole 2.73 M
|
||||
burst is resident and the cap is breached outright. Bounding the tick makes the outcome independent
|
||||
of the peer, which is the property the previous design lacked.
|
||||
|
||||
**Shipped:**
|
||||
|
||||
- `SiteEventLogOptions.MaxPurgedRowsPerTick`, default **250 000**, `0` = unlimited.
|
||||
- Ceiling on rows deleted by **one tick**, **shared** between the retention purge and the cap
|
||||
trim. Two independently-capped phases would let a tick emit twice the intended burst.
|
||||
- `0` (not a negative, which the validator rejects) is the documented escape hatch back to the
|
||||
pre-row-34 behaviour, byte-for-byte.
|
||||
- **Why 250 000:** a 4x margin under `MaxOplogRows` = 1 000 000, and that margin is shared with
|
||||
every *other* replicated table's ordinary traffic, which also occupies the same oplog. At the
|
||||
measured rates one such tick costs ~17 s of purge and ~23 s of drain.
|
||||
- `SiteEventLogOptions.PurgeCatchUpInterval`, default **5 minutes**.
|
||||
- A tick that stopped at the ceiling schedules the next tick after this instead of
|
||||
`PurgeInterval`. Without it, pacing would make retention converge in 11 **days** at the daily
|
||||
cadence — a policy that no longer enforces its own window. With it, the worst case converges in
|
||||
~11 ticks ≈ **55 minutes**, every burst bounded.
|
||||
- **Why 5 minutes rather than 1:** the interval must exceed the peer's drain time for one tick, or
|
||||
ticks outrun the drain and the backlog accumulates across them — reintroducing the very problem
|
||||
being fixed. 250 k drains in ~23 s on loopback; 5 minutes leaves ~13x headroom for a slower
|
||||
real link.
|
||||
- Validated `> 0` and `<= PurgeInterval` (a longer catch-up would *delay* convergence, inverting
|
||||
its purpose).
|
||||
- **The ceiling is not claimed when the work is finished.** An expired set that lands exactly on the
|
||||
budget is done; the retention phase runs a bounded `EXISTS` probe before flagging a catch-up tick,
|
||||
so a converged node never arms a 5-minute retry that deletes nothing.
|
||||
- **Visibility.** A ceiling-limited tick logs a Warning naming the ceiling, the rows deleted and the
|
||||
accelerated interval — **rate-limited to one line per 15 minutes**, because a large backlog
|
||||
produces a run of such ticks. Every hit increments
|
||||
`scadabridge.site.event_log.purge_ceiling_hits` (`ScadaBridgeTelemetry`, whose meter is already in
|
||||
the site `ObservedMeters` allowlist). The counter is the unthrottled signal, and its **shape** is
|
||||
the diagnosis: a run that stops = a backlog converging; hits that never stop = the site writes
|
||||
events faster than one paced tick purges, so the ceiling or the catch-up interval is mis-sized.
|
||||
- **Log wording updated.** The existing replication-backlog Information note now says the spike is
|
||||
bounded by the ceiling rather than by how much history expired.
|
||||
|
||||
**Consequence accepted:** the purge loop is now a `Task.Delay` loop rather than a `PeriodicTimer`,
|
||||
because the wait is no longer a constant. The cadence is therefore anchored to the **end** of the
|
||||
previous purge rather than to process start. At a daily interval and a measured worst-case paced
|
||||
tick of ~17 s the drift is immaterial; the topology guide and component doc say so.
|
||||
|
||||
**Explicitly not done:** no purge-exemption path, no oplog bypass, no "skip capture for deletes"
|
||||
flag. Those are ruled out by design and the measurement gives no reason to revisit that.
|
||||
|
||||
## 5. New issue found — the storage cap cannot be met on a replicated node with an undrained oplog
|
||||
|
||||
Not row 34, and not fixed here. `EventLogPurgeService.GetDatabaseSizeBytes()` measures
|
||||
`PRAGMA page_count - freelist_count` — i.e. the **whole consolidated LocalDb file**, which since
|
||||
Phase 1/2 holds ten tables *plus* `__localdb_oplog` and `__localdb_row_version`. On a replicated
|
||||
node the cap trim therefore fights itself: each 1000-row delete frees ~390 KB of `site_events` pages
|
||||
and adds ~1000 oplog rows plus ~1000 row-version tombstones to the same file. Measured (§3.4) the
|
||||
trim stopped **2.5x over a 64 MB cap** with 226 k rows still present, tripped by the existing
|
||||
"size did not decrease across an iteration" guard — which is doing its job (it prevents the table
|
||||
being wiped) but leaves the cap unenforced.
|
||||
|
||||
Severity is moderate, not acute: with a live peer the oplog drains on ack and successive ticks
|
||||
recover ground, and tombstones expire after 7 days. But two real consequences follow — a node whose
|
||||
peer is down cannot enforce its event-log storage cap at all, and the "1 GB event log" budget is in
|
||||
practice shared with replication bookkeeping that the event log does not own. A correct fix would
|
||||
measure `site_events`' own pages (e.g. `dbstat`) rather than the file, which is a change to the
|
||||
cap's semantics and deserves its own decision. Reported for the register; no row added here.
|
||||
|
||||
## 6. Verification
|
||||
|
||||
- Fail-before: the new behavioural tests do not compile against pre-fix `EventLogPurgeService`
|
||||
(`RunPurge` was `void`); with the fix restored but `PurgeBudget` neutralised to always-unlimited,
|
||||
**6 of the new tests fail** and the 19 pre-existing ones pass — so the assertions are carried by
|
||||
the ceiling, not incidentally.
|
||||
- `dotnet build ZB.MOM.WW.ScadaBridge.slnx` clean.
|
||||
- `ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests`: 93 passed / 0 failed.
|
||||
Reference in New Issue
Block a user