ScadaBridge/docs/plans/2026-08-15-site-events-purge-burst-measurement.md

site_events purge burst — measurement + pacing decision (register row 34)

Date: 2026-08-15 Row: deferred-work register #34 — "site_events purge burst is unbounded in the oplog and unmeasured at scale." Outcome: measured; pacing implemented (a per-tick delete ceiling with an accelerated catch-up tick). One separate defect found on the storage-cap path — see New issue at the end.

---

1. What the row asked

site_events lives in the consolidated LocalDb file and is CDC-replicated on a replication-enabled site. The daily retention purge (30 days) and the storage-cap trim (1 GB) slice their deletes at 1000 rows per DELETE statement, but the whole expired set went in one tick, so the oplog gained one row per deleted event with no pacing. The wire path is bounded (MaxBatchBytes 2 MB; the rig also pins MaxBatchSize: 64 to bound the DB read page) but the oplog write was not, and nothing had been measured.

The design commitment that had to survive, and did: there is no purge-exemption path. Purge deletes are captured like any other write — the same property that made ReplaceAllAsync unsafe to reintroduce. Nothing below adds an exemption or an oplog bypass; the fix bounds how much one tick deletes, not what gets captured.

2. How it was measured

A throwaway console harness (dotnet console app, scratchpad — not committed) running the real EventLogPurgeService over a real ZB.MOM.WW.LocalDb file with site_events genuinely RegisterReplicated, plus a two-node loopback replication pair for the drain leg:

Phase	What it does
calibrate	Seeds realistically-shaped rows, measures page_count/freelist_count/page_size delta ⇒ bytes per row ⇒ rows at the 1 GB cap.
burst	Seeds N expired rows, registers CDC, parks the oplog at "everything acked", runs one real RunPurge(). Records purge wall time, oplog rows, oplog payload bytes, row-version tombstones, file size before/after.
cap	Same but with rows inside retention and a small MaxStorageMb, so only the storage-cap trim runs. Run twice — replicated and not — to isolate the CDC contribution.
drain	Two full replication stacks (initiator + passive MapZbLocalDbSync) over Kestrel h2c on loopback, both nodes registered for site_events. Seeds through the normal write path so the pair converges naturally, then purges on node A and polls the O(1) backlog expression until it returns to 0 and node B's table is empty.

Row content mirrors the post-WP3.2 event stream: mostly short Info rows, 1-in-10 carrying a details payload, instance ids on 2-in-3. docker/ was not touched.

Machine: macOS (darwin 25.5.0), .NET 10, Release build, server GC, local SSD. Loopback drain numbers are therefore an upper bound on throughput — a real inter-node link is slower, and the margins below are stated with that in mind.

3. Numbers

3.1 How big the worst case actually is

calibrate: 200,000 rows
  bytes per row (incl. the four indexes)   393
  rows at the 1 GB storage cap             2,731,947

The 1 GB cap is the real bound on a single retention burst: the worst realistic case is a log at the cap in which everything has expired (a site restarted after a long outage, a clock correction, or simply the first purge after site_events retention was ever satisfied).

3.2 The unpaced retention burst

Seeded expired rows	Purge wall time	Oplog rows queued	Oplog payload	Row-version tombstones	File before → after
500,000	36.2 s	500,000	52.0 MB	500,000	254 MB → 129 MB
2,731,947 (1 GB cap)	285.1 s	2,731,947	284.1 MB	2,731,947	1.39 GB → 707 MB

The headline finding: the worst-case burst queues 2.7x LocalDb's MaxOplogRows default of 1,000,000. Crossing that ceiling is not cosmetic — OplogStore.EnforceCapsAsync, which the replication maintenance service runs every 60 s, prunes the oldest oplog rows and sets needs_snapshot = 1, the database-wide flag. The next handshake therefore serves a full snapshot of all ten registered tables. A routine retention purge could, on its own, force a full resync of the site's entire replicated state.

How much of that 2.73 M is actually resident depends on the peer — the oplog is drained concurrently, so the peak backlog is the burst minus what the peer acks while the purge runs. §3.3 measures it: on a loopback pair the peak was 391 k, not 2.73 M, because the peer kept up. That does not make the risk theoretical, it makes it conditional on peer speed, which is exactly the property a design should not depend on:

• peer down / disconnected — nothing drains, the full 2.73 M is resident. Over the cap outright.
• peer slower than ~63 % of loopback — the production deficit outruns the drain and the peak crosses 1 M. Arithmetic from §3.3: the purge produced 8,548 rows/s and the loopback peer consumed 7,585 rows/s; halve the peer's rate and the peak lands ≈1.5 M.
• peer at loopback speed — peak 391 k, under the cap, but still 391x the row-33 badge threshold.

Pacing removes the dependency entirely: a 250 k tick cannot exceed the cap at any peer speed, including a peer that is down.

Secondary costs at worst case: 285 s of purge (the write lock is taken and released 2,732 times, so recorder flushes interleave — but they contend for nearly five minutes), 284 MB of oplog payload, and 2.73 M row-version tombstones retained for TombstoneRetention (7 days). The file was still 707 MB after deleting a 1 GB event log, because the replication bookkeeping the delete generated is itself in that file.

3.3 Drain

Two-node loopback pair, MaxBatchSize: 64 (the rig's pin), MaxBatchBytes at its 2 MB default. Both nodes registered for site_events; rows are seeded through the normal write path so the pair converges naturally before the purge runs.

Rows	Insert leg (seed + replicate)	Purge	Backlog → 0	Peak backlog	Tombstones/s	Inside the 15-min dwell?
20,000	1.3 s (15.0 k/s)	0.5 s	1.0 s	17,424	19,807	yes
500,000	65.7 s (7.6 k/s)	34.4 s	46.3 s	150,912	10,795	yes
2,731,947 (1 GB cap)	996.9 s (2.7 k/s)	319.6 s	360.2 s	391,424	7,585	yes (6.0 min)

Drain is not the bottleneck, and a healthy peer blunts the burst. At every size the pump consumed tombstones almost as fast as the purge produced them, so the resident backlog stayed far below the burst size — 391 k peak on a 2.73 M burst. The tombstone leg is roughly 4x cheaper per row than the insert leg, as expected: a delete oplog entry carries a null row_json.

That is exactly why the unpaced burst was still unsafe, though. The peak is produced − drained, so it is a function of the peer's throughput, and the margin evaporates as soon as the peer is slower than loopback or absent (see §3.2). A design whose oplog-cap safety depends on the peer keeping up is one bad link away from a full ten-table resync.

Row-33 interplay (the sustained badge). The badge needs the backlog above 1000 continuously for 15 minutes.

• Paced (shipped default). A full 250 000-row tick drains in ~23 s at the measured rate; even assuming a real link is 10x slower than loopback it clears in ~4 minutes — comfortably inside the dwell, with the next tick 5 minutes away. Pacing lowers each spike and lengthens total purge duration, but every individual spike still drains between ticks, so the badge assumption holds with a large margin. Verified, not assumed: the 500 k measurement is 2x a paced tick and drained in 46 s.
• Unpaced (previous behaviour). 6.0 minutes on loopback — inside the dwell, but only 2.5x of margin, at a peak of 391 k (391x the badge threshold). At a 3x-slower link the same burst badges. So pacing does not merely preserve the row-33 assumption; it is what makes it robust.

3.4 The storage-cap trim

MaxStorageMb = 64, 400,000 rows seeded inside the retention window (so only the cap path runs):

Replicated?	Events deleted	Remaining	Measured size before → after	Reached the cap?
no	248,000	152,000	158.0 MB → 66.98 MB	yes (cap 67.11 MB)
yes	174,000	226,000	203.6 MB → 167.7 MB	no — stopped 2.5x over cap

See New issue below.

4. Decision — pace it

Pacing is warranted, on the strength of §3.2: an unbounded tick can exceed MaxOplogRows and buy a full ten-table snapshot resync. Drain speed (§3.3) was not the reason — that is healthy — but the fact that the safety margin depends on drain speed is: with the peer down, the whole 2.73 M burst is resident and the cap is breached outright. Bounding the tick makes the outcome independent of the peer, which is the property the previous design lacked.

Shipped:

• SiteEventLogOptions.MaxPurgedRowsPerTick, default 250 000, 0 = unlimited.
  • Ceiling on rows deleted by one tick, shared between the retention purge and the cap trim. Two independently-capped phases would let a tick emit twice the intended burst.
  • 0 (not a negative, which the validator rejects) is the documented escape hatch back to the pre-row-34 behaviour, byte-for-byte.
  • Why 250 000: a 4x margin under MaxOplogRows = 1 000 000, and that margin is shared with every other replicated table's ordinary traffic, which also occupies the same oplog. At the measured rates one such tick costs ~17 s of purge and ~23 s of drain.
• SiteEventLogOptions.PurgeCatchUpInterval, default 5 minutes.
  • A tick that stopped at the ceiling schedules the next tick after this instead of PurgeInterval. Without it, pacing would make retention converge in 11 days at the daily cadence — a policy that no longer enforces its own window. With it, the worst case converges in ~11 ticks ≈ 55 minutes, every burst bounded.
  • Why 5 minutes rather than 1: the interval must exceed the peer's drain time for one tick, or ticks outrun the drain and the backlog accumulates across them — reintroducing the very problem being fixed. 250 k drains in ~23 s on loopback; 5 minutes leaves ~13x headroom for a slower real link.
  • Validated > 0 and <= PurgeInterval (a longer catch-up would delay convergence, inverting its purpose).
• The ceiling is not claimed when the work is finished. An expired set that lands exactly on the budget is done; the retention phase runs a bounded EXISTS probe before flagging a catch-up tick, so a converged node never arms a 5-minute retry that deletes nothing.
• Visibility. A ceiling-limited tick logs a Warning naming the ceiling, the rows deleted and the accelerated interval — rate-limited to one line per 15 minutes, because a large backlog produces a run of such ticks. Every hit increments scadabridge.site.event_log.purge_ceiling_hits (ScadaBridgeTelemetry, whose meter is already in the site ObservedMeters allowlist). The counter is the unthrottled signal, and its shape is the diagnosis: a run that stops = a backlog converging; hits that never stop = the site writes events faster than one paced tick purges, so the ceiling or the catch-up interval is mis-sized.
• Log wording updated. The existing replication-backlog Information note now says the spike is bounded by the ceiling rather than by how much history expired.

Consequence accepted: the purge loop is now a Task.Delay loop rather than a PeriodicTimer, because the wait is no longer a constant. The cadence is therefore anchored to the end of the previous purge rather than to process start. At a daily interval and a measured worst-case paced tick of ~17 s the drift is immaterial; the topology guide and component doc say so.

Explicitly not done: no purge-exemption path, no oplog bypass, no "skip capture for deletes" flag. Those are ruled out by design and the measurement gives no reason to revisit that.

5. New issue found — the storage cap cannot be met on a replicated node with an undrained oplog

Not row 34, and not fixed here. EventLogPurgeService.GetDatabaseSizeBytes() measures PRAGMA page_count - freelist_count — i.e. the whole consolidated LocalDb file, which since Phase 1/2 holds ten tables plus __localdb_oplog and __localdb_row_version. On a replicated node the cap trim therefore fights itself: each 1000-row delete frees ~390 KB of site_events pages and adds ~1000 oplog rows plus ~1000 row-version tombstones to the same file. Measured (§3.4) the trim stopped 2.5x over a 64 MB cap with 226 k rows still present, tripped by the existing "size did not decrease across an iteration" guard — which is doing its job (it prevents the table being wiped) but leaves the cap unenforced.

Severity is moderate, not acute: with a live peer the oplog drains on ack and successive ticks recover ground, and tombstones expire after 7 days. But two real consequences follow — a node whose peer is down cannot enforce its event-log storage cap at all, and the "1 GB event log" budget is in practice shared with replication bookkeeping that the event log does not own. A correct fix would measure site_events' own pages (e.g. dbstat) rather than the file, which is a change to the cap's semantics and deserves its own decision. Reported for the register; no row added here.

Postscript (2026-08-17) — fixed; register row 81 closed. The cap now measures site_events and its four indexes only. One correction to the suggestion above: dbstat turned out to be unavailable. It is a compile-time option and the bundled SQLitePCLRaw.lib.e_sqlite3 (SQLite 3.53.3) does not enable it — PRAGMA compile_options lists no ENABLE_DBSTAT_VTAB, and neither dbstat nor sqlite_dbpage appears in pragma_module_list. The table's footprint is therefore modelled from sampled row width rather than measured, at −0.8 %…+12.8 % accuracy against the numbers in §3.4, biased high on purpose. The "size did not decrease" guard survives, re-based onto the table-scoped metric so replication bookkeeping can no longer trip it. Decision record, including the probe evidence and why a different SQLite build was rejected: docs/plans/2026-08-17-site-event-log-cap-semantics-decision.md. §3.4's replicated row is left as measured — it is the record of the defect, not of current behaviour.

6. Verification

• Fail-before: the new behavioural tests do not compile against pre-fix EventLogPurgeService (RunPurge was void); with the fix restored but PurgeBudget neutralised to always-unlimited, 6 of the new tests fail and the 19 pre-existing ones pass — so the assertions are carried by the ceiling, not incidentally.
• dotnet build ZB.MOM.WW.ScadaBridge.slnx clean.
• ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests: 93 passed / 0 failed.