Merge row 34: site_events purge burst measured (2.73M-row worst case = 2.7x MaxOplogRows) and paced — shared 250k/tick budget, 5-min catch-up, ceiling-hit counter

docs/deployment/topology-guide.md

@@ -223,19 +223,35 @@ one of the ten replicated tables, and its retention DELETE is captured by CDC ex
 ordinary write — by design, since LocalDb Phase 2 there is deliberately no purge-exemption path
 (the same property that makes a mass DELETE dangerous, which is why `ReplaceAllAsync` was deleted
 rather than reinstated). One oplog row is therefore queued per deleted event, and the backlog jumps
-by the size of the day's expired batch.
+by the size of the batch the tick deleted.
+
+**The spike is bounded per tick** (register row 34, measured:
+`docs/plans/2026-08-15-site-events-purge-burst-measurement.md`). One purge tick deletes at most
+`ScadaBridge:SiteEventLog:MaxPurgedRowsPerTick` rows — default **250 000**, across the retention
+purge and the cap trim combined — so the height of the spike is a property of the *configuration*,
+not of how much history had expired. That matters because the burst used to be unbounded: a full
+1 GB event log holds ~2.73 million rows, and expiring all of them at once queued 2 731 947 oplog
+rows, **2.7x LocalDb's `MaxOplogRows` (default 1 000 000)** — enough for the maintenance tick to
+prune the oplog and flag a *full ten-table snapshot resync*. A healthy peer blunts that (the oplog
+drains while the purge runs — measured peak was 391 000, not 2.73 M), which is precisely the problem:
+the old safety margin was a function of how fast the peer happened to be, and vanished when the peer
+was down. Set `MaxPurgedRowsPerTick` to `0` to restore the old unbounded behaviour.
 
 - **When.** Every `ScadaBridge:SiteEventLog:PurgeInterval` (default **24 h**), plus once at
-  startup. The timer is anchored to the **active node's process start**, not to a wall-clock hour,
-  so the burst lands at a different time of day after each failover or restart — do not expect it
-  at a fixed hour. The storage-cap trim (default 1 GB) can produce the same shape off-schedule, and
-  is usually the larger of the two.
+  startup — **except** after a tick that hit the ceiling, which schedules the next tick after
+  `ScadaBridge:SiteEventLog:PurgeCatchUpInterval` (default **5 minutes**) instead. So a large
+  backlog appears as a *run* of bounded spikes minutes apart rather than one huge one; a
+  worst-case 2.73 M-row backlog converges in ~11 ticks ≈ 55 minutes. The cadence is anchored to the
+  **end of the previous purge**, not to a wall-clock hour, so the burst lands at a different time of
+  day after each failover or restart — do not expect it at a fixed hour. The storage-cap trim
+  (default 1 GB) can produce the same shape off-schedule, and draws from the same per-tick budget.
 - **Where it shows.** Only on a node with replication configured — the rig's site-a. site-b/site-c
   have no capture triggers at all and report no backlog for a purge.
 - **What healthy looks like.** `LocalDbReplicationConnected` stays **true** across the spike, and
-  the backlog drains back to ~0 as the peer acks the batch — within seconds to a couple of minutes
-  depending on batch size (delta messages are bounded by `LocalDb:Replication:MaxBatchBytes`,
-  default 2 MB, and secondarily by `MaxBatchSize`). No dead letters, no schema-mismatch errors.
+  the backlog drains back to ~0 as the peer acks the batch. Measured on a loopback pair at
+  ~10 800 tombstones/second, so a full 250 000-row tick drains in well under a minute (delta
+  messages are bounded by `LocalDb:Replication:MaxBatchBytes`, default 2 MB, and secondarily by
+  `MaxBatchSize`). No dead letters, no schema-mismatch errors.
 - **What is actually wrong.** `LocalDbReplicationConnected` **false** while the backlog climbs, a
   backlog that keeps rising across successive readings rather than draining, or a backlog that
   never returns near zero between bursts. Those point at the sync stream — an ApiKey mismatch
@@ -248,13 +264,30 @@ logs an Information line next to the purge count:
 ```
 Purged 41230 events older than 30 days
 Purged 41230 site_events rows on a replication-enabled node — a transient LocalDb oplog backlog
-is expected while the deletes replicate to the peer. It drains on its own;
+of up to that many rows is expected while the deletes replicate to the peer. It drains on its own;
 LocalDbReplicationConnected staying true with the backlog returning to ~0 is the healthy signature.
+A tick deletes at most 250000 rows, so the spike is bounded by that number rather than by how much
+history had expired.
 ```
 
 An unreplicated node logs only the first line. If a backlog spike has no such line near it in the
 active node's log, the purge is *not* the explanation and the spike is worth investigating.
 
+**A tick that hit the ceiling adds a Warning**, rate-limited to one line per 15 minutes so a
+converging backlog does not flood the log:
+
+```
+Event log purge stopped at its per-tick ceiling of 250000 rows (250000 deleted); expired events
+remain and the next purge runs in 00:05:00 instead of 1.00:00:00.
+```
+
+Every hit — throttled line or not — increments the counter
+`scadabridge_site_event_log_purge_ceiling_hits`. **Read the counter's shape, not a single hit:** a
+short run of hits that stops is a backlog converging (normal, e.g. the first purge after enabling
+replication on a long-running site); hits that never stop mean the site writes events faster than
+one paced tick can purge, i.e. `MaxPurgedRowsPerTick` or `PurgeCatchUpInterval` is mis-sized for
+that site's event rate.
+
 **Two dashboard surfaces make this reviewable rather than a log-archaeology exercise** (register
 row 33):
 
@@ -263,7 +296,13 @@ row 33):
   above `ScadaBridge:HealthMonitoring:LocalDbBacklogThreshold` (default **1000**) continuously for
   `ScadaBridge:HealthMonitoring:LocalDbBacklogSustainedFor` (default **15 minutes**). The daily
   purge burst described above drains in seconds to a couple of minutes, so it never reaches the
-  dwell and never badges the site — which is the entire reason the dwell exists. A bare threshold
+  dwell and never badges the site — which is the entire reason the dwell exists. **The per-tick
+  ceiling reinforces that rather than threatening it:** pacing lowers the spike (250 000 rows
+  maximum instead of an unbounded 2.7 M) at the cost of more ticks, and a 250 000-row tick was
+  measured draining in well under a minute — even allowing 10x for a real inter-node link that is
+  ~4 minutes, against a 15-minute dwell and a 5-minute gap to the next tick. (The *unpaced* 2.73 M
+  burst drained in 6.0 minutes on loopback: inside the dwell, but with only 2.5x of margin, so a
+  materially slower link would have badged.) A bare threshold
   would fire once a day on every replicated site and be ignored within a week. The dwell is
   validated to be at least one `ReportInterval`, since a shorter window could be satisfied by a
   single sample and would collapse back into a bare threshold.

docs/plans/2026-08-15-site-events-purge-burst-measurement.md

@@ -0,0 +1,214 @@
+# `site_events` purge burst — measurement + pacing decision (register row 34)
+
+**Date:** 2026-08-15
+**Row:** deferred-work register #34 — *"`site_events` purge burst is unbounded in the oplog and unmeasured at scale."*
+**Outcome:** measured; **pacing implemented** (a per-tick delete ceiling with an accelerated
+catch-up tick). One separate defect found on the storage-cap path — see *New issue* at the end.
+
+---
+
+## 1. What the row asked
+
+`site_events` lives in the consolidated LocalDb file and is CDC-replicated on a replication-enabled
+site. The daily retention purge (30 days) and the storage-cap trim (1 GB) slice their deletes at
+1000 rows per DELETE statement, but the **whole** expired set went in one tick, so the oplog gained
+one row per deleted event with no pacing. The wire path is bounded (`MaxBatchBytes` 2 MB; the rig
+also pins `MaxBatchSize: 64` to bound the DB read page) but the oplog **write** was not, and nothing
+had been measured.
+
+The design commitment that had to survive, and did: **there is no purge-exemption path.** Purge
+deletes are captured like any other write — the same property that made `ReplaceAllAsync` unsafe to
+reintroduce. Nothing below adds an exemption or an oplog bypass; the fix bounds how much one tick
+deletes, not what gets captured.
+
+## 2. How it was measured
+
+A throwaway console harness (`dotnet` console app, scratchpad — not committed) running the **real**
+`EventLogPurgeService` over a **real** `ZB.MOM.WW.LocalDb` file with `site_events` genuinely
+`RegisterReplicated`, plus a two-node loopback replication pair for the drain leg:
+
+| Phase | What it does |
+| --- | --- |
+| `calibrate` | Seeds realistically-shaped rows, measures `page_count`/`freelist_count`/`page_size` delta ⇒ bytes per row ⇒ rows at the 1 GB cap. |
+| `burst` | Seeds N expired rows, registers CDC, parks the oplog at "everything acked", runs one real `RunPurge()`. Records purge wall time, oplog rows, oplog payload bytes, row-version tombstones, file size before/after. |
+| `cap` | Same but with rows *inside* retention and a small `MaxStorageMb`, so only the storage-cap trim runs. Run twice — replicated and not — to isolate the CDC contribution. |
+| `drain` | Two full replication stacks (initiator + passive `MapZbLocalDbSync`) over Kestrel h2c on loopback, both nodes registered for `site_events`. Seeds through the normal write path so the pair converges naturally, then purges on node A and polls the O(1) backlog expression until it returns to 0 and node B's table is empty. |
+
+Row content mirrors the post-WP3.2 event stream: mostly short Info rows, 1-in-10 carrying a details
+payload, instance ids on 2-in-3. `docker/` was not touched.
+
+Machine: macOS (darwin 25.5.0), .NET 10, Release build, server GC, local SSD. Loopback drain numbers
+are therefore an **upper bound on throughput** — a real inter-node link is slower, and the margins
+below are stated with that in mind.
+
+## 3. Numbers
+
+### 3.1 How big the worst case actually is
+
+```
+calibrate: 200,000 rows
+  bytes per row (incl. the four indexes)   393
+  rows at the 1 GB storage cap             2,731,947
+```
+
+The 1 GB cap is the real bound on a single retention burst: the worst realistic case is a log at the
+cap in which everything has expired (a site restarted after a long outage, a clock correction, or
+simply the first purge after `site_events` retention was ever satisfied).
+
+### 3.2 The unpaced retention burst
+
+| Seeded expired rows | Purge wall time | Oplog rows queued | Oplog payload | Row-version tombstones | File before → after |
+| ---: | ---: | ---: | ---: | ---: | --- |
+| 500,000 | 36.2 s | 500,000 | 52.0 MB | 500,000 | 254 MB → 129 MB |
+| **2,731,947** (1 GB cap) | **285.1 s** | **2,731,947** | **284.1 MB** | **2,731,947** | 1.39 GB → 707 MB |
+
+**The headline finding:** the worst-case burst *queues* **2.7x LocalDb's `MaxOplogRows` default of
+1,000,000**. Crossing that ceiling is not cosmetic — `OplogStore.EnforceCapsAsync`, which the
+replication maintenance service runs every 60 s, prunes the oldest oplog rows **and sets
+`needs_snapshot = 1`**, the database-wide flag. The next handshake therefore serves a **full
+snapshot of all ten registered tables**. A routine retention purge could, on its own, force a full
+resync of the site's entire replicated state.
+
+**How much of that 2.73 M is actually resident depends on the peer** — the oplog is drained
+concurrently, so the *peak backlog* is the burst minus what the peer acks while the purge runs. §3.3
+measures it: on a loopback pair the peak was 391 k, not 2.73 M, because the peer kept up. That does
+not make the risk theoretical, it makes it **conditional on peer speed**, which is exactly the
+property a design should not depend on:
+
+- **peer down / disconnected** — nothing drains, the full 2.73 M is resident. Over the cap outright.
+- **peer slower than ~63 % of loopback** — the production deficit outruns the drain and the peak
+  crosses 1 M. Arithmetic from §3.3: the purge produced 8,548 rows/s and the loopback peer consumed
+  7,585 rows/s; halve the peer's rate and the peak lands ≈1.5 M.
+- **peer at loopback speed** — peak 391 k, under the cap, but still 391x the row-33 badge threshold.
+
+Pacing removes the dependency entirely: a 250 k tick cannot exceed the cap at *any* peer speed,
+including a peer that is down.
+
+Secondary costs at worst case: 285 s of purge (the write lock is taken and released 2,732 times, so
+recorder flushes interleave — but they contend for nearly five minutes), 284 MB of oplog payload,
+and 2.73 M row-version tombstones retained for `TombstoneRetention` (7 days). The file was still
+707 MB *after* deleting a 1 GB event log, because the replication bookkeeping the delete generated
+is itself in that file.
+
+### 3.3 Drain
+
+Two-node loopback pair, `MaxBatchSize: 64` (the rig's pin), `MaxBatchBytes` at its 2 MB default.
+Both nodes registered for `site_events`; rows are seeded through the normal write path so the pair
+converges naturally before the purge runs.
+
+| Rows | Insert leg (seed + replicate) | Purge | Backlog → 0 | **Peak backlog** | Tombstones/s | Inside the 15-min dwell? |
+| ---: | ---: | ---: | ---: | ---: | ---: | --- |
+| 20,000 | 1.3 s (15.0 k/s) | 0.5 s | 1.0 s | 17,424 | 19,807 | yes |
+| 500,000 | 65.7 s (7.6 k/s) | 34.4 s | 46.3 s | 150,912 | 10,795 | yes |
+| **2,731,947** (1 GB cap) | 996.9 s (2.7 k/s) | 319.6 s | **360.2 s** | **391,424** | 7,585 | yes (6.0 min) |
+
+**Drain is not the bottleneck, and a healthy peer blunts the burst.** At every size the pump
+consumed tombstones almost as fast as the purge produced them, so the resident backlog stayed far
+below the burst size — 391 k peak on a 2.73 M burst. The tombstone leg is roughly 4x cheaper per row
+than the insert leg, as expected: a delete oplog entry carries a null `row_json`.
+
+**That is exactly why the unpaced burst was still unsafe**, though. The peak is
+`produced − drained`, so it is a function of the *peer's* throughput, and the margin evaporates as
+soon as the peer is slower than loopback or absent (see §3.2). A design whose oplog-cap safety
+depends on the peer keeping up is one bad link away from a full ten-table resync.
+
+**Row-33 interplay (the sustained badge).** The badge needs the backlog above 1000 continuously for
+15 minutes.
+
+- **Paced (shipped default).** A full 250 000-row tick drains in ~23 s at the measured rate; even
+  assuming a real link is **10x slower** than loopback it clears in ~4 minutes — comfortably inside
+  the dwell, with the next tick 5 minutes away. Pacing lowers each spike and lengthens *total*
+  purge duration, but every individual spike still drains between ticks, so the badge assumption
+  holds with a large margin. **Verified, not assumed:** the 500 k measurement is 2x a paced tick and
+  drained in 46 s.
+- **Unpaced (previous behaviour).** 6.0 minutes on loopback — inside the dwell, but only 2.5x of
+  margin, at a peak of 391 k (391x the badge threshold). At a 3x-slower link the same burst badges.
+  So pacing does not merely preserve the row-33 assumption; it is what makes it robust.
+
+### 3.4 The storage-cap trim
+
+`MaxStorageMb = 64`, 400,000 rows seeded inside the retention window (so only the cap path runs):
+
+| Replicated? | Events deleted | Remaining | Measured size before → after | Reached the cap? |
+| --- | ---: | ---: | --- | --- |
+| no | 248,000 | 152,000 | 158.0 MB → 66.98 MB | **yes** (cap 67.11 MB) |
+| yes | 174,000 | 226,000 | 203.6 MB → 167.7 MB | **no — stopped 2.5x over cap** |
+
+See *New issue* below.
+
+## 4. Decision — pace it
+
+Pacing is warranted, on the strength of §3.2: an unbounded tick can exceed `MaxOplogRows` and buy a
+full ten-table snapshot resync. Drain *speed* (§3.3) was not the reason — that is healthy — but the
+fact that the safety margin **depends on** drain speed is: with the peer down, the whole 2.73 M
+burst is resident and the cap is breached outright. Bounding the tick makes the outcome independent
+of the peer, which is the property the previous design lacked.
+
+**Shipped:**
+
+- `SiteEventLogOptions.MaxPurgedRowsPerTick`, default **250 000**, `0` = unlimited.
+  - Ceiling on rows deleted by **one tick**, **shared** between the retention purge and the cap
+    trim. Two independently-capped phases would let a tick emit twice the intended burst.
+  - `0` (not a negative, which the validator rejects) is the documented escape hatch back to the
+    pre-row-34 behaviour, byte-for-byte.
+  - **Why 250 000:** a 4x margin under `MaxOplogRows` = 1 000 000, and that margin is shared with
+    every *other* replicated table's ordinary traffic, which also occupies the same oplog. At the
+    measured rates one such tick costs ~17 s of purge and ~23 s of drain.
+- `SiteEventLogOptions.PurgeCatchUpInterval`, default **5 minutes**.
+  - A tick that stopped at the ceiling schedules the next tick after this instead of
+    `PurgeInterval`. Without it, pacing would make retention converge in 11 **days** at the daily
+    cadence — a policy that no longer enforces its own window. With it, the worst case converges in
+    ~11 ticks ≈ **55 minutes**, every burst bounded.
+  - **Why 5 minutes rather than 1:** the interval must exceed the peer's drain time for one tick, or
+    ticks outrun the drain and the backlog accumulates across them — reintroducing the very problem
+    being fixed. 250 k drains in ~23 s on loopback; 5 minutes leaves ~13x headroom for a slower
+    real link.
+  - Validated `> 0` and `<= PurgeInterval` (a longer catch-up would *delay* convergence, inverting
+    its purpose).
+- **The ceiling is not claimed when the work is finished.** An expired set that lands exactly on the
+  budget is done; the retention phase runs a bounded `EXISTS` probe before flagging a catch-up tick,
+  so a converged node never arms a 5-minute retry that deletes nothing.
+- **Visibility.** A ceiling-limited tick logs a Warning naming the ceiling, the rows deleted and the
+  accelerated interval — **rate-limited to one line per 15 minutes**, because a large backlog
+  produces a run of such ticks. Every hit increments
+  `scadabridge.site.event_log.purge_ceiling_hits` (`ScadaBridgeTelemetry`, whose meter is already in
+  the site `ObservedMeters` allowlist). The counter is the unthrottled signal, and its **shape** is
+  the diagnosis: a run that stops = a backlog converging; hits that never stop = the site writes
+  events faster than one paced tick purges, so the ceiling or the catch-up interval is mis-sized.
+- **Log wording updated.** The existing replication-backlog Information note now says the spike is
+  bounded by the ceiling rather than by how much history expired.
+
+**Consequence accepted:** the purge loop is now a `Task.Delay` loop rather than a `PeriodicTimer`,
+because the wait is no longer a constant. The cadence is therefore anchored to the **end** of the
+previous purge rather than to process start. At a daily interval and a measured worst-case paced
+tick of ~17 s the drift is immaterial; the topology guide and component doc say so.
+
+**Explicitly not done:** no purge-exemption path, no oplog bypass, no "skip capture for deletes"
+flag. Those are ruled out by design and the measurement gives no reason to revisit that.
+
+## 5. New issue found — the storage cap cannot be met on a replicated node with an undrained oplog
+
+Not row 34, and not fixed here. `EventLogPurgeService.GetDatabaseSizeBytes()` measures
+`PRAGMA page_count - freelist_count` — i.e. the **whole consolidated LocalDb file**, which since
+Phase 1/2 holds ten tables *plus* `__localdb_oplog` and `__localdb_row_version`. On a replicated
+node the cap trim therefore fights itself: each 1000-row delete frees ~390 KB of `site_events` pages
+and adds ~1000 oplog rows plus ~1000 row-version tombstones to the same file. Measured (§3.4) the
+trim stopped **2.5x over a 64 MB cap** with 226 k rows still present, tripped by the existing
+"size did not decrease across an iteration" guard — which is doing its job (it prevents the table
+being wiped) but leaves the cap unenforced.
+
+Severity is moderate, not acute: with a live peer the oplog drains on ack and successive ticks
+recover ground, and tombstones expire after 7 days. But two real consequences follow — a node whose
+peer is down cannot enforce its event-log storage cap at all, and the "1 GB event log" budget is in
+practice shared with replication bookkeeping that the event log does not own. A correct fix would
+measure `site_events`' own pages (e.g. `dbstat`) rather than the file, which is a change to the
+cap's semantics and deserves its own decision. Reported for the register; no row added here.
+
+## 6. Verification
+
+- Fail-before: the new behavioural tests do not compile against pre-fix `EventLogPurgeService`
+  (`RunPurge` was `void`); with the fix restored but `PurgeBudget` neutralised to always-unlimited,
+  **6 of the new tests fail** and the 19 pre-existing ones pass — so the assertions are carried by
+  the ceiling, not incidentally.
+- `dotnet build ZB.MOM.WW.ScadaBridge.slnx` clean.
+- `ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests`: 93 passed / 0 failed.

docs/requirements/Component-SiteEventLogging.md

@@ -94,12 +94,35 @@ Each event entry contains:
   documented trade of not configuring a peer, unchanged by this policy.
 - **Retention**: 30 days. A **daily background job** runs on the active node and deletes all events older than 30 days. Hard delete — no archival. Deletes are sliced into bounded 1000-row batches per DELETE statement rather than one unbounded statement, so a large expired backlog does not hold the write lock (and every concurrent recorder flush) for the duration of the purge.
 - **Storage cap**: A configurable maximum database size (default: 1 GB) is enforced. If the storage cap is reached before the 30-day retention window, the oldest events are purged first. This prevents disk exhaustion from alarm storms, script failure loops, or connection flapping.
+- **Per-tick delete ceiling** (`MaxPurgedRowsPerTick`, default **250 000**; `0` = unlimited): one
+  purge tick deletes at most this many rows, **shared** by the retention purge and the storage-cap
+  trim — two independently-capped phases would let a tick emit twice the intended burst. A tick that
+  stops at the ceiling schedules the next tick after `PurgeCatchUpInterval` (default **5 minutes**)
+  instead of `PurgeInterval`, so retention still converges: a worst-case backlog drains over
+  ~11 accelerated ticks (≈55 minutes) rather than 11 days at the daily cadence, with every
+  individual burst bounded. The ceiling exists because the purge is CDC-captured (next bullet) and
+  an unbounded tick could exceed LocalDb's oplog budget; measured evidence and the chosen defaults
+  are in `docs/plans/2026-08-15-site-events-purge-burst-measurement.md`. A ceiling-limited tick logs
+  a Warning (rate-limited to one line per 15 minutes) and increments
+  `scadabridge.site.event_log.purge_ceiling_hits` — the counter is the unthrottled series that
+  distinguishes a backlog converging from a site whose event rate outruns one paced tick.
+  Consequence of the accelerated retry: the purge loop is a delay loop, not a `PeriodicTimer`, so
+  the cadence is anchored to the **end** of the previous purge rather than to process start.
 - **Purge deletes are CDC-captured on a replicated site — the resulting backlog spike is expected.**
   Retention and cap deletes are ordinary row changes on a replicated table; there is deliberately no
   purge-exemption path (LocalDb Phase 2 — CDC does all three jobs, and a table-wide exemption is the
   same mechanism that made `ReplaceAllAsync` unsafe). One oplog row is queued per deleted event, so
   the site health report's `LocalDbOplogBacklog` (Prometheus `localdb_oplog_depth`) jumps by the size
-  of the batch at purge time and drains as the peer acks it. **Healthy signature:**
+  of the batch at purge time and drains as the peer acks it. **The height of that jump is bounded by
+  `MaxPurgedRowsPerTick`, not by how much history expired** — measured (register row 34): a full 1 GB
+  event log holds ~2.73 million rows at 393 bytes/row, and purging it unbounded queued 2 731 947
+  oplog rows, **2.7x LocalDb's `MaxOplogRows` default of 1 000 000**, which would make the
+  maintenance tick prune the oplog and flag a full ten-table snapshot resync. How much of that is
+  *resident* depends on the peer — the oplog drains concurrently, and a loopback peer held the same
+  burst to a 391 000 peak — so the overrun is conditional on peer throughput and unconditional when
+  the peer is down. Bounding the tick is what makes the outcome independent of the peer. Pacing the
+  tick, not exempting the purge from capture, is the fix — the exemption remains ruled out.
+  **Healthy signature:**
   `LocalDbReplicationConnected` stays true across the spike and the backlog returns to ~0; a backlog
   that keeps climbing, or climbs while `LocalDbReplicationConnected` is false, is a genuine
   replication fault and not the purge. The purge logs an Information line naming the row count and