Merge row 34: site_events purge burst measured (2.73M-row worst case = 2.7x MaxOplogRows) and paced — shared 250k/tick budget, 5-min catch-up, ceiling-hit counter

docs/deployment/topology-guide.md

@@ -223,19 +223,35 @@ one of the ten replicated tables, and its retention DELETE is captured by CDC ex
 ordinary write — by design, since LocalDb Phase 2 there is deliberately no purge-exemption path
 (the same property that makes a mass DELETE dangerous, which is why `ReplaceAllAsync` was deleted
 rather than reinstated). One oplog row is therefore queued per deleted event, and the backlog jumps
-by the size of the day's expired batch.
+by the size of the batch the tick deleted.
+
+**The spike is bounded per tick** (register row 34, measured:
+`docs/plans/2026-08-15-site-events-purge-burst-measurement.md`). One purge tick deletes at most
+`ScadaBridge:SiteEventLog:MaxPurgedRowsPerTick` rows — default **250 000**, across the retention
+purge and the cap trim combined — so the height of the spike is a property of the *configuration*,
+not of how much history had expired. That matters because the burst used to be unbounded: a full
+1 GB event log holds ~2.73 million rows, and expiring all of them at once queued 2 731 947 oplog
+rows, **2.7x LocalDb's `MaxOplogRows` (default 1 000 000)** — enough for the maintenance tick to
+prune the oplog and flag a *full ten-table snapshot resync*. A healthy peer blunts that (the oplog
+drains while the purge runs — measured peak was 391 000, not 2.73 M), which is precisely the problem:
+the old safety margin was a function of how fast the peer happened to be, and vanished when the peer
+was down. Set `MaxPurgedRowsPerTick` to `0` to restore the old unbounded behaviour.
 
 - **When.** Every `ScadaBridge:SiteEventLog:PurgeInterval` (default **24 h**), plus once at
-  startup. The timer is anchored to the **active node's process start**, not to a wall-clock hour,
-  so the burst lands at a different time of day after each failover or restart — do not expect it
-  at a fixed hour. The storage-cap trim (default 1 GB) can produce the same shape off-schedule, and
-  is usually the larger of the two.
+  startup — **except** after a tick that hit the ceiling, which schedules the next tick after
+  `ScadaBridge:SiteEventLog:PurgeCatchUpInterval` (default **5 minutes**) instead. So a large
+  backlog appears as a *run* of bounded spikes minutes apart rather than one huge one; a
+  worst-case 2.73 M-row backlog converges in ~11 ticks ≈ 55 minutes. The cadence is anchored to the
+  **end of the previous purge**, not to a wall-clock hour, so the burst lands at a different time of
+  day after each failover or restart — do not expect it at a fixed hour. The storage-cap trim
+  (default 1 GB) can produce the same shape off-schedule, and draws from the same per-tick budget.
 - **Where it shows.** Only on a node with replication configured — the rig's site-a. site-b/site-c
   have no capture triggers at all and report no backlog for a purge.
 - **What healthy looks like.** `LocalDbReplicationConnected` stays **true** across the spike, and
-  the backlog drains back to ~0 as the peer acks the batch — within seconds to a couple of minutes
-  depending on batch size (delta messages are bounded by `LocalDb:Replication:MaxBatchBytes`,
-  default 2 MB, and secondarily by `MaxBatchSize`). No dead letters, no schema-mismatch errors.
+  the backlog drains back to ~0 as the peer acks the batch. Measured on a loopback pair at
+  ~10 800 tombstones/second, so a full 250 000-row tick drains in well under a minute (delta
+  messages are bounded by `LocalDb:Replication:MaxBatchBytes`, default 2 MB, and secondarily by
+  `MaxBatchSize`). No dead letters, no schema-mismatch errors.
 - **What is actually wrong.** `LocalDbReplicationConnected` **false** while the backlog climbs, a
   backlog that keeps rising across successive readings rather than draining, or a backlog that
   never returns near zero between bursts. Those point at the sync stream — an ApiKey mismatch
@@ -248,13 +264,30 @@ logs an Information line next to the purge count:
 ```
 Purged 41230 events older than 30 days
 Purged 41230 site_events rows on a replication-enabled node — a transient LocalDb oplog backlog
-is expected while the deletes replicate to the peer. It drains on its own;
+of up to that many rows is expected while the deletes replicate to the peer. It drains on its own;
 LocalDbReplicationConnected staying true with the backlog returning to ~0 is the healthy signature.
+A tick deletes at most 250000 rows, so the spike is bounded by that number rather than by how much
+history had expired.
 ```
 
 An unreplicated node logs only the first line. If a backlog spike has no such line near it in the
 active node's log, the purge is *not* the explanation and the spike is worth investigating.
 
+**A tick that hit the ceiling adds a Warning**, rate-limited to one line per 15 minutes so a
+converging backlog does not flood the log:
+
+```
+Event log purge stopped at its per-tick ceiling of 250000 rows (250000 deleted); expired events
+remain and the next purge runs in 00:05:00 instead of 1.00:00:00.
+```
+
+Every hit — throttled line or not — increments the counter
+`scadabridge_site_event_log_purge_ceiling_hits`. **Read the counter's shape, not a single hit:** a
+short run of hits that stops is a backlog converging (normal, e.g. the first purge after enabling
+replication on a long-running site); hits that never stop mean the site writes events faster than
+one paced tick can purge, i.e. `MaxPurgedRowsPerTick` or `PurgeCatchUpInterval` is mis-sized for
+that site's event rate.
+
 **Two dashboard surfaces make this reviewable rather than a log-archaeology exercise** (register
 row 33):
 
@@ -263,7 +296,13 @@ row 33):
   above `ScadaBridge:HealthMonitoring:LocalDbBacklogThreshold` (default **1000**) continuously for
   `ScadaBridge:HealthMonitoring:LocalDbBacklogSustainedFor` (default **15 minutes**). The daily
   purge burst described above drains in seconds to a couple of minutes, so it never reaches the
-  dwell and never badges the site — which is the entire reason the dwell exists. A bare threshold
+  dwell and never badges the site — which is the entire reason the dwell exists. **The per-tick
+  ceiling reinforces that rather than threatening it:** pacing lowers the spike (250 000 rows
+  maximum instead of an unbounded 2.7 M) at the cost of more ticks, and a 250 000-row tick was
+  measured draining in well under a minute — even allowing 10x for a real inter-node link that is
+  ~4 minutes, against a 15-minute dwell and a 5-minute gap to the next tick. (The *unpaced* 2.73 M
+  burst drained in 6.0 minutes on loopback: inside the dwell, but with only 2.5x of margin, so a
+  materially slower link would have badged.) A bare threshold
   would fire once a day on every replicated site and be ignored within a week. The dwell is
   validated to be at least one `ReportInterval`, since a shorter window could be satisfied by a
   single sample and would collapse back into a bare threshold.