Measured the unbounded purge burst and found it can force a full ten-table
snapshot resync, then bounded it. No purge-exemption path was added — the
deletes are still CDC-captured like any other write, which stays ruled out
by design; what changed is how many a single tick performs.
MEASURED (harness: real EventLogPurgeService over a real LocalDb file with
site_events genuinely RegisterReplicated, plus a two-node loopback gRPC
replication pair; docker rig untouched):
393 bytes/row incl. indexes => a 1 GB event log holds ~2,731,947 rows.
Expiring all of them in ONE tick: 285 s, 2,731,947 oplog rows,
284 MB oplog payload, 2.73 M row-version tombstones (7-day retention),
i.e. 2.7x LocalDb's MaxOplogRows default of 1,000,000. Crossing that makes
the 60 s maintenance tick prune the oplog and set needs_snapshot — a FULL
snapshot of all ten replicated tables, bought by a routine retention purge.
The resident backlog is produced-minus-drained, so a healthy peer blunts it:
on loopback the same burst peaked at 391,424 and cleared in 6.0 min. That is
the problem, not the reassurance — the old safety margin was a function of
peer throughput and vanished with the peer down. A bounded tick is safe at
any peer speed.
Drain is not a bottleneck: 7.6k-19.8k tombstones/s (20k / 500k / 2.73M runs).
SHIPPED:
- SiteEventLogOptions.MaxPurgedRowsPerTick (default 250,000; 0 = unlimited
escape hatch, negatives rejected). One SHARED budget across the retention
purge and the cap trim — two independently-capped phases would let a tick
emit twice the intended burst. 4x margin under MaxOplogRows, which is also
shared with every other replicated table's ordinary traffic.
- SiteEventLogOptions.PurgeCatchUpInterval (default 5 min, validated > 0 and
<= PurgeInterval). Without it pacing would make retention converge in 11
DAYS at the daily cadence; with it the worst case converges in ~11 ticks
(~55 min), every burst bounded. 5 min rather than 1 so ticks cannot outrun
the peer's drain and re-accumulate the backlog.
- A ceiling is claimed only when work genuinely remains (bounded EXISTS probe),
so an expired set landing exactly on the budget does not arm a no-op retry.
- Visibility: rate-limited (15 min) Warning naming the ceiling and the
accelerated interval, plus an unthrottled counter
scadabridge.site.event_log.purge_ceiling_hits — its SHAPE distinguishes a
backlog converging from a site whose event rate outruns one paced tick.
- The purge loop is a delay loop rather than a PeriodicTimer, so its cadence
now anchors to the END of the previous purge. Documented.
ROW-33 INTERPLAY VERIFIED: pacing lowers each spike and lengthens total purge
duration, but a full 250k tick drains in ~23 s (~4 min even at 10x a real
link's penalty) against the 15-minute sustained dwell and a 5-minute gap to
the next tick. The UNPACED 2.73M burst had only 2.5x of margin (6.0 min), so
pacing makes the badge assumption robust rather than merely preserving it.
Docs travel together: Component-SiteEventLogging.md, topology-guide.md
("Reading the replication backlog" + the row-33 badge note), CLAUDE.md's
purge-burst sentence, and a dated measurement/decision note under docs/plans/.
Fail-before: the new behavioural tests do not compile against the pre-fix
service (RunPurge was void); with PurgeBudget neutralised to always-unlimited,
6 of the new tests fail and all 19 pre-existing ones pass.
Tests: SiteEventLogging 93/93, Commons 715/715, HealthMonitoring 115/115,
Host 505/505. Solution build clean.
The daily site_events retention purge (and the storage-cap trim) is CDC-captured
on a replication-enabled site node exactly like any other write — correct by
design, since LocalDb Phase 2 deliberately has no purge-exemption path — so the
backlog jumps by the deleted batch size at purge time. LocalDbOplogBacklog /
localdb_oplog_depth spike, drain, and an operator watching the gauge with no
context reads it as a replication fault.
Documentation + one log line, no behaviour change:
- topology-guide.md gains "Reading the replication backlog — the daily
site_events purge burst": when it fires (PurgeInterval 24h, anchored to the
active node's PROCESS START, not a wall-clock hour, so it moves after every
failover), where it shows (replicated nodes only — not rig site-b/site-c),
the healthy signature (LocalDbReplicationConnected stays true, backlog
returns to ~0) and what a genuine fault looks like instead.
- Component-SiteEventLogging.md Storage records the same under retention/purge;
Component-HealthMonitoring.md gains the two previously-undocumented
LocalDbReplicationConnected / LocalDbOplogBacklog metric rows carrying the
caveat, with cross-references both ways.
- EventLogPurgeService emits one Information line naming the row count and the
expected transient backlog when a purge deleted rows on a replication-enabled
node, so the spike is correlatable in the log. Replication-awareness comes in
as a Host-supplied SiteEventLogReplicationCheck delegate, mirroring the
existing SiteEventLogActiveNodeCheck seam: SiteLocalDbSetup.ReplicationIsConfigured
goes internal so the PeerAddress-OR-ApiKey rule stays in one place and
SiteEventLogging never learns to read LocalDb config. Unregistered ⇒ no note,
matching the default that replication is opt-in and off.
Both delete paths carry the note (a cap trim is usually the larger burst); the
predicate is try/caught since a log-wording check must never break the purge.
Tests: 5 new EventLogPurgeServiceTests cases (replicated logs it, unreplicated
does not, zero-rows does not, cap purge logs it, throwing predicate still purges
and swallows) via a local capturing ILogger. SiteEventLogging 81/81 green,
Host 490/490 green, full solution build clean (0 warnings).
Implements WP3.2 stage (b) per docs/plans/2026-08-15-site-events-policy-design.md.
- Per-run instance-script Started/Completed Info site events are now off by
default (SiteRuntimeOptions.PerRunScriptEvents=false) instead of firing on
every run, closing the dominant site_events writer. Gated at the ScriptRunLauncher
call sites (moved there from ScriptExecutionActor by WP3.1). Error-level events
(timeout/failure/stuck-watchdog/recursion-limit) remain unconditional.
- ScriptRunSummaryRecorder accumulates per-(instance, script) run counters and a
new site-only ScriptRunSummaryFlushService emits one aggregate "script" Info
site event per ScriptRunSummaryIntervalSeconds (default 300s), top-50-script
breakdown with an "others" rollup, zero-activity intervals emit nothing.
- Per-script opt-in via PerRunScriptEventScripts ("Instance/Script" exact or
"Instance/*" wildcard), matched by the new pure ScriptRunEventPolicy. All three
options are read from IOptionsMonitor<SiteRuntimeOptions> per run, so the
policy is hot-togglable without a restart.
- Fixed the stale "event log is not replicated" comment at AkkaHostedService.cs
(~905): site_events IS registered in SiteLocalDbSetup.ReplicatedTables — the
singleton is what makes queries always hit the actively-written copy;
replication is what gives the singleton history to read after a failover
(memo Decision (b)). site_events replication itself is unchanged (still
registered) and already pinned by
tests/ZB.MOM.WW.ScadaBridge.Host.Tests/SiteLocalDbCdcRegistrationTests.cs.
- Updated Component-SiteEventLogging.md (Volume Policy section, corrected
Storage/replication rationale) and Component-SiteRuntime.md (Script Run
Launch + Error Handling sections).
Organize documentation by moving requirements (HighLevelReqs, Component-*,
lmxproxy_protocol) to docs/requirements/ and test infrastructure docs to
docs/test_infra/. Updates all cross-references in README, CLAUDE.md,
infra/README, component docs, and 23 plan files.
Joseph Doherty