Joseph Doherty
d9a35846d9
Measured the unbounded purge burst and found it can force a full ten-table
snapshot resync, then bounded it. No purge-exemption path was added — the
deletes are still CDC-captured like any other write, which stays ruled out
by design; what changed is how many a single tick performs.
MEASURED (harness: real EventLogPurgeService over a real LocalDb file with
site_events genuinely RegisterReplicated, plus a two-node loopback gRPC
replication pair; docker rig untouched):
393 bytes/row incl. indexes => a 1 GB event log holds ~2,731,947 rows.
Expiring all of them in ONE tick: 285 s, 2,731,947 oplog rows,
284 MB oplog payload, 2.73 M row-version tombstones (7-day retention),
i.e. 2.7x LocalDb's MaxOplogRows default of 1,000,000. Crossing that makes
the 60 s maintenance tick prune the oplog and set needs_snapshot — a FULL
snapshot of all ten replicated tables, bought by a routine retention purge.
The resident backlog is produced-minus-drained, so a healthy peer blunts it:
on loopback the same burst peaked at 391,424 and cleared in 6.0 min. That is
the problem, not the reassurance — the old safety margin was a function of
peer throughput and vanished with the peer down. A bounded tick is safe at
any peer speed.
Drain is not a bottleneck: 7.6k-19.8k tombstones/s (20k / 500k / 2.73M runs).
SHIPPED:
- SiteEventLogOptions.MaxPurgedRowsPerTick (default 250,000; 0 = unlimited
escape hatch, negatives rejected). One SHARED budget across the retention
purge and the cap trim — two independently-capped phases would let a tick
emit twice the intended burst. 4x margin under MaxOplogRows, which is also
shared with every other replicated table's ordinary traffic.
- SiteEventLogOptions.PurgeCatchUpInterval (default 5 min, validated > 0 and
<= PurgeInterval). Without it pacing would make retention converge in 11
DAYS at the daily cadence; with it the worst case converges in ~11 ticks
(~55 min), every burst bounded. 5 min rather than 1 so ticks cannot outrun
the peer's drain and re-accumulate the backlog.
- A ceiling is claimed only when work genuinely remains (bounded EXISTS probe),
so an expired set landing exactly on the budget does not arm a no-op retry.
- Visibility: rate-limited (15 min) Warning naming the ceiling and the
accelerated interval, plus an unthrottled counter
scadabridge.site.event_log.purge_ceiling_hits — its SHAPE distinguishes a
backlog converging from a site whose event rate outruns one paced tick.
- The purge loop is a delay loop rather than a PeriodicTimer, so its cadence
now anchors to the END of the previous purge. Documented.
ROW-33 INTERPLAY VERIFIED: pacing lowers each spike and lengthens total purge
duration, but a full 250k tick drains in ~23 s (~4 min even at 10x a real
link's penalty) against the 15-minute sustained dwell and a 5-minute gap to
the next tick. The UNPACED 2.73M burst had only 2.5x of margin (6.0 min), so
pacing makes the badge assumption robust rather than merely preserving it.
Docs travel together: Component-SiteEventLogging.md, topology-guide.md
("Reading the replication backlog" + the row-33 badge note), CLAUDE.md's
purge-burst sentence, and a dated measurement/decision note under docs/plans/.
Fail-before: the new behavioural tests do not compile against the pre-fix
service (RunPurge was void); with PurgeBudget neutralised to always-unlimited,
6 of the new tests fail and all 19 pre-existing ones pass.
Tests: SiteEventLogging 93/93, Commons 715/715, HealthMonitoring 115/115,
Host 505/505. Solution build clean.
80 lines
4.5 KiB
C#
80 lines
4.5 KiB
C#
using ZB.MOM.WW.Configuration;
|
|
|
|
namespace ZB.MOM.WW.ScadaBridge.SiteEventLogging;
|
|
|
|
/// <summary>
|
|
/// Validates <see cref="SiteEventLogOptions"/> at startup. Retention/purge values
|
|
/// drive the daily purge PeriodicTimer (a zero/negative period trips
|
|
/// <see cref="ArgumentOutOfRangeException"/>), the paging bounds gate query result
|
|
/// sizes, and the write-queue capacity sizes a bounded channel; a
|
|
/// <c>MaxQueryPageSize</c> below <c>QueryPageSize</c> would silently clamp the
|
|
/// default page below itself. Registered with <c>ValidateOnStart()</c> so a bad
|
|
/// <c>ScadaBridge:SiteEventLog</c> section fails fast at boot with a clear,
|
|
/// key-naming message.
|
|
/// </summary>
|
|
public sealed class SiteEventLogOptionsValidator : OptionsValidatorBase<SiteEventLogOptions>
|
|
{
|
|
/// <inheritdoc />
|
|
protected override void Validate(ValidationBuilder builder, SiteEventLogOptions options)
|
|
{
|
|
builder.RequireThat(options.RetentionDays > 0,
|
|
$"ScadaBridge:SiteEventLog:RetentionDays must be greater than 0 " +
|
|
$"(was {options.RetentionDays}).");
|
|
|
|
builder.RequireThat(options.MaxStorageMb > 0,
|
|
$"ScadaBridge:SiteEventLog:MaxStorageMb must be greater than 0 " +
|
|
$"(was {options.MaxStorageMb}).");
|
|
|
|
builder.RequireThat(!string.IsNullOrWhiteSpace(options.DatabasePath),
|
|
"ScadaBridge:SiteEventLog:DatabasePath must be a non-empty path; " +
|
|
"it is the SQLite file backing the site event log.");
|
|
|
|
builder.RequireThat(options.QueryPageSize > 0,
|
|
$"ScadaBridge:SiteEventLog:QueryPageSize must be greater than 0 " +
|
|
$"(was {options.QueryPageSize}).");
|
|
|
|
builder.RequireThat(options.WriteQueueCapacity > 0,
|
|
$"ScadaBridge:SiteEventLog:WriteQueueCapacity must be greater than 0 " +
|
|
$"(was {options.WriteQueueCapacity}); it bounds the background write channel.");
|
|
|
|
builder.RequireThat(options.PurgeInterval > TimeSpan.Zero,
|
|
$"ScadaBridge:SiteEventLog:PurgeInterval must be a positive duration " +
|
|
$"(was {options.PurgeInterval}); it is the purge PeriodicTimer period.");
|
|
|
|
// A NEGATIVE ceiling is a config error, not a synonym for "unlimited": 0 already
|
|
// carries that meaning explicitly, and silently treating -1 the same way would hide
|
|
// a typo that removes the only bound on a purge burst's oplog cost.
|
|
builder.RequireThat(options.MaxPurgedRowsPerTick >= 0,
|
|
$"ScadaBridge:SiteEventLog:MaxPurgedRowsPerTick must be 0 (unlimited) or greater " +
|
|
$"(was {options.MaxPurgedRowsPerTick}); it bounds how many rows one purge tick deletes.");
|
|
|
|
builder.RequireThat(options.PurgeCatchUpInterval > TimeSpan.Zero,
|
|
$"ScadaBridge:SiteEventLog:PurgeCatchUpInterval must be a positive duration " +
|
|
$"(was {options.PurgeCatchUpInterval}); it is the delay before the next tick when a " +
|
|
"purge stopped at MaxPurgedRowsPerTick.");
|
|
|
|
// A catch-up longer than the ordinary interval would make hitting the ceiling SLOW the
|
|
// purge down, which inverts the whole point of the accelerated retry. Only compared when
|
|
// both are positive (each owned above), so a bad value fires exactly one message.
|
|
builder.RequireThat(
|
|
!(options.PurgeInterval > TimeSpan.Zero
|
|
&& options.PurgeCatchUpInterval > TimeSpan.Zero
|
|
&& options.PurgeCatchUpInterval > options.PurgeInterval),
|
|
$"ScadaBridge:SiteEventLog:PurgeCatchUpInterval ({options.PurgeCatchUpInterval}) must be " +
|
|
$"less than or equal to PurgeInterval ({options.PurgeInterval}): it is an ACCELERATED " +
|
|
"retry after a ceiling-limited tick, so a longer value would delay convergence instead.");
|
|
|
|
// The hard page-size ceiling clamps a caller-supplied PageSize; a ceiling
|
|
// below the default QueryPageSize would silently clamp the default below
|
|
// itself. Only meaningful when both are positive (owned above), so stay
|
|
// silent when either is non-positive rather than double-firing.
|
|
builder.RequireThat(
|
|
!(options.QueryPageSize > 0
|
|
&& options.MaxQueryPageSize > 0
|
|
&& options.MaxQueryPageSize < options.QueryPageSize),
|
|
$"ScadaBridge:SiteEventLog:MaxQueryPageSize ({options.MaxQueryPageSize}) " +
|
|
$"must be >= QueryPageSize ({options.QueryPageSize}): the ceiling clamps a caller's " +
|
|
"requested page size, so it cannot be smaller than the default page size.");
|
|
}
|
|
}
|