fix(siteeventlog): scope the storage cap to site_events' own pages (register rows 81, 84)

Row 81 — the cap could not be enforced on a replicated node. MaxStorageMb was
compared against page_count-freelist_count over the WHOLE consolidated LocalDb
file, which since Phase 1/2 holds ten tables plus __localdb_oplog /
__localdb_row_version / __localdb_snapshot_state. Every 1000-row trim batch freed
~390 KB of site_events pages and added ~1000 oplog rows + ~1000 tombstones to the
same file, so the measured size did not fall, the "size did not decrease" guard
fired, and the trim stopped 2.5x over a 64 MB cap with 226 k rows still present.
A node whose peer was down could not enforce its cap at all, and the event log's
budget was in practice shared with bookkeeping it does not own.

The cap now measures site_events AND ITS FOUR INDEXES only, so replication
bookkeeping can neither consume nor block the budget — and the cap stops being a
function of the peer's health, which was the wrong direction of coupling: an
inflated oplog is a replication fault, and letting it drive event deletion would
destroy operational history in response to one.

dbstat — the exact per-table measurement the register proposed — is NOT usable
here, verified before implementing. It is a compile-time option and the bundled
SQLitePCLRaw.lib.e_sqlite3 (SQLite 3.53.3) does not enable it: PRAGMA
compile_options lists no ENABLE_DBSTAT_VTAB, pragma_module_list returns ten
modules with neither dbstat nor sqlite_dbpage, and selecting from dbstat fails
with "no such table". Switching SQLite builds was rejected — the provider is
shared with ZB.MOM.WW.LocalDb and every other consumer, so it trades cap accuracy
for fleet-wide version skew. The footprint is therefore modelled from sampled row
width (500 rows each end, octet_length so multi-byte UTF-8 is not under-counted)
through a page-layout model; whole pages rather than a flat rate, because page
fill collapses once rows approach half a page and a linear model underestimates
that by ~40%. Clamped by whole-file logical size, a true upper bound that can only
correct an over-estimate and never binds on a replicated node.

Index pages count against the cap deliberately: measured at ~34% of the table's
footprint, and the 1 GB default was sized against a 393 B/row figure that already
included them.

Guard semantics restated — two backstops replace the single on-disk check:
(1) row width unsampleable => refuse to trim (checked before the cap comparison,
since a zero width makes "over cap" unanswerable rather than false; the old loop's
answer to an unanswerable cap was to keep deleting); (2) estimated size not
falling across a batch => stop. (2) is the old guard re-based onto the
table-scoped metric, so it can no longer be tripped by replication bookkeeping and
now means exactly one thing: events are arriving as fast as the trim removes them.
For that to be observable the loop re-reads the row count each batch, while
bytes-per-row is sampled once per tick and held fixed — re-sampling per batch
would jitter the estimate as the trim eats the head of the table and could stall
the loop through guard (2), the same self-defeating measurement as row 81 itself.

Row 84 — LogReplicationBacklogNote moved from per-phase to once per tick, driven
by the shared PurgeBudget. A tick where retention AND the cap trim both deleted
used to emit two identical Information notes; the tick total is also the more
useful number beside an oplog spike.

Measured model error: -0.8% / +8.4% / +12.8% across representative workloads,
biased high on purpose (an early cap forfeits budget; a low one fails at its only
job). Independently cross-checks row 34's harness at 397.2 vs 393 B/row.

Fail-before: reverting the estimator to whole-file semantics fails 5 of the new
tests and reproduces the original signature exactly (remaining=2000
estimate=21725184 cap=11534336 — rows destroyed, cap still unmet); restoring
per-phase notes fails the row-84 test with "collection contained 2 matching items".

No config keys added or renamed, so no rig/deploy changes. Decision record:
docs/plans/2026-08-17-site-event-log-cap-semantics-decision.md

Tests: SiteEventLogging 104/104 (was 93), Host 520/520, solution build clean.
This commit is contained in:
Joseph Doherty
2026-08-17 03:58:41 -04:00
parent 07220bc655
commit ddb3c32e06
10 changed files with 1171 additions and 46 deletions
@@ -93,7 +93,50 @@ Each event entry contains:
rig's site-b/site-c), the log stays node-local and a failover does start it fresh — the
documented trade of not configuring a peer, unchanged by this policy.
- **Retention**: 30 days. A **daily background job** runs on the active node and deletes all events older than 30 days. Hard delete — no archival. Deletes are sliced into bounded 1000-row batches per DELETE statement rather than one unbounded statement, so a large expired backlog does not hold the write lock (and every concurrent recorder flush) for the duration of the purge.
- **Storage cap**: A configurable maximum database size (default: 1 GB) is enforced. If the storage cap is reached before the 30-day retention window, the oldest events are purged first. This prevents disk exhaustion from alarm storms, script failure loops, or connection flapping.
- **Storage cap**: A configurable maximum size (default: 1 GB) is enforced. If the cap is reached
before the 30-day retention window, the oldest events are purged first. This prevents disk
exhaustion from alarm storms, script failure loops, or connection flapping.
- **The cap measures `site_events` and its four indexes — NOT the database file** (register row 81;
decision record `docs/plans/2026-08-17-site-event-log-cap-semantics-decision.md`). This matters
because the event log shares one SQLite file with nine other tables plus LocalDb's
`__localdb_oplog` / `__localdb_row_version` / `__localdb_snapshot_state` bookkeeping. The cap used
to compare `MaxStorageMb` against `page_count − freelist_count` over that whole file, which on a
replication-enabled node made the trim **unwinnable**: each 1000-row delete freed ~390 KB of event
pages while adding ~1000 oplog rows and ~1000 tombstones to the same file, the measured size did
not fall, and the progress guard stopped the trim — measured 2.5x over a 64 MB cap with 226 k rows
still present. A node whose peer was down could not enforce its cap at all. Two consequences of
the new scoping are worth internalising:
- **The cap is now independent of replication health.** It was previously a function of the
*peer's* liveness, and in the wrong direction — an inflated oplog is a replication fault, and
letting it drive event deletion would destroy operational history in response to one.
- **`MaxStorageMb` no longer bounds the data volume on its own.** LocalDb bookkeeping is separate
overhead on the same disk, bounded separately by `LocalDb:Replication:MaxOplogRows` and
`TombstoneRetention`; sizing a site's volume means adding the two. Managing that bookkeeping is
deliberately **not** this component's job — see the decision record §6.
- **The measurement is modelled, not exact.** SQLite can report per-table page usage precisely via
the `dbstat` virtual table, and that was the intended fix — but `dbstat` is a compile-time option
and the bundled `SQLitePCLRaw.lib.e_sqlite3` native library (SQLite 3.53.3) does **not** enable it;
neither `dbstat` nor `sqlite_dbpage` exists, verified by probe and pinned by
`SqliteCapabilityTests`. The table's footprint is therefore estimated from sampled row width
(500 rows from each end of the table, measured with `octet_length` so multi-byte UTF-8 is not
under-counted) fed into a page-layout model; index pages count against the cap, deliberately,
because they are ~34 % of the table's real footprint and the 1 GB default was sized against a
bytes/row figure that included them. Measured error is **−0.8 % to +12.8 %** across representative
workloads, biased high on purpose — a cap that fires slightly early forfeits budget, whereas one
that reads low fails at the only job it has. The estimate is additionally clamped by the
whole-file logical size, which is a true upper bound on any one table.
- **Two wipe-prevention backstops** guard the cap trim, replacing the single "on-disk size did not
decrease" check. (1) If row width cannot be sampled, the trim **refuses to run** and warns —
checked before the cap comparison, because a zero row width makes "are we over cap" unanswerable
rather than false, and the old loop's answer to an unanswerable cap was to keep deleting. (2) If
the *estimated table size* fails to fall across a batch, the trim stops and warns. Guard (2) is the
old check re-based onto the table-scoped metric, and its meaning is now unambiguous: replication
bookkeeping can no longer trip it, so it fires only when new events are arriving at least as fast
as the trim removes them. For that to be observable, the loop re-reads the row count from the
database after every batch, while bytes-per-row is sampled **once per tick and held fixed** —
re-sampling per batch would make the estimate jitter as the trim eats the head of the table and
could stall the loop through guard (2), which is the same self-defeating-measurement failure as
row 81 itself.
- **Per-tick delete ceiling** (`MaxPurgedRowsPerTick`, default **250 000**; `0` = unlimited): one
purge tick deletes at most this many rows, **shared** by the retention purge and the storage-cap
trim — two independently-capped phases would let a tick emit twice the intended burst. A tick that
@@ -128,7 +171,10 @@ Each event entry contains:
replication fault and not the purge. The purge logs an Information line naming the row count and
the expected transient backlog whenever it deletes rows on a replication-enabled node, so an
operator can correlate a spike with the purge that caused it — a spike with no such line nearby is
not the purge. Node-local wiring: the Host supplies the replication predicate
not the purge. That line is emitted **once per tick**, carrying the retention purge's and the cap
trim's deletes combined (register row 84 — it used to fire once per phase, so a tick in which both
deleted produced two identical notes; the tick total is also the more useful number to put beside
an oplog spike). Node-local wiring: the Host supplies the replication predicate
(`SiteLocalDbSetup.ReplicationIsConfigured` — the `PeerAddress`-OR-`ApiKey` rule) as
`SiteEventLogReplicationCheck`; this component never reads LocalDb configuration itself, and with
no predicate registered the note is suppressed. Operator detail —