fix(siteeventlog): scope the storage cap to site_events' own pages (register rows 81, 84)
Row 81 — the cap could not be enforced on a replicated node. MaxStorageMb was compared against page_count-freelist_count over the WHOLE consolidated LocalDb file, which since Phase 1/2 holds ten tables plus __localdb_oplog / __localdb_row_version / __localdb_snapshot_state. Every 1000-row trim batch freed ~390 KB of site_events pages and added ~1000 oplog rows + ~1000 tombstones to the same file, so the measured size did not fall, the "size did not decrease" guard fired, and the trim stopped 2.5x over a 64 MB cap with 226 k rows still present. A node whose peer was down could not enforce its cap at all, and the event log's budget was in practice shared with bookkeeping it does not own. The cap now measures site_events AND ITS FOUR INDEXES only, so replication bookkeeping can neither consume nor block the budget — and the cap stops being a function of the peer's health, which was the wrong direction of coupling: an inflated oplog is a replication fault, and letting it drive event deletion would destroy operational history in response to one. dbstat — the exact per-table measurement the register proposed — is NOT usable here, verified before implementing. It is a compile-time option and the bundled SQLitePCLRaw.lib.e_sqlite3 (SQLite 3.53.3) does not enable it: PRAGMA compile_options lists no ENABLE_DBSTAT_VTAB, pragma_module_list returns ten modules with neither dbstat nor sqlite_dbpage, and selecting from dbstat fails with "no such table". Switching SQLite builds was rejected — the provider is shared with ZB.MOM.WW.LocalDb and every other consumer, so it trades cap accuracy for fleet-wide version skew. The footprint is therefore modelled from sampled row width (500 rows each end, octet_length so multi-byte UTF-8 is not under-counted) through a page-layout model; whole pages rather than a flat rate, because page fill collapses once rows approach half a page and a linear model underestimates that by ~40%. Clamped by whole-file logical size, a true upper bound that can only correct an over-estimate and never binds on a replicated node. Index pages count against the cap deliberately: measured at ~34% of the table's footprint, and the 1 GB default was sized against a 393 B/row figure that already included them. Guard semantics restated — two backstops replace the single on-disk check: (1) row width unsampleable => refuse to trim (checked before the cap comparison, since a zero width makes "over cap" unanswerable rather than false; the old loop's answer to an unanswerable cap was to keep deleting); (2) estimated size not falling across a batch => stop. (2) is the old guard re-based onto the table-scoped metric, so it can no longer be tripped by replication bookkeeping and now means exactly one thing: events are arriving as fast as the trim removes them. For that to be observable the loop re-reads the row count each batch, while bytes-per-row is sampled once per tick and held fixed — re-sampling per batch would jitter the estimate as the trim eats the head of the table and could stall the loop through guard (2), the same self-defeating measurement as row 81 itself. Row 84 — LogReplicationBacklogNote moved from per-phase to once per tick, driven by the shared PurgeBudget. A tick where retention AND the cap trim both deleted used to emit two identical Information notes; the tick total is also the more useful number beside an oplog spike. Measured model error: -0.8% / +8.4% / +12.8% across representative workloads, biased high on purpose (an early cap forfeits budget; a low one fails at its only job). Independently cross-checks row 34's harness at 397.2 vs 393 B/row. Fail-before: reverting the estimator to whole-file semantics fails 5 of the new tests and reproduces the original signature exactly (remaining=2000 estimate=21725184 cap=11534336 — rows destroyed, cap still unmet); restoring per-phase notes fails the row-84 test with "collection contained 2 matching items". No config keys added or renamed, so no rig/deploy changes. Decision record: docs/plans/2026-08-17-site-event-log-cap-semantics-decision.md Tests: SiteEventLogging 104/104 (was 93), Host 520/520, solution build clean.
This commit is contained in:
@@ -93,7 +93,50 @@ Each event entry contains:
|
||||
rig's site-b/site-c), the log stays node-local and a failover does start it fresh — the
|
||||
documented trade of not configuring a peer, unchanged by this policy.
|
||||
- **Retention**: 30 days. A **daily background job** runs on the active node and deletes all events older than 30 days. Hard delete — no archival. Deletes are sliced into bounded 1000-row batches per DELETE statement rather than one unbounded statement, so a large expired backlog does not hold the write lock (and every concurrent recorder flush) for the duration of the purge.
|
||||
- **Storage cap**: A configurable maximum database size (default: 1 GB) is enforced. If the storage cap is reached before the 30-day retention window, the oldest events are purged first. This prevents disk exhaustion from alarm storms, script failure loops, or connection flapping.
|
||||
- **Storage cap**: A configurable maximum size (default: 1 GB) is enforced. If the cap is reached
|
||||
before the 30-day retention window, the oldest events are purged first. This prevents disk
|
||||
exhaustion from alarm storms, script failure loops, or connection flapping.
|
||||
- **The cap measures `site_events` and its four indexes — NOT the database file** (register row 81;
|
||||
decision record `docs/plans/2026-08-17-site-event-log-cap-semantics-decision.md`). This matters
|
||||
because the event log shares one SQLite file with nine other tables plus LocalDb's
|
||||
`__localdb_oplog` / `__localdb_row_version` / `__localdb_snapshot_state` bookkeeping. The cap used
|
||||
to compare `MaxStorageMb` against `page_count − freelist_count` over that whole file, which on a
|
||||
replication-enabled node made the trim **unwinnable**: each 1000-row delete freed ~390 KB of event
|
||||
pages while adding ~1000 oplog rows and ~1000 tombstones to the same file, the measured size did
|
||||
not fall, and the progress guard stopped the trim — measured 2.5x over a 64 MB cap with 226 k rows
|
||||
still present. A node whose peer was down could not enforce its cap at all. Two consequences of
|
||||
the new scoping are worth internalising:
|
||||
- **The cap is now independent of replication health.** It was previously a function of the
|
||||
*peer's* liveness, and in the wrong direction — an inflated oplog is a replication fault, and
|
||||
letting it drive event deletion would destroy operational history in response to one.
|
||||
- **`MaxStorageMb` no longer bounds the data volume on its own.** LocalDb bookkeeping is separate
|
||||
overhead on the same disk, bounded separately by `LocalDb:Replication:MaxOplogRows` and
|
||||
`TombstoneRetention`; sizing a site's volume means adding the two. Managing that bookkeeping is
|
||||
deliberately **not** this component's job — see the decision record §6.
|
||||
- **The measurement is modelled, not exact.** SQLite can report per-table page usage precisely via
|
||||
the `dbstat` virtual table, and that was the intended fix — but `dbstat` is a compile-time option
|
||||
and the bundled `SQLitePCLRaw.lib.e_sqlite3` native library (SQLite 3.53.3) does **not** enable it;
|
||||
neither `dbstat` nor `sqlite_dbpage` exists, verified by probe and pinned by
|
||||
`SqliteCapabilityTests`. The table's footprint is therefore estimated from sampled row width
|
||||
(500 rows from each end of the table, measured with `octet_length` so multi-byte UTF-8 is not
|
||||
under-counted) fed into a page-layout model; index pages count against the cap, deliberately,
|
||||
because they are ~34 % of the table's real footprint and the 1 GB default was sized against a
|
||||
bytes/row figure that included them. Measured error is **−0.8 % to +12.8 %** across representative
|
||||
workloads, biased high on purpose — a cap that fires slightly early forfeits budget, whereas one
|
||||
that reads low fails at the only job it has. The estimate is additionally clamped by the
|
||||
whole-file logical size, which is a true upper bound on any one table.
|
||||
- **Two wipe-prevention backstops** guard the cap trim, replacing the single "on-disk size did not
|
||||
decrease" check. (1) If row width cannot be sampled, the trim **refuses to run** and warns —
|
||||
checked before the cap comparison, because a zero row width makes "are we over cap" unanswerable
|
||||
rather than false, and the old loop's answer to an unanswerable cap was to keep deleting. (2) If
|
||||
the *estimated table size* fails to fall across a batch, the trim stops and warns. Guard (2) is the
|
||||
old check re-based onto the table-scoped metric, and its meaning is now unambiguous: replication
|
||||
bookkeeping can no longer trip it, so it fires only when new events are arriving at least as fast
|
||||
as the trim removes them. For that to be observable, the loop re-reads the row count from the
|
||||
database after every batch, while bytes-per-row is sampled **once per tick and held fixed** —
|
||||
re-sampling per batch would make the estimate jitter as the trim eats the head of the table and
|
||||
could stall the loop through guard (2), which is the same self-defeating-measurement failure as
|
||||
row 81 itself.
|
||||
- **Per-tick delete ceiling** (`MaxPurgedRowsPerTick`, default **250 000**; `0` = unlimited): one
|
||||
purge tick deletes at most this many rows, **shared** by the retention purge and the storage-cap
|
||||
trim — two independently-capped phases would let a tick emit twice the intended burst. A tick that
|
||||
@@ -128,7 +171,10 @@ Each event entry contains:
|
||||
replication fault and not the purge. The purge logs an Information line naming the row count and
|
||||
the expected transient backlog whenever it deletes rows on a replication-enabled node, so an
|
||||
operator can correlate a spike with the purge that caused it — a spike with no such line nearby is
|
||||
not the purge. Node-local wiring: the Host supplies the replication predicate
|
||||
not the purge. That line is emitted **once per tick**, carrying the retention purge's and the cap
|
||||
trim's deletes combined (register row 84 — it used to fire once per phase, so a tick in which both
|
||||
deleted produced two identical notes; the tick total is also the more useful number to put beside
|
||||
an oplog spike). Node-local wiring: the Host supplies the replication predicate
|
||||
(`SiteLocalDbSetup.ReplicationIsConfigured` — the `PeerAddress`-OR-`ApiKey` rule) as
|
||||
`SiteEventLogReplicationCheck`; this component never reads LocalDb configuration itself, and with
|
||||
no predicate registered the note is suppressed. Operator detail —
|
||||
|
||||
Reference in New Issue
Block a user