merge(agent-b): rows 81+84 — storage cap now measures site_events' own footprint (sampled page-layout model; dbstat probed unavailable and pinned); backlog note once per tick
This commit is contained in:
@@ -93,7 +93,50 @@ Each event entry contains:
|
||||
rig's site-b/site-c), the log stays node-local and a failover does start it fresh — the
|
||||
documented trade of not configuring a peer, unchanged by this policy.
|
||||
- **Retention**: 30 days. A **daily background job** runs on the active node and deletes all events older than 30 days. Hard delete — no archival. Deletes are sliced into bounded 1000-row batches per DELETE statement rather than one unbounded statement, so a large expired backlog does not hold the write lock (and every concurrent recorder flush) for the duration of the purge.
|
||||
- **Storage cap**: A configurable maximum database size (default: 1 GB) is enforced. If the storage cap is reached before the 30-day retention window, the oldest events are purged first. This prevents disk exhaustion from alarm storms, script failure loops, or connection flapping.
|
||||
- **Storage cap**: A configurable maximum size (default: 1 GB) is enforced. If the cap is reached
|
||||
before the 30-day retention window, the oldest events are purged first. This prevents disk
|
||||
exhaustion from alarm storms, script failure loops, or connection flapping.
|
||||
- **The cap measures `site_events` and its four indexes — NOT the database file** (register row 81;
|
||||
decision record `docs/plans/2026-08-17-site-event-log-cap-semantics-decision.md`). This matters
|
||||
because the event log shares one SQLite file with nine other tables plus LocalDb's
|
||||
`__localdb_oplog` / `__localdb_row_version` / `__localdb_snapshot_state` bookkeeping. The cap used
|
||||
to compare `MaxStorageMb` against `page_count − freelist_count` over that whole file, which on a
|
||||
replication-enabled node made the trim **unwinnable**: each 1000-row delete freed ~390 KB of event
|
||||
pages while adding ~1000 oplog rows and ~1000 tombstones to the same file, the measured size did
|
||||
not fall, and the progress guard stopped the trim — measured 2.5x over a 64 MB cap with 226 k rows
|
||||
still present. A node whose peer was down could not enforce its cap at all. Two consequences of
|
||||
the new scoping are worth internalising:
|
||||
- **The cap is now independent of replication health.** It was previously a function of the
|
||||
*peer's* liveness, and in the wrong direction — an inflated oplog is a replication fault, and
|
||||
letting it drive event deletion would destroy operational history in response to one.
|
||||
- **`MaxStorageMb` no longer bounds the data volume on its own.** LocalDb bookkeeping is separate
|
||||
overhead on the same disk, bounded separately by `LocalDb:Replication:MaxOplogRows` and
|
||||
`TombstoneRetention`; sizing a site's volume means adding the two. Managing that bookkeeping is
|
||||
deliberately **not** this component's job — see the decision record §6.
|
||||
- **The measurement is modelled, not exact.** SQLite can report per-table page usage precisely via
|
||||
the `dbstat` virtual table, and that was the intended fix — but `dbstat` is a compile-time option
|
||||
and the bundled `SQLitePCLRaw.lib.e_sqlite3` native library (SQLite 3.53.3) does **not** enable it;
|
||||
neither `dbstat` nor `sqlite_dbpage` exists, verified by probe and pinned by
|
||||
`SqliteCapabilityTests`. The table's footprint is therefore estimated from sampled row width
|
||||
(500 rows from each end of the table, measured with `octet_length` so multi-byte UTF-8 is not
|
||||
under-counted) fed into a page-layout model; index pages count against the cap, deliberately,
|
||||
because they are ~34 % of the table's real footprint and the 1 GB default was sized against a
|
||||
bytes/row figure that included them. Measured error is **−0.8 % to +12.8 %** across representative
|
||||
workloads, biased high on purpose — a cap that fires slightly early forfeits budget, whereas one
|
||||
that reads low fails at the only job it has. The estimate is additionally clamped by the
|
||||
whole-file logical size, which is a true upper bound on any one table.
|
||||
- **Two wipe-prevention backstops** guard the cap trim, replacing the single "on-disk size did not
|
||||
decrease" check. (1) If row width cannot be sampled, the trim **refuses to run** and warns —
|
||||
checked before the cap comparison, because a zero row width makes "are we over cap" unanswerable
|
||||
rather than false, and the old loop's answer to an unanswerable cap was to keep deleting. (2) If
|
||||
the *estimated table size* fails to fall across a batch, the trim stops and warns. Guard (2) is the
|
||||
old check re-based onto the table-scoped metric, and its meaning is now unambiguous: replication
|
||||
bookkeeping can no longer trip it, so it fires only when new events are arriving at least as fast
|
||||
as the trim removes them. For that to be observable, the loop re-reads the row count from the
|
||||
database after every batch, while bytes-per-row is sampled **once per tick and held fixed** —
|
||||
re-sampling per batch would make the estimate jitter as the trim eats the head of the table and
|
||||
could stall the loop through guard (2), which is the same self-defeating-measurement failure as
|
||||
row 81 itself.
|
||||
- **Per-tick delete ceiling** (`MaxPurgedRowsPerTick`, default **250 000**; `0` = unlimited): one
|
||||
purge tick deletes at most this many rows, **shared** by the retention purge and the storage-cap
|
||||
trim — two independently-capped phases would let a tick emit twice the intended burst. A tick that
|
||||
@@ -128,7 +171,10 @@ Each event entry contains:
|
||||
replication fault and not the purge. The purge logs an Information line naming the row count and
|
||||
the expected transient backlog whenever it deletes rows on a replication-enabled node, so an
|
||||
operator can correlate a spike with the purge that caused it — a spike with no such line nearby is
|
||||
not the purge. Node-local wiring: the Host supplies the replication predicate
|
||||
not the purge. That line is emitted **once per tick**, carrying the retention purge's and the cap
|
||||
trim's deletes combined (register row 84 — it used to fire once per phase, so a tick in which both
|
||||
deleted produced two identical notes; the tick total is also the more useful number to put beside
|
||||
an oplog spike). Node-local wiring: the Host supplies the replication predicate
|
||||
(`SiteLocalDbSetup.ReplicationIsConfigured` — the `PeerAddress`-OR-`ApiKey` rule) as
|
||||
`SiteEventLogReplicationCheck`; this component never reads LocalDb configuration itself, and with
|
||||
no predicate registered the note is suppressed. Operator detail —
|
||||
|
||||
Reference in New Issue
Block a user