# `site_events` storage cap — what the cap measures (register row 81) **Date:** 2026-08-17 **Row:** deferred-work register #81 — *"the storage cap fights its own oplog."* **Outcome:** cap semantics changed. `ScadaBridge:SiteEventLog:MaxStorageMb` now bounds **`site_events` and its four indexes**, not the whole consolidated LocalDb file. `dbstat` — the exact measurement the register proposed — turned out to be **unavailable in the SQLite this solution ships**, so the table's footprint is *modelled* from sampled row width. This memo records the probe that ruled `dbstat` out, the model, its measured accuracy, and the new meaning of the wipe-prevention guard. --- ## 1. The defect Since LocalDb Phase 1/2 the site keeps ten tables in one SQLite file at `LocalDb:Path`, alongside LocalDb's own `__localdb_oplog`, `__localdb_row_version` and `__localdb_snapshot_state` bookkeeping. `EventLogPurgeService.GetDatabaseSizeBytes()` measured `page_count − freelist_count` over **that whole file** and compared it to `MaxStorageMb`. On a replication-enabled node the cap trim therefore fought itself. Every 1000-row batch freed ~390 KB of `site_events` pages *and* pushed ~1000 rows into `__localdb_oplog` plus ~1000 row-version tombstones into the same file. The measured size did not fall, the existing "size did not decrease across an iteration" guard fired, and the trim stopped — with the cap unmet. Measured 2026-08-16 during the row-34 work (`docs/plans/2026-08-15-site-events-purge-burst-measurement.md` §3.4/§5), 64 MB cap, 400 000 rows: | Replicated? | Events deleted | Remaining | Size before → after | Reached the cap? | | --- | ---: | ---: | --- | --- | | no | 248 000 | 152 000 | 158.0 MB → 66.98 MB | **yes** (cap 67.11 MB) | | yes | 174 000 | 226 000 | 203.6 MB → 167.7 MB | **no — stopped 2.5x over cap** | Two consequences, both real: 1. **A node whose peer is down cannot enforce its event-log cap at all.** Nothing drains the oplog, so the file never shrinks and the guard stops the trim every tick, forever. 2. **The "1 GB event log" budget was in practice shared with replication bookkeeping the event log does not own.** Worse, the direction of the coupling is backwards: a *replication fault* (peer down, oplog piling up) would, under a naive "just keep deleting" fix, cause operational event history to be destroyed. The cap must not be a function of the peer's health. The old guard was not the bug — it was doing its job, preventing the table being wiped in pursuit of an unreachable target. The bug is that the target was unreachable, because the metric was wrong. ## 2. `dbstat` is not available — the decision this memo turns on The register proposed `SELECT sum(pgsize) FROM dbstat WHERE name='site_events'`. `dbstat` is a **compile-time option** (`SQLITE_ENABLE_DBSTAT_VTAB`), and it was verified before any implementation was written. It is **not enabled** in the native library this solution ships (`SQLitePCLRaw.lib.e_sqlite3`, the security-pinned 2.1.12 build referenced by both `ZB.MOM.WW.ScadaBridge.SiteEventLogging` and its test project). Probed live against a real file, SQLite **3.53.3**: ``` PRAGMA compile_options -> no ENABLE_DBSTAT_VTAB (ENABLE_FTS3/4/5, RTREE, GEOPOLY, SESSION, PREUPDATE_HOOK, … present) pragma_module_list -> 10 modules: fts3, fts3tokenize, fts4, fts4aux, fts5, fts5vocab, geopoly, pragma_module_list, rtree, rtree_i32 SELECT sum(pgsize) FROM dbstat -> SQLite Error 1: 'no such table: dbstat' SELECT … FROM dbstat('main') WHERE name=… -> SQLite Error 1: 'no such table: dbstat' ``` `sqlite_dbpage` — the other route to per-table page attribution — is absent for the same reason. The module list is non-empty, so this is a genuine absence and not an empty-result artefact. **This is pinned by a committed test**, `SqliteCapabilityTests.DbStat_IsNotAvailable_…`, written to fail if `dbstat` ever appears. If it does, the right move is to delete the model in §3 and measure exactly; the test's own comment says so. Alternatives considered and rejected: - **Switch to a SQLite build that enables `dbstat`** (system `libsqlite3` via `SQLitePCLRaw.bundle_sqlite3`, or a vendored native build). Rejected: the provider is shared with `ZB.MOM.WW.LocalDb` and every other SQLite consumer in the product, so this trades a storage-cap accuracy problem for fleet-wide version skew across macOS dev boxes and the Debian-based runtime image — a disproportionate blast radius for a coarse safety valve. - **Give `site_events` its own file again.** Reverses LocalDb Phase 1/2 consolidation and removes the table from CDC replication. Out of the question. - **Maintain a running byte counter in the writer, or via triggers.** The writer is not the only thing that inserts into `site_events` (replication applies remote rows), a counter does not survive a restart, and adding our own triggers to a `RegisterReplicated` table is hazardous — LocalDb builds its capture triggers from the column list at registration time and extra triggers would also fire on replicated applies. - **Keep the whole-file metric and simply stop re-measuring mid-loop** (compute a delete plan once and execute it). This makes the trim terminate, but it makes the *event log* pay for bookkeeping bytes: an oplog inflated by a down peer would drive aggressive deletion of operational history. That is consequence 2 above, made worse rather than fixed. ## 3. What the cap measures now `EventLogPurgeService.EstimateSiteEventsSize()` returns the live row count and modelled bytes per row. `MaxStorageMb` is compared against `rowCount × bytesPerRow`. Row width is **sampled** — `SizeSampleRows` (500) rows from each end of the table, head and tail. The head is what the oldest-first trim will actually delete; the tail catches an event stream whose shape has drifted. Widths are taken with `octet_length()` (SQLite 3.43+), not `length()`, which counts characters and would under-size any multi-byte UTF-8 message; a second committed test pins that function's availability. The table term is computed as **whole pages**, `ceil(rows / rowsPerPage) × pageSize`, rather than as a flat bytes-per-row rate: ``` usablePerPage = (pageSize - 35) * 0.78 // max-local-payload rule × measured mean page fill avgCellBytes = avgPayloadBytes + 16 // payload-length varint, rowid varint, cell pointer rowsPerPage = max(1, floor(usablePerPage / avgCellBytes)) tableBytes = ceil(rowCount / rowsPerPage) * pageSize indexBytes = rowCount * (avgIndexedBytes + 8 * 4) / 0.75 estimate = min(tableBytes + indexBytes, wholeFileLogicalBytes) ``` Whole pages rather than a rate because that term **dominates once rows approach half a page**: a 2 KB row cannot be packed two-to-a-page, so each one wastes the remainder. A linear model underestimates that case by ~40 % (measured), and underestimating is the failure direction that matters for a cap. **Index pages count against the cap — deliberately.** Measured, the four indexes are ~34 % of `site_events`' total footprint (397.2 vs 260.9 bytes/row with and without them, same 200 000 rows), and the 1 GB default was sized against the 393 bytes/row figure from row 34, which already included them. Excluding index pages would silently raise real disk consumption by about a third at an unchanged setting. The index term is driven by the *sampled* widths of the four indexed columns, so an event stream that leaves `instance_id` null is charged less than one that fills it. **Clamped by the file.** The whole-file logical size is a true upper bound on any single table's pages, so the model is capped by it. This can only ever correct an over-estimate downward, and it cannot reintroduce the row-81 defect: on a replicated node the file is *larger* than `site_events`, never smaller, so the clamp does not bind there. `GetDatabaseSizeBytes()` survives, renamed in meaning rather than in signature: it is the whole consolidated file's logical size, used for the clamp and as a diagnostic. It is no longer what the cap compares against. ### 3.1 Measured accuracy Calibration seeded realistic rows into a real file and measured `page_count`/`freelist_count`/ `page_size`. Two independent cross-checks against row 34's harness: the baseline workload came out at **397.2 bytes/row** against row 34's independently measured **393**, and the row-count-at-1 GB figure follows. | Workload | rows | actual bytes/row | payload bytes/row | | --- | ---: | ---: | ---: | | 1-in-10 details @200 B (baseline) | 200 000 | 397.2 | 187.9 | | same, **no indexes** | 200 000 | 260.9 | 187.9 | | no details at all | 200 000 | 373.5 | 167.7 | | details on every row @200 B | 200 000 | 597.7 | 367.7 | | details on every row @2 KB | 50 000 | 4295.2 | 2215.7 | | 1-in-10 details @4 KB | 50 000 | 824.7 | 583.0 | | small table | 5 000 | 396.5 | 189.6 | The shipped model, scored against a **differential** measurement of the table's real page cost (file measured with ballast only, events added, growth attributed to `site_events`) — this is `EstimateSiteEventsSize_TracksTheRealFootprint`, run on every build: | Workload | estimated | actual | error | | --- | ---: | ---: | ---: | | narrow rows, no details | 3 155 626 | 2 912 256 | **+8.4 %** | | typical rows, 200 B details | 3 992 096 | 3 538 944 | **+12.8 %** | | wide rows, 2 KB details | 12 748 048 | 12 845 056 | **−0.8 %** | The test asserts the band **−5 % … +25 %**. The bias is intentionally to the high side: a cap that fires slightly early forfeits a slice of the budget, whereas one that reads low lets the log overrun the disk allowance the cap exists to defend. Against the previous behaviour's **+250 %** error on a replicated node this is not a close call. **Known limitation.** The model samples 1000 rows out of a table that may hold millions. An event stream whose row width changes sharply *in the middle* of the history — neither at the head nor the tail — is mis-sized until the trim reaches it. Self-correcting in practice, since the cap is re-evaluated every tick and the head sample advances as the oldest rows go. ## 4. The wipe-prevention guard, restated The register asked whether the "size did not decrease" guard should survive. It should, and it does, but its meaning has changed and it has gained a sibling. There are now **two** backstops: 1. **Unusable row width → refuse to trim.** Checked *before* the cap comparison, because a zero bytes-per-row makes "is `site_events` over cap" unanswerable rather than false. Logs a Warning naming the row count and deletes nothing. The old code had no analogue: its answer to an unanswerable cap was to keep deleting until the file shrank, which is exactly how a cap that can never be met empties a table. 2. **Estimated size did not decrease across a batch → stop.** The old guard, re-based onto the table-scoped metric. Crucially it **can no longer be tripped by replication bookkeeping** — the metric excludes it — so when it fires it now means precisely one thing: the table is not shrinking even though deletes are landing, i.e. **new events are arriving at least as fast as the trim removes them.** That is a real operational condition and the Warning says so. For this to be observable rather than tautological, the loop re-reads the **row count** from the database after each batch instead of decrementing a local counter, so concurrent recorder writes are visible. Bytes-per-row, by contrast, is sampled **once per tick and held fixed** — re-sampling every batch would make the estimate jitter as the trim consumes the head of the table, and a sample that happened to widen could stall the loop through this very guard. That is the same class of self-defeating measurement as row 81 itself, and it is deliberately avoided. Both backstops are covered by tests that pin a degenerate estimate through `internal virtual` seams (`EstimateSiteEventsSize`, `GetSiteEventsRowCount`). ## 5. Behaviour on an UNREPLICATED node Materially unchanged, as intended — but not bit-identical, and the delta is worth stating. On a node with no CDC triggers there is no oplog and no tombstone table, so whole-file logical size was already ≈ `site_events` + the nine other (largely small) tables. The cap therefore used to fire at roughly the right place, and the row-34 measurement confirms it: the unreplicated trim reached 66.98 MB against a 67.11 MB cap. What changes: - **Slightly more headroom.** The other nine tables no longer count against the event log's budget. On a rig-shaped site those tables are small relative to a 1 GB event log, so the shift is a few percent at most. - **Model error replaces measurement error.** Where the old number was exact-but-wrong-scoped, the new one is right-scoped but carries the +8 %/+13 % conservative bias of §3.1. Net effect on an unreplicated node: the cap fires marginally earlier than before at the same setting. Neither warrants a config change. Sites that were happy with `MaxStorageMb: 1024` remain so. ## 6. Explicitly out of scope: LocalDb's own bookkeeping This change makes the cap ignore `__localdb_oplog`, `__localdb_row_version` and `__localdb_snapshot_state`. It does **not** attempt to manage them, and should not. Their size is governed by LocalDb's own machinery — `LocalDb:Replication:MaxOplogRows` (default 1 000 000) and `TombstoneRetention` (default 7 days) — and row 34's pacing (`MaxPurgedRowsPerTick`) is what keeps a purge burst from blowing through the oplog cap and forcing a snapshot resync. Those are the correct controls and they already exist. A second, competing bound applied from the event log would only obscure which subsystem is responsible for a full disk. The honest statement of the budget is therefore: **`MaxStorageMb` bounds `site_events` and its indexes; LocalDb replication bookkeeping is separate overhead on the same volume, bounded separately.** Sizing a site's data volume means adding the two, and the topology guide now says so. Row 34's measurement gives the scale — a 2.73 M-row purge left 707 MB of bookkeeping in the file after deleting a 1 GB event log — and that overhead is transient, draining as the peer acks and as tombstones expire. ## 7. Verification - **`dbstat` unavailability probed before any implementation was written**, on a real file, with the result recorded in §2 and pinned by `SqliteCapabilityTests`. - **Fail-before, row 81.** With `EstimateSiteEventsSize` reverted to pre-row-81 semantics (whole file attributed to `site_events`) and nothing else changed, **5 of the new tests fail** and all 99 others pass. `PurgeByStorageCap_WithLargeBallastInTheSameFile_StillEnforces…` reproduces the original signature exactly: `remaining=2000 estimate=21725184 cap=11534336` — 4000 of 6000 rows destroyed and the cap **still unmet**. - **Fail-before, row 84.** With per-phase emission restored, `RunPurge_WhenBothPhasesDeleteRows_LogsTheBacklogNoteOnceForTheWholeTick` fails with `Assert.Single() Failure: The collection contained 2 matching items`. - `dotnet build ZB.MOM.WW.ScadaBridge.slnx` — clean, 0 warnings. - `ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests`: **104 passed / 0 failed** (93 before this change). - The docker rig was not touched. ## 8. Register row 84, folded in `LogReplicationBacklogNote` was emitted by each purge phase, so a tick in which the retention purge *and* the cap trim both deleted rows produced two identical Information notes. It is now emitted once from `RunPurge`, driven by the tick's shared `PurgeBudget`. The budget is the natural source: it already holds exactly "rows this tick deleted", which is also the more useful number to put next to an oplog spike — the tick's whole contribution rather than one phase's share. `PurgeByRetention` keeps its own "Purged N events older than D days" line; only the replication note moved.