ScadaBridge/docs/plans/2026-08-17-site-event-log-cap-semantics-decision.md

site_events storage cap — what the cap measures (register row 81)

Date: 2026-08-17 Row: deferred-work register #81 — "the storage cap fights its own oplog." Outcome: cap semantics changed. ScadaBridge:SiteEventLog:MaxStorageMb now bounds site_events and its four indexes, not the whole consolidated LocalDb file. dbstat — the exact measurement the register proposed — turned out to be unavailable in the SQLite this solution ships, so the table's footprint is modelled from sampled row width. This memo records the probe that ruled dbstat out, the model, its measured accuracy, and the new meaning of the wipe-prevention guard.

---

1. The defect

Since LocalDb Phase 1/2 the site keeps ten tables in one SQLite file at LocalDb:Path, alongside LocalDb's own __localdb_oplog, __localdb_row_version and __localdb_snapshot_state bookkeeping. EventLogPurgeService.GetDatabaseSizeBytes() measured page_count − freelist_count over that whole file and compared it to MaxStorageMb.

On a replication-enabled node the cap trim therefore fought itself. Every 1000-row batch freed ~390 KB of site_events pages and pushed ~1000 rows into __localdb_oplog plus ~1000 row-version tombstones into the same file. The measured size did not fall, the existing "size did not decrease across an iteration" guard fired, and the trim stopped — with the cap unmet.

Measured 2026-08-16 during the row-34 work (docs/plans/2026-08-15-site-events-purge-burst-measurement.md §3.4/§5), 64 MB cap, 400 000 rows:

Replicated?	Events deleted	Remaining	Size before → after	Reached the cap?
no	248 000	152 000	158.0 MB → 66.98 MB	yes (cap 67.11 MB)
yes	174 000	226 000	203.6 MB → 167.7 MB	no — stopped 2.5x over cap

Two consequences, both real:

1. A node whose peer is down cannot enforce its event-log cap at all. Nothing drains the oplog, so the file never shrinks and the guard stops the trim every tick, forever.
2. The "1 GB event log" budget was in practice shared with replication bookkeeping the event log does not own. Worse, the direction of the coupling is backwards: a replication fault (peer down, oplog piling up) would, under a naive "just keep deleting" fix, cause operational event history to be destroyed. The cap must not be a function of the peer's health.

The old guard was not the bug — it was doing its job, preventing the table being wiped in pursuit of an unreachable target. The bug is that the target was unreachable, because the metric was wrong.

2. dbstat is not available — the decision this memo turns on

The register proposed SELECT sum(pgsize) FROM dbstat WHERE name='site_events'. dbstat is a compile-time option (SQLITE_ENABLE_DBSTAT_VTAB), and it was verified before any implementation was written. It is not enabled in the native library this solution ships (SQLitePCLRaw.lib.e_sqlite3, the security-pinned 2.1.12 build referenced by both ZB.MOM.WW.ScadaBridge.SiteEventLogging and its test project).

Probed live against a real file, SQLite 3.53.3:

PRAGMA compile_options   -> no ENABLE_DBSTAT_VTAB
                            (ENABLE_FTS3/4/5, RTREE, GEOPOLY, SESSION, PREUPDATE_HOOK, … present)
pragma_module_list       -> 10 modules: fts3, fts3tokenize, fts4, fts4aux, fts5, fts5vocab,
                            geopoly, pragma_module_list, rtree, rtree_i32
SELECT sum(pgsize) FROM dbstat            -> SQLite Error 1: 'no such table: dbstat'
SELECT … FROM dbstat('main') WHERE name=… -> SQLite Error 1: 'no such table: dbstat'

sqlite_dbpage — the other route to per-table page attribution — is absent for the same reason. The module list is non-empty, so this is a genuine absence and not an empty-result artefact.

This is pinned by a committed test, SqliteCapabilityTests.DbStat_IsNotAvailable_…, written to fail if dbstat ever appears. If it does, the right move is to delete the model in §3 and measure exactly; the test's own comment says so.

Alternatives considered and rejected:

• Switch to a SQLite build that enables dbstat (system libsqlite3 via SQLitePCLRaw.bundle_sqlite3, or a vendored native build). Rejected: the provider is shared with ZB.MOM.WW.LocalDb and every other SQLite consumer in the product, so this trades a storage-cap accuracy problem for fleet-wide version skew across macOS dev boxes and the Debian-based runtime image — a disproportionate blast radius for a coarse safety valve.
• Give site_events its own file again. Reverses LocalDb Phase 1/2 consolidation and removes the table from CDC replication. Out of the question.
• Maintain a running byte counter in the writer, or via triggers. The writer is not the only thing that inserts into site_events (replication applies remote rows), a counter does not survive a restart, and adding our own triggers to a RegisterReplicated table is hazardous — LocalDb builds its capture triggers from the column list at registration time and extra triggers would also fire on replicated applies.
• Keep the whole-file metric and simply stop re-measuring mid-loop (compute a delete plan once and execute it). This makes the trim terminate, but it makes the event log pay for bookkeeping bytes: an oplog inflated by a down peer would drive aggressive deletion of operational history. That is consequence 2 above, made worse rather than fixed.

3. What the cap measures now

EventLogPurgeService.EstimateSiteEventsSize() returns the live row count and modelled bytes per row. MaxStorageMb is compared against rowCount × bytesPerRow.

Row width is sampled — SizeSampleRows (500) rows from each end of the table, head and tail. The head is what the oldest-first trim will actually delete; the tail catches an event stream whose shape has drifted. Widths are taken with octet_length() (SQLite 3.43+), not length(), which counts characters and would under-size any multi-byte UTF-8 message; a second committed test pins that function's availability.

The table term is computed as whole pages, ceil(rows / rowsPerPage) × pageSize, rather than as a flat bytes-per-row rate:

usablePerPage = (pageSize - 35) * 0.78          // max-local-payload rule × measured mean page fill
avgCellBytes  = avgPayloadBytes + 16            // payload-length varint, rowid varint, cell pointer
rowsPerPage   = max(1, floor(usablePerPage / avgCellBytes))
tableBytes    = ceil(rowCount / rowsPerPage) * pageSize
indexBytes    = rowCount * (avgIndexedBytes + 8 * 4) / 0.75
estimate      = min(tableBytes + indexBytes, wholeFileLogicalBytes)

Whole pages rather than a rate because that term dominates once rows approach half a page: a 2 KB row cannot be packed two-to-a-page, so each one wastes the remainder. A linear model underestimates that case by ~40 % (measured), and underestimating is the failure direction that matters for a cap.

Index pages count against the cap — deliberately. Measured, the four indexes are ~34 % of site_events' total footprint (397.2 vs 260.9 bytes/row with and without them, same 200 000 rows), and the 1 GB default was sized against the 393 bytes/row figure from row 34, which already included them. Excluding index pages would silently raise real disk consumption by about a third at an unchanged setting. The index term is driven by the sampled widths of the four indexed columns, so an event stream that leaves instance_id null is charged less than one that fills it.

Clamped by the file. The whole-file logical size is a true upper bound on any single table's pages, so the model is capped by it. This can only ever correct an over-estimate downward, and it cannot reintroduce the row-81 defect: on a replicated node the file is larger than site_events, never smaller, so the clamp does not bind there.

GetDatabaseSizeBytes() survives, renamed in meaning rather than in signature: it is the whole consolidated file's logical size, used for the clamp and as a diagnostic. It is no longer what the cap compares against.

3.1 Measured accuracy

Calibration seeded realistic rows into a real file and measured page_count/freelist_count/ page_size. Two independent cross-checks against row 34's harness: the baseline workload came out at 397.2 bytes/row against row 34's independently measured 393, and the row-count-at-1 GB figure follows.

Workload	rows	actual bytes/row	payload bytes/row
1-in-10 details @200 B (baseline)	200 000	397.2	187.9
same, no indexes	200 000	260.9	187.9
no details at all	200 000	373.5	167.7
details on every row @200 B	200 000	597.7	367.7
details on every row @2 KB	50 000	4295.2	2215.7
1-in-10 details @4 KB	50 000	824.7	583.0
small table	5 000	396.5	189.6

The shipped model, scored against a differential measurement of the table's real page cost (file measured with ballast only, events added, growth attributed to site_events) — this is EstimateSiteEventsSize_TracksTheRealFootprint, run on every build:

Workload	estimated	actual	error
narrow rows, no details	3 155 626	2 912 256	+8.4 %
typical rows, 200 B details	3 992 096	3 538 944	+12.8 %
wide rows, 2 KB details	12 748 048	12 845 056	−0.8 %

The test asserts the band −5 % … +25 %. The bias is intentionally to the high side: a cap that fires slightly early forfeits a slice of the budget, whereas one that reads low lets the log overrun the disk allowance the cap exists to defend. Against the previous behaviour's +250 % error on a replicated node this is not a close call.

Known limitation. The model samples 1000 rows out of a table that may hold millions. An event stream whose row width changes sharply in the middle of the history — neither at the head nor the tail — is mis-sized until the trim reaches it. Self-correcting in practice, since the cap is re-evaluated every tick and the head sample advances as the oldest rows go.

3.2 Recalibration for the composite indexes (#123, 2026-11-10)

#123 replaced the single-column event_type and severity indexes with (event_type, timestamp) and (severity, timestamp). There are still four secondary indexes, so IndexedColumnCount = 4 stands. The sampler's indexed_bytes now counts timestamp three times: once for its own index and once for each composite. The other constants are unchanged.

The model was re-scored with EstimateSiteEventsSize_TracksTheRealFootprint:

Workload	estimated	actual	error	before #123
narrow rows, no details	3 859 626	3 551 232	+8.7 %	+8.4 %
typical rows, 200 B details	4 520 096	4 005 888	+12.8 %	+12.8 %
wide rows, 2 KB details	13 012 048	13 078 528	−0.5 %	−0.8 %

The band and the high bias are unchanged. SiteEventsSchemaCalibrationTests pins the new index set and the columns each index covers.

4. The wipe-prevention guard, restated

The register asked whether the "size did not decrease" guard should survive. It should, and it does, but its meaning has changed and it has gained a sibling. There are now two backstops:

1. Unusable row width → refuse to trim. Checked before the cap comparison, because a zero bytes-per-row makes "is site_events over cap" unanswerable rather than false. Logs a Warning naming the row count and deletes nothing. The old code had no analogue: its answer to an unanswerable cap was to keep deleting until the file shrank, which is exactly how a cap that can never be met empties a table.

2. Estimated size did not decrease across a batch → stop. The old guard, re-based onto the table-scoped metric. Crucially it can no longer be tripped by replication bookkeeping — the metric excludes it — so when it fires it now means precisely one thing: the table is not shrinking even though deletes are landing, i.e. new events are arriving at least as fast as the trim removes them. That is a real operational condition and the Warning says so.

   For this to be observable rather than tautological, the loop re-reads the row count from the database after each batch instead of decrementing a local counter, so concurrent recorder writes are visible. Bytes-per-row, by contrast, is sampled once per tick and held fixed — re-sampling every batch would make the estimate jitter as the trim consumes the head of the table, and a sample that happened to widen could stall the loop through this very guard. That is the same class of self-defeating measurement as row 81 itself, and it is deliberately avoided.

Both backstops are covered by tests that pin a degenerate estimate through internal virtual seams (EstimateSiteEventsSize, GetSiteEventsRowCount).

5. Behaviour on an UNREPLICATED node

Materially unchanged, as intended — but not bit-identical, and the delta is worth stating.

On a node with no CDC triggers there is no oplog and no tombstone table, so whole-file logical size was already ≈ site_events + the nine other (largely small) tables. The cap therefore used to fire at roughly the right place, and the row-34 measurement confirms it: the unreplicated trim reached 66.98 MB against a 67.11 MB cap.

What changes:

• Slightly more headroom. The other nine tables no longer count against the event log's budget. On a rig-shaped site those tables are small relative to a 1 GB event log, so the shift is a few percent at most.
• Model error replaces measurement error. Where the old number was exact-but-wrong-scoped, the new one is right-scoped but carries the +8 %/+13 % conservative bias of §3.1. Net effect on an unreplicated node: the cap fires marginally earlier than before at the same setting.

Neither warrants a config change. Sites that were happy with MaxStorageMb: 1024 remain so.

6. Explicitly out of scope: LocalDb's own bookkeeping

This change makes the cap ignore __localdb_oplog, __localdb_row_version and __localdb_snapshot_state. It does not attempt to manage them, and should not.

Their size is governed by LocalDb's own machinery — LocalDb:Replication:MaxOplogRows (default 1 000 000) and TombstoneRetention (default 7 days) — and row 34's pacing (MaxPurgedRowsPerTick) is what keeps a purge burst from blowing through the oplog cap and forcing a snapshot resync. Those are the correct controls and they already exist. A second, competing bound applied from the event log would only obscure which subsystem is responsible for a full disk.

The honest statement of the budget is therefore: MaxStorageMb bounds site_events and its indexes; LocalDb replication bookkeeping is separate overhead on the same volume, bounded separately. Sizing a site's data volume means adding the two, and the topology guide now says so. Row 34's measurement gives the scale — a 2.73 M-row purge left 707 MB of bookkeeping in the file after deleting a 1 GB event log — and that overhead is transient, draining as the peer acks and as tombstones expire.

7. Verification

• dbstat unavailability probed before any implementation was written, on a real file, with the result recorded in §2 and pinned by SqliteCapabilityTests.
• Fail-before, row 81. With EstimateSiteEventsSize reverted to pre-row-81 semantics (whole file attributed to site_events) and nothing else changed, 5 of the new tests fail and all 99 others pass. PurgeByStorageCap_WithLargeBallastInTheSameFile_StillEnforces… reproduces the original signature exactly: remaining=2000 estimate=21725184 cap=11534336 — 4000 of 6000 rows destroyed and the cap still unmet.
• Fail-before, row 84. With per-phase emission restored, RunPurge_WhenBothPhasesDeleteRows_LogsTheBacklogNoteOnceForTheWholeTick fails with Assert.Single() Failure: The collection contained 2 matching items.
• dotnet build ZB.MOM.WW.ScadaBridge.slnx — clean, 0 warnings.
• ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests: 104 passed / 0 failed (93 before this change).
• The docker rig was not touched.

8. Register row 84, folded in

LogReplicationBacklogNote was emitted by each purge phase, so a tick in which the retention purge and the cap trim both deleted rows produced two identical Information notes. It is now emitted once from RunPurge, driven by the tick's shared PurgeBudget.

The budget is the natural source: it already holds exactly "rows this tick deleted", which is also the more useful number to put next to an oplog spike — the tick's whole contribution rather than one phase's share. PurgeByRetention keeps its own "Purged N events older than D days" line; only the replication note moved.