ScadaBridge/docs/plans/2026-08-17-site-event-log-cap-semantics-decision.md

# `site_events` storage cap — what the cap measures (register row 81)

**Date:** 2026-08-17
**Row:** deferred-work register #81 — *"the storage cap fights its own oplog."*
**Outcome:** cap semantics changed. `ScadaBridge:SiteEventLog:MaxStorageMb` now bounds
**`site_events` and its four indexes**, not the whole consolidated LocalDb file. `dbstat` — the
exact measurement the register proposed — turned out to be **unavailable in the SQLite this
solution ships**, so the table's footprint is *modelled* from sampled row width. This memo records
the probe that ruled `dbstat` out, the model, its measured accuracy, and the new meaning of the
wipe-prevention guard.

---

## 1. The defect

Since LocalDb Phase 1/2 the site keeps ten tables in one SQLite file at `LocalDb:Path`, alongside
LocalDb's own `__localdb_oplog`, `__localdb_row_version` and `__localdb_snapshot_state`
bookkeeping. `EventLogPurgeService.GetDatabaseSizeBytes()` measured
`page_count − freelist_count` over **that whole file** and compared it to `MaxStorageMb`.

On a replication-enabled node the cap trim therefore fought itself. Every 1000-row batch freed
~390 KB of `site_events` pages *and* pushed ~1000 rows into `__localdb_oplog` plus ~1000
row-version tombstones into the same file. The measured size did not fall, the existing
"size did not decrease across an iteration" guard fired, and the trim stopped — with the cap
unmet.

Measured 2026-08-16 during the row-34 work
(`docs/plans/2026-08-15-site-events-purge-burst-measurement.md` §3.4/§5), 64 MB cap, 400 000 rows:

| Replicated? | Events deleted | Remaining | Size before → after | Reached the cap? |
| --- | ---: | ---: | --- | --- |
| no | 248 000 | 152 000 | 158.0 MB → 66.98 MB | **yes** (cap 67.11 MB) |
| yes | 174 000 | 226 000 | 203.6 MB → 167.7 MB | **no — stopped 2.5x over cap** |

Two consequences, both real:

1. **A node whose peer is down cannot enforce its event-log cap at all.** Nothing drains the
   oplog, so the file never shrinks and the guard stops the trim every tick, forever.
2. **The "1 GB event log" budget was in practice shared with replication bookkeeping the event
   log does not own.** Worse, the direction of the coupling is backwards: a *replication fault*
   (peer down, oplog piling up) would, under a naive "just keep deleting" fix, cause operational
   event history to be destroyed. The cap must not be a function of the peer's health.

The old guard was not the bug — it was doing its job, preventing the table being wiped in pursuit
of an unreachable target. The bug is that the target was unreachable, because the metric was
wrong.

## 2. `dbstat` is not available — the decision this memo turns on

The register proposed `SELECT sum(pgsize) FROM dbstat WHERE name='site_events'`. `dbstat` is a
**compile-time option** (`SQLITE_ENABLE_DBSTAT_VTAB`), and it was verified before any
implementation was written. It is **not enabled** in the native library this solution ships
(`SQLitePCLRaw.lib.e_sqlite3`, the security-pinned 2.1.12 build referenced by both
`ZB.MOM.WW.ScadaBridge.SiteEventLogging` and its test project).

Probed live against a real file, SQLite **3.53.3**:

```
PRAGMA compile_options   -> no ENABLE_DBSTAT_VTAB
                            (ENABLE_FTS3/4/5, RTREE, GEOPOLY, SESSION, PREUPDATE_HOOK, … present)
pragma_module_list       -> 10 modules: fts3, fts3tokenize, fts4, fts4aux, fts5, fts5vocab,
                            geopoly, pragma_module_list, rtree, rtree_i32
SELECT sum(pgsize) FROM dbstat            -> SQLite Error 1: 'no such table: dbstat'
SELECT … FROM dbstat('main') WHERE name=… -> SQLite Error 1: 'no such table: dbstat'
```

`sqlite_dbpage` — the other route to per-table page attribution — is absent for the same reason.
The module list is non-empty, so this is a genuine absence and not an empty-result artefact.

**This is pinned by a committed test**, `SqliteCapabilityTests.DbStat_IsNotAvailable_…`, written to
fail if `dbstat` ever appears. If it does, the right move is to delete the model in §3 and measure
exactly; the test's own comment says so.

Alternatives considered and rejected:

- **Switch to a SQLite build that enables `dbstat`** (system `libsqlite3` via
  `SQLitePCLRaw.bundle_sqlite3`, or a vendored native build). Rejected: the provider is shared with
  `ZB.MOM.WW.LocalDb` and every other SQLite consumer in the product, so this trades a storage-cap
  accuracy problem for fleet-wide version skew across macOS dev boxes and the Debian-based runtime
  image — a disproportionate blast radius for a coarse safety valve.
- **Give `site_events` its own file again.** Reverses LocalDb Phase 1/2 consolidation and removes
  the table from CDC replication. Out of the question.
- **Maintain a running byte counter in the writer, or via triggers.** The writer is not the only
  thing that inserts into `site_events` (replication applies remote rows), a counter does not
  survive a restart, and adding our own triggers to a `RegisterReplicated` table is hazardous —
  LocalDb builds its capture triggers from the column list at registration time and extra triggers
  would also fire on replicated applies.
- **Keep the whole-file metric and simply stop re-measuring mid-loop** (compute a delete plan once
  and execute it). This makes the trim terminate, but it makes the *event log* pay for bookkeeping
  bytes: an oplog inflated by a down peer would drive aggressive deletion of operational history.
  That is consequence 2 above, made worse rather than fixed.

## 3. What the cap measures now

`EventLogPurgeService.EstimateSiteEventsSize()` returns the live row count and modelled bytes per
row. `MaxStorageMb` is compared against `rowCount × bytesPerRow`.

Row width is **sampled** — `SizeSampleRows` (500) rows from each end of the table, head and tail.
The head is what the oldest-first trim will actually delete; the tail catches an event stream whose
shape has drifted. Widths are taken with `octet_length()` (SQLite 3.43+), not `length()`, which
counts characters and would under-size any multi-byte UTF-8 message; a second committed test pins
that function's availability.

The table term is computed as **whole pages**, `ceil(rows / rowsPerPage) × pageSize`, rather than
as a flat bytes-per-row rate:

```
usablePerPage = (pageSize - 35) * 0.78          // max-local-payload rule × measured mean page fill
avgCellBytes  = avgPayloadBytes + 16            // payload-length varint, rowid varint, cell pointer
rowsPerPage   = max(1, floor(usablePerPage / avgCellBytes))
tableBytes    = ceil(rowCount / rowsPerPage) * pageSize
indexBytes    = rowCount * (avgIndexedBytes + 8 * 4) / 0.75
estimate      = min(tableBytes + indexBytes, wholeFileLogicalBytes)
```

Whole pages rather than a rate because that term **dominates once rows approach half a page**: a
2 KB row cannot be packed two-to-a-page, so each one wastes the remainder. A linear model
underestimates that case by ~40 % (measured), and underestimating is the failure direction that
matters for a cap.

**Index pages count against the cap — deliberately.** Measured, the four indexes are ~34 % of
`site_events`' total footprint (397.2 vs 260.9 bytes/row with and without them, same 200 000 rows),
and the 1 GB default was sized against the 393 bytes/row figure from row 34, which already included
them. Excluding index pages would silently raise real disk consumption by about a third at an
unchanged setting. The index term is driven by the *sampled* widths of the four indexed columns, so
an event stream that leaves `instance_id` null is charged less than one that fills it.

**Clamped by the file.** The whole-file logical size is a true upper bound on any single table's
pages, so the model is capped by it. This can only ever correct an over-estimate downward, and it
cannot reintroduce the row-81 defect: on a replicated node the file is *larger* than
`site_events`, never smaller, so the clamp does not bind there.

`GetDatabaseSizeBytes()` survives, renamed in meaning rather than in signature: it is the whole
consolidated file's logical size, used for the clamp and as a diagnostic. It is no longer what the
cap compares against.

### 3.1 Measured accuracy

Calibration seeded realistic rows into a real file and measured `page_count`/`freelist_count`/
`page_size`. Two independent cross-checks against row 34's harness: the baseline workload came out
at **397.2 bytes/row** against row 34's independently measured **393**, and the row-count-at-1 GB
figure follows.

| Workload | rows | actual bytes/row | payload bytes/row |
| --- | ---: | ---: | ---: |
| 1-in-10 details @200 B (baseline) | 200 000 | 397.2 | 187.9 |
| same, **no indexes** | 200 000 | 260.9 | 187.9 |
| no details at all | 200 000 | 373.5 | 167.7 |
| details on every row @200 B | 200 000 | 597.7 | 367.7 |
| details on every row @2 KB | 50 000 | 4295.2 | 2215.7 |
| 1-in-10 details @4 KB | 50 000 | 824.7 | 583.0 |
| small table | 5 000 | 396.5 | 189.6 |

The shipped model, scored against a **differential** measurement of the table's real page cost
(file measured with ballast only, events added, growth attributed to `site_events`) — this is
`EstimateSiteEventsSize_TracksTheRealFootprint`, run on every build:

| Workload | estimated | actual | error |
| --- | ---: | ---: | ---: |
| narrow rows, no details | 3 155 626 | 2 912 256 | **+8.4 %** |
| typical rows, 200 B details | 3 992 096 | 3 538 944 | **+12.8 %** |
| wide rows, 2 KB details | 12 748 048 | 12 845 056 | **−0.8 %** |

The test asserts the band **−5 % … +25 %**. The bias is intentionally to the high side: a cap that
fires slightly early forfeits a slice of the budget, whereas one that reads low lets the log
overrun the disk allowance the cap exists to defend. Against the previous behaviour's **+250 %**
error on a replicated node this is not a close call.

**Known limitation.** The model samples 1000 rows out of a table that may hold millions. An event
stream whose row width changes sharply *in the middle* of the history — neither at the head nor the
tail — is mis-sized until the trim reaches it. Self-correcting in practice, since the cap is
re-evaluated every tick and the head sample advances as the oldest rows go.

### 3.2 Recalibration for the composite indexes (#123, 2026-11-10)

#123 replaced the single-column `event_type` and `severity` indexes with `(event_type, timestamp)`
and `(severity, timestamp)`. There are still four secondary indexes, so `IndexedColumnCount = 4`
stands. The sampler's `indexed_bytes` now counts `timestamp` three times: once for its own index
and once for each composite. The other constants are unchanged.

The model was re-scored with `EstimateSiteEventsSize_TracksTheRealFootprint`:

| Workload | estimated | actual | error | before #123 |
| --- | ---: | ---: | ---: | ---: |
| narrow rows, no details | 3 859 626 | 3 551 232 | **+8.7 %** | +8.4 % |
| typical rows, 200 B details | 4 520 096 | 4 005 888 | **+12.8 %** | +12.8 % |
| wide rows, 2 KB details | 13 012 048 | 13 078 528 | **−0.5 %** | −0.8 % |

The band and the high bias are unchanged. `SiteEventsSchemaCalibrationTests` pins the new index
set and the columns each index covers.

## 4. The wipe-prevention guard, restated

The register asked whether the "size did not decrease" guard should survive. It should, and it
does, but its meaning has changed and it has gained a sibling. There are now **two** backstops:

1. **Unusable row width → refuse to trim.** Checked *before* the cap comparison, because a zero
   bytes-per-row makes "is `site_events` over cap" unanswerable rather than false. Logs a Warning
   naming the row count and deletes nothing. The old code had no analogue: its answer to an
   unanswerable cap was to keep deleting until the file shrank, which is exactly how a cap that can
   never be met empties a table.

2. **Estimated size did not decrease across a batch → stop.** The old guard, re-based onto the
   table-scoped metric. Crucially it **can no longer be tripped by replication bookkeeping** — the
   metric excludes it — so when it fires it now means precisely one thing: the table is not
   shrinking even though deletes are landing, i.e. **new events are arriving at least as fast as
   the trim removes them.** That is a real operational condition and the Warning says so.

   For this to be observable rather than tautological, the loop re-reads the **row count** from the
   database after each batch instead of decrementing a local counter, so concurrent recorder writes
   are visible. Bytes-per-row, by contrast, is sampled **once per tick and held fixed** — re-sampling
   every batch would make the estimate jitter as the trim consumes the head of the table, and a
   sample that happened to widen could stall the loop through this very guard. That is the same
   class of self-defeating measurement as row 81 itself, and it is deliberately avoided.

Both backstops are covered by tests that pin a degenerate estimate through `internal virtual`
seams (`EstimateSiteEventsSize`, `GetSiteEventsRowCount`).

## 5. Behaviour on an UNREPLICATED node

Materially unchanged, as intended — but not bit-identical, and the delta is worth stating.

On a node with no CDC triggers there is no oplog and no tombstone table, so whole-file logical size
was already ≈ `site_events` + the nine other (largely small) tables. The cap therefore used to fire
at roughly the right place, and the row-34 measurement confirms it: the unreplicated trim reached
66.98 MB against a 67.11 MB cap.

What changes:

- **Slightly more headroom.** The other nine tables no longer count against the event log's budget.
  On a rig-shaped site those tables are small relative to a 1 GB event log, so the shift is a few
  percent at most.
- **Model error replaces measurement error.** Where the old number was exact-but-wrong-scoped, the
  new one is right-scoped but carries the +8 %/+13 % conservative bias of §3.1. Net effect on an
  unreplicated node: the cap fires marginally earlier than before at the same setting.

Neither warrants a config change. Sites that were happy with `MaxStorageMb: 1024` remain so.

## 6. Explicitly out of scope: LocalDb's own bookkeeping

This change makes the cap ignore `__localdb_oplog`, `__localdb_row_version` and
`__localdb_snapshot_state`. It does **not** attempt to manage them, and should not.

Their size is governed by LocalDb's own machinery — `LocalDb:Replication:MaxOplogRows` (default
1 000 000) and `TombstoneRetention` (default 7 days) — and row 34's pacing
(`MaxPurgedRowsPerTick`) is what keeps a purge burst from blowing through the oplog cap and forcing
a snapshot resync. Those are the correct controls and they already exist. A second, competing
bound applied from the event log would only obscure which subsystem is responsible for a full disk.

The honest statement of the budget is therefore: **`MaxStorageMb` bounds `site_events` and its
indexes; LocalDb replication bookkeeping is separate overhead on the same volume, bounded
separately.** Sizing a site's data volume means adding the two, and the topology guide now says so.
Row 34's measurement gives the scale — a 2.73 M-row purge left 707 MB of bookkeeping in the file
after deleting a 1 GB event log — and that overhead is transient, draining as the peer acks and as
tombstones expire.

## 7. Verification

- **`dbstat` unavailability probed before any implementation was written**, on a real file, with
  the result recorded in §2 and pinned by `SqliteCapabilityTests`.
- **Fail-before, row 81.** With `EstimateSiteEventsSize` reverted to pre-row-81 semantics (whole
  file attributed to `site_events`) and nothing else changed, **5 of the new tests fail** and all
  99 others pass. `PurgeByStorageCap_WithLargeBallastInTheSameFile_StillEnforces…` reproduces the
  original signature exactly: `remaining=2000 estimate=21725184 cap=11534336` — 4000 of 6000 rows
  destroyed and the cap **still unmet**.
- **Fail-before, row 84.** With per-phase emission restored,
  `RunPurge_WhenBothPhasesDeleteRows_LogsTheBacklogNoteOnceForTheWholeTick` fails with
  `Assert.Single() Failure: The collection contained 2 matching items`.
- `dotnet build ZB.MOM.WW.ScadaBridge.slnx` — clean, 0 warnings.
- `ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests`: **104 passed / 0 failed** (93 before this change).
- The docker rig was not touched.

## 8. Register row 84, folded in

`LogReplicationBacklogNote` was emitted by each purge phase, so a tick in which the retention purge
*and* the cap trim both deleted rows produced two identical Information notes. It is now emitted
once from `RunPurge`, driven by the tick's shared `PurgeBudget`.

The budget is the natural source: it already holds exactly "rows this tick deleted", which is also
the more useful number to put next to an oplog spike — the tick's whole contribution rather than
one phase's share. `PurgeByRetention` keeps its own "Purged N events older than D days" line; only
the replication note moved.