merge(agent-b): rows 81+84 — storage cap now measures site_events' own footprint (sampled page-layout model; dbstat probed unavailable and pinned); backlog note once per tick
This commit is contained in:
Joseph Doherty
@@ -204,6 +204,18 @@ practice shared with replication bookkeeping that the event log does not own. A
|
||||
measure `site_events`' own pages (e.g. `dbstat`) rather than the file, which is a change to the
|
||||
cap's semantics and deserves its own decision. Reported for the register; no row added here.
|
||||
|
||||
> **Postscript (2026-08-17) — fixed; register row 81 closed.** The cap now measures `site_events`
|
||||
> and its four indexes only. One correction to the suggestion above: **`dbstat` turned out to be
|
||||
> unavailable.** It is a compile-time option and the bundled `SQLitePCLRaw.lib.e_sqlite3`
|
||||
> (SQLite 3.53.3) does not enable it — `PRAGMA compile_options` lists no `ENABLE_DBSTAT_VTAB`, and
|
||||
> neither `dbstat` nor `sqlite_dbpage` appears in `pragma_module_list`. The table's footprint is
|
||||
> therefore *modelled* from sampled row width rather than measured, at −0.8 %…+12.8 % accuracy
|
||||
> against the numbers in §3.4, biased high on purpose. The "size did not decrease" guard survives,
|
||||
> re-based onto the table-scoped metric so replication bookkeeping can no longer trip it. Decision
|
||||
> record, including the probe evidence and why a different SQLite build was rejected:
|
||||
> `docs/plans/2026-08-17-site-event-log-cap-semantics-decision.md`. §3.4's replicated row is left
|
||||
> as measured — it is the record of the defect, not of current behaviour.
|
||||
|
||||
## 6. Verification
|
||||
|
||||
- Fail-before: the new behavioural tests do not compile against pre-fix `EventLogPurgeService`
|
||||
|
||||
@@ -0,0 +1,264 @@
|
||||
# `site_events` storage cap — what the cap measures (register row 81)
|
||||
|
||||
**Date:** 2026-08-17
|
||||
**Row:** deferred-work register #81 — *"the storage cap fights its own oplog."*
|
||||
**Outcome:** cap semantics changed. `ScadaBridge:SiteEventLog:MaxStorageMb` now bounds
|
||||
**`site_events` and its four indexes**, not the whole consolidated LocalDb file. `dbstat` — the
|
||||
exact measurement the register proposed — turned out to be **unavailable in the SQLite this
|
||||
solution ships**, so the table's footprint is *modelled* from sampled row width. This memo records
|
||||
the probe that ruled `dbstat` out, the model, its measured accuracy, and the new meaning of the
|
||||
wipe-prevention guard.
|
||||
|
||||
---
|
||||
|
||||
## 1. The defect
|
||||
|
||||
Since LocalDb Phase 1/2 the site keeps ten tables in one SQLite file at `LocalDb:Path`, alongside
|
||||
LocalDb's own `__localdb_oplog`, `__localdb_row_version` and `__localdb_snapshot_state`
|
||||
bookkeeping. `EventLogPurgeService.GetDatabaseSizeBytes()` measured
|
||||
`page_count − freelist_count` over **that whole file** and compared it to `MaxStorageMb`.
|
||||
|
||||
On a replication-enabled node the cap trim therefore fought itself. Every 1000-row batch freed
|
||||
~390 KB of `site_events` pages *and* pushed ~1000 rows into `__localdb_oplog` plus ~1000
|
||||
row-version tombstones into the same file. The measured size did not fall, the existing
|
||||
"size did not decrease across an iteration" guard fired, and the trim stopped — with the cap
|
||||
unmet.
|
||||
|
||||
Measured 2026-08-16 during the row-34 work
|
||||
(`docs/plans/2026-08-15-site-events-purge-burst-measurement.md` §3.4/§5), 64 MB cap, 400 000 rows:
|
||||
|
||||
| Replicated? | Events deleted | Remaining | Size before → after | Reached the cap? |
|
||||
| --- | ---: | ---: | --- | --- |
|
||||
| no | 248 000 | 152 000 | 158.0 MB → 66.98 MB | **yes** (cap 67.11 MB) |
|
||||
| yes | 174 000 | 226 000 | 203.6 MB → 167.7 MB | **no — stopped 2.5x over cap** |
|
||||
|
||||
Two consequences, both real:
|
||||
|
||||
1. **A node whose peer is down cannot enforce its event-log cap at all.** Nothing drains the
|
||||
oplog, so the file never shrinks and the guard stops the trim every tick, forever.
|
||||
2. **The "1 GB event log" budget was in practice shared with replication bookkeeping the event
|
||||
log does not own.** Worse, the direction of the coupling is backwards: a *replication fault*
|
||||
(peer down, oplog piling up) would, under a naive "just keep deleting" fix, cause operational
|
||||
event history to be destroyed. The cap must not be a function of the peer's health.
|
||||
|
||||
The old guard was not the bug — it was doing its job, preventing the table being wiped in pursuit
|
||||
of an unreachable target. The bug is that the target was unreachable, because the metric was
|
||||
wrong.
|
||||
|
||||
## 2. `dbstat` is not available — the decision this memo turns on
|
||||
|
||||
The register proposed `SELECT sum(pgsize) FROM dbstat WHERE name='site_events'`. `dbstat` is a
|
||||
**compile-time option** (`SQLITE_ENABLE_DBSTAT_VTAB`), and it was verified before any
|
||||
implementation was written. It is **not enabled** in the native library this solution ships
|
||||
(`SQLitePCLRaw.lib.e_sqlite3`, the security-pinned 2.1.12 build referenced by both
|
||||
`ZB.MOM.WW.ScadaBridge.SiteEventLogging` and its test project).
|
||||
|
||||
Probed live against a real file, SQLite **3.53.3**:
|
||||
|
||||
```
|
||||
PRAGMA compile_options -> no ENABLE_DBSTAT_VTAB
|
||||
(ENABLE_FTS3/4/5, RTREE, GEOPOLY, SESSION, PREUPDATE_HOOK, … present)
|
||||
pragma_module_list -> 10 modules: fts3, fts3tokenize, fts4, fts4aux, fts5, fts5vocab,
|
||||
geopoly, pragma_module_list, rtree, rtree_i32
|
||||
SELECT sum(pgsize) FROM dbstat -> SQLite Error 1: 'no such table: dbstat'
|
||||
SELECT … FROM dbstat('main') WHERE name=… -> SQLite Error 1: 'no such table: dbstat'
|
||||
```
|
||||
|
||||
`sqlite_dbpage` — the other route to per-table page attribution — is absent for the same reason.
|
||||
The module list is non-empty, so this is a genuine absence and not an empty-result artefact.
|
||||
|
||||
**This is pinned by a committed test**, `SqliteCapabilityTests.DbStat_IsNotAvailable_…`, written to
|
||||
fail if `dbstat` ever appears. If it does, the right move is to delete the model in §3 and measure
|
||||
exactly; the test's own comment says so.
|
||||
|
||||
Alternatives considered and rejected:
|
||||
|
||||
- **Switch to a SQLite build that enables `dbstat`** (system `libsqlite3` via
|
||||
`SQLitePCLRaw.bundle_sqlite3`, or a vendored native build). Rejected: the provider is shared with
|
||||
`ZB.MOM.WW.LocalDb` and every other SQLite consumer in the product, so this trades a storage-cap
|
||||
accuracy problem for fleet-wide version skew across macOS dev boxes and the Debian-based runtime
|
||||
image — a disproportionate blast radius for a coarse safety valve.
|
||||
- **Give `site_events` its own file again.** Reverses LocalDb Phase 1/2 consolidation and removes
|
||||
the table from CDC replication. Out of the question.
|
||||
- **Maintain a running byte counter in the writer, or via triggers.** The writer is not the only
|
||||
thing that inserts into `site_events` (replication applies remote rows), a counter does not
|
||||
survive a restart, and adding our own triggers to a `RegisterReplicated` table is hazardous —
|
||||
LocalDb builds its capture triggers from the column list at registration time and extra triggers
|
||||
would also fire on replicated applies.
|
||||
- **Keep the whole-file metric and simply stop re-measuring mid-loop** (compute a delete plan once
|
||||
and execute it). This makes the trim terminate, but it makes the *event log* pay for bookkeeping
|
||||
bytes: an oplog inflated by a down peer would drive aggressive deletion of operational history.
|
||||
That is consequence 2 above, made worse rather than fixed.
|
||||
|
||||
## 3. What the cap measures now
|
||||
|
||||
`EventLogPurgeService.EstimateSiteEventsSize()` returns the live row count and modelled bytes per
|
||||
row. `MaxStorageMb` is compared against `rowCount × bytesPerRow`.
|
||||
|
||||
Row width is **sampled** — `SizeSampleRows` (500) rows from each end of the table, head and tail.
|
||||
The head is what the oldest-first trim will actually delete; the tail catches an event stream whose
|
||||
shape has drifted. Widths are taken with `octet_length()` (SQLite 3.43+), not `length()`, which
|
||||
counts characters and would under-size any multi-byte UTF-8 message; a second committed test pins
|
||||
that function's availability.
|
||||
|
||||
The table term is computed as **whole pages**, `ceil(rows / rowsPerPage) × pageSize`, rather than
|
||||
as a flat bytes-per-row rate:
|
||||
|
||||
```
|
||||
usablePerPage = (pageSize - 35) * 0.78 // max-local-payload rule × measured mean page fill
|
||||
avgCellBytes = avgPayloadBytes + 16 // payload-length varint, rowid varint, cell pointer
|
||||
rowsPerPage = max(1, floor(usablePerPage / avgCellBytes))
|
||||
tableBytes = ceil(rowCount / rowsPerPage) * pageSize
|
||||
indexBytes = rowCount * (avgIndexedBytes + 8 * 4) / 0.75
|
||||
estimate = min(tableBytes + indexBytes, wholeFileLogicalBytes)
|
||||
```
|
||||
|
||||
Whole pages rather than a rate because that term **dominates once rows approach half a page**: a
|
||||
2 KB row cannot be packed two-to-a-page, so each one wastes the remainder. A linear model
|
||||
underestimates that case by ~40 % (measured), and underestimating is the failure direction that
|
||||
matters for a cap.
|
||||
|
||||
**Index pages count against the cap — deliberately.** Measured, the four indexes are ~34 % of
|
||||
`site_events`' total footprint (397.2 vs 260.9 bytes/row with and without them, same 200 000 rows),
|
||||
and the 1 GB default was sized against the 393 bytes/row figure from row 34, which already included
|
||||
them. Excluding index pages would silently raise real disk consumption by about a third at an
|
||||
unchanged setting. The index term is driven by the *sampled* widths of the four indexed columns, so
|
||||
an event stream that leaves `instance_id` null is charged less than one that fills it.
|
||||
|
||||
**Clamped by the file.** The whole-file logical size is a true upper bound on any single table's
|
||||
pages, so the model is capped by it. This can only ever correct an over-estimate downward, and it
|
||||
cannot reintroduce the row-81 defect: on a replicated node the file is *larger* than
|
||||
`site_events`, never smaller, so the clamp does not bind there.
|
||||
|
||||
`GetDatabaseSizeBytes()` survives, renamed in meaning rather than in signature: it is the whole
|
||||
consolidated file's logical size, used for the clamp and as a diagnostic. It is no longer what the
|
||||
cap compares against.
|
||||
|
||||
### 3.1 Measured accuracy
|
||||
|
||||
Calibration seeded realistic rows into a real file and measured `page_count`/`freelist_count`/
|
||||
`page_size`. Two independent cross-checks against row 34's harness: the baseline workload came out
|
||||
at **397.2 bytes/row** against row 34's independently measured **393**, and the row-count-at-1 GB
|
||||
figure follows.
|
||||
|
||||
| Workload | rows | actual bytes/row | payload bytes/row |
|
||||
| --- | ---: | ---: | ---: |
|
||||
| 1-in-10 details @200 B (baseline) | 200 000 | 397.2 | 187.9 |
|
||||
| same, **no indexes** | 200 000 | 260.9 | 187.9 |
|
||||
| no details at all | 200 000 | 373.5 | 167.7 |
|
||||
| details on every row @200 B | 200 000 | 597.7 | 367.7 |
|
||||
| details on every row @2 KB | 50 000 | 4295.2 | 2215.7 |
|
||||
| 1-in-10 details @4 KB | 50 000 | 824.7 | 583.0 |
|
||||
| small table | 5 000 | 396.5 | 189.6 |
|
||||
|
||||
The shipped model, scored against a **differential** measurement of the table's real page cost
|
||||
(file measured with ballast only, events added, growth attributed to `site_events`) — this is
|
||||
`EstimateSiteEventsSize_TracksTheRealFootprint`, run on every build:
|
||||
|
||||
| Workload | estimated | actual | error |
|
||||
| --- | ---: | ---: | ---: |
|
||||
| narrow rows, no details | 3 155 626 | 2 912 256 | **+8.4 %** |
|
||||
| typical rows, 200 B details | 3 992 096 | 3 538 944 | **+12.8 %** |
|
||||
| wide rows, 2 KB details | 12 748 048 | 12 845 056 | **−0.8 %** |
|
||||
|
||||
The test asserts the band **−5 % … +25 %**. The bias is intentionally to the high side: a cap that
|
||||
fires slightly early forfeits a slice of the budget, whereas one that reads low lets the log
|
||||
overrun the disk allowance the cap exists to defend. Against the previous behaviour's **+250 %**
|
||||
error on a replicated node this is not a close call.
|
||||
|
||||
**Known limitation.** The model samples 1000 rows out of a table that may hold millions. An event
|
||||
stream whose row width changes sharply *in the middle* of the history — neither at the head nor the
|
||||
tail — is mis-sized until the trim reaches it. Self-correcting in practice, since the cap is
|
||||
re-evaluated every tick and the head sample advances as the oldest rows go.
|
||||
|
||||
## 4. The wipe-prevention guard, restated
|
||||
|
||||
The register asked whether the "size did not decrease" guard should survive. It should, and it
|
||||
does, but its meaning has changed and it has gained a sibling. There are now **two** backstops:
|
||||
|
||||
1. **Unusable row width → refuse to trim.** Checked *before* the cap comparison, because a zero
|
||||
bytes-per-row makes "is `site_events` over cap" unanswerable rather than false. Logs a Warning
|
||||
naming the row count and deletes nothing. The old code had no analogue: its answer to an
|
||||
unanswerable cap was to keep deleting until the file shrank, which is exactly how a cap that can
|
||||
never be met empties a table.
|
||||
|
||||
2. **Estimated size did not decrease across a batch → stop.** The old guard, re-based onto the
|
||||
table-scoped metric. Crucially it **can no longer be tripped by replication bookkeeping** — the
|
||||
metric excludes it — so when it fires it now means precisely one thing: the table is not
|
||||
shrinking even though deletes are landing, i.e. **new events are arriving at least as fast as
|
||||
the trim removes them.** That is a real operational condition and the Warning says so.
|
||||
|
||||
For this to be observable rather than tautological, the loop re-reads the **row count** from the
|
||||
database after each batch instead of decrementing a local counter, so concurrent recorder writes
|
||||
are visible. Bytes-per-row, by contrast, is sampled **once per tick and held fixed** — re-sampling
|
||||
every batch would make the estimate jitter as the trim consumes the head of the table, and a
|
||||
sample that happened to widen could stall the loop through this very guard. That is the same
|
||||
class of self-defeating measurement as row 81 itself, and it is deliberately avoided.
|
||||
|
||||
Both backstops are covered by tests that pin a degenerate estimate through `internal virtual`
|
||||
seams (`EstimateSiteEventsSize`, `GetSiteEventsRowCount`).
|
||||
|
||||
## 5. Behaviour on an UNREPLICATED node
|
||||
|
||||
Materially unchanged, as intended — but not bit-identical, and the delta is worth stating.
|
||||
|
||||
On a node with no CDC triggers there is no oplog and no tombstone table, so whole-file logical size
|
||||
was already ≈ `site_events` + the nine other (largely small) tables. The cap therefore used to fire
|
||||
at roughly the right place, and the row-34 measurement confirms it: the unreplicated trim reached
|
||||
66.98 MB against a 67.11 MB cap.
|
||||
|
||||
What changes:
|
||||
|
||||
- **Slightly more headroom.** The other nine tables no longer count against the event log's budget.
|
||||
On a rig-shaped site those tables are small relative to a 1 GB event log, so the shift is a few
|
||||
percent at most.
|
||||
- **Model error replaces measurement error.** Where the old number was exact-but-wrong-scoped, the
|
||||
new one is right-scoped but carries the +8 %/+13 % conservative bias of §3.1. Net effect on an
|
||||
unreplicated node: the cap fires marginally earlier than before at the same setting.
|
||||
|
||||
Neither warrants a config change. Sites that were happy with `MaxStorageMb: 1024` remain so.
|
||||
|
||||
## 6. Explicitly out of scope: LocalDb's own bookkeeping
|
||||
|
||||
This change makes the cap ignore `__localdb_oplog`, `__localdb_row_version` and
|
||||
`__localdb_snapshot_state`. It does **not** attempt to manage them, and should not.
|
||||
|
||||
Their size is governed by LocalDb's own machinery — `LocalDb:Replication:MaxOplogRows` (default
|
||||
1 000 000) and `TombstoneRetention` (default 7 days) — and row 34's pacing
|
||||
(`MaxPurgedRowsPerTick`) is what keeps a purge burst from blowing through the oplog cap and forcing
|
||||
a snapshot resync. Those are the correct controls and they already exist. A second, competing
|
||||
bound applied from the event log would only obscure which subsystem is responsible for a full disk.
|
||||
|
||||
The honest statement of the budget is therefore: **`MaxStorageMb` bounds `site_events` and its
|
||||
indexes; LocalDb replication bookkeeping is separate overhead on the same volume, bounded
|
||||
separately.** Sizing a site's data volume means adding the two, and the topology guide now says so.
|
||||
Row 34's measurement gives the scale — a 2.73 M-row purge left 707 MB of bookkeeping in the file
|
||||
after deleting a 1 GB event log — and that overhead is transient, draining as the peer acks and as
|
||||
tombstones expire.
|
||||
|
||||
## 7. Verification
|
||||
|
||||
- **`dbstat` unavailability probed before any implementation was written**, on a real file, with
|
||||
the result recorded in §2 and pinned by `SqliteCapabilityTests`.
|
||||
- **Fail-before, row 81.** With `EstimateSiteEventsSize` reverted to pre-row-81 semantics (whole
|
||||
file attributed to `site_events`) and nothing else changed, **5 of the new tests fail** and all
|
||||
99 others pass. `PurgeByStorageCap_WithLargeBallastInTheSameFile_StillEnforces…` reproduces the
|
||||
original signature exactly: `remaining=2000 estimate=21725184 cap=11534336` — 4000 of 6000 rows
|
||||
destroyed and the cap **still unmet**.
|
||||
- **Fail-before, row 84.** With per-phase emission restored,
|
||||
`RunPurge_WhenBothPhasesDeleteRows_LogsTheBacklogNoteOnceForTheWholeTick` fails with
|
||||
`Assert.Single() Failure: The collection contained 2 matching items`.
|
||||
- `dotnet build ZB.MOM.WW.ScadaBridge.slnx` — clean, 0 warnings.
|
||||
- `ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests`: **104 passed / 0 failed** (93 before this change).
|
||||
- The docker rig was not touched.
|
||||
|
||||
## 8. Register row 84, folded in
|
||||
|
||||
`LogReplicationBacklogNote` was emitted by each purge phase, so a tick in which the retention purge
|
||||
*and* the cap trim both deleted rows produced two identical Information notes. It is now emitted
|
||||
once from `RunPurge`, driven by the tick's shared `PurgeBudget`.
|
||||
|
||||
The budget is the natural source: it already holds exactly "rows this tick deleted", which is also
|
||||
the more useful number to put next to an oplog spike — the tick's whole contribution rather than
|
||||
one phase's share. `PurgeByRetention` keeps its own "Purged N events older than D days" line; only
|
||||
the replication note moved.
|
||||
Reference in New Issue
Block a user