fix(siteeventlog): scope the storage cap to site_events' own pages (register rows 81, 84)
Row 81 — the cap could not be enforced on a replicated node. MaxStorageMb was compared against page_count-freelist_count over the WHOLE consolidated LocalDb file, which since Phase 1/2 holds ten tables plus __localdb_oplog / __localdb_row_version / __localdb_snapshot_state. Every 1000-row trim batch freed ~390 KB of site_events pages and added ~1000 oplog rows + ~1000 tombstones to the same file, so the measured size did not fall, the "size did not decrease" guard fired, and the trim stopped 2.5x over a 64 MB cap with 226 k rows still present. A node whose peer was down could not enforce its cap at all, and the event log's budget was in practice shared with bookkeeping it does not own. The cap now measures site_events AND ITS FOUR INDEXES only, so replication bookkeeping can neither consume nor block the budget — and the cap stops being a function of the peer's health, which was the wrong direction of coupling: an inflated oplog is a replication fault, and letting it drive event deletion would destroy operational history in response to one. dbstat — the exact per-table measurement the register proposed — is NOT usable here, verified before implementing. It is a compile-time option and the bundled SQLitePCLRaw.lib.e_sqlite3 (SQLite 3.53.3) does not enable it: PRAGMA compile_options lists no ENABLE_DBSTAT_VTAB, pragma_module_list returns ten modules with neither dbstat nor sqlite_dbpage, and selecting from dbstat fails with "no such table". Switching SQLite builds was rejected — the provider is shared with ZB.MOM.WW.LocalDb and every other consumer, so it trades cap accuracy for fleet-wide version skew. The footprint is therefore modelled from sampled row width (500 rows each end, octet_length so multi-byte UTF-8 is not under-counted) through a page-layout model; whole pages rather than a flat rate, because page fill collapses once rows approach half a page and a linear model underestimates that by ~40%. Clamped by whole-file logical size, a true upper bound that can only correct an over-estimate and never binds on a replicated node. Index pages count against the cap deliberately: measured at ~34% of the table's footprint, and the 1 GB default was sized against a 393 B/row figure that already included them. Guard semantics restated — two backstops replace the single on-disk check: (1) row width unsampleable => refuse to trim (checked before the cap comparison, since a zero width makes "over cap" unanswerable rather than false; the old loop's answer to an unanswerable cap was to keep deleting); (2) estimated size not falling across a batch => stop. (2) is the old guard re-based onto the table-scoped metric, so it can no longer be tripped by replication bookkeeping and now means exactly one thing: events are arriving as fast as the trim removes them. For that to be observable the loop re-reads the row count each batch, while bytes-per-row is sampled once per tick and held fixed — re-sampling per batch would jitter the estimate as the trim eats the head of the table and could stall the loop through guard (2), the same self-defeating measurement as row 81 itself. Row 84 — LogReplicationBacklogNote moved from per-phase to once per tick, driven by the shared PurgeBudget. A tick where retention AND the cap trim both deleted used to emit two identical Information notes; the tick total is also the more useful number beside an oplog spike. Measured model error: -0.8% / +8.4% / +12.8% across representative workloads, biased high on purpose (an early cap forfeits budget; a low one fails at its only job). Independently cross-checks row 34's harness at 397.2 vs 393 B/row. Fail-before: reverting the estimator to whole-file semantics fails 5 of the new tests and reproduces the original signature exactly (remaining=2000 estimate=21725184 cap=11534336 — rows destroyed, cap still unmet); restoring per-phase notes fails the row-84 test with "collection contained 2 matching items". No config keys added or renamed, so no rig/deploy changes. Decision record: docs/plans/2026-08-17-site-event-log-cap-semantics-decision.md Tests: SiteEventLogging 104/104 (was 93), Host 520/520, solution build clean.
This commit is contained in:
Joseph Doherty
@@ -273,6 +273,13 @@ history had expired.
|
||||
An unreplicated node logs only the first line. If a backlog spike has no such line near it in the
|
||||
active node's log, the purge is *not* the explanation and the spike is worth investigating.
|
||||
|
||||
The second line is emitted **once per purge tick**, and its count covers the retention purge and
|
||||
the storage-cap trim together. If you are comparing it against the `Purged N events older than…`
|
||||
line above it, expect the two numbers to differ whenever the cap trim also ran — the note reports
|
||||
the tick's whole contribution to the oplog, which is what a backlog spike is made of. (Through
|
||||
2026-08-16 the note was emitted per phase, so a tick that ran both produced two near-identical
|
||||
lines; register row 84.)
|
||||
|
||||
**A tick that hit the ceiling adds a Warning**, rate-limited to one line per 15 minutes so a
|
||||
converging backlog does not flood the log:
|
||||
|
||||
@@ -513,6 +520,33 @@ hardening and needs no change to the key design.
|
||||
| Site node | 2 cores | 4 GB | 20 GB | SQLite databases grow with S&F |
|
||||
| SQL Server | 4 cores | 16 GB | 100 GB | Shared across central cluster |
|
||||
|
||||
#### Sizing a site's data volume — `MaxStorageMb` is not the whole story
|
||||
|
||||
`ScadaBridge:SiteEventLog:MaxStorageMb` (default **1 GB**) bounds **`site_events` and its four
|
||||
indexes**, and nothing else. It is deliberately *not* a cap on the consolidated LocalDb file
|
||||
(register row 81 — decision record
|
||||
`docs/plans/2026-08-17-site-event-log-cap-semantics-decision.md`). Before that change it did
|
||||
measure the whole file, which on a replicated node made the cap unenforceable: the trim's own CDC
|
||||
oplog rows and tombstones grew the file as fast as the deletes shrank it, and the trim stopped
|
||||
2.5x over cap. Tying the event log's budget to the peer's liveness was the defect, not a feature
|
||||
worth preserving.
|
||||
|
||||
So budget the volume as the **sum** of:
|
||||
|
||||
- **`MaxStorageMb`** — the event log proper.
|
||||
- **LocalDb replication bookkeeping** — `__localdb_oplog` and `__localdb_row_version`, bounded
|
||||
separately by `LocalDb:Replication:MaxOplogRows` (default 1 000 000) and `TombstoneRetention`
|
||||
(default 7 days). This is transient: it drains as the peer acks and as tombstones expire, but it
|
||||
is not small during a burst. Measured worst case (register row 34): purging a full 1 GB event log
|
||||
left **707 MB** of bookkeeping in the file. Unreplicated sites carry none of this — they have no
|
||||
capture triggers at all.
|
||||
- **The other eight consolidated tables** plus the separate, unreplicated audit database at
|
||||
`AuditLog:SiteWriter:DatabasePath`.
|
||||
|
||||
The 20 GB figure above has ample headroom for all three at default settings. Raising
|
||||
`MaxStorageMb` on a replicated site should be paired with a look at the oplog settings, not done
|
||||
in isolation.
|
||||
|
||||
### Network Bandwidth
|
||||
|
||||
- Health reports: ~1 KB per site per 30 seconds = negligible
|
||||
|
||||
@@ -204,6 +204,18 @@ practice shared with replication bookkeeping that the event log does not own. A
|
||||
measure `site_events`' own pages (e.g. `dbstat`) rather than the file, which is a change to the
|
||||
cap's semantics and deserves its own decision. Reported for the register; no row added here.
|
||||
|
||||
> **Postscript (2026-08-17) — fixed; register row 81 closed.** The cap now measures `site_events`
|
||||
> and its four indexes only. One correction to the suggestion above: **`dbstat` turned out to be
|
||||
> unavailable.** It is a compile-time option and the bundled `SQLitePCLRaw.lib.e_sqlite3`
|
||||
> (SQLite 3.53.3) does not enable it — `PRAGMA compile_options` lists no `ENABLE_DBSTAT_VTAB`, and
|
||||
> neither `dbstat` nor `sqlite_dbpage` appears in `pragma_module_list`. The table's footprint is
|
||||
> therefore *modelled* from sampled row width rather than measured, at −0.8 %…+12.8 % accuracy
|
||||
> against the numbers in §3.4, biased high on purpose. The "size did not decrease" guard survives,
|
||||
> re-based onto the table-scoped metric so replication bookkeeping can no longer trip it. Decision
|
||||
> record, including the probe evidence and why a different SQLite build was rejected:
|
||||
> `docs/plans/2026-08-17-site-event-log-cap-semantics-decision.md`. §3.4's replicated row is left
|
||||
> as measured — it is the record of the defect, not of current behaviour.
|
||||
|
||||
## 6. Verification
|
||||
|
||||
- Fail-before: the new behavioural tests do not compile against pre-fix `EventLogPurgeService`
|
||||
|
||||
@@ -0,0 +1,264 @@
|
||||
# `site_events` storage cap — what the cap measures (register row 81)
|
||||
|
||||
**Date:** 2026-08-17
|
||||
**Row:** deferred-work register #81 — *"the storage cap fights its own oplog."*
|
||||
**Outcome:** cap semantics changed. `ScadaBridge:SiteEventLog:MaxStorageMb` now bounds
|
||||
**`site_events` and its four indexes**, not the whole consolidated LocalDb file. `dbstat` — the
|
||||
exact measurement the register proposed — turned out to be **unavailable in the SQLite this
|
||||
solution ships**, so the table's footprint is *modelled* from sampled row width. This memo records
|
||||
the probe that ruled `dbstat` out, the model, its measured accuracy, and the new meaning of the
|
||||
wipe-prevention guard.
|
||||
|
||||
---
|
||||
|
||||
## 1. The defect
|
||||
|
||||
Since LocalDb Phase 1/2 the site keeps ten tables in one SQLite file at `LocalDb:Path`, alongside
|
||||
LocalDb's own `__localdb_oplog`, `__localdb_row_version` and `__localdb_snapshot_state`
|
||||
bookkeeping. `EventLogPurgeService.GetDatabaseSizeBytes()` measured
|
||||
`page_count − freelist_count` over **that whole file** and compared it to `MaxStorageMb`.
|
||||
|
||||
On a replication-enabled node the cap trim therefore fought itself. Every 1000-row batch freed
|
||||
~390 KB of `site_events` pages *and* pushed ~1000 rows into `__localdb_oplog` plus ~1000
|
||||
row-version tombstones into the same file. The measured size did not fall, the existing
|
||||
"size did not decrease across an iteration" guard fired, and the trim stopped — with the cap
|
||||
unmet.
|
||||
|
||||
Measured 2026-08-16 during the row-34 work
|
||||
(`docs/plans/2026-08-15-site-events-purge-burst-measurement.md` §3.4/§5), 64 MB cap, 400 000 rows:
|
||||
|
||||
| Replicated? | Events deleted | Remaining | Size before → after | Reached the cap? |
|
||||
| --- | ---: | ---: | --- | --- |
|
||||
| no | 248 000 | 152 000 | 158.0 MB → 66.98 MB | **yes** (cap 67.11 MB) |
|
||||
| yes | 174 000 | 226 000 | 203.6 MB → 167.7 MB | **no — stopped 2.5x over cap** |
|
||||
|
||||
Two consequences, both real:
|
||||
|
||||
1. **A node whose peer is down cannot enforce its event-log cap at all.** Nothing drains the
|
||||
oplog, so the file never shrinks and the guard stops the trim every tick, forever.
|
||||
2. **The "1 GB event log" budget was in practice shared with replication bookkeeping the event
|
||||
log does not own.** Worse, the direction of the coupling is backwards: a *replication fault*
|
||||
(peer down, oplog piling up) would, under a naive "just keep deleting" fix, cause operational
|
||||
event history to be destroyed. The cap must not be a function of the peer's health.
|
||||
|
||||
The old guard was not the bug — it was doing its job, preventing the table being wiped in pursuit
|
||||
of an unreachable target. The bug is that the target was unreachable, because the metric was
|
||||
wrong.
|
||||
|
||||
## 2. `dbstat` is not available — the decision this memo turns on
|
||||
|
||||
The register proposed `SELECT sum(pgsize) FROM dbstat WHERE name='site_events'`. `dbstat` is a
|
||||
**compile-time option** (`SQLITE_ENABLE_DBSTAT_VTAB`), and it was verified before any
|
||||
implementation was written. It is **not enabled** in the native library this solution ships
|
||||
(`SQLitePCLRaw.lib.e_sqlite3`, the security-pinned 2.1.12 build referenced by both
|
||||
`ZB.MOM.WW.ScadaBridge.SiteEventLogging` and its test project).
|
||||
|
||||
Probed live against a real file, SQLite **3.53.3**:
|
||||
|
||||
```
|
||||
PRAGMA compile_options -> no ENABLE_DBSTAT_VTAB
|
||||
(ENABLE_FTS3/4/5, RTREE, GEOPOLY, SESSION, PREUPDATE_HOOK, … present)
|
||||
pragma_module_list -> 10 modules: fts3, fts3tokenize, fts4, fts4aux, fts5, fts5vocab,
|
||||
geopoly, pragma_module_list, rtree, rtree_i32
|
||||
SELECT sum(pgsize) FROM dbstat -> SQLite Error 1: 'no such table: dbstat'
|
||||
SELECT … FROM dbstat('main') WHERE name=… -> SQLite Error 1: 'no such table: dbstat'
|
||||
```
|
||||
|
||||
`sqlite_dbpage` — the other route to per-table page attribution — is absent for the same reason.
|
||||
The module list is non-empty, so this is a genuine absence and not an empty-result artefact.
|
||||
|
||||
**This is pinned by a committed test**, `SqliteCapabilityTests.DbStat_IsNotAvailable_…`, written to
|
||||
fail if `dbstat` ever appears. If it does, the right move is to delete the model in §3 and measure
|
||||
exactly; the test's own comment says so.
|
||||
|
||||
Alternatives considered and rejected:
|
||||
|
||||
- **Switch to a SQLite build that enables `dbstat`** (system `libsqlite3` via
|
||||
`SQLitePCLRaw.bundle_sqlite3`, or a vendored native build). Rejected: the provider is shared with
|
||||
`ZB.MOM.WW.LocalDb` and every other SQLite consumer in the product, so this trades a storage-cap
|
||||
accuracy problem for fleet-wide version skew across macOS dev boxes and the Debian-based runtime
|
||||
image — a disproportionate blast radius for a coarse safety valve.
|
||||
- **Give `site_events` its own file again.** Reverses LocalDb Phase 1/2 consolidation and removes
|
||||
the table from CDC replication. Out of the question.
|
||||
- **Maintain a running byte counter in the writer, or via triggers.** The writer is not the only
|
||||
thing that inserts into `site_events` (replication applies remote rows), a counter does not
|
||||
survive a restart, and adding our own triggers to a `RegisterReplicated` table is hazardous —
|
||||
LocalDb builds its capture triggers from the column list at registration time and extra triggers
|
||||
would also fire on replicated applies.
|
||||
- **Keep the whole-file metric and simply stop re-measuring mid-loop** (compute a delete plan once
|
||||
and execute it). This makes the trim terminate, but it makes the *event log* pay for bookkeeping
|
||||
bytes: an oplog inflated by a down peer would drive aggressive deletion of operational history.
|
||||
That is consequence 2 above, made worse rather than fixed.
|
||||
|
||||
## 3. What the cap measures now
|
||||
|
||||
`EventLogPurgeService.EstimateSiteEventsSize()` returns the live row count and modelled bytes per
|
||||
row. `MaxStorageMb` is compared against `rowCount × bytesPerRow`.
|
||||
|
||||
Row width is **sampled** — `SizeSampleRows` (500) rows from each end of the table, head and tail.
|
||||
The head is what the oldest-first trim will actually delete; the tail catches an event stream whose
|
||||
shape has drifted. Widths are taken with `octet_length()` (SQLite 3.43+), not `length()`, which
|
||||
counts characters and would under-size any multi-byte UTF-8 message; a second committed test pins
|
||||
that function's availability.
|
||||
|
||||
The table term is computed as **whole pages**, `ceil(rows / rowsPerPage) × pageSize`, rather than
|
||||
as a flat bytes-per-row rate:
|
||||
|
||||
```
|
||||
usablePerPage = (pageSize - 35) * 0.78 // max-local-payload rule × measured mean page fill
|
||||
avgCellBytes = avgPayloadBytes + 16 // payload-length varint, rowid varint, cell pointer
|
||||
rowsPerPage = max(1, floor(usablePerPage / avgCellBytes))
|
||||
tableBytes = ceil(rowCount / rowsPerPage) * pageSize
|
||||
indexBytes = rowCount * (avgIndexedBytes + 8 * 4) / 0.75
|
||||
estimate = min(tableBytes + indexBytes, wholeFileLogicalBytes)
|
||||
```
|
||||
|
||||
Whole pages rather than a rate because that term **dominates once rows approach half a page**: a
|
||||
2 KB row cannot be packed two-to-a-page, so each one wastes the remainder. A linear model
|
||||
underestimates that case by ~40 % (measured), and underestimating is the failure direction that
|
||||
matters for a cap.
|
||||
|
||||
**Index pages count against the cap — deliberately.** Measured, the four indexes are ~34 % of
|
||||
`site_events`' total footprint (397.2 vs 260.9 bytes/row with and without them, same 200 000 rows),
|
||||
and the 1 GB default was sized against the 393 bytes/row figure from row 34, which already included
|
||||
them. Excluding index pages would silently raise real disk consumption by about a third at an
|
||||
unchanged setting. The index term is driven by the *sampled* widths of the four indexed columns, so
|
||||
an event stream that leaves `instance_id` null is charged less than one that fills it.
|
||||
|
||||
**Clamped by the file.** The whole-file logical size is a true upper bound on any single table's
|
||||
pages, so the model is capped by it. This can only ever correct an over-estimate downward, and it
|
||||
cannot reintroduce the row-81 defect: on a replicated node the file is *larger* than
|
||||
`site_events`, never smaller, so the clamp does not bind there.
|
||||
|
||||
`GetDatabaseSizeBytes()` survives, renamed in meaning rather than in signature: it is the whole
|
||||
consolidated file's logical size, used for the clamp and as a diagnostic. It is no longer what the
|
||||
cap compares against.
|
||||
|
||||
### 3.1 Measured accuracy
|
||||
|
||||
Calibration seeded realistic rows into a real file and measured `page_count`/`freelist_count`/
|
||||
`page_size`. Two independent cross-checks against row 34's harness: the baseline workload came out
|
||||
at **397.2 bytes/row** against row 34's independently measured **393**, and the row-count-at-1 GB
|
||||
figure follows.
|
||||
|
||||
| Workload | rows | actual bytes/row | payload bytes/row |
|
||||
| --- | ---: | ---: | ---: |
|
||||
| 1-in-10 details @200 B (baseline) | 200 000 | 397.2 | 187.9 |
|
||||
| same, **no indexes** | 200 000 | 260.9 | 187.9 |
|
||||
| no details at all | 200 000 | 373.5 | 167.7 |
|
||||
| details on every row @200 B | 200 000 | 597.7 | 367.7 |
|
||||
| details on every row @2 KB | 50 000 | 4295.2 | 2215.7 |
|
||||
| 1-in-10 details @4 KB | 50 000 | 824.7 | 583.0 |
|
||||
| small table | 5 000 | 396.5 | 189.6 |
|
||||
|
||||
The shipped model, scored against a **differential** measurement of the table's real page cost
|
||||
(file measured with ballast only, events added, growth attributed to `site_events`) — this is
|
||||
`EstimateSiteEventsSize_TracksTheRealFootprint`, run on every build:
|
||||
|
||||
| Workload | estimated | actual | error |
|
||||
| --- | ---: | ---: | ---: |
|
||||
| narrow rows, no details | 3 155 626 | 2 912 256 | **+8.4 %** |
|
||||
| typical rows, 200 B details | 3 992 096 | 3 538 944 | **+12.8 %** |
|
||||
| wide rows, 2 KB details | 12 748 048 | 12 845 056 | **−0.8 %** |
|
||||
|
||||
The test asserts the band **−5 % … +25 %**. The bias is intentionally to the high side: a cap that
|
||||
fires slightly early forfeits a slice of the budget, whereas one that reads low lets the log
|
||||
overrun the disk allowance the cap exists to defend. Against the previous behaviour's **+250 %**
|
||||
error on a replicated node this is not a close call.
|
||||
|
||||
**Known limitation.** The model samples 1000 rows out of a table that may hold millions. An event
|
||||
stream whose row width changes sharply *in the middle* of the history — neither at the head nor the
|
||||
tail — is mis-sized until the trim reaches it. Self-correcting in practice, since the cap is
|
||||
re-evaluated every tick and the head sample advances as the oldest rows go.
|
||||
|
||||
## 4. The wipe-prevention guard, restated
|
||||
|
||||
The register asked whether the "size did not decrease" guard should survive. It should, and it
|
||||
does, but its meaning has changed and it has gained a sibling. There are now **two** backstops:
|
||||
|
||||
1. **Unusable row width → refuse to trim.** Checked *before* the cap comparison, because a zero
|
||||
bytes-per-row makes "is `site_events` over cap" unanswerable rather than false. Logs a Warning
|
||||
naming the row count and deletes nothing. The old code had no analogue: its answer to an
|
||||
unanswerable cap was to keep deleting until the file shrank, which is exactly how a cap that can
|
||||
never be met empties a table.
|
||||
|
||||
2. **Estimated size did not decrease across a batch → stop.** The old guard, re-based onto the
|
||||
table-scoped metric. Crucially it **can no longer be tripped by replication bookkeeping** — the
|
||||
metric excludes it — so when it fires it now means precisely one thing: the table is not
|
||||
shrinking even though deletes are landing, i.e. **new events are arriving at least as fast as
|
||||
the trim removes them.** That is a real operational condition and the Warning says so.
|
||||
|
||||
For this to be observable rather than tautological, the loop re-reads the **row count** from the
|
||||
database after each batch instead of decrementing a local counter, so concurrent recorder writes
|
||||
are visible. Bytes-per-row, by contrast, is sampled **once per tick and held fixed** — re-sampling
|
||||
every batch would make the estimate jitter as the trim consumes the head of the table, and a
|
||||
sample that happened to widen could stall the loop through this very guard. That is the same
|
||||
class of self-defeating measurement as row 81 itself, and it is deliberately avoided.
|
||||
|
||||
Both backstops are covered by tests that pin a degenerate estimate through `internal virtual`
|
||||
seams (`EstimateSiteEventsSize`, `GetSiteEventsRowCount`).
|
||||
|
||||
## 5. Behaviour on an UNREPLICATED node
|
||||
|
||||
Materially unchanged, as intended — but not bit-identical, and the delta is worth stating.
|
||||
|
||||
On a node with no CDC triggers there is no oplog and no tombstone table, so whole-file logical size
|
||||
was already ≈ `site_events` + the nine other (largely small) tables. The cap therefore used to fire
|
||||
at roughly the right place, and the row-34 measurement confirms it: the unreplicated trim reached
|
||||
66.98 MB against a 67.11 MB cap.
|
||||
|
||||
What changes:
|
||||
|
||||
- **Slightly more headroom.** The other nine tables no longer count against the event log's budget.
|
||||
On a rig-shaped site those tables are small relative to a 1 GB event log, so the shift is a few
|
||||
percent at most.
|
||||
- **Model error replaces measurement error.** Where the old number was exact-but-wrong-scoped, the
|
||||
new one is right-scoped but carries the +8 %/+13 % conservative bias of §3.1. Net effect on an
|
||||
unreplicated node: the cap fires marginally earlier than before at the same setting.
|
||||
|
||||
Neither warrants a config change. Sites that were happy with `MaxStorageMb: 1024` remain so.
|
||||
|
||||
## 6. Explicitly out of scope: LocalDb's own bookkeeping
|
||||
|
||||
This change makes the cap ignore `__localdb_oplog`, `__localdb_row_version` and
|
||||
`__localdb_snapshot_state`. It does **not** attempt to manage them, and should not.
|
||||
|
||||
Their size is governed by LocalDb's own machinery — `LocalDb:Replication:MaxOplogRows` (default
|
||||
1 000 000) and `TombstoneRetention` (default 7 days) — and row 34's pacing
|
||||
(`MaxPurgedRowsPerTick`) is what keeps a purge burst from blowing through the oplog cap and forcing
|
||||
a snapshot resync. Those are the correct controls and they already exist. A second, competing
|
||||
bound applied from the event log would only obscure which subsystem is responsible for a full disk.
|
||||
|
||||
The honest statement of the budget is therefore: **`MaxStorageMb` bounds `site_events` and its
|
||||
indexes; LocalDb replication bookkeeping is separate overhead on the same volume, bounded
|
||||
separately.** Sizing a site's data volume means adding the two, and the topology guide now says so.
|
||||
Row 34's measurement gives the scale — a 2.73 M-row purge left 707 MB of bookkeeping in the file
|
||||
after deleting a 1 GB event log — and that overhead is transient, draining as the peer acks and as
|
||||
tombstones expire.
|
||||
|
||||
## 7. Verification
|
||||
|
||||
- **`dbstat` unavailability probed before any implementation was written**, on a real file, with
|
||||
the result recorded in §2 and pinned by `SqliteCapabilityTests`.
|
||||
- **Fail-before, row 81.** With `EstimateSiteEventsSize` reverted to pre-row-81 semantics (whole
|
||||
file attributed to `site_events`) and nothing else changed, **5 of the new tests fail** and all
|
||||
99 others pass. `PurgeByStorageCap_WithLargeBallastInTheSameFile_StillEnforces…` reproduces the
|
||||
original signature exactly: `remaining=2000 estimate=21725184 cap=11534336` — 4000 of 6000 rows
|
||||
destroyed and the cap **still unmet**.
|
||||
- **Fail-before, row 84.** With per-phase emission restored,
|
||||
`RunPurge_WhenBothPhasesDeleteRows_LogsTheBacklogNoteOnceForTheWholeTick` fails with
|
||||
`Assert.Single() Failure: The collection contained 2 matching items`.
|
||||
- `dotnet build ZB.MOM.WW.ScadaBridge.slnx` — clean, 0 warnings.
|
||||
- `ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests`: **104 passed / 0 failed** (93 before this change).
|
||||
- The docker rig was not touched.
|
||||
|
||||
## 8. Register row 84, folded in
|
||||
|
||||
`LogReplicationBacklogNote` was emitted by each purge phase, so a tick in which the retention purge
|
||||
*and* the cap trim both deleted rows produced two identical Information notes. It is now emitted
|
||||
once from `RunPurge`, driven by the tick's shared `PurgeBudget`.
|
||||
|
||||
The budget is the natural source: it already holds exactly "rows this tick deleted", which is also
|
||||
the more useful number to put next to an oplog spike — the tick's whole contribution rather than
|
||||
one phase's share. `PurgeByRetention` keeps its own "Purged N events older than D days" line; only
|
||||
the replication note moved.
|
||||
@@ -93,7 +93,50 @@ Each event entry contains:
|
||||
rig's site-b/site-c), the log stays node-local and a failover does start it fresh — the
|
||||
documented trade of not configuring a peer, unchanged by this policy.
|
||||
- **Retention**: 30 days. A **daily background job** runs on the active node and deletes all events older than 30 days. Hard delete — no archival. Deletes are sliced into bounded 1000-row batches per DELETE statement rather than one unbounded statement, so a large expired backlog does not hold the write lock (and every concurrent recorder flush) for the duration of the purge.
|
||||
- **Storage cap**: A configurable maximum database size (default: 1 GB) is enforced. If the storage cap is reached before the 30-day retention window, the oldest events are purged first. This prevents disk exhaustion from alarm storms, script failure loops, or connection flapping.
|
||||
- **Storage cap**: A configurable maximum size (default: 1 GB) is enforced. If the cap is reached
|
||||
before the 30-day retention window, the oldest events are purged first. This prevents disk
|
||||
exhaustion from alarm storms, script failure loops, or connection flapping.
|
||||
- **The cap measures `site_events` and its four indexes — NOT the database file** (register row 81;
|
||||
decision record `docs/plans/2026-08-17-site-event-log-cap-semantics-decision.md`). This matters
|
||||
because the event log shares one SQLite file with nine other tables plus LocalDb's
|
||||
`__localdb_oplog` / `__localdb_row_version` / `__localdb_snapshot_state` bookkeeping. The cap used
|
||||
to compare `MaxStorageMb` against `page_count − freelist_count` over that whole file, which on a
|
||||
replication-enabled node made the trim **unwinnable**: each 1000-row delete freed ~390 KB of event
|
||||
pages while adding ~1000 oplog rows and ~1000 tombstones to the same file, the measured size did
|
||||
not fall, and the progress guard stopped the trim — measured 2.5x over a 64 MB cap with 226 k rows
|
||||
still present. A node whose peer was down could not enforce its cap at all. Two consequences of
|
||||
the new scoping are worth internalising:
|
||||
- **The cap is now independent of replication health.** It was previously a function of the
|
||||
*peer's* liveness, and in the wrong direction — an inflated oplog is a replication fault, and
|
||||
letting it drive event deletion would destroy operational history in response to one.
|
||||
- **`MaxStorageMb` no longer bounds the data volume on its own.** LocalDb bookkeeping is separate
|
||||
overhead on the same disk, bounded separately by `LocalDb:Replication:MaxOplogRows` and
|
||||
`TombstoneRetention`; sizing a site's volume means adding the two. Managing that bookkeeping is
|
||||
deliberately **not** this component's job — see the decision record §6.
|
||||
- **The measurement is modelled, not exact.** SQLite can report per-table page usage precisely via
|
||||
the `dbstat` virtual table, and that was the intended fix — but `dbstat` is a compile-time option
|
||||
and the bundled `SQLitePCLRaw.lib.e_sqlite3` native library (SQLite 3.53.3) does **not** enable it;
|
||||
neither `dbstat` nor `sqlite_dbpage` exists, verified by probe and pinned by
|
||||
`SqliteCapabilityTests`. The table's footprint is therefore estimated from sampled row width
|
||||
(500 rows from each end of the table, measured with `octet_length` so multi-byte UTF-8 is not
|
||||
under-counted) fed into a page-layout model; index pages count against the cap, deliberately,
|
||||
because they are ~34 % of the table's real footprint and the 1 GB default was sized against a
|
||||
bytes/row figure that included them. Measured error is **−0.8 % to +12.8 %** across representative
|
||||
workloads, biased high on purpose — a cap that fires slightly early forfeits budget, whereas one
|
||||
that reads low fails at the only job it has. The estimate is additionally clamped by the
|
||||
whole-file logical size, which is a true upper bound on any one table.
|
||||
- **Two wipe-prevention backstops** guard the cap trim, replacing the single "on-disk size did not
|
||||
decrease" check. (1) If row width cannot be sampled, the trim **refuses to run** and warns —
|
||||
checked before the cap comparison, because a zero row width makes "are we over cap" unanswerable
|
||||
rather than false, and the old loop's answer to an unanswerable cap was to keep deleting. (2) If
|
||||
the *estimated table size* fails to fall across a batch, the trim stops and warns. Guard (2) is the
|
||||
old check re-based onto the table-scoped metric, and its meaning is now unambiguous: replication
|
||||
bookkeeping can no longer trip it, so it fires only when new events are arriving at least as fast
|
||||
as the trim removes them. For that to be observable, the loop re-reads the row count from the
|
||||
database after every batch, while bytes-per-row is sampled **once per tick and held fixed** —
|
||||
re-sampling per batch would make the estimate jitter as the trim eats the head of the table and
|
||||
could stall the loop through guard (2), which is the same self-defeating-measurement failure as
|
||||
row 81 itself.
|
||||
- **Per-tick delete ceiling** (`MaxPurgedRowsPerTick`, default **250 000**; `0` = unlimited): one
|
||||
purge tick deletes at most this many rows, **shared** by the retention purge and the storage-cap
|
||||
trim — two independently-capped phases would let a tick emit twice the intended burst. A tick that
|
||||
@@ -128,7 +171,10 @@ Each event entry contains:
|
||||
replication fault and not the purge. The purge logs an Information line naming the row count and
|
||||
the expected transient backlog whenever it deletes rows on a replication-enabled node, so an
|
||||
operator can correlate a spike with the purge that caused it — a spike with no such line nearby is
|
||||
not the purge. Node-local wiring: the Host supplies the replication predicate
|
||||
not the purge. That line is emitted **once per tick**, carrying the retention purge's and the cap
|
||||
trim's deletes combined (register row 84 — it used to fire once per phase, so a tick in which both
|
||||
deleted produced two identical notes; the tick total is also the more useful number to put beside
|
||||
an oplog spike). Node-local wiring: the Host supplies the replication predicate
|
||||
(`SiteLocalDbSetup.ReplicationIsConfigured` — the `PeerAddress`-OR-`ApiKey` rule) as
|
||||
`SiteEventLogReplicationCheck`; this component never reads LocalDb configuration itself, and with
|
||||
no predicate registered the note is suppressed. Operator detail —
|
||||
|
||||
Reference in New Issue
Block a user