fix(siteeventlog): scope the storage cap to site_events' own pages (register rows 81, 84)

Row 81 — the cap could not be enforced on a replicated node. MaxStorageMb was
compared against page_count-freelist_count over the WHOLE consolidated LocalDb
file, which since Phase 1/2 holds ten tables plus __localdb_oplog /
__localdb_row_version / __localdb_snapshot_state. Every 1000-row trim batch freed
~390 KB of site_events pages and added ~1000 oplog rows + ~1000 tombstones to the
same file, so the measured size did not fall, the "size did not decrease" guard
fired, and the trim stopped 2.5x over a 64 MB cap with 226 k rows still present.
A node whose peer was down could not enforce its cap at all, and the event log's
budget was in practice shared with bookkeeping it does not own.

The cap now measures site_events AND ITS FOUR INDEXES only, so replication
bookkeeping can neither consume nor block the budget — and the cap stops being a
function of the peer's health, which was the wrong direction of coupling: an
inflated oplog is a replication fault, and letting it drive event deletion would
destroy operational history in response to one.

dbstat — the exact per-table measurement the register proposed — is NOT usable
here, verified before implementing. It is a compile-time option and the bundled
SQLitePCLRaw.lib.e_sqlite3 (SQLite 3.53.3) does not enable it: PRAGMA
compile_options lists no ENABLE_DBSTAT_VTAB, pragma_module_list returns ten
modules with neither dbstat nor sqlite_dbpage, and selecting from dbstat fails
with "no such table". Switching SQLite builds was rejected — the provider is
shared with ZB.MOM.WW.LocalDb and every other consumer, so it trades cap accuracy
for fleet-wide version skew. The footprint is therefore modelled from sampled row
width (500 rows each end, octet_length so multi-byte UTF-8 is not under-counted)
through a page-layout model; whole pages rather than a flat rate, because page
fill collapses once rows approach half a page and a linear model underestimates
that by ~40%. Clamped by whole-file logical size, a true upper bound that can only
correct an over-estimate and never binds on a replicated node.

Index pages count against the cap deliberately: measured at ~34% of the table's
footprint, and the 1 GB default was sized against a 393 B/row figure that already
included them.

Guard semantics restated — two backstops replace the single on-disk check:
(1) row width unsampleable => refuse to trim (checked before the cap comparison,
since a zero width makes "over cap" unanswerable rather than false; the old loop's
answer to an unanswerable cap was to keep deleting); (2) estimated size not
falling across a batch => stop. (2) is the old guard re-based onto the
table-scoped metric, so it can no longer be tripped by replication bookkeeping and
now means exactly one thing: events are arriving as fast as the trim removes them.
For that to be observable the loop re-reads the row count each batch, while
bytes-per-row is sampled once per tick and held fixed — re-sampling per batch
would jitter the estimate as the trim eats the head of the table and could stall
the loop through guard (2), the same self-defeating measurement as row 81 itself.

Row 84 — LogReplicationBacklogNote moved from per-phase to once per tick, driven
by the shared PurgeBudget. A tick where retention AND the cap trim both deleted
used to emit two identical Information notes; the tick total is also the more
useful number beside an oplog spike.

Measured model error: -0.8% / +8.4% / +12.8% across representative workloads,
biased high on purpose (an early cap forfeits budget; a low one fails at its only
job). Independently cross-checks row 34's harness at 397.2 vs 393 B/row.

Fail-before: reverting the estimator to whole-file semantics fails 5 of the new
tests and reproduces the original signature exactly (remaining=2000
estimate=21725184 cap=11534336 — rows destroyed, cap still unmet); restoring
per-phase notes fails the row-84 test with "collection contained 2 matching items".

No config keys added or renamed, so no rig/deploy changes. Decision record:
docs/plans/2026-08-17-site-event-log-cap-semantics-decision.md

Tests: SiteEventLogging 104/104 (was 93), Host 520/520, solution build clean.
This commit is contained in:
Joseph Doherty
2026-08-17 03:58:41 -04:00
parent 07220bc655
commit ddb3c32e06
10 changed files with 1171 additions and 46 deletions
+34
View File
@@ -273,6 +273,13 @@ history had expired.
An unreplicated node logs only the first line. If a backlog spike has no such line near it in the
active node's log, the purge is *not* the explanation and the spike is worth investigating.
The second line is emitted **once per purge tick**, and its count covers the retention purge and
the storage-cap trim together. If you are comparing it against the `Purged N events older than…`
line above it, expect the two numbers to differ whenever the cap trim also ran — the note reports
the tick's whole contribution to the oplog, which is what a backlog spike is made of. (Through
2026-08-16 the note was emitted per phase, so a tick that ran both produced two near-identical
lines; register row 84.)
**A tick that hit the ceiling adds a Warning**, rate-limited to one line per 15 minutes so a
converging backlog does not flood the log:
@@ -513,6 +520,33 @@ hardening and needs no change to the key design.
| Site node | 2 cores | 4 GB | 20 GB | SQLite databases grow with S&F |
| SQL Server | 4 cores | 16 GB | 100 GB | Shared across central cluster |
#### Sizing a site's data volume — `MaxStorageMb` is not the whole story
`ScadaBridge:SiteEventLog:MaxStorageMb` (default **1 GB**) bounds **`site_events` and its four
indexes**, and nothing else. It is deliberately *not* a cap on the consolidated LocalDb file
(register row 81 — decision record
`docs/plans/2026-08-17-site-event-log-cap-semantics-decision.md`). Before that change it did
measure the whole file, which on a replicated node made the cap unenforceable: the trim's own CDC
oplog rows and tombstones grew the file as fast as the deletes shrank it, and the trim stopped
2.5x over cap. Tying the event log's budget to the peer's liveness was the defect, not a feature
worth preserving.
So budget the volume as the **sum** of:
- **`MaxStorageMb`** — the event log proper.
- **LocalDb replication bookkeeping** — `__localdb_oplog` and `__localdb_row_version`, bounded
separately by `LocalDb:Replication:MaxOplogRows` (default 1 000 000) and `TombstoneRetention`
(default 7 days). This is transient: it drains as the peer acks and as tombstones expire, but it
is not small during a burst. Measured worst case (register row 34): purging a full 1 GB event log
left **707 MB** of bookkeeping in the file. Unreplicated sites carry none of this — they have no
capture triggers at all.
- **The other eight consolidated tables** plus the separate, unreplicated audit database at
`AuditLog:SiteWriter:DatabasePath`.
The 20 GB figure above has ample headroom for all three at default settings. Raising
`MaxStorageMb` on a replicated site should be paired with a look at the oplog settings, not done
in isolation.
### Network Bandwidth
- Health reports: ~1 KB per site per 30 seconds = negligible