perf(localdb): dereg API, O(1) backlog depth, byte-budget batching, prepare-once apply (0.2.0)

WP3.3 of the ScadaBridge architectural-review remediation. Six changes, all
wire-compatible with 0.1.x peers — no proto change, no lib_schema_version bump. The
on-disk bookkeeping schema goes to v2 and is upgraded in place when a file is opened.

DeregisterReplicated(table). There was no way to stop capturing once a file had been
registered, so a node that does not replicate kept paying three triggers inside every
write transaction, appending to an oplog nothing would ever drain. Drops the triggers and
prunes the table's oplog + row-version rows in one transaction; idempotent, and
deliberately usable on a table this process never registered, since stale triggers left by
an EARLIER build are the primary case. Deregistration must be symmetric across a pair, and
already is enforced: the handshake compares registered-table digests fail-closed, so a node
that drops a table its peer still replicates stops syncing loudly rather than diverging.

RegisterReplicated(table, baselineExistingRows: true). Closes the late-opt-in hole: the
snapshot streamer pages from __localdb_row_version, which only triggers populate, so a node
that started replicating LATE never sent its pre-existing rows — silently, forever. Seeding
alone would not have fixed it, because a node whose oplog starts at seq 1 presents no gap at
the handshake and is told it needs no snapshot; the seed therefore also flags needs_snapshot,
and the existing resync machinery carries the rows. The floor is HLC 0 stamped with this
node's id: every genuine HLC is a UTC millisecond shifted left 16 bits and so is strictly
greater, which makes a baselined row lose every comparison against a real remote write and
win only where the peer has no version of that key at all. Two nodes baselining the same key
both hold 0 and resolve by the node-id tie-break — convergent, and nothing better is
available for two rows of unordered antiquity. Opt-in, so the documented CDC contract is
unchanged unless asked. Pruning the ledger on deregister is safe precisely because
re-registration can re-baseline; the two APIs are a pair.

O(1) backlog depth. GetOplogDepth ran COUNT(*) over the entire unacked backlog, once a
minute from maintenance AND on every metrics scrape — a full scan of a million rows exactly
when a node is already behind. Now MAX(seq) - MAX(last_acked_seq, MIN(seq) - 1), floored at
0: three b-tree seeks, measured at 58 VM steps against 100,020 for the count on a 50k
backlog. Exact, because seq is AUTOINCREMENT and every prune path removes a prefix (ack
pruning by watermark, the row cap by count, the age cap by an HLC cutoff that rises with
seq). The MIN clamp is what keeps it exact after a cap prune has moved the horizon above a
stalled watermark — without it the caps would re-flag on every tick forever.

Byte-budget batching. MaxBatchSize was the only cap on an outbound message, which made it a
proxy for a byte budget that consumers had to tune against their widest row: ScadaBridge
pinned it to 16 because 70 KB config_json times the 500 default is ~35 MB against a 4 MB
gRPC limit. MaxBatchBytes (default 2 MB) now bounds delta and snapshot messages by summed
serialized size, with the row count demoted to a secondary cap. A row larger than the budget
is sent alone rather than stalling the stream on it. Since more, smaller messages would
otherwise mean more apply transactions and more acks, inbound delta batches and acks ALREADY
queued are coalesced — one transaction and one ack per burst, bounded by the sender's own
caps, and a no-op under a trickle.

LwwApplier prepares each statement once. It went through the generic transaction façade,
which builds a fresh SqliteCommand per call: three SQL parses per applied row, on the path
that drains a reconnect backlog. Statements are now held for the batch and rebound.

Dropped __localdb_oplog_hlc. It indexed (table_name, pk_json, hlc), which no query has ever
used — every oplog access path is by seq or a full hlc-predicate delete — while costing a
b-tree insert inside every captured consumer write.

184/184 tests pass (was 149). New coverage: schema upgrade drops the index and preserves
identity/watermarks/data, deregistration idempotency and trigger absence and cross-process
cleanup, baseline seeding at the floor with LWW safety proved through the real applier,
depth asserted equal to the counting query across every prune scenario plus a constant-time
assertion over a 200k backlog, byte-budget chunking including the one-giant-row edge on both
the delta and snapshot paths, prepare-once statement counting, and coalescing that must not
swallow the message that ends its run.

Version 0.2.0.

ZB.MOM.WW.LocalDb/README.md

@@ -10,16 +10,38 @@ Change capture uses SQLite triggers writing a single HLC-stamped oplog inside th
 consumer's own write transaction (crash-consistent, non-bypassable). Convergence is
 last-writer-wins over a hybrid logical clock. All timestamps are UTC.
 
-## Packages (all `0.1.3`)
+## Packages (all `0.2.0`)
 
 | Package | Contents |
 |---|---|
-| `ZB.MOM.WW.LocalDb` | Embedded DB host: `ILocalDb` SQL surface (`ExecuteAsync` / `QueryAsync` / transactions), `RegisterReplicated(table)` (PK validation + trigger install + oplog schema), `HybridLogicalClock`, `AddZbLocalDb()` DI. |
+| `ZB.MOM.WW.LocalDb` | Embedded DB host: `ILocalDb` SQL surface (`ExecuteAsync` / `QueryAsync` / transactions), `RegisterReplicated(table[, baselineExistingRows])` (PK validation + trigger install + oplog schema), `DeregisterReplicated(table)` (trigger removal + oplog/ledger prune), `HybridLogicalClock`, `AddZbLocalDb()` DI. |
 | `ZB.MOM.WW.LocalDb.Contracts` | `localdb_sync.v1` proto + generated gRPC types (packable; a future peer could be non-.NET). |
 | `ZB.MOM.WW.LocalDb.Replication` | gRPC sync engine: passive `LocalDbSyncService`, initiating `SyncBackgroundService`, `MaintenanceBackgroundService`, LWW apply, watermark acks + pruning, snapshot resync, `ISyncStatus`, `LocalDbMetrics`, `AddZbLocalDbReplication()` / `MapZbLocalDbSync()`. |
 
 > Status: **published to the Gitea feed; adopted by ScadaBridge (Phases 1+2) and OtOpcUa (Phase 1).**
 >
+> - `0.2.0` (2026-08-14) — performance + lifecycle work from the ScadaBridge architectural review.
+>   All of it is wire-compatible with `0.1.x` peers (no proto change, no `lib_schema_version` bump);
+>   the on-disk bookkeeping schema goes to **v2** and is upgraded in place on open.
+>   - **`DeregisterReplicated(table)`** — drops a table's capture triggers and prunes its oplog and
+>     row-version rows. A node that does not replicate should not be paying for capture on every
+>     write, and until now there was no way to stop once a file had been registered. Idempotent, and
+>     usable on triggers left behind by an earlier build.
+>   - **`RegisterReplicated(table, baselineExistingRows: true)`** — closes the late-opt-in hole: rows
+>     that predate registration were invisible to the peer forever, because the snapshot streamer
+>     pages from the row-version ledger and CDC never wrote them there. Opting in seeds the ledger at
+>     an LWW-safe floor (HLC `0`, this node's id) and flags a snapshot resync so they actually
+>     travel. Default is still off — the documented CDC contract is unchanged unless you ask.
+>   - **O(1) backlog depth** — `MAX(seq) - MAX(last_acked_seq, MIN(seq) - 1)` replaces a
+>     `COUNT(*)` over the whole unacked backlog that ran once a minute AND on every metrics scrape
+>     (measured 58 VM steps vs 100,020 on a 50k backlog).
+>   - **Byte-budget batching** — `MaxBatchBytes` (default 2 MB) bounds delta and snapshot messages by
+>     summed serialized size, with `MaxBatchSize` demoted to a secondary row cap. Consumers no longer
+>     have to tune the row count against their widest row to stay under the gRPC message limit.
+>     Inbound delta batches and acks already queued are coalesced into one transaction / one ack.
+>   - **`LwwApplier` prepares once per statement** instead of three SQL parses per applied row.
+>   - **Dropped `__localdb_oplog_hlc`** — an index no query ever used, costing a b-tree insert inside
+>     every captured consumer write. Existing files lose it on the next open.
 > - `0.1.3` (2026-07-21) — fixes the other half of the rebuilt-peer story: a rebuilt peer's OWN
 >   writes were silently dropped. `last_applied_remote_seq` is a watermark in the *peer's* seq
 >   space, and a rebuilt peer numbers from 1 again — so the healthy node advertised the old peer's
@@ -107,7 +129,8 @@ node still pushes its own deltas back. Only one side needs `PeerAddress`.
 | `PeerAddress` | `""` | gRPC address to dial. Empty ⇒ passive (initiator idles). |
 | `ApiKey` | `null` | Bearer token the initiator presents on its channel; `null` ⇒ no key header. |
 | `FlushInterval` | `250 ms` | How often the change pump flushes pending oplog deltas. |
-| `MaxBatchSize` | `500` | Max oplog entries per outbound batch (also the snapshot page size). |
+| `MaxBatchBytes` | `2 MB` | **Primary** batch cap: summed serialized bytes per outbound delta/snapshot message. Sized under the 4 MB gRPC default; a single row above the budget is still sent, alone. |
+| `MaxBatchSize` | `500` | **Secondary** row-count cap per outbound batch (also the snapshot DB page size). |
 | `MaxOplogRows` | `1_000_000` | Backlog ceiling; exceeding it flags a snapshot resync and prunes oldest rows. |
 | `MaxOplogAge` | `7 days` | Age ceiling; an unpruned entry older than this flags a snapshot resync. |
 | `TombstoneRetention` | `7 days` | How long deleted-row tombstones are kept before pruning. |
@@ -161,8 +184,13 @@ All options are validated at startup (`ZB.MOM.WW.Configuration` validators).
   still hold a BLOB value at runtime, which then surfaces as a write-time error, not a
   registration error.
 - **CDC semantics** — rows that existed **before** `RegisterReplicated` are not captured
-  and are not snapshotted; only changes after registration are replicated. Re-baselining
-  pre-existing rows requires re-writing them.
+  and are not snapshotted; only changes after registration are replicated. **Opt out with
+  `RegisterReplicated(table, baselineExistingRows: true)`**, which seeds the row-version
+  ledger for existing rows at the baseline floor (HLC `0`) and flags a snapshot resync so
+  they reach the peer. The floor is below every genuine stamp, so a baselined row loses
+  every comparison against a real remote write and wins only where the peer has no version
+  of that key at all. Two nodes that baseline the same key both hold HLC `0` and resolve by
+  the node-id tie-break — convergent, but arbitrary as to which content survives.
 - **REAL ±Infinity** is carried in SQLite's JSON dialect on the wire (fine for the
   SQLite↔SQLite peers this library targets).
 - `AddZbLocalDb` / `AddZbLocalDbReplication` are **first-registration-wins** (idempotent).
@@ -198,6 +226,16 @@ provider is wired or the poll threw, so a DB failure never reads as "0 backlog")
 
 - When the oplog row/age caps are exceeded, maintenance prunes to the ceiling and flags
   `needs_snapshot` (disk safety over delta continuity); the peer then snapshot-resyncs.
+- **Turning a table off** — `DeregisterReplicated(table)` drops its capture triggers and prunes
+  its oplog + ledger rows. It must be applied **symmetrically**: the handshake compares
+  registered-table digests fail-closed, so a node that drops a table its peer still
+  replicates stops syncing with a schema-mismatch error. Deregister on both nodes, or on
+  neither. Re-enabling later should pass `baselineExistingRows: true`, or the rows that
+  accumulated while capture was off stay invisible to the peer.
+- **On-disk schema upgrades** are applied in place when the file is opened, in one
+  transaction, and every step is safe to re-run. `schema_version` in `__localdb_meta` is
+  local-file state only — it is NOT the wire `lib_schema_version`, so a pair mid-upgrade
+  keeps syncing.
 - The **`__localdb_dead_letter`** table holds poison rows (bad JSON, null-row non-tombstone
   entries) and over-drift rejects (when `FailClosedOnDrift`). Inspect it to diagnose
   rejects.
@@ -208,13 +246,14 @@ provider is wired or the poll threw, so a DB failure never reads as "0 backlog")
 
 ## Testing
 
-144 tests, all offline / no live dependencies — unit (HLC ordering + restart
+184 tests, all offline / no live dependencies — unit (HLC ordering + restart
 persistence, trigger SQL generation, LWW decision table, watermark/pruning, digest
-computation), integration over two real SQLite DBs on an in-memory (and real) gRPC pipe
-(bidirectional convergence, delete/update races, offline-accumulate → reconnect,
-prune → snapshot resync), plus seeded randomized property tests (random interleaved op
-sequences → both DBs identical after quiesce). Family conventions apply
-(`TreatWarningsAsErrors`).
+computation, schema-version upgrade, deregistration idempotency, baseline-floor LWW
+safety, O(1) depth vs a seeded backlog, byte-budget chunking), integration over two real
+SQLite DBs on an in-memory (and real) gRPC pipe (bidirectional convergence, delete/update
+races, offline-accumulate → reconnect, prune → snapshot resync), plus seeded randomized
+property tests (random interleaved op sequences → both DBs identical after quiesce).
+Family conventions apply (`TreatWarningsAsErrors`).
 
 Build / test from this directory: