feat(localdb): per-table needs_snapshot (0.3.0, on-disk schema v3)
Baselining one table used to set a single per-DATABASE needs_snapshot bit, and a snapshot streams every registered table in full — so opting one table into replication late re-sent all of them, in both directions at once on a pair enabling replication late. On a database holding a high-volume table (an event log at its retention cap) that is the largest transfer this library performs. 0.2.1 documented the cost and deferred the fix as needing an on-disk change. Schema v3 adds __localdb_snapshot_state (table_name PK, needs_snapshot), created by the shared DDL with IF NOT EXISTS so an older file gains it on open; the v2->v3 transition is a version stamp only. RegisterReplicated(..., baselineExistingRows: true) now flags that table alone. The database-wide flag survives unchanged for the paths that genuinely reach every table — a cap prune or an impossible peer ack loses delta continuity for the oplog as a whole. DeregisterReplicated drops a table's pending flag with its ledger rows, since nothing would ever stream (and so clear) it afterwards. Wire compatibility is additive and negotiated by CAPABILITY, not version: lib_schema_version stays 1 because ValidateHandshake compares it fail-closed for equality, so bumping it would stop 0.1.x/0.2.x pairs syncing outright. Handshake gains supports_partial_snapshot (field 5) and SnapshotBegin gains a tables scope (field 2) where EMPTY means all tables — precisely what a pre-0.3.0 sender emits and what a pre-0.3.0 receiver assumes. A peer that does not advertise the capability is served a full snapshot: a scoped one would make it advance last_applied_remote_seq to the snapshot's as-of and silently discard every delta below it for the tables left out. For the same reason a partial snapshot moves no watermark on either side. The receiver keeps last_applied_remote_seq (carrying only the HLC across, which is MAX-guarded), and the sender keeps _sentThruSeq, so the deltas the snapshot overlaps are still pumped and merged by LWW. Crash safety: snapshot debt now clears on the receiver's new SnapshotAck rather than on enqueue, so an ungraceful exit part-way through re-streams the same scope on reconnect. The ack rides the unbounded CONTROL lane like DeltaAck, so it composes with 0.2.1's bounded 64-message inbox and its out-of-handshake snapshot. Against a pre-0.3.0 peer the 0.2.x clear-on-enqueue behaviour is kept — there is no ack to wait for — with the handshake's horizon gap check as the net. Tests 199/199 (was 188): single-table baseline streams only that table, mixed- version interop (peer without the capability gets a full snapshot), debt standing until SnapshotAck, partial snapshot not advancing the receiver's watermark, no SnapshotAck emitted to an old peer, end-to-end over two real databases, plus v2->v3 upgrade and per-table flag isolation. README, ILocalDb remarks (which named this as the follow-up) and the umbrella CLAUDE.md index updated.
This commit is contained in:
Joseph Doherty
+56
-17
@@ -10,7 +10,7 @@ Change capture uses SQLite triggers writing a single HLC-stamped oplog inside th
|
||||
consumer's own write transaction (crash-consistent, non-bypassable). Convergence is
|
||||
last-writer-wins over a hybrid logical clock. All timestamps are UTC.
|
||||
|
||||
## Packages (all `0.2.1`)
|
||||
## Packages (all `0.3.0`)
|
||||
|
||||
| Package | Contents |
|
||||
|---|---|
|
||||
@@ -20,6 +20,30 @@ last-writer-wins over a hybrid logical clock. All timestamps are UTC.
|
||||
|
||||
> Status: **published to the Gitea feed; adopted by ScadaBridge (Phases 1+2) and OtOpcUa (Phase 1).**
|
||||
>
|
||||
> - `0.3.0` (2026-08-15) — **the snapshot-resync flag is per-table.** On-disk bookkeeping schema
|
||||
> goes to **v3** (new `__localdb_snapshot_state`, upgraded in place on open); wire-compatible with
|
||||
> `0.1.x`/`0.2.x` peers, and `lib_schema_version` deliberately stays `1` because it is compared
|
||||
> fail-closed for equality.
|
||||
> - **`RegisterReplicated(table, baselineExistingRows: true)` now flags a resync for THAT TABLE.**
|
||||
> Through `0.2.x` the flag was one bit for the whole database, so baselining one table
|
||||
> re-streamed every registered table in full — and on a pair enabling replication late, both
|
||||
> nodes did that to each other at once. A snapshot now streams only the flagged tables.
|
||||
> - **The database-wide flag survives, unchanged, for the cases that genuinely need it**: an oplog
|
||||
> cap prune or an impossible peer ack loses deltas for the oplog as a whole, so nothing short of
|
||||
> every table is sound. What moved is only where a *baseline* records its debt.
|
||||
> - **Additive wire extension, negotiated by capability, not by version.** `Handshake` gains
|
||||
> `supports_partial_snapshot`; `SnapshotBegin` gains a `tables` scope where **empty means "all
|
||||
> tables"** — which is precisely what a pre-`0.3.0` sender emits and what a pre-`0.3.0` receiver
|
||||
> assumes. A peer that does not advertise the capability is served a **full** snapshot, because a
|
||||
> scoped one would make it advance `last_applied_remote_seq` past deltas it never received.
|
||||
> - **A partial snapshot moves no watermark on either side.** The receiver keeps
|
||||
> `last_applied_remote_seq` (it is one number for the whole peer oplog, and the untouched tables'
|
||||
> deltas still sit below the snapshot's as-of); the sender keeps pumping from where it was. The
|
||||
> overlap re-delivers rows the snapshot already carried, which LWW absorbs.
|
||||
> - **Snapshot debt now clears on acknowledgement, not on enqueue** (new `SnapshotAck`, exchanged
|
||||
> only between `0.3.0` peers). A crash part-way through a snapshot re-streams the same scope on
|
||||
> reconnect instead of losing it. Against a pre-`0.3.0` peer the `0.2.x` clear-on-enqueue
|
||||
> behaviour is kept — there is no ack to wait for.
|
||||
> - `0.2.1` (2026-08-14) — two adversarial-review findings against `0.2.0`. Wire-compatible with
|
||||
> `0.1.x`/`0.2.0` peers; no proto change, no `lib_schema_version` bump, no on-disk schema change.
|
||||
> - **`DeregisterReplicated` no longer leaves the HLC without a crash anchor.** Clock recovery on
|
||||
@@ -177,10 +201,15 @@ All options are validated at startup (`ZB.MOM.WW.Configuration` validators).
|
||||
delta/snapshot batches), so receiving never blocks on the ability to ack.
|
||||
- **Resync** — when a peer's watermark falls below the pruned oplog horizon, it is sent a
|
||||
full snapshot of the registered tables (streamed, keyset-paged) and deltas resume above
|
||||
the snapshot's as-of seq. **Each node decides from its OWN state** whether it owes the
|
||||
peer a snapshot. The snapshot is streamed *after* the handshake, alongside a running
|
||||
receive loop, so a node sending one still applies what the peer sends it — a mutual
|
||||
snapshot exchanges under flow control instead of being buffered whole in memory.
|
||||
the snapshot's as-of seq. When instead a single table was *baselined*, only that table is
|
||||
streamed and **no watermark moves on either side** — a partial snapshot speaks for nothing
|
||||
outside its own tables, so the deltas it overlaps are still pumped and merged by LWW.
|
||||
**Each node decides from its OWN state** whether it owes the peer a snapshot, and a peer
|
||||
that does not advertise `supports_partial_snapshot` is sent a full one. The snapshot is
|
||||
streamed *after* the handshake, alongside a running receive loop, so a node sending one
|
||||
still applies what the peer sends it — a mutual snapshot exchanges under flow control
|
||||
instead of being buffered whole in memory. Between `0.3.0` peers the sender's flags are
|
||||
cleared by the receiver's `SnapshotAck`, so a crash mid-snapshot re-streams it.
|
||||
- **Maintenance** — a background service on **both** roles enforces the oplog row/age caps
|
||||
and prunes expired tombstones every `MaintenanceInterval`.
|
||||
|
||||
@@ -209,17 +238,18 @@ All options are validated at startup (`ZB.MOM.WW.Configuration` validators).
|
||||
- **CDC semantics** — rows that existed **before** `RegisterReplicated` are not captured
|
||||
and are not snapshotted; only changes after registration are replicated. **Opt out with
|
||||
`RegisterReplicated(table, baselineExistingRows: true)`**, which seeds the row-version
|
||||
ledger for existing rows at the baseline floor (HLC `0`) and flags a snapshot resync so
|
||||
they reach the peer. The floor is below every genuine stamp, so a baselined row loses
|
||||
ledger for existing rows at the baseline floor (HLC `0`) and flags a snapshot resync
|
||||
**for that table** so they reach the peer. The floor is below every genuine stamp, so a baselined row loses
|
||||
every comparison against a real remote write and wins only where the peer has no version
|
||||
of that key at all. Two nodes that baseline the same key both hold HLC `0` and resolve by
|
||||
the node-id tie-break — convergent, but arbitrary as to which content survives.
|
||||
- **The snapshot flag is per-database, not per-table.** Baselining one table flags a resync
|
||||
that streams **every** registered table in full, and on a pair enabling replication late
|
||||
both nodes do that to each other simultaneously — the largest transfer this library
|
||||
performs. Bounded (batch budget out, bounded inbox in), but **purge or trim high-volume
|
||||
tables such as event logs before enabling replication late**, and expect one slow first
|
||||
sync. A per-table flag needs an on-disk schema change and is not in `0.2.x`.
|
||||
- **A snapshot still ships whole tables.** The flag is per-table as of `0.3.0`, so baselining
|
||||
one table streams that table and no other — but *that* table travels in full, and two
|
||||
cases still stream **everything**: an oplog cap prune / impossible peer ack (delta
|
||||
continuity is lost for the oplog as a whole), and any peer older than `0.3.0` (which
|
||||
cannot be told a snapshot is partial). So **purge or trim high-volume tables such as event
|
||||
logs before baselining them**, and expect one slow first sync. Bounded either way (batch
|
||||
budget out, bounded inbox in).
|
||||
- **REAL ±Infinity** is carried in SQLite's JSON dialect on the wire (fine for the
|
||||
SQLite↔SQLite peers this library targets).
|
||||
- `AddZbLocalDb` / `AddZbLocalDbReplication` are **first-registration-wins** (idempotent).
|
||||
@@ -256,8 +286,15 @@ provider is wired or the poll threw, so a DB failure never reads as "0 backlog")
|
||||
|
||||
## Ops notes
|
||||
|
||||
- When the oplog row/age caps are exceeded, maintenance prunes to the ceiling and flags
|
||||
`needs_snapshot` (disk safety over delta continuity); the peer then snapshot-resyncs.
|
||||
- When the oplog row/age caps are exceeded, maintenance prunes to the ceiling and flags the
|
||||
**database-wide** `needs_snapshot` (disk safety over delta continuity); the peer then gets a
|
||||
full snapshot. Per-table debt lives separately in `__localdb_snapshot_state` and is what
|
||||
baselining writes.
|
||||
- **Upgrading a pair to `0.3.0`** needs no coordination: the two nodes negotiate scoped snapshots
|
||||
by capability, so a `0.3.0` node paired with a `0.2.x` one simply keeps sending full snapshots
|
||||
until both are upgraded. Rolling the file forward is one-way — a `0.2.x` build reopening a v3
|
||||
file leaves `__localdb_snapshot_state` in place and ignores it, so any per-table debt recorded
|
||||
by `0.3.0` is invisible to it (roll forward again, or re-baseline).
|
||||
- **Turning a table off** — `DeregisterReplicated(table)` drops its capture triggers and prunes
|
||||
its oplog + ledger rows. It must be applied **symmetrically**: the handshake compares
|
||||
registered-table digests fail-closed, so a node that drops a table its peer still
|
||||
@@ -281,14 +318,16 @@ provider is wired or the poll threw, so a DB failure never reads as "0 backlog")
|
||||
|
||||
## Testing
|
||||
|
||||
188 tests, all offline / no live dependencies — unit (HLC ordering + restart
|
||||
199 tests, all offline / no live dependencies — unit (HLC ordering + restart
|
||||
persistence, deregistration's HLC anchor surviving an ungraceful reopen, trigger SQL
|
||||
generation, LWW decision table, watermark/pruning, digest computation, schema-version
|
||||
upgrade, deregistration idempotency, baseline-floor LWW safety, O(1) depth vs a seeded
|
||||
backlog, byte-budget chunking, bounded-inbox backpressure under a slow reader), integration
|
||||
over two real SQLite DBs on an in-memory (and real) gRPC pipe (bidirectional convergence,
|
||||
delete/update races, offline-accumulate → reconnect, prune → snapshot resync, mutual
|
||||
snapshot over a capacity-1 transport), plus seeded randomized property tests (random interleaved op sequences → both DBs identical after quiesce).
|
||||
snapshot over a capacity-1 transport, single-table baseline streaming only that table,
|
||||
mixed-version interop where a peer without `supports_partial_snapshot` gets a full one,
|
||||
snapshot debt surviving until the peer's `SnapshotAck`), plus seeded randomized property tests (random interleaved op sequences → both DBs identical after quiesce).
|
||||
Family conventions apply (`TreatWarningsAsErrors`).
|
||||
|
||||
Build / test from this directory:
|
||||
|
||||
Reference in New Issue
Block a user