feat(localdb): per-table needs_snapshot (0.3.0, on-disk schema v3)

Baselining one table used to set a single per-DATABASE needs_snapshot bit, and a
snapshot streams every registered table in full — so opting one table into
replication late re-sent all of them, in both directions at once on a pair
enabling replication late. On a database holding a high-volume table (an event
log at its retention cap) that is the largest transfer this library performs.
0.2.1 documented the cost and deferred the fix as needing an on-disk change.

Schema v3 adds __localdb_snapshot_state (table_name PK, needs_snapshot), created
by the shared DDL with IF NOT EXISTS so an older file gains it on open; the v2->v3
transition is a version stamp only. RegisterReplicated(..., baselineExistingRows:
true) now flags that table alone. The database-wide flag survives unchanged for
the paths that genuinely reach every table — a cap prune or an impossible peer ack
loses delta continuity for the oplog as a whole. DeregisterReplicated drops a
table's pending flag with its ledger rows, since nothing would ever stream (and so
clear) it afterwards.

Wire compatibility is additive and negotiated by CAPABILITY, not version:
lib_schema_version stays 1 because ValidateHandshake compares it fail-closed for
equality, so bumping it would stop 0.1.x/0.2.x pairs syncing outright. Handshake
gains supports_partial_snapshot (field 5) and SnapshotBegin gains a tables scope
(field 2) where EMPTY means all tables — precisely what a pre-0.3.0 sender emits
and what a pre-0.3.0 receiver assumes. A peer that does not advertise the
capability is served a full snapshot: a scoped one would make it advance
last_applied_remote_seq to the snapshot's as-of and silently discard every delta
below it for the tables left out.

For the same reason a partial snapshot moves no watermark on either side. The
receiver keeps last_applied_remote_seq (carrying only the HLC across, which is
MAX-guarded), and the sender keeps _sentThruSeq, so the deltas the snapshot
overlaps are still pumped and merged by LWW.

Crash safety: snapshot debt now clears on the receiver's new SnapshotAck rather
than on enqueue, so an ungraceful exit part-way through re-streams the same scope
on reconnect. The ack rides the unbounded CONTROL lane like DeltaAck, so it
composes with 0.2.1's bounded 64-message inbox and its out-of-handshake snapshot.
Against a pre-0.3.0 peer the 0.2.x clear-on-enqueue behaviour is kept — there is
no ack to wait for — with the handshake's horizon gap check as the net.

Tests 199/199 (was 188): single-table baseline streams only that table, mixed-
version interop (peer without the capability gets a full snapshot), debt standing
until SnapshotAck, partial snapshot not advancing the receiver's watermark, no
SnapshotAck emitted to an old peer, end-to-end over two real databases, plus v2->v3
upgrade and per-table flag isolation.

README, ILocalDb remarks (which named this as the follow-up) and the umbrella
CLAUDE.md index updated.
This commit is contained in:
Joseph Doherty
2026-08-15 03:34:54 -04:00
parent f627cee9c0
commit cea3930b94
16 changed files with 791 additions and 96 deletions
@@ -291,6 +291,45 @@ public sealed class RegistrationTests : IDisposable
// Without the flag the seeding would be inert: no gap exists at the handshake, so the peer
// would be told it needs no snapshot and the baselined rows would never travel.
Assert.True(await NeedsSnapshot(db));
// ...and it is flagged for THIS TABLE ONLY. Through 0.2.x this set the one database-wide
// bit, so baselining `orders` re-streamed every other registered table with it.
Assert.False(await NeedsFullSnapshot(db));
}
[Fact]
public async Task Register_WithBaseline_FlagsOnlyTheBaselinedTable()
{
// The per-table flag's reason to exist: opting one table into replication late must not
// schedule a full re-stream of its unrelated neighbours.
using var db = Create();
await CreateOrders(db);
await db.ExecuteAsync("CREATE TABLE widgets (id INTEGER PRIMARY KEY, name TEXT)");
await db.ExecuteAsync("INSERT INTO orders (id, sku, qty) VALUES (1,'A',1)");
await db.ExecuteAsync("INSERT INTO widgets (id, name) VALUES (1,'W')");
db.RegisterReplicated("widgets");
db.RegisterReplicated("orders", baselineExistingRows: true);
Assert.True(await NeedsSnapshot(db, "orders"));
Assert.False(await NeedsSnapshot(db, "widgets"));
Assert.False(await NeedsFullSnapshot(db));
}
[Fact]
public async Task Deregister_DropsThePendingSnapshotFlag()
{
// A flag naming a table nothing streams any more can never clear itself — the streamer skips
// unregistered tables — so it would sit in the file and re-appear as unasked-for debt if the
// table were registered again.
using var db = Create();
await CreateOrders(db);
await db.ExecuteAsync("INSERT INTO orders (id, sku, qty) VALUES (1,'A',1)");
db.RegisterReplicated("orders", baselineExistingRows: true);
Assert.True(await NeedsSnapshot(db));
db.DeregisterReplicated("orders");
Assert.False(await NeedsSnapshot(db));
}
[Fact]
@@ -301,7 +340,8 @@ public sealed class RegistrationTests : IDisposable
await db.ExecuteAsync("INSERT INTO orders (id, sku, qty) VALUES (1,'A',1)");
db.RegisterReplicated("orders", baselineExistingRows: true);
await db.ExecuteAsync("UPDATE __localdb_peer_state SET needs_snapshot = 0 WHERE id = 1");
// Stand in for the snapshot having been delivered and acknowledged.
await db.ExecuteAsync("DELETE FROM __localdb_snapshot_state");
db.RegisterReplicated("orders", baselineExistingRows: true);
@@ -480,7 +520,17 @@ public sealed class RegistrationTests : IDisposable
private static async Task<long> CountAsync(SqliteLocalDb db, string sql) =>
(await db.QueryAsync(sql, r => r.GetInt64(0)))[0];
private static async Task<bool> NeedsSnapshot(SqliteLocalDb db) =>
/// <summary>Whether <paramref name="table"/> individually owes the peer a snapshot (schema v3).</summary>
private static async Task<bool> NeedsSnapshot(SqliteLocalDb db, string table = "orders") =>
(await db.QueryAsync(
"SELECT COUNT(*) FROM __localdb_snapshot_state WHERE table_name = @t AND needs_snapshot <> 0",
r => r.GetInt64(0), new { t = table }))[0] != 0;
/// <summary>
/// The DATABASE-WIDE flag — "every registered table". Baselining must never set it: that is the
/// whole point of the per-table flags, and setting it would restore the 0.2.x behaviour silently.
/// </summary>
private static async Task<bool> NeedsFullSnapshot(SqliteLocalDb db) =>
(await db.QueryAsync(
"SELECT needs_snapshot FROM __localdb_peer_state WHERE id = 1", r => r.GetInt64(0)))[0] != 0;
}