fix(localdb): a rebuilt peer's own writes were silently dropped

The other half of the rebuilt-peer story, and only reachable once 0.1.2 made
back-fill work: with the wiped node repopulated and writing again, its writes
never reached its peer.

last_applied_remote_seq is a watermark in the PEER's seq space, and a rebuilt
peer is a new database whose oplog numbers from 1. Two things then went wrong,
and both had to be fixed — the first alone leaves data stuck:

- The healthy node kept advertising the OLD peer's watermark, and the sending
  side seeds its pump from exactly that number, so the rebuilt node skipped its
  entire oplog. The watermark (and the observed peer clock) is now reset when
  the peer's node id changes. last_acked_seq is deliberately NOT reset: it
  describes our own oplog's pruned horizon and is what tells a rebuilt peer it
  needs a snapshot — clearing it would turn 0.1.2's back-fill back off.

- A peer claiming to have applied more of our stream than we have ever produced
  can only be remembering a previous incarnation of us. That claim is now
  rejected in favour of starting from the beginning; LWW makes the re-send
  harmless. This mirrors the clamp RecordPeerAckAsync already applies to acks.

Found on the OtOpcUa docker-dev rig, where the healthy node held watermark 10
while the rebuilt peer's oplog was seqs 1-3 and its last_acked stayed 0 — the
peer never accepted a single one. Covered by a test that converges with one
peer, replaces it with a fresh database, and requires a write on the new peer
to arrive: RED before this fix, green after. 149/149 pass.

Version 0.1.3.

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
This commit is contained in:
Joseph Doherty
2026-07-21 02:06:03 -04:00
parent 45d08929e7
commit 3b7489a7b0
5 changed files with 136 additions and 4 deletions
@@ -46,6 +46,14 @@ public sealed class SnapshotResyncTests : IDisposable
await db.ExecuteAsync(OrdersSql);
db.RegisterReplicated("orders");
return NewSessionOver(db, options);
}
// A fresh session stack over an EXISTING database. Reconnecting after a session ends needs a new
// session (RunAsync is single-use), so any scenario with two connection episodes over one node's
// data — a peer being replaced between them — goes through here.
private static Side NewSessionOver(SqliteLocalDb db, ReplicationOptions? options = null)
{
options ??= new ReplicationOptions { FlushInterval = TimeSpan.FromMilliseconds(20) };
var store = new OplogStore(db, options);
var applier = new LwwApplier(db);
@@ -198,6 +206,59 @@ public sealed class SnapshotResyncTests : IDisposable
await SwallowAsync(runB);
}
[Fact]
public async Task RebuiltPeer_OwnWritesReachTheHealthyNode_DespiteTheOldPeersWatermark()
{
// The mirror image of the wiped-peer snapshot gap, found on the OtOpcUa docker-dev rig once
// back-fill started working and the rebuilt node could be observed writing again.
//
// last_applied_remote_seq is a watermark in the PEER's seq space. A rebuilt peer is a new
// database, so its seqs restart at 1 — but the healthy node kept the old peer's watermark
// and discarded everything at or below it. The rebuilt node's own writes were silently
// dropped until its counter climbed past a number that has nothing to do with it. Silent is
// the operative word: both nodes cache after every apply, so the pair looks converged right
// up until the moment only the rebuilt node applies something.
var a = await NewSideAsync();
var bOld = await NewSideAsync();
// Phase 1: converge with the ORIGINAL peer, so A's inbound watermark advances well past the
// seq numbers a rebuilt peer would start from.
for (var i = 1; i <= 5; i++)
await bOld.Db.ExecuteAsync("INSERT INTO orders (id, sku, qty) VALUES (@id, 'OLD', @id)", new { id = i });
var (d1A, d1B) = DuplexPair();
using (var cts1 = new CancellationTokenSource(RunTimeout))
{
var run1A = a.Session.RunAsync(d1A, cts1.Token);
var run1B = bOld.Session.RunAsync(d1B, cts1.Token);
await WaitForAsync(async () => (await ReadOrders(a.Db)).Count == 5, RunTimeout);
cts1.Cancel();
await SwallowAsync(run1A);
await SwallowAsync(run1B);
}
Assert.True(await LastAppliedRemoteSeq(a.Db) >= 5);
// Phase 2: B is rebuilt — a new database, a new node id, and an oplog numbering from 1. Its
// write must still reach A.
var bNew = await NewSideAsync();
await bNew.Db.ExecuteAsync("INSERT INTO orders (id, sku, qty) VALUES (99, 'REBUILT', 99)");
var a2 = NewSessionOver(a.Db);
var (d2A, d2B) = DuplexPair();
using var cts2 = new CancellationTokenSource(RunTimeout);
var run2A = a2.Session.RunAsync(d2A, cts2.Token);
var run2B = bNew.Session.RunAsync(d2B, cts2.Token);
await WaitForAsync(
async () => (await ReadOrders(a.Db)).Any(r => r.Id == 99 && r.Sku == "REBUILT"),
RunTimeout);
cts2.Cancel();
await SwallowAsync(run2A);
await SwallowAsync(run2B);
}
[Fact]
public async Task PrunedPeer_GetsSnapshot_ThenConverges()
{