fix(localdb): a rebuilt peer's own writes were silently dropped

The other half of the rebuilt-peer story, and only reachable once 0.1.2 made
back-fill work: with the wiped node repopulated and writing again, its writes
never reached its peer.

last_applied_remote_seq is a watermark in the PEER's seq space, and a rebuilt
peer is a new database whose oplog numbers from 1. Two things then went wrong,
and both had to be fixed — the first alone leaves data stuck:

- The healthy node kept advertising the OLD peer's watermark, and the sending
  side seeds its pump from exactly that number, so the rebuilt node skipped its
  entire oplog. The watermark (and the observed peer clock) is now reset when
  the peer's node id changes. last_acked_seq is deliberately NOT reset: it
  describes our own oplog's pruned horizon and is what tells a rebuilt peer it
  needs a snapshot — clearing it would turn 0.1.2's back-fill back off.

- A peer claiming to have applied more of our stream than we have ever produced
  can only be remembering a previous incarnation of us. That claim is now
  rejected in favour of starting from the beginning; LWW makes the re-send
  harmless. This mirrors the clamp RecordPeerAckAsync already applies to acks.

Found on the OtOpcUa docker-dev rig, where the healthy node held watermark 10
while the rebuilt peer's oplog was seqs 1-3 and its last_acked stayed 0 — the
peer never accepted a single one. Covered by a test that converges with one
peer, replaces it with a fresh database, and requires a write on the new peer
to arrive: RED before this fix, green after. 149/149 pass.

Version 0.1.3.

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
This commit is contained in:
Joseph Doherty
2026-07-21 02:06:03 -04:00
parent 45d08929e7
commit 3b7489a7b0
5 changed files with 136 additions and 4 deletions
@@ -97,6 +97,20 @@ internal sealed class OplogStore(ILocalDb db, ReplicationOptions options, Func<D
return (long)cmd.ExecuteScalar()!;
}
/// <summary>
/// The highest seq this node has ever produced: the oplog's max, or — once acked rows have been
/// pruned away — the ack watermark that pruned them. Nothing we emit can exceed it, so a peer
/// claiming to have applied more than this is not talking about the stream we are producing now.
/// </summary>
public async Task<long> GetKnownMaxSeqAsync(CancellationToken ct = default)
{
var rows = await db.QueryAsync(
"SELECT MAX((SELECT COALESCE(MAX(seq), 0) FROM __localdb_oplog), " +
"(SELECT last_acked_seq FROM __localdb_peer_state WHERE id = 1))",
static r => r.GetInt64(0), null, ct);
return rows[0];
}
/// <summary>The highest oplog seq present, or 0 when the oplog is empty.</summary>
public async Task<long> GetMaxSeqAsync(CancellationToken ct = default)
{
@@ -160,6 +174,28 @@ internal sealed class OplogStore(ILocalDb db, ReplicationOptions options, Func<D
public Task SetPeerNodeIdAsync(string peerNodeId, CancellationToken ct = default) =>
db.ExecuteAsync("UPDATE __localdb_peer_state SET peer_node_id = @v WHERE id = 1", new { v = peerNodeId }, ct);
/// <summary>
/// Forgets everything we knew about the previous peer's stream: the inbound seq watermark and
/// its observed clock. Called when the peer's node id changes.
/// </summary>
/// <remarks>
/// <para>
/// <c>last_applied_remote_seq</c> is a watermark in the PEER's seq space, and a rebuilt peer is
/// a new database whose oplog numbers from 1. Carrying the old peer's watermark across the
/// swap makes the new peer's early writes look already-applied, so they are dropped — silently,
/// and for as many writes as the old watermark was high.
/// </para>
/// <para>
/// <c>last_acked_seq</c> is deliberately NOT reset: it describes OUR oplog (what has been
/// pruned), not the peer's, and it is what tells a rebuilt peer's handshake that it needs a
/// snapshot. Clearing it would make an already-pruned node look like it had never written
/// anything and turn the back-fill back off.
/// </para>
/// </remarks>
public Task ResetInboundWatermarkAsync(CancellationToken ct = default) =>
db.ExecuteAsync(
"UPDATE __localdb_peer_state SET last_applied_remote_seq = 0, last_seen_hlc = 0 WHERE id = 1", null, ct);
/// <summary>Sets or clears the snapshot-resync-required flag.</summary>
public Task SetNeedsSnapshotAsync(bool value, CancellationToken ct = default) =>
db.ExecuteAsync("UPDATE __localdb_peer_state SET needs_snapshot = @v WHERE id = 1", new { v = value ? 1 : 0 }, ct);