fix(localdb): keep a tombstone until the peer holds it — no resurrection past TombstoneRetention (scadaproj#16)

0.3.7 (unpublished), owner-chosen option A. A tombstone is now pruned only when
it is past TombstoneRetention AND the peer is known to hold it:
- held while database-wide or per-table snapshot debt is outstanding (the
  snapshot is built from the row-version ledger, so a pruned tombstone is a
  delete it can no longer carry — the issue's second path);
- a local tombstone held while its delete is still in the unacked oplog.
Delivery (delta ack covering the delete, or snapshot debt discharged) restamps
tombstone_utc so retention runs from delivery. New optional
LocalDb:Replication:TombstoneHardCap (default off, >= TombstoneRetention)
bounds an undelivered tombstone's life. No proto, lib_schema_version or
on-disk schema change.

Regression test TombstoneResurrectionTests (two real nodes over loopback gRPC):
offline update on B, newer delete on A, A's clock +8 days (oplog age cap +
tombstone prune) -> reconnect. Fails on 0.3.6 (row resurrected), passes now;
plus a no-prune positive control and a pruned-after-delivery control, and five
controlled-clock OplogStore tests. Each guard clause mutation-killed. 259/259.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
This commit is contained in:
Joseph Doherty
2026-11-09 22:27:46 -05:00
parent 3215d51154
commit be5e9fd436
11 changed files with 423 additions and 18 deletions
@@ -249,6 +249,20 @@ public sealed class ConvergenceFixture : IAsyncDisposable
public Task<bool> EnforceCapsAAsync() =>
new OplogStore(_dbA, InitiatorOptions).EnforceCapsAsync();
/// <summary>
/// Runs one maintenance pass (caps, then — unless <paramref name="pruneTombstones"/> is false —
/// tombstone pruning) on node A with A's wall clock advanced by <paramref name="clockOffset"/>,
/// i.e. what <see cref="MaintenanceBackgroundService"/> does on its tick once that much time has
/// passed. Lets a test model "node B was offline for N days" without waiting N days.
/// </summary>
public async Task RunMaintenanceAAsync(TimeSpan clockOffset, bool pruneTombstones = true)
{
var store = new OplogStore(_dbA, InitiatorOptions, () => DateTimeOffset.UtcNow + clockOffset);
await store.EnforceCapsAsync();
if (pruneTombstones)
await store.PruneTombstonesAsync();
}
// ---- internals --------------------------------------------------------------------------
private static SqliteLocalDb NewDb(string path)