fix(localdb): keep a tombstone until the peer holds it — no resurrection past TombstoneRetention (scadaproj#16)

0.3.7 (unpublished), owner-chosen option A. A tombstone is now pruned only when
it is past TombstoneRetention AND the peer is known to hold it:
- held while database-wide or per-table snapshot debt is outstanding (the
  snapshot is built from the row-version ledger, so a pruned tombstone is a
  delete it can no longer carry — the issue's second path);
- a local tombstone held while its delete is still in the unacked oplog.
Delivery (delta ack covering the delete, or snapshot debt discharged) restamps
tombstone_utc so retention runs from delivery. New optional
LocalDb:Replication:TombstoneHardCap (default off, >= TombstoneRetention)
bounds an undelivered tombstone's life. No proto, lib_schema_version or
on-disk schema change.

Regression test TombstoneResurrectionTests (two real nodes over loopback gRPC):
offline update on B, newer delete on A, A's clock +8 days (oplog age cap +
tombstone prune) -> reconnect. Fails on 0.3.6 (row resurrected), passes now;
plus a no-prune positive control and a pruned-after-delivery control, and five
controlled-clock OplogStore tests. Each guard clause mutation-killed. 259/259.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
This commit is contained in:
Joseph Doherty
2026-11-09 22:27:46 -05:00
parent 3215d51154
commit be5e9fd436
11 changed files with 423 additions and 18 deletions
@@ -89,6 +89,8 @@ public sealed class MaintenanceTests : IDisposable
var (db, store) = await NewDbAsync(options);
await db.ExecuteAsync("INSERT INTO orders (id, sku, qty) VALUES (1, 'DOOMED', 1)");
await db.ExecuteAsync("DELETE FROM orders WHERE id = 1");
// The peer has the delete (an unacked tombstone is never pruned on age — scadaproj#16).
await store.RecordPeerAckAsync(await store.GetMaxSeqAsync());
// Backdate the delete-trigger's tombstone far past any retention window.
var updated = await db.ExecuteAsync(
"UPDATE __localdb_row_version SET tombstone_utc = '2000-01-01T00:00:00.000Z' WHERE is_tombstone = 1");
@@ -147,6 +149,29 @@ public sealed class MaintenanceTests : IDisposable
Assert.Contains(result.Failures, f => f.Contains("MaintenanceInterval", StringComparison.Ordinal));
}
[Fact]
public void Validator_TombstoneHardCapBelowRetention_Fails_AtOrAboveAndUnset_Pass()
{
var validator = new ReplicationOptionsValidator();
var below = validator.Validate(null, new ReplicationOptions { TombstoneHardCap = TimeSpan.FromDays(6) });
Assert.True(below.Failed);
Assert.Contains(below.Failures, f => f.Contains("TombstoneHardCap", StringComparison.Ordinal));
Assert.True(validator.Validate(null, new ReplicationOptions { TombstoneHardCap = TimeSpan.FromDays(7) }).Succeeded);
Assert.True(validator.Validate(null, new ReplicationOptions()).Succeeded);
}
[Fact]
public void TombstoneHardCap_BindsFromConfiguration_DefaultsToOff()
{
var bound = new ReplicationOptions();
new ConfigurationBuilder()
.AddInMemoryCollection(new Dictionary<string, string?> { ["TombstoneHardCap"] = "30.00:00:00" })
.Build().Bind(bound);
Assert.Equal(TimeSpan.FromDays(30), bound.TombstoneHardCap);
Assert.Null(new ReplicationOptions().TombstoneHardCap);
}
[Fact]
public void AddZbLocalDbReplication_RegistersMaintenance_BothRoles()
{