fix(localdb): keep a tombstone until the peer holds it — no resurrection past TombstoneRetention (scadaproj#16)

0.3.7 (unpublished), owner-chosen option A. A tombstone is now pruned only when it is past TombstoneRetention AND the peer is known to hold it: - held while database-wide or per-table snapshot debt is outstanding (the snapshot is built from the row-version ledger, so a pruned tombstone is a delete it can no longer carry — the issue's second path); - a local tombstone held while its delete is still in the unacked oplog. Delivery (delta ack covering the delete, or snapshot debt discharged) restamps tombstone_utc so retention runs from delivery. New optional LocalDb:Replication:TombstoneHardCap (default off, >= TombstoneRetention) bounds an undelivered tombstone's life. No proto, lib_schema_version or on-disk schema change. Regression test TombstoneResurrectionTests (two real nodes over loopback gRPC): offline update on B, newer delete on A, A's clock +8 days (oplog age cap + tombstone prune) -> reconnect. Fails on 0.3.6 (row resurrected), passes now; plus a no-prune positive control and a pruned-after-delivery control, and five controlled-clock OplogStore tests. Each guard clause mutation-killed. 259/259. Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
2026-11-09 22:27:46 -05:00
parent 3215d51154
commit be5e9fd436
11 changed files with 423 additions and 18 deletions
@@ -2,7 +2,7 @@

 Shared embedded SQLite cache + 2-node LWW sync library for the **ZB.MOM.WW SCADA family** (3 packages; consumed by ScadaBridge and OtOpcUa). Build/test/pack from this directory: `dotnet build ZB.MOM.WW.LocalDb.slnx`, `dotnet test ZB.MOM.WW.LocalDb.slnx` (fully offline), `dotnet pack ZB.MOM.WW.LocalDb.slnx -c Release -o artifacts`. Design: `../docs/plans/2026-07-17-localdb-design.md`; adoption designs `../docs/plans/2026-07-19-scadabridge-localdb-design.md` + `../docs/plans/2026-07-20-otopcua-localdb-design.md`.

-**Backlog / open work:** lives in Gitea issues on `dohertj2/scadaproj` (`tea issues list --repo dohertj2/scadaproj`) — currently #16 (resurrection guard past `TombstoneRetention`). The history below is a record, not a status tracker; file new follow-ups as issues.
+**Backlog / open work:** lives in Gitea issues on `dohertj2/scadaproj` (`tea issues list --repo dohertj2/scadaproj`) — #16 (resurrection past `TombstoneRetention`) was fixed in `0.3.7`. The history below is a record, not a status tracker; file new follow-ups as issues.

 **`0.3.5` (unpublished, scadaproj#26):** `Microsoft.Data.Sqlite` 10.0.7 -> 10.0.13 (efcore#39009 pool fault, which a LocalDb consumer is exposed to); `SQLitePCLRaw.lib.e_sqlite3` transitive pin 2.1.12 -> 2.1.13 (10.0.13 floors 2.1.13). 216/216; no wire/schema change.

@@ -16,6 +16,8 @@ Shared embedded SQLite cache + 2-node LWW sync library for the **ZB.MOM.WW SCADA
 - **#50:** snapshot-path dead-letters (poison, drift, FK orphan, cap spill) now count in `localdb.sync.dead_lettered` (`SnapshotApplier` takes `LocalDbMetrics`, wired by `SyncSessionFactory`). New option **`LocalDb:Replication:MaxSnapshotDeferredRows`** (default 100,000, validated > 0) caps snapshot rows held for a missing FK parent; past it the oldest are re-applied once and dead-lettered if still orphaned. **Open note (cross-repo, #50):** once ScadaBridge and OtOpcUa pin 0.3.6, their own `LocalDbSyncAuthInterceptor`s duplicate the library check — they differ only in refusing every stream when no key is set. Each app decides whether to keep that stricter rule or delete its interceptor; nothing in this library changes for either choice.
 249/249 tests.

+**`0.3.7` (unpublished, scadaproj#16 — owner chose option A):** a tombstone is pruned only when it is past `TombstoneRetention` **and** the peer holds it. `OplogStore.PruneTombstonesAsync` holds every tombstone while database-wide (`__localdb_peer_state.needs_snapshot`) or per-table (`__localdb_snapshot_state`) snapshot debt is outstanding, and a LOCAL tombstone while its delete is still unacked (`hlc >=` the first unacked oplog row's HLC — ledger and oplog share the HLC, local HLCs rise with seq). Delivery restamps `tombstone_utc` (in `RecordPeerAckAsync` for tombstones in the newly acked range; in `ClearSnapshotDebtAsync` for the snapshot's scope), so retention runs from delivery and the returning peer's stale queued writes get a full retention to drain. Remote-origin tombstones age on retention alone. New optional **`LocalDb:Replication:TombstoneHardCap`** (default `null` = off, validated ≥ `TombstoneRetention`) bounds an undelivered tombstone's life — past it the old risk returns. **The snapshot path (issue's second note) is covered by the debt hold:** the cap prune sets `needs_snapshot` in the same transaction that drops undelivered oplog rows, so the ledger tombstone the snapshot is built from survives until that snapshot is acknowledged. Cost: ledger growth for the length of an outage. Not covered: a peer restored from an old backup of its own file. No proto, `lib_schema_version` or on-disk schema change. Regression test `TombstoneResurrectionTests.PeerOfflinePastRetention_DeleteStillWinsOnBothNodes` (fails on 0.3.6 with the row resurrected, `A=1`) + positive control without the prune + pruned-after-delivery control; 5 `OplogStoreTests` with a controlled clock; every guard clause mutation-killed. 259/259.
+
 ## Index history (migrated from the scadaproj root CLAUDE.md)

 > Moved verbatim out of the scadaproj root `CLAUDE.md` on 2026-09-03 so it loads only when needed. **This file is now the authoritative place for this history — append new entries here and keep the root index row short.**
@@ -5,7 +5,7 @@
    <Nullable>enable</Nullable>
    <ImplicitUsings>enable</ImplicitUsings>
    <LangVersion>latest</LangVersion>
-    <Version>0.3.6</Version>
+    <Version>0.3.7</Version>
    <ManagePackageVersionsCentrally>true</ManagePackageVersionsCentrally>
    <TreatWarningsAsErrors>true</TreatWarningsAsErrors>
    <!-- NU1510 (package-pruning advice): the Replication and Tests projects carry a
@@ -10,7 +10,7 @@ Change capture uses SQLite triggers writing a single HLC-stamped oplog inside th
 consumer's own write transaction (crash-consistent, non-bypassable). Convergence is
 last-writer-wins over a hybrid logical clock. All timestamps are UTC.

-## Packages (all `0.3.6`; the feed carries up to `0.3.4` — `0.3.5` and `0.3.6` are unpublished)
+## Packages (all `0.3.7`; the feed carries up to `0.3.4` — `0.3.5` through `0.3.7` are unpublished)

 | Package | Contents |
 |---|---|
@@ -21,6 +21,15 @@ last-writer-wins over a hybrid logical clock. All timestamps are UTC.
 > Status: **published to the Gitea feed (through `0.3.4`); adopted by ScadaBridge (Phases 1+2) and
 > OtOpcUa (Phases 1+2), both pinned to `0.3.4`.**
 >
+> - `0.3.7` (unpublished) — **a deleted row no longer comes back when the peer was offline longer
+>   than `TombstoneRetention`** (scadaproj#16). Tombstones used to be pruned on age alone; once the
+>   delete's oplog row had also aged out, the snapshot (built from the row-version ledger) could not
+>   carry the delete, and the returning peer's older update re-created the row on **both** nodes. A
+>   tombstone is now pruned only when it is past retention **and** the peer holds it — no outstanding
+>   snapshot debt (database-wide or for its table) and, for a local delete, no longer in the unacked
+>   oplog — and its retention clock restarts when it is delivered (delta ack or snapshot discharge).
+>   New optional `TombstoneHardCap` (default off) bounds how long an undelivered tombstone is kept.
+>   Wire-compatible: no proto, `lib_schema_version` or on-disk schema change.
 > - `0.3.6` (unpublished) — code-review fixes (scadaproj#36–#39, #46, #50). Wire-compatible: no proto
 >   change, no `lib_schema_version` bump, no on-disk schema change.
 >   - The passive endpoint enforces `ApiKey` itself (Bearer, constant-time, before the handshake).
@@ -256,7 +265,8 @@ node still pushes its own deltas back. Only one side needs `PeerAddress`.
 | `MaxBatchSize` | `500` | **Secondary** row-count cap per outbound batch (also the snapshot DB page size). |
 | `MaxOplogRows` | `1_000_000` | Backlog ceiling; exceeding it flags a snapshot resync and prunes oldest rows. |
 | `MaxOplogAge` | `7 days` | Age ceiling; an unpruned entry older than this flags a snapshot resync. |
-| `TombstoneRetention` | `7 days` | How long deleted-row tombstones are kept before pruning. |
+| `TombstoneRetention` | `7 days` | How long a deleted-row tombstone is kept **after the peer has received it** (`0.3.7`: pruned only once past this AND delivered — delta acked or snapshot discharged; the clock restarts at delivery). An undelivered tombstone is kept for the whole outage. |
+| `TombstoneHardCap` | `null` (off) | Optional absolute ceiling on keeping an **undelivered** tombstone (`0.3.7`). Bounds ledger growth when the peer is gone for good; a peer that returns after more than this can resurrect deleted rows, so rebuild it instead. Must be ≥ `TombstoneRetention`. |
 | `ReconnectBackoffMax` | `60 s` | Upper bound on exponential reconnect backoff (validator caps ≤ 1 day). |
 | `MaintenanceInterval` | `60 s` | How often maintenance enforces caps + prunes expired tombstones. |
 | `MaxSnapshotDeferredRows` | `100_000` | Ceiling on snapshot rows held in memory while their FK parent has not arrived (`0.3.6`). Past it the oldest are dead-lettered (and counted). |
@@ -290,7 +300,8 @@ All options are validated at startup (`ZB.MOM.WW.Configuration` validators).
  instead of being buffered whole in memory. Between `0.3.0` peers the sender's flags are
  cleared by the receiver's `SnapshotAck`, so a crash mid-snapshot re-streams it.
 - **Maintenance** — a background service on **both** roles enforces the oplog row/age caps
-  and prunes expired tombstones every `MaintenanceInterval`.
+  and prunes expired tombstones every `MaintenanceInterval` — only those the peer is known to
+  hold (`0.3.7`; see Ops notes).

 ## Semantics worth knowing

@@ -406,6 +417,19 @@ provider is wired or the poll threw, so a DB failure never reads as "0 backlog")

 ## Ops notes

+- **Tombstones and long outages** (`0.3.7`, scadaproj#16). A tombstone is what lets LWW reject an
+  older write of a deleted row, so it is kept until the peer has it: while the database or its
+  table owes the peer a snapshot (the snapshot is built from the ledger, so a pruned tombstone is a
+  delete it can no longer carry), and while a local delete is still in the unacked oplog. Delivery
+  (a delta ack covering the delete, or the snapshot debt being discharged) restamps
+  `tombstone_utc`, and `TombstoneRetention` runs from there — the returning peer's own stale writes
+  get a full retention to drain. The cost: **while the peer is down, every delete's tombstone stays
+  in `__localdb_row_version`** (high-churn tables such as a store-and-forward queue grow the ledger
+  for the whole outage). Set `TombstoneHardCap` to bound that; past it the pre-`0.3.7` resurrection
+  risk returns, so treat a peer offline longer than the cap as one to rebuild, not reconnect. Not
+  covered by any setting: a peer **restored from an old backup** of its own file presents stale rows
+  under its old identity and watermarks — rebuild it empty instead.
+
 - When the oplog row/age caps are exceeded, maintenance prunes to the ceiling and flags the
  **database-wide** `needs_snapshot` (disk safety over delta continuity); the peer then gets a
  full snapshot. Per-table debt lives separately in `__localdb_snapshot_state` and is what
@@ -451,13 +475,14 @@ provider is wired or the poll threw, so a DB failure never reads as "0 backlog")

 ## Testing

-249 tests, all offline / no live dependencies — unit (HLC ordering + restart
+259 tests, all offline / no live dependencies — unit (HLC ordering + restart
 persistence, deregistration's HLC anchor surviving an ungraceful reopen, trigger SQL
 generation, LWW decision table, watermark/pruning, digest computation, schema-version
 upgrade, deregistration idempotency, baseline-floor LWW safety, O(1) depth vs a seeded
 backlog, byte-budget chunking, bounded-inbox backpressure under a slow reader), integration
 over two real SQLite DBs on an in-memory (and real) gRPC pipe (bidirectional convergence,
-delete/update races, offline-accumulate → reconnect, prune → snapshot resync, mutual
+delete/update races, offline-accumulate → reconnect, prune → snapshot resync, a peer offline
+past `TombstoneRetention` not resurrecting a newer delete, mutual
 snapshot over a capacity-1 transport, single-table baseline streaming only that table,
 mixed-version interop where a peer without `supports_partial_snapshot` gets a full one,
 snapshot debt surviving until the peer's `SnapshotAck`), plus seeded randomized property tests (random interleaved op sequences → both DBs identical after quiesce).
@@ -56,20 +56,93 @@ internal sealed class OplogStore(ILocalDb db, ReplicationOptions options, Func<D
        // safe only because needs_snapshot = 1 commits in the SAME transaction, forcing a resync.
        if (ackedSeq > knownMax)
            await tx.ExecuteAsync("UPDATE __localdb_peer_state SET needs_snapshot = 1 WHERE id = 1", null, ct);
+        var now = FormatUtc(_utcNow());
+        var acked = Math.Min(ackedSeq, knownMax);
+        // scadaproj#16: the peer now holds every local tombstone in the newly acked range, so their
+        // retention clock starts HERE rather than at the delete. Must run before the prune below
+        // deletes the oplog rows that identify them, and before last_acked_seq moves (it bounds the
+        // range from below).
+        await tx.ExecuteAsync(RestampDeliveredTombstonesSql, new { acked, now }, ct);
        await tx.ExecuteAsync(
            "UPDATE __localdb_peer_state SET last_acked_seq = MAX(last_acked_seq, @acked), last_sync_utc = @now WHERE id = 1",
-            new { acked = Math.Min(ackedSeq, knownMax), now = FormatUtc(_utcNow()) }, ct);
+            new { acked, now }, ct);
        await tx.ExecuteAsync(
            "DELETE FROM __localdb_oplog WHERE seq <= (SELECT last_acked_seq FROM __localdb_peer_state WHERE id = 1)",
            null, ct);
        await tx.CommitAsync(ct);
    }

-    /// <summary>Deletes row-version tombstones older than <see cref="ReplicationOptions.TombstoneRetention"/>; live rows and fresh tombstones are kept.</summary>
-    public Task PruneTombstonesAsync(CancellationToken ct = default) =>
-        db.ExecuteAsync(
-            "DELETE FROM __localdb_row_version WHERE is_tombstone = 1 AND tombstone_utc < @cutoff",
-            new { cutoff = FormatUtc(_utcNow() - options.TombstoneRetention) }, ct);
+    // Local tombstones whose delete is in the acked range (old last_acked_seq, @acked]. A local
+    // tombstone's ledger row and its oplog row share (table_name, pk_json); a later re-delete of the
+    // same key that is still unacked is restamped too, which is harmless — the prune below still
+    // holds it on the unacked-oplog floor. Never moves the stamp backwards.
+    private const string RestampDeliveredTombstonesSql =
+        "UPDATE __localdb_row_version SET tombstone_utc = @now " +
+        "WHERE is_tombstone = 1 AND (tombstone_utc IS NULL OR tombstone_utc < @now) " +
+        "AND (table_name, pk_json) IN (" +
+        "SELECT table_name, pk_json FROM __localdb_oplog WHERE is_tombstone = 1 " +
+        "AND seq > (SELECT last_acked_seq FROM __localdb_peer_state WHERE id = 1) AND seq <= @acked)";
+
+    // A tombstone the peer may not hold yet. Each clause is evaluated once per statement (they are
+    // uncorrelated scalar subqueries), so the prune stays one pass over the tombstones.
+    //  - Database-wide snapshot debt: a cap prune (or an impossible ack) dropped oplog rows the peer
+    //    never received, and the snapshot that replaces them is built from this ledger — a pruned
+    //    tombstone is a delete the snapshot can no longer carry.
+    //  - Per-table snapshot debt: the same, scoped to the baselined table.
+    //  - A LOCAL tombstone at or above the unacked-oplog floor: its delete is still waiting in the
+    //    oplog. The ledger row carries the same HLC as its oplog row and local HLCs rise with seq, so
+    //    "hlc >= HLC of the first unacked oplog row" is exactly "not yet acked" (equal is held, the
+    //    conservative side; no unacked row at all ⇒ the COALESCE ceiling ⇒ nothing held). Remote-origin
+    //    tombstones came FROM the peer and are never in our oplog.
+    private const string HeldTombstonePredicate =
+        "(SELECT needs_snapshot FROM __localdb_peer_state WHERE id = 1) <> 0 " +
+        "OR table_name IN (SELECT table_name FROM __localdb_snapshot_state WHERE needs_snapshot <> 0) " +
+        "OR (node_id = (SELECT node_id FROM __localdb_meta WHERE id = 1) " +
+        "AND hlc >= COALESCE((SELECT hlc FROM __localdb_oplog " +
+        "WHERE seq > (SELECT last_acked_seq FROM __localdb_peer_state WHERE id = 1) ORDER BY seq LIMIT 1), " +
+        "9223372036854775807))";
+
+    /// <summary>
+    /// Deletes row-version tombstones that are past <see cref="ReplicationOptions.TombstoneRetention"/>
+    /// <b>and</b> known to be held by the peer; returns how many were removed.
+    /// </summary>
+    /// <remarks>
+    /// <para>
+    /// <b>Why retention alone is not enough (scadaproj#16).</b> A tombstone is the only thing that
+    /// lets LWW reject an OLDER write of a deleted row. Up to 0.3.6 it was pruned on age alone, so a
+    /// peer offline longer than the retention came back to a node that had forgotten the delete: the
+    /// delete's oplog row had been cap-pruned too, the snapshot (built from this ledger) no longer
+    /// carried it, and the peer's stale update re-created the row on both nodes.
+    /// </para>
+    /// <para>
+    /// <b>The rule.</b> A tombstone is kept while the peer might not hold it — see
+    /// <see cref="HeldTombstonePredicate"/>: outstanding snapshot debt, or a local delete still in the
+    /// unacked oplog. Once the peer does hold it (delta ack, or snapshot debt discharged) its
+    /// <c>tombstone_utc</c> is restamped to that moment, so retention is measured from DELIVERY: the
+    /// peer's own stale writes, queued before it received the delete, have a full retention to drain
+    /// before the tombstone stops being able to reject them. Remote-origin tombstones need no
+    /// delivery — the peer wrote them — and age from the moment they were applied here.
+    /// </para>
+    /// <para>
+    /// <b>Cost.</b> While the peer is down nothing local is delivered, so tombstones accumulate in the
+    /// ledger for the whole outage. <see cref="ReplicationOptions.TombstoneHardCap"/> bounds that when
+    /// set, at the price of reintroducing the resurrection risk for an outage longer than the cap.
+    /// </para>
+    /// </remarks>
+    public Task<int> PruneTombstonesAsync(CancellationToken ct = default)
+    {
+        var now = _utcNow();
+        var cutoff = FormatUtc(now - options.TombstoneRetention);
+        if (options.TombstoneHardCap is { } hardCap)
+            return db.ExecuteAsync(
+                "DELETE FROM __localdb_row_version WHERE is_tombstone = 1 AND (tombstone_utc < @hardCutoff " +
+                $"OR (tombstone_utc < @cutoff AND NOT ({HeldTombstonePredicate})))",
+                new { cutoff, hardCutoff = FormatUtc(now - hardCap) }, ct);
+        return db.ExecuteAsync(
+            "DELETE FROM __localdb_row_version WHERE is_tombstone = 1 " +
+            $"AND tombstone_utc < @cutoff AND NOT ({HeldTombstonePredicate})",
+            new { cutoff }, ct);
+    }

    /// <summary>
    /// The unacked backlog — oplog entries above the peer's last-acked watermark — in O(1).
@@ -293,16 +366,28 @@ internal sealed class OplogStore(ILocalDb db, ReplicationOptions options, Func<D
    public async Task ClearSnapshotDebtAsync(IReadOnlyList<string> tables, CancellationToken ct = default)
    {
        await using var tx = await db.BeginTransactionAsync(ct);
+        var now = FormatUtc(_utcNow());
        if (tables.Count == 0)
        {
            await tx.ExecuteAsync("UPDATE __localdb_peer_state SET needs_snapshot = 0 WHERE id = 1", null, ct);
            await tx.ExecuteAsync("DELETE FROM __localdb_snapshot_state", null, ct);
+            // scadaproj#16: the snapshot carried every tombstone; retention restarts at delivery.
+            await tx.ExecuteAsync(
+                "UPDATE __localdb_row_version SET tombstone_utc = @now " +
+                "WHERE is_tombstone = 1 AND (tombstone_utc IS NULL OR tombstone_utc < @now)",
+                new { now }, ct);
        }
        else
        {
            foreach (var table in tables)
+            {
                await tx.ExecuteAsync(
                    "DELETE FROM __localdb_snapshot_state WHERE table_name = @t", new { t = table }, ct);
+                await tx.ExecuteAsync(
+                    "UPDATE __localdb_row_version SET tombstone_utc = @now WHERE table_name = @t " +
+                    "AND is_tombstone = 1 AND (tombstone_utc IS NULL OR tombstone_utc < @now)",
+                    new { t = table, now }, ct);
+            }
        }
        await tx.CommitAsync(ct);
    }
@@ -8,7 +8,7 @@ namespace ZB.MOM.WW.LocalDb.Replication;
 /// <summary>
 /// Periodic oplog maintenance: every <see cref="ReplicationOptions.MaintenanceInterval"/> it
 /// enforces the backlog/age caps (which flag <c>needs_snapshot</c> when exceeded) and prunes
-/// expired tombstone row-versions. Runs on BOTH roles — a passive node with a long-dead peer is
+/// expired tombstone row-versions the peer is known to hold (see <c>OplogStore.PruneTombstonesAsync</c>). Runs on BOTH roles — a passive node with a long-dead peer is
 /// exactly the case where the oplog would otherwise grow unbounded. A failing tick is logged and
 /// skipped; maintenance never dies.
 /// </summary>
@@ -48,7 +48,7 @@ public sealed class MaintenanceBackgroundService : BackgroundService
            try
            {
                var snapshotFlagged = await _store.EnforceCapsAsync(stoppingToken);
-                await _store.PruneTombstonesAsync(stoppingToken);
+                var pruned = await _store.PruneTombstonesAsync(stoppingToken);

                if (snapshotFlagged && !_snapshotFlagWarned)
                {
@@ -61,7 +61,9 @@ public sealed class MaintenanceBackgroundService : BackgroundService
                    _snapshotFlagWarned = false;
                }

-                _logger.LogDebug("Maintenance tick complete (snapshotFlagged={SnapshotFlagged}).", snapshotFlagged);
+                _logger.LogDebug(
+                    "Maintenance tick complete (snapshotFlagged={SnapshotFlagged}, tombstonesPruned={TombstonesPruned}).",
+                    snapshotFlagged, pruned);
            }
            catch (OperationCanceledException) when (stoppingToken.IsCancellationRequested)
            {
@@ -66,9 +66,32 @@ public sealed class ReplicationOptions
    /// <summary>Age ceiling: an unpruned oplog entry older than this flags a snapshot resync.</summary>
    public TimeSpan MaxOplogAge { get; set; } = TimeSpan.FromDays(7);

-    /// <summary>How long deleted-row tombstones are retained before pruning.</summary>
+    /// <summary>
+    /// How long a deleted-row tombstone is retained once the peer holds it, before pruning.
+    /// </summary>
+    /// <remarks>
+    /// Since 0.3.7 (scadaproj#16) a tombstone is pruned only when it is past this retention AND the
+    /// peer has received it (its delete acked, or the snapshot that carries it acknowledged), and the
+    /// retention clock restarts at that delivery. Up to 0.3.6 it was pruned on age alone, which let a
+    /// peer offline longer than the retention resurrect the deleted row. A tombstone the peer has
+    /// not received is kept for as long as the outage lasts — bounded only by
+    /// <see cref="TombstoneHardCap"/>.
+    /// </remarks>
    public TimeSpan TombstoneRetention { get; set; } = TimeSpan.FromDays(7);

+    /// <summary>
+    /// Optional absolute ceiling on how long a tombstone the peer has NOT received is kept. Default
+    /// <c>null</c> (off): an undelivered tombstone is kept until the peer receives it, however long
+    /// that takes. Must be at least <see cref="TombstoneRetention"/> when set.
+    /// </summary>
+    /// <remarks>
+    /// Bounds ledger growth on a node whose peer is gone for good (high-churn tables such as a
+    /// store-and-forward queue add one tombstone per delete for the whole outage). The trade is
+    /// explicit: a peer that comes back after more than this can resurrect rows deleted while it was
+    /// away, exactly as before 0.3.7 — rebuild such a peer instead of reconnecting it.
+    /// </remarks>
+    public TimeSpan? TombstoneHardCap { get; set; }
+
    /// <summary>Upper bound on the exponential reconnect backoff.</summary>
    public TimeSpan ReconnectBackoffMax { get; set; } = TimeSpan.FromSeconds(60);

@@ -36,6 +36,9 @@ public sealed class ReplicationOptionsValidator : IValidateOptions<ReplicationOp
            failures.Add($"LocalDb:Replication:MaxOplogAge must be greater than zero; got {options.MaxOplogAge}.");
        if (options.TombstoneRetention <= TimeSpan.Zero)
            failures.Add($"LocalDb:Replication:TombstoneRetention must be greater than zero; got {options.TombstoneRetention}.");
+        if (options.TombstoneHardCap is { } hardCap && hardCap < options.TombstoneRetention)
+            failures.Add(
+                $"LocalDb:Replication:TombstoneHardCap must be at least TombstoneRetention ({options.TombstoneRetention}) when set; got {hardCap}.");
        if (options.ReconnectBackoffMax <= TimeSpan.Zero)
            failures.Add($"LocalDb:Replication:ReconnectBackoffMax must be greater than zero; got {options.ReconnectBackoffMax}.");
        else if (options.ReconnectBackoffMax > ReconnectBackoffCeiling)
@@ -249,6 +249,20 @@ public sealed class ConvergenceFixture : IAsyncDisposable
    public Task<bool> EnforceCapsAAsync() =>
        new OplogStore(_dbA, InitiatorOptions).EnforceCapsAsync();

+    /// <summary>
+    /// Runs one maintenance pass (caps, then — unless <paramref name="pruneTombstones"/> is false —
+    /// tombstone pruning) on node A with A's wall clock advanced by <paramref name="clockOffset"/>,
+    /// i.e. what <see cref="MaintenanceBackgroundService"/> does on its tick once that much time has
+    /// passed. Lets a test model "node B was offline for N days" without waiting N days.
+    /// </summary>
+    public async Task RunMaintenanceAAsync(TimeSpan clockOffset, bool pruneTombstones = true)
+    {
+        var store = new OplogStore(_dbA, InitiatorOptions, () => DateTimeOffset.UtcNow + clockOffset);
+        await store.EnforceCapsAsync();
+        if (pruneTombstones)
+            await store.PruneTombstonesAsync();
+    }
+
    // ---- internals --------------------------------------------------------------------------

    private static SqliteLocalDb NewDb(string path)
@@ -0,0 +1,116 @@
+namespace ZB.MOM.WW.LocalDb.Tests.Convergence;
+
+/// <summary>
+/// scadaproj#16 — a deleted row must not come back when the peer was offline longer than
+/// <c>TombstoneRetention</c>. Driven over the real loopback gRPC pair (<see cref="ConvergenceFixture"/>).
+/// </summary>
+/// <remarks>
+/// The scenario: both nodes hold row 1; while the transport is down B updates it and A then deletes
+/// it (A's delete is the newer HLC, so the delete must win everywhere). Eight days pass on A — the
+/// oplog age cap drops A's delete from the oplog (flagging a snapshot resync) and maintenance runs
+/// its tombstone prune. Before the fix the prune removed A's tombstone too, so A's snapshot had
+/// nothing to say about row 1 and B's older update re-created it on A: resurrected on both nodes.
+/// </remarks>
+[Collection("Convergence")]
+public sealed class TombstoneResurrectionTests
+{
+    private static readonly TimeSpan EightDays = TimeSpan.FromDays(8);
+
+    [Fact]
+    public async Task PeerOfflinePastRetention_DeleteStillWinsOnBothNodes()
+    {
+        await using var fx = await OfflineUpdateThenNewerDeleteAsync();
+
+        // The maintenance tick 8 days later: age cap + tombstone prune, as the background service runs them.
+        await fx.RunMaintenanceAAsync(EightDays);
+        Assert.True(await IsTombstonedAsync(fx.A, 1),
+            "A's undelivered tombstone was pruned while B had never seen it (scadaproj#16).");
+
+        await fx.RestartTransportAsync();
+        await fx.AssertDataConvergedAsync();
+
+        Assert.Equal(0, await ConvergenceFixture.CountOrdersAsync(fx.A));
+        Assert.Equal(0, await ConvergenceFixture.CountOrdersAsync(fx.B));
+        Assert.True(await IsTombstonedAsync(fx.B, 1), "B should hold A's tombstone after the snapshot.");
+    }
+
+    // Positive control: the same scenario with the tombstone prune skipped must converge on "deleted".
+    // If it did not, the test above could pass or fail for reasons unrelated to tombstone retention.
+    [Fact]
+    public async Task Control_SameScenarioWithoutTombstonePrune_DeleteWins()
+    {
+        await using var fx = await OfflineUpdateThenNewerDeleteAsync();
+
+        await fx.RunMaintenanceAAsync(EightDays, pruneTombstones: false);
+
+        await fx.RestartTransportAsync();
+        await fx.AssertDataConvergedAsync();
+
+        Assert.Equal(0, await ConvergenceFixture.CountOrdersAsync(fx.A));
+        Assert.Equal(0, await ConvergenceFixture.CountOrdersAsync(fx.B));
+    }
+
+    // Retention still does its job once the peer HAS the tombstone: after the reconnect the pair has
+    // exchanged it, so a prune a full retention later must remove it on A.
+    [Fact]
+    public async Task AfterPeerReceivesTombstone_ItIsPrunedOneRetentionLater()
+    {
+        await using var fx = await OfflineUpdateThenNewerDeleteAsync();
+        await fx.RunMaintenanceAAsync(EightDays);
+        await fx.RestartTransportAsync();
+        await fx.AssertDataConvergedAsync();
+        Assert.Equal(0, await ConvergenceFixture.CountOrdersAsync(fx.B));
+        // The snapshot debt clears on B's SnapshotAck, which can trail the data by a moment.
+        await WaitForNoSnapshotDebtAsync(fx.A);
+
+        // Delivery restarts A's retention clock (proved with a controlled clock in OplogStoreTests);
+        // one retention past it, the delivered tombstone goes.
+        await fx.RunMaintenanceAAsync(EightDays);
+        Assert.False(await HasRowVersionAsync(fx.A, 1), "A never pruned a delivered tombstone.");
+    }
+
+    private static async Task<ConvergenceFixture> OfflineUpdateThenNewerDeleteAsync()
+    {
+        var fx = new ConvergenceFixture();
+        await ConvergenceFixture.UpsertAsync(fx.A, 1, "seed", 1);
+        await fx.StartAsync();
+        await fx.AssertConvergedAsync();
+
+        await fx.KillTransportAsync();
+        await ConvergenceFixture.UpsertAsync(fx.B, 1, "b-old", 2);
+        // Wall-clock gap so A's delete is the strictly newer HLC regardless of millisecond ties.
+        await Task.Delay(20);
+        await ConvergenceFixture.DeleteAsync(fx.A, 1);
+        return fx;
+    }
+
+    private static async Task WaitForNoSnapshotDebtAsync(ILocalDb db)
+    {
+        var deadline = DateTime.UtcNow + TimeSpan.FromSeconds(30);
+        while (DateTime.UtcNow < deadline)
+        {
+            var flag = await db.QueryAsync(
+                "SELECT needs_snapshot FROM __localdb_peer_state WHERE id = 1", static r => r.GetInt64(0));
+            if (flag[0] == 0)
+                return;
+            await Task.Delay(25);
+        }
+        Assert.Fail("A's snapshot debt was never discharged.");
+    }
+
+    private static async Task<bool> IsTombstonedAsync(ILocalDb db, long id)
+    {
+        var rows = await db.QueryAsync(
+            "SELECT is_tombstone FROM __localdb_row_version WHERE table_name = 'orders' AND pk_json = @pk",
+            static r => r.GetInt64(0), new { pk = $"{{\"id\":{id}}}" });
+        return rows.Count == 1 && rows[0] == 1;
+    }
+
+    private static async Task<bool> HasRowVersionAsync(ILocalDb db, long id)
+    {
+        var rows = await db.QueryAsync(
+            "SELECT COUNT(*) FROM __localdb_row_version WHERE table_name = 'orders' AND pk_json = @pk",
+            static r => r.GetInt64(0), new { pk = $"{{\"id\":{id}}}" });
+        return rows[0] == 1;
+    }
+}
@@ -89,6 +89,8 @@ public sealed class MaintenanceTests : IDisposable
        var (db, store) = await NewDbAsync(options);
        await db.ExecuteAsync("INSERT INTO orders (id, sku, qty) VALUES (1, 'DOOMED', 1)");
        await db.ExecuteAsync("DELETE FROM orders WHERE id = 1");
+        // The peer has the delete (an unacked tombstone is never pruned on age — scadaproj#16).
+        await store.RecordPeerAckAsync(await store.GetMaxSeqAsync());
        // Backdate the delete-trigger's tombstone far past any retention window.
        var updated = await db.ExecuteAsync(
            "UPDATE __localdb_row_version SET tombstone_utc = '2000-01-01T00:00:00.000Z' WHERE is_tombstone = 1");
@@ -147,6 +149,29 @@ public sealed class MaintenanceTests : IDisposable
        Assert.Contains(result.Failures, f => f.Contains("MaintenanceInterval", StringComparison.Ordinal));
    }

+    [Fact]
+    public void Validator_TombstoneHardCapBelowRetention_Fails_AtOrAboveAndUnset_Pass()
+    {
+        var validator = new ReplicationOptionsValidator();
+        var below = validator.Validate(null, new ReplicationOptions { TombstoneHardCap = TimeSpan.FromDays(6) });
+        Assert.True(below.Failed);
+        Assert.Contains(below.Failures, f => f.Contains("TombstoneHardCap", StringComparison.Ordinal));
+
+        Assert.True(validator.Validate(null, new ReplicationOptions { TombstoneHardCap = TimeSpan.FromDays(7) }).Succeeded);
+        Assert.True(validator.Validate(null, new ReplicationOptions()).Succeeded);
+    }
+
+    [Fact]
+    public void TombstoneHardCap_BindsFromConfiguration_DefaultsToOff()
+    {
+        var bound = new ReplicationOptions();
+        new ConfigurationBuilder()
+            .AddInMemoryCollection(new Dictionary<string, string?> { ["TombstoneHardCap"] = "30.00:00:00" })
+            .Build().Bind(bound);
+        Assert.Equal(TimeSpan.FromDays(30), bound.TombstoneHardCap);
+        Assert.Null(new ReplicationOptions().TombstoneHardCap);
+    }
+
    [Fact]
    public void AddZbLocalDbReplication_RegistersMaintenance_BothRoles()
    {
@@ -122,6 +122,113 @@ public sealed class OplogStoreTests : IDisposable
        Assert.True(peer.NeedsSnapshot);
    }

+    // ---- scadaproj#16: a tombstone is pruned only once the peer holds it --------------------
+
+    private static async Task<long> TombstoneCount(SqliteLocalDb db) =>
+        (await db.QueryAsync(
+            "SELECT COUNT(*) FROM __localdb_row_version WHERE is_tombstone = 1", static r => r.GetInt64(0)))[0];
+
+    [Fact]
+    public async Task Prune_KeepsUndeliveredLocalTombstonePastRetention_ThenAgesItFromDelivery()
+    {
+        using var db = await NewOrdersDb();
+        await db.ExecuteAsync("INSERT INTO orders (id, sku, qty) VALUES (1, 'A', 1)");
+        await db.ExecuteAsync("DELETE FROM orders WHERE id = 1");
+        var now = DateTimeOffset.UtcNow;
+        var store = new OplogStore(db, new ReplicationOptions(), () => now);
+
+        // Peer offline 8 days: past the 7-day retention, but the delete is still unacked.
+        now += TimeSpan.FromDays(8);
+        Assert.Equal(0, await store.PruneTombstonesAsync(default));
+        Assert.Equal(1, await TombstoneCount(db));
+
+        // The peer comes back and acks it: retention restarts at the ack, not at the delete.
+        await store.RecordPeerAckAsync(await store.GetMaxSeqAsync(), default);
+        now += TimeSpan.FromDays(6);
+        Assert.Equal(0, await store.PruneTombstonesAsync(default));
+        Assert.Equal(1, await TombstoneCount(db));
+
+        now += TimeSpan.FromDays(2);
+        Assert.Equal(1, await store.PruneTombstonesAsync(default));
+        Assert.Equal(0, await TombstoneCount(db));
+    }
+
+    [Fact]
+    public async Task Prune_KeepsTombstonesWhileDatabaseWideSnapshotDebtIsOutstanding()
+    {
+        using var db = await NewOrdersDb();
+        await db.ExecuteAsync("INSERT INTO orders (id, sku, qty) VALUES (1, 'A', 1)");
+        await db.ExecuteAsync("DELETE FROM orders WHERE id = 1");
+        var now = DateTimeOffset.UtcNow;
+        var store = new OplogStore(db, new ReplicationOptions(), () => now);
+
+        // The age cap drops the delete from the oplog (never delivered) and flags a snapshot — the
+        // snapshot is built from the ledger, so the tombstone is the only carrier of the delete.
+        now += TimeSpan.FromDays(8);
+        Assert.True(await store.EnforceCapsAsync(default));
+        Assert.Equal(0, await store.GetMaxSeqAsync());
+        Assert.Equal(0, await store.PruneTombstonesAsync(default));
+        Assert.Equal(1, await TombstoneCount(db));
+
+        // The snapshot is acknowledged: debt cleared, tombstone restamped, aged from there.
+        await store.ClearSnapshotDebtAsync([], default);
+        now += TimeSpan.FromDays(6);
+        Assert.Equal(0, await store.PruneTombstonesAsync(default));
+        now += TimeSpan.FromDays(2);
+        Assert.Equal(1, await store.PruneTombstonesAsync(default));
+    }
+
+    [Fact]
+    public async Task Prune_KeepsTombstonesOfATableWithPerTableSnapshotDebt()
+    {
+        using var db = await NewOrdersDb();
+        await db.ExecuteAsync("INSERT INTO orders (id, sku, qty) VALUES (1, 'A', 1)");
+        await db.ExecuteAsync("DELETE FROM orders WHERE id = 1");
+        var now = DateTimeOffset.UtcNow;
+        var store = new OplogStore(db, new ReplicationOptions(), () => now);
+        await store.RecordPeerAckAsync(await store.GetMaxSeqAsync(), default);
+        await db.ExecuteAsync(
+            "INSERT INTO __localdb_snapshot_state (table_name, needs_snapshot) VALUES ('orders', 1)");
+
+        now += TimeSpan.FromDays(8);
+        Assert.Equal(0, await store.PruneTombstonesAsync(default));
+
+        await store.ClearSnapshotDebtAsync(["orders"], default);
+        now += TimeSpan.FromDays(8);
+        Assert.Equal(1, await store.PruneTombstonesAsync(default));
+    }
+
+    [Fact]
+    public async Task Prune_RemoteOriginTombstone_AgesOnRetentionAlone_EvenWithLocalBacklog()
+    {
+        using var db = await NewOrdersDb();
+        // A local write the peer has not acked — it must not hold the PEER's own delete.
+        await db.ExecuteAsync("INSERT INTO orders (id, sku, qty) VALUES (9, 'LOCAL', 1)");
+        await db.ExecuteAsync(
+            "INSERT INTO __localdb_row_version (table_name, pk_json, hlc, node_id, is_tombstone, tombstone_utc) " +
+            "VALUES ('orders', '{\"id\":1}', 9223372036854775000, 'the-peer', 1, @utc)",
+            new { utc = DateTimeOffset.UtcNow.UtcDateTime.ToString(OplogStore.TombstoneUtcFormat) });
+        var now = DateTimeOffset.UtcNow + TimeSpan.FromDays(8);
+        var store = new OplogStore(db, new ReplicationOptions(), () => now);
+
+        Assert.Equal(1, await store.PruneTombstonesAsync(default));
+    }
+
+    [Fact]
+    public async Task Prune_HardCap_DropsAnUndeliveredTombstoneOnceExceeded()
+    {
+        using var db = await NewOrdersDb();
+        await db.ExecuteAsync("INSERT INTO orders (id, sku, qty) VALUES (1, 'A', 1)");
+        await db.ExecuteAsync("DELETE FROM orders WHERE id = 1");
+        var now = DateTimeOffset.UtcNow;
+        var store = new OplogStore(db, new ReplicationOptions { TombstoneHardCap = TimeSpan.FromDays(30) }, () => now);
+
+        now += TimeSpan.FromDays(29);
+        Assert.Equal(0, await store.PruneTombstonesAsync(default));
+        now += TimeSpan.FromDays(2);
+        Assert.Equal(1, await store.PruneTombstonesAsync(default));
+    }
+
    [Fact]
    public async Task Prune_DropsExpiredTombstoneRowVersions_KeepsLive()
    {
@@ -129,10 +236,13 @@ public sealed class OplogStoreTests : IDisposable
        await db.ExecuteAsync("INSERT INTO orders (id, sku, qty) VALUES (1,'A',1),(2,'B',2),(3,'C',3)");
        await db.ExecuteAsync("DELETE FROM orders WHERE id = 1"); // will be backdated -> expired
        await db.ExecuteAsync("DELETE FROM orders WHERE id = 3"); // fresh tombstone -> kept
+        var store = new OplogStore(db, new ReplicationOptions());
+        // The peer has both deletes (scadaproj#16: an undelivered tombstone is never pruned on age);
+        // backdate AFTER the ack, which restamps delivered tombstones.
+        await store.RecordPeerAckAsync(await store.GetMaxSeqAsync(), default);
        await db.ExecuteAsync(
            "UPDATE __localdb_row_version SET tombstone_utc = '2000-01-01T00:00:00.000Z' WHERE pk_json = '{\"id\":1}'");

-        var store = new OplogStore(db, new ReplicationOptions());
        await store.PruneTombstonesAsync(default);

        var rv = await db.QueryAsync(