fix(localdb): keep a tombstone until the peer holds it — no resurrection past TombstoneRetention (scadaproj#16)

0.3.7 (unpublished), owner-chosen option A. A tombstone is now pruned only when
it is past TombstoneRetention AND the peer is known to hold it:
- held while database-wide or per-table snapshot debt is outstanding (the
  snapshot is built from the row-version ledger, so a pruned tombstone is a
  delete it can no longer carry — the issue's second path);
- a local tombstone held while its delete is still in the unacked oplog.
Delivery (delta ack covering the delete, or snapshot debt discharged) restamps
tombstone_utc so retention runs from delivery. New optional
LocalDb:Replication:TombstoneHardCap (default off, >= TombstoneRetention)
bounds an undelivered tombstone's life. No proto, lib_schema_version or
on-disk schema change.

Regression test TombstoneResurrectionTests (two real nodes over loopback gRPC):
offline update on B, newer delete on A, A's clock +8 days (oplog age cap +
tombstone prune) -> reconnect. Fails on 0.3.6 (row resurrected), passes now;
plus a no-prune positive control and a pruned-after-delivery control, and five
controlled-clock OplogStore tests. Each guard clause mutation-killed. 259/259.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
This commit is contained in:
Joseph Doherty
2026-11-09 22:27:46 -05:00
parent 3215d51154
commit be5e9fd436
11 changed files with 423 additions and 18 deletions
+30 -5
View File
@@ -10,7 +10,7 @@ Change capture uses SQLite triggers writing a single HLC-stamped oplog inside th
consumer's own write transaction (crash-consistent, non-bypassable). Convergence is
last-writer-wins over a hybrid logical clock. All timestamps are UTC.
## Packages (all `0.3.6`; the feed carries up to `0.3.4` — `0.3.5` and `0.3.6` are unpublished)
## Packages (all `0.3.7`; the feed carries up to `0.3.4` — `0.3.5` through `0.3.7` are unpublished)
| Package | Contents |
|---|---|
@@ -21,6 +21,15 @@ last-writer-wins over a hybrid logical clock. All timestamps are UTC.
> Status: **published to the Gitea feed (through `0.3.4`); adopted by ScadaBridge (Phases 1+2) and
> OtOpcUa (Phases 1+2), both pinned to `0.3.4`.**
>
> - `0.3.7` (unpublished) — **a deleted row no longer comes back when the peer was offline longer
> than `TombstoneRetention`** (scadaproj#16). Tombstones used to be pruned on age alone; once the
> delete's oplog row had also aged out, the snapshot (built from the row-version ledger) could not
> carry the delete, and the returning peer's older update re-created the row on **both** nodes. A
> tombstone is now pruned only when it is past retention **and** the peer holds it — no outstanding
> snapshot debt (database-wide or for its table) and, for a local delete, no longer in the unacked
> oplog — and its retention clock restarts when it is delivered (delta ack or snapshot discharge).
> New optional `TombstoneHardCap` (default off) bounds how long an undelivered tombstone is kept.
> Wire-compatible: no proto, `lib_schema_version` or on-disk schema change.
> - `0.3.6` (unpublished) — code-review fixes (scadaproj#36–#39, #46, #50). Wire-compatible: no proto
> change, no `lib_schema_version` bump, no on-disk schema change.
> - The passive endpoint enforces `ApiKey` itself (Bearer, constant-time, before the handshake).
@@ -256,7 +265,8 @@ node still pushes its own deltas back. Only one side needs `PeerAddress`.
| `MaxBatchSize` | `500` | **Secondary** row-count cap per outbound batch (also the snapshot DB page size). |
| `MaxOplogRows` | `1_000_000` | Backlog ceiling; exceeding it flags a snapshot resync and prunes oldest rows. |
| `MaxOplogAge` | `7 days` | Age ceiling; an unpruned entry older than this flags a snapshot resync. |
| `TombstoneRetention` | `7 days` | How long deleted-row tombstones are kept before pruning. |
| `TombstoneRetention` | `7 days` | How long a deleted-row tombstone is kept **after the peer has received it** (`0.3.7`: pruned only once past this AND delivered — delta acked or snapshot discharged; the clock restarts at delivery). An undelivered tombstone is kept for the whole outage. |
| `TombstoneHardCap` | `null` (off) | Optional absolute ceiling on keeping an **undelivered** tombstone (`0.3.7`). Bounds ledger growth when the peer is gone for good; a peer that returns after more than this can resurrect deleted rows, so rebuild it instead. Must be ≥ `TombstoneRetention`. |
| `ReconnectBackoffMax` | `60 s` | Upper bound on exponential reconnect backoff (validator caps ≤ 1 day). |
| `MaintenanceInterval` | `60 s` | How often maintenance enforces caps + prunes expired tombstones. |
| `MaxSnapshotDeferredRows` | `100_000` | Ceiling on snapshot rows held in memory while their FK parent has not arrived (`0.3.6`). Past it the oldest are dead-lettered (and counted). |
@@ -290,7 +300,8 @@ All options are validated at startup (`ZB.MOM.WW.Configuration` validators).
instead of being buffered whole in memory. Between `0.3.0` peers the sender's flags are
cleared by the receiver's `SnapshotAck`, so a crash mid-snapshot re-streams it.
- **Maintenance** — a background service on **both** roles enforces the oplog row/age caps
and prunes expired tombstones every `MaintenanceInterval`.
and prunes expired tombstones every `MaintenanceInterval` — only those the peer is known to
hold (`0.3.7`; see Ops notes).
## Semantics worth knowing
@@ -406,6 +417,19 @@ provider is wired or the poll threw, so a DB failure never reads as "0 backlog")
## Ops notes
- **Tombstones and long outages** (`0.3.7`, scadaproj#16). A tombstone is what lets LWW reject an
older write of a deleted row, so it is kept until the peer has it: while the database or its
table owes the peer a snapshot (the snapshot is built from the ledger, so a pruned tombstone is a
delete it can no longer carry), and while a local delete is still in the unacked oplog. Delivery
(a delta ack covering the delete, or the snapshot debt being discharged) restamps
`tombstone_utc`, and `TombstoneRetention` runs from there — the returning peer's own stale writes
get a full retention to drain. The cost: **while the peer is down, every delete's tombstone stays
in `__localdb_row_version`** (high-churn tables such as a store-and-forward queue grow the ledger
for the whole outage). Set `TombstoneHardCap` to bound that; past it the pre-`0.3.7` resurrection
risk returns, so treat a peer offline longer than the cap as one to rebuild, not reconnect. Not
covered by any setting: a peer **restored from an old backup** of its own file presents stale rows
under its old identity and watermarks — rebuild it empty instead.
- When the oplog row/age caps are exceeded, maintenance prunes to the ceiling and flags the
**database-wide** `needs_snapshot` (disk safety over delta continuity); the peer then gets a
full snapshot. Per-table debt lives separately in `__localdb_snapshot_state` and is what
@@ -451,13 +475,14 @@ provider is wired or the poll threw, so a DB failure never reads as "0 backlog")
## Testing
249 tests, all offline / no live dependencies — unit (HLC ordering + restart
259 tests, all offline / no live dependencies — unit (HLC ordering + restart
persistence, deregistration's HLC anchor surviving an ungraceful reopen, trigger SQL
generation, LWW decision table, watermark/pruning, digest computation, schema-version
upgrade, deregistration idempotency, baseline-floor LWW safety, O(1) depth vs a seeded
backlog, byte-budget chunking, bounded-inbox backpressure under a slow reader), integration
over two real SQLite DBs on an in-memory (and real) gRPC pipe (bidirectional convergence,
delete/update races, offline-accumulate → reconnect, prune → snapshot resync, mutual
delete/update races, offline-accumulate → reconnect, prune → snapshot resync, a peer offline
past `TombstoneRetention` not resurrecting a newer delete, mutual
snapshot over a capacity-1 transport, single-table baseline streaming only that table,
mixed-version interop where a peer without `supports_partial_snapshot` gets a full one,
snapshot debt surviving until the peer's `SnapshotAck`), plus seeded randomized property tests (random interleaved op sequences → both DBs identical after quiesce).