From be5e9fd436d91f879579b5d61e2591fa56c4126c Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Mon, 9 Nov 2026 22:27:46 -0500 Subject: [PATCH] =?UTF-8?q?fix(localdb):=20keep=20a=20tombstone=20until=20?= =?UTF-8?q?the=20peer=20holds=20it=20=E2=80=94=20no=20resurrection=20past?= =?UTF-8?q?=20TombstoneRetention=20(scadaproj#16)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 0.3.7 (unpublished), owner-chosen option A. A tombstone is now pruned only when it is past TombstoneRetention AND the peer is known to hold it: - held while database-wide or per-table snapshot debt is outstanding (the snapshot is built from the row-version ledger, so a pruned tombstone is a delete it can no longer carry — the issue's second path); - a local tombstone held while its delete is still in the unacked oplog. Delivery (delta ack covering the delete, or snapshot debt discharged) restamps tombstone_utc so retention runs from delivery. New optional LocalDb:Replication:TombstoneHardCap (default off, >= TombstoneRetention) bounds an undelivered tombstone's life. No proto, lib_schema_version or on-disk schema change. Regression test TombstoneResurrectionTests (two real nodes over loopback gRPC): offline update on B, newer delete on A, A's clock +8 days (oplog age cap + tombstone prune) -> reconnect. Fails on 0.3.6 (row resurrected), passes now; plus a no-prune positive control and a pruned-after-delivery control, and five controlled-clock OplogStore tests. Each guard clause mutation-killed. 259/259. Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV --- ZB.MOM.WW.LocalDb/CLAUDE.md | 4 +- ZB.MOM.WW.LocalDb/Directory.Build.props | 2 +- ZB.MOM.WW.LocalDb/README.md | 35 +++++- .../Internal/OplogStore.cs | 97 ++++++++++++++- .../MaintenanceBackgroundService.cs | 8 +- .../ReplicationOptions.cs | 25 +++- .../ReplicationOptionsValidator.cs | 3 + .../Convergence/ConvergenceFixture.cs | 14 +++ .../Convergence/TombstoneResurrectionTests.cs | 116 ++++++++++++++++++ .../MaintenanceTests.cs | 25 ++++ .../OplogStoreTests.cs | 112 ++++++++++++++++- 11 files changed, 423 insertions(+), 18 deletions(-) create mode 100644 ZB.MOM.WW.LocalDb/tests/ZB.MOM.WW.LocalDb.Tests/Convergence/TombstoneResurrectionTests.cs diff --git a/ZB.MOM.WW.LocalDb/CLAUDE.md b/ZB.MOM.WW.LocalDb/CLAUDE.md index 6ff118b..93b9a62 100644 --- a/ZB.MOM.WW.LocalDb/CLAUDE.md +++ b/ZB.MOM.WW.LocalDb/CLAUDE.md @@ -2,7 +2,7 @@ Shared embedded SQLite cache + 2-node LWW sync library for the **ZB.MOM.WW SCADA family** (3 packages; consumed by ScadaBridge and OtOpcUa). Build/test/pack from this directory: `dotnet build ZB.MOM.WW.LocalDb.slnx`, `dotnet test ZB.MOM.WW.LocalDb.slnx` (fully offline), `dotnet pack ZB.MOM.WW.LocalDb.slnx -c Release -o artifacts`. Design: `../docs/plans/2026-07-17-localdb-design.md`; adoption designs `../docs/plans/2026-07-19-scadabridge-localdb-design.md` + `../docs/plans/2026-07-20-otopcua-localdb-design.md`. -**Backlog / open work:** lives in Gitea issues on `dohertj2/scadaproj` (`tea issues list --repo dohertj2/scadaproj`) — currently #16 (resurrection guard past `TombstoneRetention`). The history below is a record, not a status tracker; file new follow-ups as issues. +**Backlog / open work:** lives in Gitea issues on `dohertj2/scadaproj` (`tea issues list --repo dohertj2/scadaproj`) — #16 (resurrection past `TombstoneRetention`) was fixed in `0.3.7`. The history below is a record, not a status tracker; file new follow-ups as issues. **`0.3.5` (unpublished, scadaproj#26):** `Microsoft.Data.Sqlite` 10.0.7 -> 10.0.13 (efcore#39009 pool fault, which a LocalDb consumer is exposed to); `SQLitePCLRaw.lib.e_sqlite3` transitive pin 2.1.12 -> 2.1.13 (10.0.13 floors 2.1.13). 216/216; no wire/schema change. @@ -16,6 +16,8 @@ Shared embedded SQLite cache + 2-node LWW sync library for the **ZB.MOM.WW SCADA - **#50:** snapshot-path dead-letters (poison, drift, FK orphan, cap spill) now count in `localdb.sync.dead_lettered` (`SnapshotApplier` takes `LocalDbMetrics`, wired by `SyncSessionFactory`). New option **`LocalDb:Replication:MaxSnapshotDeferredRows`** (default 100,000, validated > 0) caps snapshot rows held for a missing FK parent; past it the oldest are re-applied once and dead-lettered if still orphaned. **Open note (cross-repo, #50):** once ScadaBridge and OtOpcUa pin 0.3.6, their own `LocalDbSyncAuthInterceptor`s duplicate the library check — they differ only in refusing every stream when no key is set. Each app decides whether to keep that stricter rule or delete its interceptor; nothing in this library changes for either choice. 249/249 tests. +**`0.3.7` (unpublished, scadaproj#16 — owner chose option A):** a tombstone is pruned only when it is past `TombstoneRetention` **and** the peer holds it. `OplogStore.PruneTombstonesAsync` holds every tombstone while database-wide (`__localdb_peer_state.needs_snapshot`) or per-table (`__localdb_snapshot_state`) snapshot debt is outstanding, and a LOCAL tombstone while its delete is still unacked (`hlc >=` the first unacked oplog row's HLC — ledger and oplog share the HLC, local HLCs rise with seq). Delivery restamps `tombstone_utc` (in `RecordPeerAckAsync` for tombstones in the newly acked range; in `ClearSnapshotDebtAsync` for the snapshot's scope), so retention runs from delivery and the returning peer's stale queued writes get a full retention to drain. Remote-origin tombstones age on retention alone. New optional **`LocalDb:Replication:TombstoneHardCap`** (default `null` = off, validated ≥ `TombstoneRetention`) bounds an undelivered tombstone's life — past it the old risk returns. **The snapshot path (issue's second note) is covered by the debt hold:** the cap prune sets `needs_snapshot` in the same transaction that drops undelivered oplog rows, so the ledger tombstone the snapshot is built from survives until that snapshot is acknowledged. Cost: ledger growth for the length of an outage. Not covered: a peer restored from an old backup of its own file. No proto, `lib_schema_version` or on-disk schema change. Regression test `TombstoneResurrectionTests.PeerOfflinePastRetention_DeleteStillWinsOnBothNodes` (fails on 0.3.6 with the row resurrected, `A=1`) + positive control without the prune + pruned-after-delivery control; 5 `OplogStoreTests` with a controlled clock; every guard clause mutation-killed. 259/259. + ## Index history (migrated from the scadaproj root CLAUDE.md) > Moved verbatim out of the scadaproj root `CLAUDE.md` on 2026-09-03 so it loads only when needed. **This file is now the authoritative place for this history — append new entries here and keep the root index row short.** diff --git a/ZB.MOM.WW.LocalDb/Directory.Build.props b/ZB.MOM.WW.LocalDb/Directory.Build.props index 4552f01..6d4965f 100644 --- a/ZB.MOM.WW.LocalDb/Directory.Build.props +++ b/ZB.MOM.WW.LocalDb/Directory.Build.props @@ -5,7 +5,7 @@ enable enable latest - 0.3.6 + 0.3.7 true true