fix(localdb): apply upserts on the primary key, not INSERT OR REPLACE (scadaproj#38)

INSERT OR REPLACE resolved a secondary UNIQUE conflict by deleting a
different row (another PK) with capture suppressed and its ledger left live,
so a pair diverged silently (LocalDb-003). The applier now uses
INSERT ... ON CONFLICT(<pk>) DO UPDATE (DO NOTHING when every column is PK):
a clash on another unique key aborts the entry, which is dead-lettered, and
each node keeps its own row. Side effect: a remote update of a parent with
ON DELETE RESTRICT children now applies instead of being dead-lettered.

Also waits for the snapshot watermark in WipedPeer_SnapshotLiftsItsWatermark,
which raced SnapshotComplete under full-suite load.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
This commit is contained in:
Joseph Doherty
2026-11-09 14:58:34 -05:00
parent 4af0ea8152
commit 15a6c7c710
10 changed files with 250 additions and 22 deletions
+1
View File
@@ -9,6 +9,7 @@ Shared embedded SQLite cache + 2-node LWW sync library for the **ZB.MOM.WW SCADA
**`0.3.6` (unpublished, scadaproj#36–#39):** code-review fixes, all wire-compatible (no proto, `lib_schema_version` or on-disk schema change).
- **#36 LocalDb-001:** the passive `LocalDbSyncService` now enforces `LocalDb:Replication:ApiKey` itself (Bearer, constant-time, checked before the handshake and before newest-wins takeover; `Unauthenticated` otherwise). Blank key = unchanged behaviour plus a startup warning. ScadaBridge's and OtOpcUa's own `LocalDbSyncAuthInterceptor`s use the same key and header, so they keep working; they are now redundant except for their fail-closed-when-blank rule.
- **#37 LocalDb-002:** snapshots deliver FK-linked rows in any order. `SnapshotStreamer` sends referenced tables first (Kahn order, name tie-break); `SnapshotApplier` defers rows that fail only SQLITE_CONSTRAINT_FOREIGNKEY (`LwwApplier.ApplyBatchAsync(..., deferForeignKeyViolations: true)`), retries them as rows land, and dead-letters true orphans at `SnapshotComplete`. Before, a child that arrived first was dead-lettered and acked over permanently.
- **#38 LocalDb-003:** the applier upserts with `INSERT … ON CONFLICT(<pk>) DO UPDATE` (`DO NOTHING` when every column is PK) instead of `INSERT OR REPLACE`. A secondary-`UNIQUE` clash is now a dead-letter instead of a silent delete of another row; as a side effect a remote update of a parent with `ON DELETE RESTRICT` children applies instead of being dead-lettered.
## Index history (migrated from the scadaproj root CLAUDE.md)
+9 -1
View File
@@ -297,7 +297,8 @@ All options are validated at startup (`ZB.MOM.WW.Configuration` validators).
registration error.
- **A replicated table may not be the parent of a row-mutating `ON DELETE` foreign key**
(`CASCADE` / `SET NULL` / `SET DEFAULT` from any other table, or from itself). Applying a
remote row uses `INSERT OR REPLACE`, whose implicit delete fires that action with change
remote tombstone deletes the row (and up to `0.3.5` applying any remote row used
`INSERT OR REPLACE`, whose implicit delete did the same), which fires that action with change
capture suppressed — the child rows would change on the applying node only and never
replicate, diverging the pair undetectably. `RegisterReplicated` refuses such a table
(`0.3.3`); use `ON DELETE NO ACTION`/`RESTRICT` and delete children explicitly, or leave
@@ -313,6 +314,13 @@ All options are validated at startup (`ZB.MOM.WW.Configuration` validators).
receiver still loses self-referencing rows that arrive child-first, and a pre-`0.3.6`
sender's name order relies on the `0.3.6` receiver's deferral — so upgrade both nodes before
adding FK-linked tables. Deferred rows are held in memory until the snapshot completes.
- **Secondary `UNIQUE` keys must be unique across both nodes.** Remote rows are applied with
`INSERT … ON CONFLICT(<pk>) DO UPDATE` (`0.3.6`), so only the row with the same primary key
is touched. If the same natural key is created under different primary keys on the two nodes
(e.g. during a partition), each node keeps its own row and dead-letters the peer's — visible in
`__localdb_dead_letter`, but the pair holds different rows until the conflict is resolved by
hand. Up to `0.3.5` `INSERT OR REPLACE` silently deleted the local row instead and the pair
diverged undetectably. Prefer GUID or node-scoped natural keys.
- **CDC semantics** — rows that existed **before** `RegisterReplicated` are not captured
and are not snapshotted; only changes after registration are replicated. **Opt out with
`RegisterReplicated(table, baselineExistingRows: true)`**, which seeds the row-version
@@ -362,11 +362,27 @@ internal sealed class LwwApplier(SqliteLocalDb db, Func<DateTimeOffset>? utcNow
return $"DELETE FROM {Ident(table.Name)} WHERE {pkMatch}";
}
/// <summary>
/// An upsert keyed on the PRIMARY KEY only (LocalDb-003). <c>INSERT OR REPLACE</c> resolved
/// a conflict on ANY unique constraint by deleting the conflicting row — on a secondary
/// <c>UNIQUE</c> that is a different row (another primary key), removed with capture
/// suppressed, its ledger row left saying "live", and nothing shipped to the peer: silent
/// divergence. <c>ON CONFLICT(pk) DO UPDATE</c> touches only the row this entry names; a
/// clash on any other unique key aborts the statement, which the caller dead-letters, so it
/// is visible. Updating in place also stops a parent's update from deleting it first, which
/// an <c>ON DELETE RESTRICT</c> child refused.
/// </summary>
private static string BuildUpsert(ReplicatedTable table)
{
var columns = string.Join(", ", table.Columns.Select(c => Ident(c.Name)));
var values = string.Join(", ", table.Columns.Select(c => $"json_extract(@row, {JsonPath(c.Name)})"));
return $"INSERT OR REPLACE INTO {Ident(table.Name)} ({columns}) VALUES ({values})";
var pk = string.Join(", ", table.PkColumns.Select(Ident));
var nonPk = table.Columns
.Where(c => !table.PkColumns.Contains(c.Name, StringComparer.OrdinalIgnoreCase))
.Select(c => $"{Ident(c.Name)} = excluded.{Ident(c.Name)}")
.ToList();
var action = nonPk.Count == 0 ? "DO NOTHING" : "DO UPDATE SET " + string.Join(", ", nonPk);
return $"INSERT INTO {Ident(table.Name)} ({columns}) VALUES ({values}) ON CONFLICT ({pk}) {action}";
}
// Local copy of TriggerSqlGenerator's convention (that helper lives in the core assembly).
@@ -44,13 +44,20 @@ public interface ILocalDb
/// </summary>
/// <remarks>
/// <b>Replicated tables may not be the parent of a cascading foreign key.</b> The apply path
/// writes remote rows with <c>INSERT OR REPLACE</c>; REPLACE deletes the conflicting row first,
/// and with <c>PRAGMA foreign_keys=ON</c> that delete fires the child's <c>ON DELETE</c> action
/// while change capture is suppressed. The child rows would be mutated on the applying node only,
/// deletes rows for remote tombstones (and, before 0.3.6, wrote remote rows with
/// <c>INSERT OR REPLACE</c>, which deleted the conflicting row first); with
/// <c>PRAGMA foreign_keys=ON</c> that delete fires the child's <c>ON DELETE</c> action while
/// change capture is suppressed. The child rows would be mutated on the applying node only,
/// never captured and never shipped, diverging the pair with nothing able to detect it. The check
/// runs at registration and inspects the whole database's foreign keys; SQLite cannot add a
/// foreign key to an existing table, so a later-created child table is caught on the next
/// registration (i.e. the next startup), not at the moment of the DDL.
/// <para>
/// <b>Secondary <c>UNIQUE</c> keys must be unique across BOTH nodes.</b> Since 0.3.6 a remote row
/// is upserted on its primary key only; if it collides with a different local row on another
/// unique column, the remote row is dead-lettered (the local row is kept) and the pair holds
/// different rows until someone resolves it. Earlier versions silently deleted the local row.
/// </para>
/// </remarks>
ReplicatedTable RegisterReplicated(string tableName);
@@ -301,10 +301,11 @@ internal sealed class SqliteLocalDb : ILocalDb, IDisposable
/// </summary>
/// <remarks>
/// <para>
/// <b>Why this is a replication hazard, not a schema opinion.</b> The apply path writes a remote
/// row with <c>INSERT OR REPLACE</c>, and REPLACE resolves a primary-key conflict by DELETING the
/// existing row before inserting the new one. Connections run <c>PRAGMA foreign_keys=ON</c>, and
/// that implicit delete fires foreign-key actions (verified behaviour, not inference). It runs
/// <b>Why this is a replication hazard, not a schema opinion.</b> The apply path deletes the row
/// for a remote tombstone (and up to 0.3.5 wrote remote rows with <c>INSERT OR REPLACE</c>, whose
/// primary-key conflict resolution DELETES the existing row first). Connections run
/// <c>PRAGMA foreign_keys=ON</c>, and such a delete fires foreign-key actions (verified behaviour,
/// not inference). It runs
/// with the capture guard raised, so the resulting child-row deletes/nulls are invisible to the
/// oplog: they happen on this node and are never shipped. The peer, applying the same parent row,
/// keeps its children — and the pair has silently diverged with nothing to detect it, since LWW
@@ -348,8 +349,8 @@ internal sealed class SqliteLocalDb : ILocalDb, IDisposable
throw new LocalDbRegistrationException(
$"Cannot register table '{tableName}': it is the parent of a foreign key with a row-mutating " +
$"ON DELETE action ({string.Join(", ", offenders)}). Replication applies remote rows with " +
"INSERT OR REPLACE, whose implicit delete fires that action with change capture suppressed — " +
$"ON DELETE action ({string.Join(", ", offenders)}). Replication deletes rows for remote " +
"tombstones, which fires that action with change capture suppressed — " +
"the child rows would be changed on this node only and never replicated, diverging the pair " +
"undetectably. Use ON DELETE NO ACTION/RESTRICT and delete children explicitly, or leave this " +
"table unreplicated.");
@@ -0,0 +1,170 @@
using System.Threading.Channels;
using Microsoft.Extensions.Logging.Abstractions;
using ZB.MOM.WW.LocalDb.Contracts;
using ZB.MOM.WW.LocalDb.Internal;
using ZB.MOM.WW.LocalDb.Replication;
using ZB.MOM.WW.LocalDb.Replication.Internal;
namespace ZB.MOM.WW.LocalDb.Tests;
/// <summary>
/// LocalDb-003 (scadaproj#38): the apply path upserts on the PRIMARY KEY only. <c>INSERT OR REPLACE</c>
/// also resolved a secondary <c>UNIQUE</c> conflict by deleting the OTHER row — a different primary
/// key, with no capture, no ledger change and nothing shipped — so a pair diverged silently.
/// </summary>
public sealed class ApplyUpsertTests : IDisposable
{
private static readonly TimeSpan RunTimeout = TimeSpan.FromSeconds(15);
private readonly List<string> _paths = [];
private readonly List<IDisposable> _disposables = [];
public void Dispose()
{
foreach (var d in _disposables)
d.Dispose();
Microsoft.Data.Sqlite.SqliteConnection.ClearAllPools();
foreach (var p in _paths)
{
if (File.Exists(p)) File.Delete(p);
if (File.Exists(p + "-wal")) File.Delete(p + "-wal");
if (File.Exists(p + "-shm")) File.Delete(p + "-shm");
}
}
private sealed record Side(SqliteLocalDb Db, SyncSession Session);
private async Task<SqliteLocalDb> NewDbAsync(params string[] ddl)
{
var path = Path.Combine(Path.GetTempPath(), Guid.NewGuid() + ".db");
_paths.Add(path);
var db = new SqliteLocalDb(new LocalDbOptions { Path = path });
_disposables.Add(db);
foreach (var statement in ddl)
await db.ExecuteAsync(statement);
return db;
}
private static Side SessionOver(SqliteLocalDb db)
{
var options = new ReplicationOptions { FlushInterval = TimeSpan.FromMilliseconds(20) };
var store = new OplogStore(db, options);
var applier = new LwwApplier(db);
var session = new SyncSession(db, store, applier, options, NullLogger.Instance);
session.SnapshotSender = new SnapshotStreamer(db, store, options, NullLogger.Instance).SendAsync;
session.SnapshotApplier = new SnapshotApplier(db, applier, store, NullLogger.Instance);
return new Side(db, session);
}
private static (SyncDuplex A, SyncDuplex B) DuplexPair()
{
var aToB = Channel.CreateUnbounded<SyncMessage>();
var bToA = Channel.CreateUnbounded<SyncMessage>();
return (
new SyncDuplex { Send = (m, ct) => aToB.Writer.WriteAsync(m, ct).AsTask(), Inbox = bToA.Reader },
new SyncDuplex { Send = (m, ct) => bToA.Writer.WriteAsync(m, ct).AsTask(), Inbox = aToB.Reader });
}
private static async Task<List<long>> Ids(SqliteLocalDb db, string table) =>
(await db.QueryAsync($"SELECT id FROM {table} ORDER BY id", r => r.GetInt64(0))).ToList();
private static async Task<long> DeadLetterCount(SqliteLocalDb db) =>
(await db.QueryAsync("SELECT COUNT(*) FROM __localdb_dead_letter", r => r.GetInt64(0)))[0];
private static async Task<List<string>> LiveLedger(SqliteLocalDb db, string table) =>
(await db.QueryAsync(
"SELECT pk_json FROM __localdb_row_version WHERE table_name = @t AND is_tombstone = 0 ORDER BY pk_json",
r => r.GetString(0), new { t = table })).ToList();
private static async Task WaitForAsync(Func<Task<bool>> predicate, TimeSpan timeout)
{
var deadline = DateTime.UtcNow + timeout;
while (DateTime.UtcNow < deadline)
{
if (await predicate()) return;
await Task.Delay(20);
}
throw new TimeoutException("Condition not reached within " + timeout);
}
private static async Task SwallowAsync(Task task)
{
try { await task; }
catch (OperationCanceledException) { }
}
[Fact]
public async Task SecondaryUniqueConflict_DoesNotDeleteAnotherRow_AndIsDeadLettered()
{
const string ddl = "CREATE TABLE t (id INTEGER PRIMARY KEY, code TEXT UNIQUE)";
var dbA = await NewDbAsync(ddl);
var dbB = await NewDbAsync(ddl);
dbA.RegisterReplicated("t");
dbB.RegisterReplicated("t");
// The same natural key created on each node during a partition.
await dbA.ExecuteAsync("INSERT INTO t (id, code) VALUES (1, 'X')");
await dbB.ExecuteAsync("INSERT INTO t (id, code) VALUES (2, 'X')");
var a = SessionOver(dbA);
var b = SessionOver(dbB);
var (da, dbx) = DuplexPair();
using var cts = new CancellationTokenSource(RunTimeout);
var runA = a.Session.RunAsync(da, cts.Token);
var runB = b.Session.RunAsync(dbx, cts.Token);
await WaitForAsync(async () => await DeadLetterCount(dbA) == 1 && await DeadLetterCount(dbB) == 1, RunTimeout);
// Each node keeps its own row: nothing was deleted behind the ledger's back, and the
// conflict is visible as a dead-letter instead of a silent loss.
Assert.Equal(new[] { 1L }, await Ids(dbA, "t"));
Assert.Equal(new[] { 2L }, await Ids(dbB, "t"));
Assert.Equal(new[] { "{\"id\":1}" }, await LiveLedger(dbA, "t"));
Assert.Equal(new[] { "{\"id\":2}" }, await LiveLedger(dbB, "t"));
cts.Cancel();
await SwallowAsync(runA);
await SwallowAsync(runB);
}
[Fact]
public async Task RemoteUpdate_OfAParentWithRestrictChildren_Applies()
{
// REPLACE deleted the parent before re-inserting it, which an ON DELETE RESTRICT child
// refuses outright — so every remote update of such a parent was dead-lettered. An upsert
// updates in place.
string[] ddl =
[
"CREATE TABLE parent (id INTEGER PRIMARY KEY, name TEXT)",
"CREATE TABLE child (id INTEGER PRIMARY KEY, parent_id INTEGER REFERENCES parent(id) ON DELETE RESTRICT)",
];
var db = await NewDbAsync(ddl);
db.RegisterReplicated("parent");
db.RegisterReplicated("child");
await db.ExecuteAsync("INSERT INTO parent (id, name) VALUES (1, 'old')");
await db.ExecuteAsync("INSERT INTO child (id, parent_id) VALUES (10, 1)");
var result = await new LwwApplier(db).ApplyBatchAsync(
[new OplogEntryRecord(1, "parent", "{\"id\":1}", "{\"id\":1,\"name\":\"new\"}", long.MaxValue / 2, "peer", false)],
CancellationToken.None);
Assert.Equal(1, result.Applied);
Assert.Equal(0, result.DeadLettered);
var name = (await db.QueryAsync("SELECT name FROM parent WHERE id = 1", r => r.GetString(0)))[0];
Assert.Equal("new", name);
}
[Fact]
public async Task TableWhosePrimaryKeyIsEveryColumn_StillApplies()
{
var db = await NewDbAsync("CREATE TABLE link (a INTEGER, b INTEGER, PRIMARY KEY (a, b))");
db.RegisterReplicated("link");
var applier = new LwwApplier(db);
var entry = new OplogEntryRecord(1, "link", "{\"a\":1,\"b\":2}", "{\"a\":1,\"b\":2}", 1000, "peer", false);
Assert.Equal(1, (await applier.ApplyBatchAsync([entry], CancellationToken.None)).Applied);
// Re-applying a newer version of the same key is a conflict on the PK with nothing to update.
Assert.Equal(1, (await applier.ApplyBatchAsync([entry with { Hlc = 2000 }], CancellationToken.None)).Applied);
Assert.Equal(1L, (await db.QueryAsync("SELECT COUNT(*) FROM link", r => r.GetInt64(0)))[0]);
}
}
@@ -73,7 +73,8 @@ public sealed class RegistrationTests : IDisposable
[Fact]
public async Task Register_CascadeDeleteParent_Throws()
{
// The apply path writes remote rows with INSERT OR REPLACE, whose implicit delete fires
// The apply path deletes rows for remote tombstones (and up to 0.3.5 wrote every remote row
// with INSERT OR REPLACE, whose implicit delete did the same); that delete fires
// ON DELETE CASCADE with capture suppressed — the child rows would die on this node only.
using var db = Create();
await CreateOrders(db);
@@ -317,7 +317,11 @@ public sealed class SnapshotResyncTests : IDisposable
{
var run1A = a.Session.RunAsync(d1A, cts1.Token);
var run1B = b.Session.RunAsync(d1B, cts1.Token);
await WaitForAsync(async () => (await ReadOrders(b.Db)).Count == 3, RunTimeout);
// Rows land per SnapshotBatch; the watermark moves on the later SnapshotComplete. Waiting
// on the rows alone let cts1 cancel the session between the two (an intermittent failure
// of the assert below under full-suite load), so wait for the watermark to move at all.
await WaitForAsync(
async () => (await ReadOrders(b.Db)).Count == 3 && await LastAppliedRemoteSeq(b.Db) > 0, RunTimeout);
cts1.Cancel();
await SwallowAsync(run1A);
await SwallowAsync(run1B);
+24 -3
View File
@@ -10,7 +10,7 @@
| Last reviewed | 2026-11-09 |
| Reviewer | Claude |
| Commit reviewed | `91fb2b4` |
| Open findings | 5 |
| Open findings | 4 |
## Summary
@@ -231,7 +231,7 @@ README §Limitations documents the mixed-version caveat: a pre-0.3.6 receiver st
| Severity | Medium |
| Priority | P2 |
| Category | Correctness & logic bugs |
| Status | Open |
| Status | Resolved |
| Location | `ZB.MOM.WW.LocalDb/src/ZB.MOM.WW.LocalDb.Replication/Internal/LwwApplier.cs:340` |
**Description**
@@ -273,7 +273,28 @@ that secondary unique keys must be globally unique across both nodes.
**Resolution**
_Unresolved._
Resolved 2026-11-09 (scadaproj#38, LocalDb `0.3.6`, unpublished). Re-verified with a 2-node test that runs the
review's partition scenario: A ended with only row 2 and B with only row 1, and there were no dead-letters. The
applier now upserts on the primary key only, via `LwwApplier.BuildUpsert`:
`INSERT … ON CONFLICT(<pk cols>) DO UPDATE SET <non-pk> = excluded.<non-pk>`, or `DO NOTHING` when every column is
part of the PK. A clash on any other unique key aborts the statement. The existing per-entry savepoint rolls that
entry back and dead-letters it, so each node keeps its own row and its ledger stays consistent with its table. The
divergence is now visible in `__localdb_dead_letter`. It is still not auto-resolved, which the docs state.
The review's other option was to refuse registration of tables with a secondary `UNIQUE`. It was not taken, because
it could break an existing consumer's boot.
The change also fixes a defect found while testing: a remote update of a parent that has `ON DELETE RESTRICT`
children used to be dead-lettered, because REPLACE deletes before it re-inserts. It now applies.
The `RegisterReplicated` and README docs now state that secondary unique keys must be unique across both nodes. The
cascading-FK registration rationale now cites tombstone deletes rather than REPLACE.
Tests (`ApplyUpsertTests`): the 2-node secondary-UNIQUE conflict and the RESTRICT-parent update both fail pre-fix.
The all-columns-PK table is a guard. The commit also hardens
`SnapshotResyncTests.WipedPeer_SnapshotLiftsItsWatermark_…`. That test cancelled after the rows landed but before
`SnapshotComplete` had moved the watermark, and it failed intermittently in one of 4 full-suite runs. It now also
waits for the watermark.
### LocalDb-004 — `FailClosedOnDrift` is bypassed on the snapshot path, so a future-stamped snapshot row moves the local HLC forward permanently
+5 -6
View File
@@ -40,7 +40,7 @@ code-reviews/
## Summary
10 of 10 libraries reviewed. 21 pending findings across all libraries.
10 of 10 libraries reviewed. 20 pending findings across all libraries.
Coverage: Secrets, LocalDb, GalaxyRepository and Overview first reviewed, and Auth, Health,
Theme, Audit, Configuration and Telemetry re-reviewed, on 2026-11-09 (#15). Each finding also
@@ -50,9 +50,9 @@ carries a P1/P2/P3 priority (P1 = Critical/High, P2 = Medium, P3 = Low).
|----------|---------------|
| Critical | 0 |
| High | 0 |
| Medium | 2 |
| Medium | 1 |
| Low | 19 |
| **Total** | **21** |
| **Total** | **20** |
## Library Status
@@ -63,7 +63,7 @@ carries a P1/P2/P3 priority (P1 = Critical/High, P2 = Medium, P3 = Low).
| [Configuration](Configuration/findings.md) | 2026-11-09 | `91fb2b4` | 0/0/0/0 | 0 | 4 |
| [GalaxyRepository](GalaxyRepository/findings.md) | 2026-11-09 | `91fb2b4` | 0/0/0/0 | 0 | 8 |
| [Health](Health/findings.md) | 2026-11-09 | `91fb2b4` | 0/0/0/0 | 0 | 7 |
| [LocalDb](LocalDb/findings.md) | 2026-11-09 | `91fb2b4` | 0/0/2/3 | 5 | 7 |
| [LocalDb](LocalDb/findings.md) | 2026-11-09 | `91fb2b4` | 0/0/1/3 | 4 | 7 |
| [Overview](Overview/findings.md) | 2026-11-09 | `91fb2b4` | 0/0/0/0 | 0 | 5 |
| [Secrets](Secrets/findings.md) | 2026-11-09 | `91fb2b4` | 0/0/0/13 | 13 | 19 |
| [Telemetry](Telemetry/findings.md) | 2026-11-09 | `91fb2b4` | 0/0/0/0 | 0 | 11 |
@@ -84,11 +84,10 @@ _None open._
_None open._
### Medium (2)
### Medium (1)
| ID | Library | Title |
|----|---------|-------|
| LocalDb-003 | [LocalDb](LocalDb/findings.md) | `INSERT OR REPLACE` apply silently deletes a different row on a secondary `UNIQUE` conflict, so the pair diverges |
| LocalDb-004 | [LocalDb](LocalDb/findings.md) | `FailClosedOnDrift` is bypassed on the snapshot path, so a future-stamped snapshot row moves the local HLC forward permanently |
### Low (19)