fix(localdb): apply upserts on the primary key, not INSERT OR REPLACE (scadaproj#38)

INSERT OR REPLACE resolved a secondary UNIQUE conflict by deleting a
different row (another PK) with capture suppressed and its ledger left live,
so a pair diverged silently (LocalDb-003). The applier now uses
INSERT ... ON CONFLICT(<pk>) DO UPDATE (DO NOTHING when every column is PK):
a clash on another unique key aborts the entry, which is dead-lettered, and
each node keeps its own row. Side effect: a remote update of a parent with
ON DELETE RESTRICT children now applies instead of being dead-lettered.

Also waits for the snapshot watermark in WipedPeer_SnapshotLiftsItsWatermark,
which raced SnapshotComplete under full-suite load.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
This commit is contained in:
Joseph Doherty
2026-11-09 14:58:34 -05:00
parent 4af0ea8152
commit 15a6c7c710
10 changed files with 250 additions and 22 deletions
@@ -362,11 +362,27 @@ internal sealed class LwwApplier(SqliteLocalDb db, Func<DateTimeOffset>? utcNow
return $"DELETE FROM {Ident(table.Name)} WHERE {pkMatch}";
}
/// <summary>
/// An upsert keyed on the PRIMARY KEY only (LocalDb-003). <c>INSERT OR REPLACE</c> resolved
/// a conflict on ANY unique constraint by deleting the conflicting row — on a secondary
/// <c>UNIQUE</c> that is a different row (another primary key), removed with capture
/// suppressed, its ledger row left saying "live", and nothing shipped to the peer: silent
/// divergence. <c>ON CONFLICT(pk) DO UPDATE</c> touches only the row this entry names; a
/// clash on any other unique key aborts the statement, which the caller dead-letters, so it
/// is visible. Updating in place also stops a parent's update from deleting it first, which
/// an <c>ON DELETE RESTRICT</c> child refused.
/// </summary>
private static string BuildUpsert(ReplicatedTable table)
{
var columns = string.Join(", ", table.Columns.Select(c => Ident(c.Name)));
var values = string.Join(", ", table.Columns.Select(c => $"json_extract(@row, {JsonPath(c.Name)})"));
return $"INSERT OR REPLACE INTO {Ident(table.Name)} ({columns}) VALUES ({values})";
var pk = string.Join(", ", table.PkColumns.Select(Ident));
var nonPk = table.Columns
.Where(c => !table.PkColumns.Contains(c.Name, StringComparer.OrdinalIgnoreCase))
.Select(c => $"{Ident(c.Name)} = excluded.{Ident(c.Name)}")
.ToList();
var action = nonPk.Count == 0 ? "DO NOTHING" : "DO UPDATE SET " + string.Join(", ", nonPk);
return $"INSERT INTO {Ident(table.Name)} ({columns}) VALUES ({values}) ON CONFLICT ({pk}) {action}";
}
// Local copy of TriggerSqlGenerator's convention (that helper lives in the core assembly).
@@ -44,13 +44,20 @@ public interface ILocalDb
/// </summary>
/// <remarks>
/// <b>Replicated tables may not be the parent of a cascading foreign key.</b> The apply path
/// writes remote rows with <c>INSERT OR REPLACE</c>; REPLACE deletes the conflicting row first,
/// and with <c>PRAGMA foreign_keys=ON</c> that delete fires the child's <c>ON DELETE</c> action
/// while change capture is suppressed. The child rows would be mutated on the applying node only,
/// deletes rows for remote tombstones (and, before 0.3.6, wrote remote rows with
/// <c>INSERT OR REPLACE</c>, which deleted the conflicting row first); with
/// <c>PRAGMA foreign_keys=ON</c> that delete fires the child's <c>ON DELETE</c> action while
/// change capture is suppressed. The child rows would be mutated on the applying node only,
/// never captured and never shipped, diverging the pair with nothing able to detect it. The check
/// runs at registration and inspects the whole database's foreign keys; SQLite cannot add a
/// foreign key to an existing table, so a later-created child table is caught on the next
/// registration (i.e. the next startup), not at the moment of the DDL.
/// <para>
/// <b>Secondary <c>UNIQUE</c> keys must be unique across BOTH nodes.</b> Since 0.3.6 a remote row
/// is upserted on its primary key only; if it collides with a different local row on another
/// unique column, the remote row is dead-lettered (the local row is kept) and the pair holds
/// different rows until someone resolves it. Earlier versions silently deleted the local row.
/// </para>
/// </remarks>
ReplicatedTable RegisterReplicated(string tableName);
@@ -301,10 +301,11 @@ internal sealed class SqliteLocalDb : ILocalDb, IDisposable
/// </summary>
/// <remarks>
/// <para>
/// <b>Why this is a replication hazard, not a schema opinion.</b> The apply path writes a remote
/// row with <c>INSERT OR REPLACE</c>, and REPLACE resolves a primary-key conflict by DELETING the
/// existing row before inserting the new one. Connections run <c>PRAGMA foreign_keys=ON</c>, and
/// that implicit delete fires foreign-key actions (verified behaviour, not inference). It runs
/// <b>Why this is a replication hazard, not a schema opinion.</b> The apply path deletes the row
/// for a remote tombstone (and up to 0.3.5 wrote remote rows with <c>INSERT OR REPLACE</c>, whose
/// primary-key conflict resolution DELETES the existing row first). Connections run
/// <c>PRAGMA foreign_keys=ON</c>, and such a delete fires foreign-key actions (verified behaviour,
/// not inference). It runs
/// with the capture guard raised, so the resulting child-row deletes/nulls are invisible to the
/// oplog: they happen on this node and are never shipped. The peer, applying the same parent row,
/// keeps its children — and the pair has silently diverged with nothing to detect it, since LWW
@@ -348,8 +349,8 @@ internal sealed class SqliteLocalDb : ILocalDb, IDisposable
throw new LocalDbRegistrationException(
$"Cannot register table '{tableName}': it is the parent of a foreign key with a row-mutating " +
$"ON DELETE action ({string.Join(", ", offenders)}). Replication applies remote rows with " +
"INSERT OR REPLACE, whose implicit delete fires that action with change capture suppressed — " +
$"ON DELETE action ({string.Join(", ", offenders)}). Replication deletes rows for remote " +
"tombstones, which fires that action with change capture suppressed — " +
"the child rows would be changed on this node only and never replicated, diverging the pair " +
"undetectably. Use ON DELETE NO ACTION/RESTRICT and delete children explicitly, or leave this " +
"table unreplicated.");