fix(localdb): count snapshot dead-letters and cap held-back snapshot rows

- SnapshotApplier now takes LocalDbMetrics (wired by SyncSessionFactory) and
  adds every row it dead-letters - poison, drift reject, FK orphan, retry
  failure, cap spill - to localdb.sync.dead_lettered. Before, only the delta
  path recorded the metric.
- New ReplicationOptions.MaxSnapshotDeferredRows (default 100,000, validated
  > 0) bounds the rows held for a missing FK parent; past it the oldest get
  one more apply with deferral off and are dead-lettered if still orphaned.
  An older name-ordered sender could otherwise defer a whole child table.

Unit + 2-node (SyncSessionFactory) tests fail pre-fix. No wire change.

(scadaproj#50)

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
This commit is contained in:
Joseph Doherty
2026-11-09 15:12:41 -05:00
parent fd46999946
commit b2adddbc8e
7 changed files with 376 additions and 7 deletions
@@ -103,6 +103,28 @@ public sealed class ReplicationOptions
/// <summary>How often the maintenance service enforces the oplog caps and prunes expired tombstones.</summary>
public TimeSpan MaintenanceInterval { get; set; } = TimeSpan.FromSeconds(60);
/// <summary>
/// Ceiling on the snapshot rows a receiver holds in memory while their foreign-key parent has not
/// arrived yet. Default 100,000. Past it the OLDEST held rows are applied once more and, if their
/// parent is still missing, dead-lettered.
/// </summary>
/// <remarks>
/// <para>
/// Since 0.3.6 a snapshot row that fails only its FK check is deferred and retried as the rest of
/// the snapshot lands, instead of being dead-lettered. A 0.3.6 sender streams parent tables first,
/// so in practice only self-referencing rows whose parent sorts later wait; but an OLDER sender
/// streams tables by name, and a child table that sorts before its parent then defers every one of
/// its rows. This bounds that memory.
/// </para>
/// <para>
/// <b>A spilled row is lost to this snapshot</b>, exactly like an orphan dead-lettered at
/// <c>SnapshotComplete</c>: the sender's debt clears on the ack, so it is not re-sent. It is
/// visible in <c>__localdb_dead_letter</c> and in <c>localdb.sync.dead_lettered</c>. Upgrade the
/// sender (parents first) or raise this cap if spills appear.
/// </para>
/// </remarks>
public int MaxSnapshotDeferredRows { get; set; } = 100_000;
/// <summary>Maximum tolerated HLC drift where a peer's clock runs ahead of ours.</summary>
public TimeSpan MaxHlcDriftAhead { get; set; } = TimeSpan.FromMinutes(5);