Joseph Doherty
b2adddbc8e
- SnapshotApplier now takes LocalDbMetrics (wired by SyncSessionFactory) and adds every row it dead-letters - poison, drift reject, FK orphan, retry failure, cap spill - to localdb.sync.dead_lettered. Before, only the delta path recorded the metric. - New ReplicationOptions.MaxSnapshotDeferredRows (default 100,000, validated > 0) bounds the rows held for a missing FK parent; past it the oldest get one more apply with deferral off and are dead-lettered if still orphaned. An older name-ordered sender could otherwise defer a whole child table. Unit + 2-node (SyncSessionFactory) tests fail pre-fix. No wire change. (scadaproj#50) Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
293 lines
13 KiB
C#
293 lines
13 KiB
C#
using System.Diagnostics.Metrics;
|
|
using System.Threading.Channels;
|
|
using Microsoft.Extensions.Logging.Abstractions;
|
|
using ZB.MOM.WW.LocalDb.Contracts;
|
|
using ZB.MOM.WW.LocalDb.Internal;
|
|
using ZB.MOM.WW.LocalDb.Replication;
|
|
using ZB.MOM.WW.LocalDb.Replication.Internal;
|
|
|
|
namespace ZB.MOM.WW.LocalDb.Tests;
|
|
|
|
/// <summary>
|
|
/// scadaproj#50, two follow-ups to the 0.3.6 snapshot fixes:
|
|
/// <list type="bullet">
|
|
/// <item>rows the SNAPSHOT path dead-letters (FK orphans from #37, drift rejects from #39, poison)
|
|
/// were never counted in <c>localdb.sync.dead_lettered</c> — only the delta path recorded it;</item>
|
|
/// <item>snapshot rows held back for a missing FK parent were kept in memory without bound, which
|
|
/// against an older child-first sender could be a whole table. They are now capped by
|
|
/// <see cref="ReplicationOptions.MaxSnapshotDeferredRows"/> and spilled to the dead-letter table
|
|
/// past it.</item>
|
|
/// </list>
|
|
/// </summary>
|
|
public sealed class SnapshotDeadLetterTests : IDisposable
|
|
{
|
|
private const string DeadLettered = "localdb.sync.dead_lettered";
|
|
private static readonly TimeSpan RunTimeout = TimeSpan.FromSeconds(15);
|
|
|
|
private readonly List<string> _paths = [];
|
|
private readonly List<IDisposable> _disposables = [];
|
|
|
|
public void Dispose()
|
|
{
|
|
foreach (var d in _disposables)
|
|
d.Dispose();
|
|
Microsoft.Data.Sqlite.SqliteConnection.ClearAllPools();
|
|
foreach (var p in _paths)
|
|
{
|
|
if (File.Exists(p)) File.Delete(p);
|
|
if (File.Exists(p + "-wal")) File.Delete(p + "-wal");
|
|
if (File.Exists(p + "-shm")) File.Delete(p + "-shm");
|
|
}
|
|
}
|
|
|
|
private async Task<SqliteLocalDb> NewDbAsync()
|
|
{
|
|
var path = Path.Combine(Path.GetTempPath(), Guid.NewGuid() + ".db");
|
|
_paths.Add(path);
|
|
var db = new SqliteLocalDb(new LocalDbOptions { Path = path });
|
|
_disposables.Add(db);
|
|
await db.ExecuteAsync("CREATE TABLE z_parent (id INTEGER PRIMARY KEY, name TEXT)");
|
|
await db.ExecuteAsync("CREATE TABLE a_child (id INTEGER PRIMARY KEY, parent_id INTEGER REFERENCES z_parent(id))");
|
|
db.RegisterReplicated("z_parent");
|
|
db.RegisterReplicated("a_child");
|
|
return db;
|
|
}
|
|
|
|
private static SnapshotApplier ApplierFor(SqliteLocalDb db, ReplicationOptions options, LocalDbMetrics metrics) =>
|
|
new(db, new LwwApplier(db), new OplogStore(db, options), NullLogger.Instance, options, utcNow: null, metrics);
|
|
|
|
private static SnapshotBatch Children(params (long Id, long ParentId)[] rows)
|
|
{
|
|
var batch = new SnapshotBatch { TableName = "a_child" };
|
|
foreach (var (id, parent) in rows)
|
|
batch.Rows.Add(new SnapshotRow
|
|
{
|
|
PkJson = $"{{\"id\":{id}}}", RowJson = $"{{\"id\":{id},\"parent_id\":{parent}}}", Hlc = 1000 + id, NodeId = "peer",
|
|
});
|
|
return batch;
|
|
}
|
|
|
|
private static SnapshotBatch Parent(long id)
|
|
{
|
|
var batch = new SnapshotBatch { TableName = "z_parent" };
|
|
batch.Rows.Add(new SnapshotRow { PkJson = $"{{\"id\":{id}}}", RowJson = $"{{\"id\":{id},\"name\":\"p\"}}", Hlc = 1000 + id, NodeId = "peer" });
|
|
return batch;
|
|
}
|
|
|
|
private static async Task<long> Scalar(SqliteLocalDb db, string sql) =>
|
|
(await db.QueryAsync(sql, r => r.GetInt64(0)))[0];
|
|
|
|
private static Task<long> DeadLetterRows(SqliteLocalDb db) => Scalar(db, "SELECT COUNT(*) FROM __localdb_dead_letter");
|
|
|
|
private static async Task WaitForAsync(Func<Task<bool>> predicate, TimeSpan timeout)
|
|
{
|
|
var deadline = DateTime.UtcNow + timeout;
|
|
while (DateTime.UtcNow < deadline)
|
|
{
|
|
if (await predicate()) return;
|
|
await Task.Delay(20);
|
|
}
|
|
throw new TimeoutException("Condition not reached within " + timeout);
|
|
}
|
|
|
|
private static async Task SwallowAsync(Task task)
|
|
{
|
|
try { await task; }
|
|
catch (OperationCanceledException) { }
|
|
}
|
|
|
|
// ---- metric ---------------------------------------------------------------------------------
|
|
|
|
[Fact]
|
|
public async Task SnapshotOrphan_IsCountedInTheDeadLetterMetric()
|
|
{
|
|
using var metrics = new LocalDbMetrics();
|
|
using var collector = new Collector(metrics.Meter);
|
|
var db = await NewDbAsync();
|
|
var applier = ApplierFor(db, new ReplicationOptions(), metrics);
|
|
|
|
await applier.OnBeginAsync(new SnapshotBegin { AsOfSeq = 5 }, default);
|
|
await applier.OnBatchAsync(Children((10, 99)), default);
|
|
await applier.OnCompleteAsync(new SnapshotComplete { AsOfSeq = 5 }, default);
|
|
|
|
Assert.Equal(1, await DeadLetterRows(db));
|
|
Assert.Equal(1, collector.Sum(DeadLettered));
|
|
}
|
|
|
|
[Fact]
|
|
public async Task SnapshotDriftReject_IsCountedInTheDeadLetterMetric()
|
|
{
|
|
using var metrics = new LocalDbMetrics();
|
|
using var collector = new Collector(metrics.Meter);
|
|
var db = await NewDbAsync();
|
|
var applier = ApplierFor(db, new ReplicationOptions { FailClosedOnDrift = true }, metrics);
|
|
|
|
var batch = Parent(1);
|
|
batch.Rows[0].Hlc = DateTimeOffset.UtcNow.AddDays(365).ToUnixTimeMilliseconds() << 16;
|
|
await applier.OnBeginAsync(new SnapshotBegin { AsOfSeq = 5 }, default);
|
|
await applier.OnBatchAsync(batch, default);
|
|
await applier.OnCompleteAsync(new SnapshotComplete { AsOfSeq = 5 }, default);
|
|
|
|
Assert.Equal(1, await DeadLetterRows(db));
|
|
Assert.Equal(1, collector.Sum(DeadLettered));
|
|
}
|
|
|
|
[Fact]
|
|
public async Task SnapshotPoisonRow_IsCountedInTheDeadLetterMetric()
|
|
{
|
|
using var metrics = new LocalDbMetrics();
|
|
using var collector = new Collector(metrics.Meter);
|
|
var db = await NewDbAsync();
|
|
var applier = ApplierFor(db, new ReplicationOptions(), metrics);
|
|
|
|
// A live row with no row data is poison on any path.
|
|
var batch = new SnapshotBatch { TableName = "z_parent" };
|
|
batch.Rows.Add(new SnapshotRow { PkJson = "{\"id\":1}", Hlc = 1001, NodeId = "peer" });
|
|
await applier.OnBeginAsync(new SnapshotBegin { AsOfSeq = 5 }, default);
|
|
await applier.OnBatchAsync(batch, default);
|
|
await applier.OnCompleteAsync(new SnapshotComplete { AsOfSeq = 5 }, default);
|
|
|
|
Assert.Equal(1, await DeadLetterRows(db));
|
|
Assert.Equal(1, collector.Sum(DeadLettered));
|
|
}
|
|
|
|
[Fact]
|
|
public async Task TwoNodes_SnapshotOrphan_IsCountedByTheProductionWiring()
|
|
{
|
|
// The sender holds an orphan child (written with FK enforcement off on that connection), so a
|
|
// wiped peer receives a row whose parent never arrives. Both sessions are built by
|
|
// SyncSessionFactory, exactly as the hosts build them, so this also proves the factory hands
|
|
// the metrics to the snapshot applier.
|
|
var options = new ReplicationOptions { FlushInterval = TimeSpan.FromMilliseconds(20) };
|
|
var dbA = await NewDbAsync();
|
|
await dbA.ExecuteAsync("INSERT INTO z_parent (id, name) VALUES (1, 'p')");
|
|
await dbA.ExecuteAsync("INSERT INTO a_child (id, parent_id) VALUES (10, 1)");
|
|
using (var conn = dbA.CreateConnection())
|
|
{
|
|
using var cmd = conn.CreateCommand();
|
|
cmd.CommandText = "PRAGMA foreign_keys=OFF; INSERT INTO a_child (id, parent_id) VALUES (11, 99);";
|
|
cmd.ExecuteNonQuery();
|
|
}
|
|
var storeA = new OplogStore(dbA, options);
|
|
await storeA.RecordPeerAckAsync(await storeA.GetMaxSeqAsync());
|
|
|
|
var dbB = await NewDbAsync();
|
|
using var metricsB = new LocalDbMetrics();
|
|
using var collector = new Collector(metricsB.Meter);
|
|
|
|
var a = SyncSessionFactory.Create(dbA, options, NullLoggerFactory.Instance);
|
|
var b = SyncSessionFactory.Create(dbB, options, NullLoggerFactory.Instance, metricsB);
|
|
var aToB = Channel.CreateUnbounded<SyncMessage>();
|
|
var bToA = Channel.CreateUnbounded<SyncMessage>();
|
|
var da = new SyncDuplex { Send = (m, ct) => aToB.Writer.WriteAsync(m, ct).AsTask(), Inbox = bToA.Reader };
|
|
var dbx = new SyncDuplex { Send = (m, ct) => bToA.Writer.WriteAsync(m, ct).AsTask(), Inbox = aToB.Reader };
|
|
using var cts = new CancellationTokenSource(RunTimeout);
|
|
var runA = a.RunAsync(da, cts.Token);
|
|
var runB = b.RunAsync(dbx, cts.Token);
|
|
|
|
await WaitForAsync(async () =>
|
|
await Scalar(dbB, "SELECT last_applied_remote_seq FROM __localdb_peer_state") > 0, RunTimeout);
|
|
|
|
Assert.Equal(1, await Scalar(dbB, "SELECT COUNT(*) FROM a_child"));
|
|
Assert.Equal(1, await DeadLetterRows(dbB));
|
|
Assert.Equal(1, collector.Sum(DeadLettered));
|
|
|
|
cts.Cancel();
|
|
await SwallowAsync(runA);
|
|
await SwallowAsync(runB);
|
|
}
|
|
|
|
// ---- cap on held-back rows -------------------------------------------------------------------
|
|
|
|
[Fact]
|
|
public async Task DeferredRows_PastTheCap_AreSpilledToDeadLetter_BeforeComplete()
|
|
{
|
|
using var metrics = new LocalDbMetrics();
|
|
using var collector = new Collector(metrics.Meter);
|
|
var db = await NewDbAsync();
|
|
var applier = ApplierFor(db, new ReplicationOptions { MaxSnapshotDeferredRows = 2 }, metrics);
|
|
|
|
await applier.OnBeginAsync(new SnapshotBegin { AsOfSeq = 5 }, default);
|
|
await applier.OnBatchAsync(Children((10, 1), (11, 1)), default);
|
|
// At the cap, nothing is spilled yet.
|
|
Assert.Equal(0, await DeadLetterRows(db));
|
|
Assert.Equal(2, applier.DeferredCount);
|
|
|
|
// One more held-back row: the OLDEST (10) is spilled, so memory stays at the cap.
|
|
await applier.OnBatchAsync(Children((12, 1)), default);
|
|
Assert.Equal(2, applier.DeferredCount);
|
|
Assert.Equal(1, await DeadLetterRows(db));
|
|
Assert.Equal(1, await Scalar(db, "SELECT COUNT(*) FROM __localdb_dead_letter WHERE pk_json = '{\"id\":10}'"));
|
|
Assert.Equal(1, collector.Sum(DeadLettered));
|
|
|
|
// The rows still held land when their parent arrives; the spilled one stays dead-lettered.
|
|
await applier.OnBatchAsync(Parent(1), default);
|
|
await applier.OnCompleteAsync(new SnapshotComplete { AsOfSeq = 5 }, default);
|
|
|
|
Assert.Equal(0, applier.DeferredCount);
|
|
Assert.Equal(new long[] { 11, 12 },
|
|
(await db.QueryAsync("SELECT id FROM a_child ORDER BY id", r => r.GetInt64(0))).ToArray());
|
|
Assert.Equal(1, await DeadLetterRows(db));
|
|
Assert.Equal(1, collector.Sum(DeadLettered));
|
|
}
|
|
|
|
[Fact]
|
|
public async Task DeferredRows_PastTheCap_ThatHaveBecomeValid_AreAppliedNotDeadLettered()
|
|
{
|
|
// The spill re-applies with deferral off first, so a row whose parent HAS landed by then is
|
|
// applied rather than dead-lettered.
|
|
var db = await NewDbAsync();
|
|
using var metrics = new LocalDbMetrics();
|
|
var applier = ApplierFor(db, new ReplicationOptions { MaxSnapshotDeferredRows = 1 }, metrics);
|
|
|
|
await applier.OnBeginAsync(new SnapshotBegin { AsOfSeq = 5 }, default);
|
|
await applier.OnBatchAsync(Children((10, 1)), default);
|
|
// Insert the parent locally outside the snapshot, so nothing in the applier retries row 10.
|
|
await db.ExecuteAsync("INSERT INTO z_parent (id, name) VALUES (1, 'local')");
|
|
await applier.OnBatchAsync(Children((11, 2)), default);
|
|
|
|
Assert.Equal(0, await DeadLetterRows(db));
|
|
Assert.Equal(1, await Scalar(db, "SELECT COUNT(*) FROM a_child WHERE id = 10"));
|
|
Assert.Equal(1, applier.DeferredCount);
|
|
}
|
|
|
|
[Fact]
|
|
public void Validator_RejectsANonPositiveCap()
|
|
{
|
|
var result = new ReplicationOptionsValidator().Validate(null, new ReplicationOptions { MaxSnapshotDeferredRows = 0 });
|
|
Assert.True(result.Failed);
|
|
Assert.Contains(result.Failures!, f => f.Contains("MaxSnapshotDeferredRows", StringComparison.Ordinal));
|
|
}
|
|
|
|
private sealed class Collector : IDisposable
|
|
{
|
|
private readonly MeterListener _listener;
|
|
private readonly Dictionary<string, long> _sums = new(StringComparer.Ordinal);
|
|
private readonly Lock _lock = new();
|
|
|
|
public Collector(Meter meter)
|
|
{
|
|
_listener = new MeterListener
|
|
{
|
|
InstrumentPublished = (inst, l) =>
|
|
{
|
|
if (ReferenceEquals(inst.Meter, meter))
|
|
l.EnableMeasurementEvents(inst);
|
|
},
|
|
};
|
|
_listener.SetMeasurementEventCallback<long>((inst, val, _, _) =>
|
|
{
|
|
lock (_lock) _sums[inst.Name] = _sums.GetValueOrDefault(inst.Name) + val;
|
|
});
|
|
_listener.Start();
|
|
}
|
|
|
|
public long Sum(string name)
|
|
{
|
|
lock (_lock) return _sums.GetValueOrDefault(name);
|
|
}
|
|
|
|
public void Dispose() => _listener.Dispose();
|
|
}
|
|
}
|