perf(localdb): dereg API, O(1) backlog depth, byte-budget batching, prepare-once apply (0.2.0)
WP3.3 of the ScadaBridge architectural-review remediation. Six changes, all wire-compatible with 0.1.x peers — no proto change, no lib_schema_version bump. The on-disk bookkeeping schema goes to v2 and is upgraded in place when a file is opened. DeregisterReplicated(table). There was no way to stop capturing once a file had been registered, so a node that does not replicate kept paying three triggers inside every write transaction, appending to an oplog nothing would ever drain. Drops the triggers and prunes the table's oplog + row-version rows in one transaction; idempotent, and deliberately usable on a table this process never registered, since stale triggers left by an EARLIER build are the primary case. Deregistration must be symmetric across a pair, and already is enforced: the handshake compares registered-table digests fail-closed, so a node that drops a table its peer still replicates stops syncing loudly rather than diverging. RegisterReplicated(table, baselineExistingRows: true). Closes the late-opt-in hole: the snapshot streamer pages from __localdb_row_version, which only triggers populate, so a node that started replicating LATE never sent its pre-existing rows — silently, forever. Seeding alone would not have fixed it, because a node whose oplog starts at seq 1 presents no gap at the handshake and is told it needs no snapshot; the seed therefore also flags needs_snapshot, and the existing resync machinery carries the rows. The floor is HLC 0 stamped with this node's id: every genuine HLC is a UTC millisecond shifted left 16 bits and so is strictly greater, which makes a baselined row lose every comparison against a real remote write and win only where the peer has no version of that key at all. Two nodes baselining the same key both hold 0 and resolve by the node-id tie-break — convergent, and nothing better is available for two rows of unordered antiquity. Opt-in, so the documented CDC contract is unchanged unless asked. Pruning the ledger on deregister is safe precisely because re-registration can re-baseline; the two APIs are a pair. O(1) backlog depth. GetOplogDepth ran COUNT(*) over the entire unacked backlog, once a minute from maintenance AND on every metrics scrape — a full scan of a million rows exactly when a node is already behind. Now MAX(seq) - MAX(last_acked_seq, MIN(seq) - 1), floored at 0: three b-tree seeks, measured at 58 VM steps against 100,020 for the count on a 50k backlog. Exact, because seq is AUTOINCREMENT and every prune path removes a prefix (ack pruning by watermark, the row cap by count, the age cap by an HLC cutoff that rises with seq). The MIN clamp is what keeps it exact after a cap prune has moved the horizon above a stalled watermark — without it the caps would re-flag on every tick forever. Byte-budget batching. MaxBatchSize was the only cap on an outbound message, which made it a proxy for a byte budget that consumers had to tune against their widest row: ScadaBridge pinned it to 16 because 70 KB config_json times the 500 default is ~35 MB against a 4 MB gRPC limit. MaxBatchBytes (default 2 MB) now bounds delta and snapshot messages by summed serialized size, with the row count demoted to a secondary cap. A row larger than the budget is sent alone rather than stalling the stream on it. Since more, smaller messages would otherwise mean more apply transactions and more acks, inbound delta batches and acks ALREADY queued are coalesced — one transaction and one ack per burst, bounded by the sender's own caps, and a no-op under a trickle. LwwApplier prepares each statement once. It went through the generic transaction façade, which builds a fresh SqliteCommand per call: three SQL parses per applied row, on the path that drains a reconnect backlog. Statements are now held for the batch and rebound. Dropped __localdb_oplog_hlc. It indexed (table_name, pk_json, hlc), which no query has ever used — every oplog access path is by seq or a full hlc-predicate delete — while costing a b-tree insert inside every captured consumer write. 184/184 tests pass (was 149). New coverage: schema upgrade drops the index and preserves identity/watermarks/data, deregistration idempotency and trigger absence and cross-process cleanup, baseline seeding at the floor with LWW safety proved through the real applier, depth asserted equal to the counting query across every prune scenario plus a constant-time assertion over a 200k backlog, byte-budget chunking including the one-giant-row edge on both the delta and snapshot paths, prepare-once statement counting, and coalescing that must not swallow the message that ends its run. Version 0.2.0.
This commit is contained in:
Joseph Doherty
@@ -16,10 +16,24 @@ internal sealed record ApplyResult(long AppliedThruSeq, int Applied, int Discard
|
||||
/// crash-replay of the batch is a no-op (LWW's strict-greater compare loses on equal versions).
|
||||
/// This never writes <c>__localdb_oplog</c>: the 2-node topology does not re-forward applied deltas.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// Every statement is issued through a <see cref="PreparedStatements"/> set held for the life of the
|
||||
/// batch, so each distinct statement is prepared ONCE and then rebound per row. The previous shape
|
||||
/// went through the generic <c>ILocalDbTransaction</c> façade, which builds a fresh
|
||||
/// <see cref="SqliteCommand"/> per call — three SQL parses per applied row (version lookup, the
|
||||
/// table's delete/upsert, the row-version write), on the path that drains a reconnect backlog.
|
||||
/// </remarks>
|
||||
internal sealed class LwwApplier(SqliteLocalDb db, Func<DateTimeOffset>? utcNow = null)
|
||||
{
|
||||
private readonly Func<DateTimeOffset> _utcNow = utcNow ?? (() => DateTimeOffset.UtcNow);
|
||||
|
||||
/// <summary>
|
||||
/// Distinct statements prepared by the most recent <see cref="ApplyBatchAsync"/>. Test-only
|
||||
/// observation: it is the direct evidence that statement count tracks the number of distinct
|
||||
/// STATEMENTS rather than the number of rows.
|
||||
/// </summary>
|
||||
internal int LastBatchStatementCount { get; private set; }
|
||||
|
||||
public async Task<ApplyResult> ApplyBatchAsync(IReadOnlyList<OplogEntryRecord> entries, CancellationToken ct)
|
||||
{
|
||||
if (entries.Count == 0)
|
||||
@@ -42,13 +56,14 @@ internal sealed class LwwApplier(SqliteLocalDb db, Func<DateTimeOffset>? utcNow
|
||||
if (entry.Hlc > maxHlc) maxHlc = entry.Hlc;
|
||||
}
|
||||
|
||||
var sqlByTable = new Dictionary<string, (string Delete, string Upsert)>(StringComparer.Ordinal);
|
||||
var applied = 0;
|
||||
var discarded = 0;
|
||||
var deadLettered = 0;
|
||||
|
||||
await using var tx = await db.BeginTransactionAsync(ct);
|
||||
await tx.ExecuteAsync("UPDATE __localdb_applying SET applying = 1 WHERE id = 1", null, ct);
|
||||
await using var tx = await db.BeginPreparedTransactionAsync(ct);
|
||||
using var sql = new PreparedStatements(tx.Connection, tx.Sqlite);
|
||||
|
||||
await sql.SetApplyingAsync(1, ct);
|
||||
|
||||
foreach (var entry in entries)
|
||||
{
|
||||
@@ -56,19 +71,16 @@ internal sealed class LwwApplier(SqliteLocalDb db, Func<DateTimeOffset>? utcNow
|
||||
// KEY table that silently inserts a phantom auto-rowid row — so it is poison, not a win.
|
||||
if (!entry.IsTombstone && entry.RowJson is null)
|
||||
{
|
||||
await DeadLetterAsync(tx, entry, "non-tombstone entry with null row_json", utcNow, ct);
|
||||
await sql.DeadLetterAsync(entry, "non-tombstone entry with null row_json", utcNow, ct);
|
||||
deadLettered++;
|
||||
continue;
|
||||
}
|
||||
|
||||
var local = await tx.QueryAsync(
|
||||
"SELECT hlc, node_id FROM __localdb_row_version WHERE table_name = @t AND pk_json = @pk",
|
||||
static r => (Hlc: r.GetInt64(0), NodeId: r.GetString(1)),
|
||||
new { t = entry.TableName, pk = entry.PkJson }, ct);
|
||||
var local = await sql.ReadLocalVersionAsync(entry.TableName, entry.PkJson, ct);
|
||||
|
||||
var remoteWins = local.Count == 0
|
||||
|| entry.Hlc > local[0].Hlc
|
||||
|| (entry.Hlc == local[0].Hlc && string.CompareOrdinal(entry.NodeId, local[0].NodeId) > 0);
|
||||
var remoteWins = local is null
|
||||
|| entry.Hlc > local.Value.Hlc
|
||||
|| (entry.Hlc == local.Value.Hlc && string.CompareOrdinal(entry.NodeId, local.Value.NodeId) > 0);
|
||||
|
||||
if (!remoteWins)
|
||||
{
|
||||
@@ -76,32 +88,15 @@ internal sealed class LwwApplier(SqliteLocalDb db, Func<DateTimeOffset>? utcNow
|
||||
continue;
|
||||
}
|
||||
|
||||
if (!sqlByTable.TryGetValue(entry.TableName, out var sql))
|
||||
{
|
||||
sql = BuildTableSql(db.ReplicatedTables[entry.TableName]);
|
||||
sqlByTable[entry.TableName] = sql;
|
||||
}
|
||||
|
||||
try
|
||||
{
|
||||
var table = db.ReplicatedTables[entry.TableName];
|
||||
if (entry.IsTombstone)
|
||||
await tx.ExecuteAsync(sql.Delete, new { pk = entry.PkJson }, ct);
|
||||
await sql.DeleteAsync(table, entry.PkJson, ct);
|
||||
else
|
||||
await tx.ExecuteAsync(sql.Upsert, new { row = entry.RowJson }, ct);
|
||||
await sql.UpsertAsync(table, entry.RowJson!, ct);
|
||||
|
||||
await tx.ExecuteAsync(
|
||||
"INSERT OR REPLACE INTO __localdb_row_version " +
|
||||
"(table_name, pk_json, hlc, node_id, is_tombstone, tombstone_utc) " +
|
||||
"VALUES (@t, @pk, @hlc, @node, @tomb, @tut)",
|
||||
new
|
||||
{
|
||||
t = entry.TableName,
|
||||
pk = entry.PkJson,
|
||||
hlc = entry.Hlc,
|
||||
node = entry.NodeId,
|
||||
tomb = entry.IsTombstone ? 1 : 0,
|
||||
tut = entry.IsTombstone ? utcNow : null,
|
||||
}, ct);
|
||||
await sql.WriteRowVersionAsync(entry, utcNow, ct);
|
||||
applied++;
|
||||
}
|
||||
catch (SqliteException ex)
|
||||
@@ -110,59 +105,180 @@ internal sealed class LwwApplier(SqliteLocalDb db, Func<DateTimeOffset>? utcNow
|
||||
// malformed row_json) are treated as per-entry poison — SQLite aborts just the
|
||||
// statement, not the transaction, so the batch continues after dead-lettering.
|
||||
// Any other exception propagates and rolls back the whole batch (fail-closed).
|
||||
await DeadLetterAsync(tx, entry, ex.Message, utcNow, ct);
|
||||
await sql.DeadLetterAsync(entry, ex.Message, utcNow, ct);
|
||||
deadLettered++;
|
||||
}
|
||||
}
|
||||
|
||||
await tx.ExecuteAsync(
|
||||
"UPDATE __localdb_peer_state SET " +
|
||||
"last_applied_remote_seq = MAX(last_applied_remote_seq, @maxSeq), " +
|
||||
"last_seen_hlc = MAX(last_seen_hlc, @maxHlc), last_sync_utc = @utc WHERE id = 1",
|
||||
new { maxSeq, maxHlc, utc = utcNow }, ct);
|
||||
await sql.AdvancePeerStateAsync(maxSeq, maxHlc, utcNow, ct);
|
||||
await sql.SetApplyingAsync(0, ct);
|
||||
|
||||
LastBatchStatementCount = sql.StatementCount;
|
||||
|
||||
await tx.ExecuteAsync("UPDATE __localdb_applying SET applying = 0 WHERE id = 1", null, ct);
|
||||
await tx.CommitAsync(ct);
|
||||
|
||||
db.Clock.Observe(maxHlc);
|
||||
return new ApplyResult(maxSeq, applied, discarded, deadLettered);
|
||||
}
|
||||
|
||||
private static Task DeadLetterAsync(
|
||||
ILocalDbTransaction tx, OplogEntryRecord entry, string error, string utcNow, CancellationToken ct) =>
|
||||
tx.ExecuteAsync(
|
||||
"INSERT INTO __localdb_dead_letter " +
|
||||
"(received_utc, table_name, pk_json, row_json, hlc, node_id, error) " +
|
||||
"VALUES (@utc, @t, @pk, @row, @hlc, @node, @err)",
|
||||
new
|
||||
{
|
||||
utc = utcNow,
|
||||
t = entry.TableName,
|
||||
pk = entry.PkJson,
|
||||
row = entry.RowJson,
|
||||
hlc = entry.Hlc,
|
||||
node = entry.NodeId,
|
||||
err = error,
|
||||
}, ct);
|
||||
|
||||
private static (string Delete, string Upsert) BuildTableSql(ReplicatedTable table)
|
||||
/// <summary>
|
||||
/// The statement set one apply-batch runs on: each distinct statement is built and prepared on
|
||||
/// first use, then rebound and re-executed per row. Statements whose SQL depends on the table
|
||||
/// (delete / upsert) are cached per table; the rest are shared across all of them.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// Reusing the same <see cref="SqliteCommand"/> with unchanged <c>CommandText</c> is what makes
|
||||
/// this prepare-once: Microsoft.Data.Sqlite holds the compiled <c>sqlite3_stmt</c> on the command
|
||||
/// and only recompiles when the text changes. <see cref="SqliteCommand.Prepare"/> is called
|
||||
/// explicitly anyway, so the intent is stated rather than inferred from an implementation detail.
|
||||
/// </remarks>
|
||||
private sealed class PreparedStatements(SqliteConnection connection, SqliteTransaction transaction) : IDisposable
|
||||
{
|
||||
var name = Ident(table.Name);
|
||||
private readonly List<SqliteCommand> _owned = [];
|
||||
private readonly Dictionary<string, SqliteCommand> _deletes = new(StringComparer.Ordinal);
|
||||
private readonly Dictionary<string, SqliteCommand> _upserts = new(StringComparer.Ordinal);
|
||||
|
||||
var pkMatch = string.Join(" AND ", table.PkColumns.Select(p => $"{Ident(p)} = json_extract(@pk, {JsonPath(p)})"));
|
||||
var delete = $"DELETE FROM {name} WHERE {pkMatch}";
|
||||
private SqliteCommand? _applying;
|
||||
private SqliteCommand? _lookup;
|
||||
private SqliteCommand? _rowVersion;
|
||||
private SqliteCommand? _deadLetter;
|
||||
private SqliteCommand? _peerState;
|
||||
|
||||
var columns = string.Join(", ", table.Columns.Select(c => Ident(c.Name)));
|
||||
var values = string.Join(", ", table.Columns.Select(c => $"json_extract(@row, {JsonPath(c.Name)})"));
|
||||
var upsert = $"INSERT OR REPLACE INTO {name} ({columns}) VALUES ({values})";
|
||||
/// <summary>How many distinct statements this set has prepared.</summary>
|
||||
public int StatementCount => _owned.Count;
|
||||
|
||||
return (delete, upsert);
|
||||
public Task SetApplyingAsync(int value, CancellationToken ct)
|
||||
{
|
||||
var cmd = _applying ??= Create(
|
||||
"UPDATE __localdb_applying SET applying = @applying WHERE id = 1", "@applying");
|
||||
cmd.Parameters["@applying"].Value = value;
|
||||
return cmd.ExecuteNonQueryAsync(ct);
|
||||
}
|
||||
|
||||
public async Task<(long Hlc, string NodeId)?> ReadLocalVersionAsync(
|
||||
string tableName, string pkJson, CancellationToken ct)
|
||||
{
|
||||
var cmd = _lookup ??= Create(
|
||||
"SELECT hlc, node_id FROM __localdb_row_version WHERE table_name = @t AND pk_json = @pk",
|
||||
"@t", "@pk");
|
||||
cmd.Parameters["@t"].Value = tableName;
|
||||
cmd.Parameters["@pk"].Value = pkJson;
|
||||
|
||||
await using var reader = await cmd.ExecuteReaderAsync(ct);
|
||||
if (!await reader.ReadAsync(ct))
|
||||
return null;
|
||||
return (reader.GetInt64(0), reader.GetString(1));
|
||||
}
|
||||
|
||||
public Task DeleteAsync(ReplicatedTable table, string pkJson, CancellationToken ct)
|
||||
{
|
||||
if (!_deletes.TryGetValue(table.Name, out var cmd))
|
||||
_deletes[table.Name] = cmd = Create(BuildDelete(table), "@pk");
|
||||
cmd.Parameters["@pk"].Value = pkJson;
|
||||
return cmd.ExecuteNonQueryAsync(ct);
|
||||
}
|
||||
|
||||
public Task UpsertAsync(ReplicatedTable table, string rowJson, CancellationToken ct)
|
||||
{
|
||||
if (!_upserts.TryGetValue(table.Name, out var cmd))
|
||||
_upserts[table.Name] = cmd = Create(BuildUpsert(table), "@row");
|
||||
cmd.Parameters["@row"].Value = rowJson;
|
||||
return cmd.ExecuteNonQueryAsync(ct);
|
||||
}
|
||||
|
||||
public Task WriteRowVersionAsync(OplogEntryRecord entry, string utcNow, CancellationToken ct)
|
||||
{
|
||||
var cmd = _rowVersion ??= Create(
|
||||
"INSERT OR REPLACE INTO __localdb_row_version " +
|
||||
"(table_name, pk_json, hlc, node_id, is_tombstone, tombstone_utc) " +
|
||||
"VALUES (@t, @pk, @hlc, @node, @tomb, @tut)",
|
||||
"@t", "@pk", "@hlc", "@node", "@tomb", "@tut");
|
||||
cmd.Parameters["@t"].Value = entry.TableName;
|
||||
cmd.Parameters["@pk"].Value = entry.PkJson;
|
||||
cmd.Parameters["@hlc"].Value = entry.Hlc;
|
||||
cmd.Parameters["@node"].Value = entry.NodeId;
|
||||
cmd.Parameters["@tomb"].Value = entry.IsTombstone ? 1 : 0;
|
||||
cmd.Parameters["@tut"].Value = entry.IsTombstone ? utcNow : DBNull.Value;
|
||||
return cmd.ExecuteNonQueryAsync(ct);
|
||||
}
|
||||
|
||||
public Task DeadLetterAsync(OplogEntryRecord entry, string error, string utcNow, CancellationToken ct)
|
||||
{
|
||||
var cmd = _deadLetter ??= Create(
|
||||
"INSERT INTO __localdb_dead_letter " +
|
||||
"(received_utc, table_name, pk_json, row_json, hlc, node_id, error) " +
|
||||
"VALUES (@utc, @t, @pk, @row, @hlc, @node, @err)",
|
||||
"@utc", "@t", "@pk", "@row", "@hlc", "@node", "@err");
|
||||
cmd.Parameters["@utc"].Value = utcNow;
|
||||
cmd.Parameters["@t"].Value = entry.TableName;
|
||||
cmd.Parameters["@pk"].Value = entry.PkJson;
|
||||
cmd.Parameters["@row"].Value = (object?)entry.RowJson ?? DBNull.Value;
|
||||
cmd.Parameters["@hlc"].Value = entry.Hlc;
|
||||
cmd.Parameters["@node"].Value = entry.NodeId;
|
||||
cmd.Parameters["@err"].Value = error;
|
||||
return cmd.ExecuteNonQueryAsync(ct);
|
||||
}
|
||||
|
||||
public Task AdvancePeerStateAsync(long maxSeq, long maxHlc, string utcNow, CancellationToken ct)
|
||||
{
|
||||
var cmd = _peerState ??= Create(
|
||||
"UPDATE __localdb_peer_state SET " +
|
||||
"last_applied_remote_seq = MAX(last_applied_remote_seq, @maxSeq), " +
|
||||
"last_seen_hlc = MAX(last_seen_hlc, @maxHlc), last_sync_utc = @utc WHERE id = 1",
|
||||
"@maxSeq", "@maxHlc", "@utc");
|
||||
cmd.Parameters["@maxSeq"].Value = maxSeq;
|
||||
cmd.Parameters["@maxHlc"].Value = maxHlc;
|
||||
cmd.Parameters["@utc"].Value = utcNow;
|
||||
return cmd.ExecuteNonQueryAsync(ct);
|
||||
}
|
||||
|
||||
private SqliteCommand Create(string sql, params string[] parameterNames)
|
||||
{
|
||||
var cmd = connection.CreateCommand();
|
||||
try
|
||||
{
|
||||
cmd.Transaction = transaction;
|
||||
cmd.CommandText = sql;
|
||||
foreach (var name in parameterNames)
|
||||
cmd.Parameters.Add(new SqliteParameter(name, DBNull.Value));
|
||||
cmd.Prepare();
|
||||
_owned.Add(cmd);
|
||||
return cmd;
|
||||
}
|
||||
catch
|
||||
{
|
||||
cmd.Dispose();
|
||||
throw;
|
||||
}
|
||||
}
|
||||
|
||||
public void Dispose()
|
||||
{
|
||||
foreach (var cmd in _owned)
|
||||
cmd.Dispose();
|
||||
_owned.Clear();
|
||||
}
|
||||
|
||||
private static string BuildDelete(ReplicatedTable table)
|
||||
{
|
||||
var pkMatch = string.Join(
|
||||
" AND ", table.PkColumns.Select(p => $"{Ident(p)} = json_extract(@pk, {JsonPath(p)})"));
|
||||
return $"DELETE FROM {Ident(table.Name)} WHERE {pkMatch}";
|
||||
}
|
||||
|
||||
private static string BuildUpsert(ReplicatedTable table)
|
||||
{
|
||||
var columns = string.Join(", ", table.Columns.Select(c => Ident(c.Name)));
|
||||
var values = string.Join(", ", table.Columns.Select(c => $"json_extract(@row, {JsonPath(c.Name)})"));
|
||||
return $"INSERT OR REPLACE INTO {Ident(table.Name)} ({columns}) VALUES ({values})";
|
||||
}
|
||||
|
||||
// Local copy of TriggerSqlGenerator's convention (that helper lives in the core assembly).
|
||||
private static string Ident(string name) => "\"" + name.Replace("\"", "\"\"") + "\"";
|
||||
|
||||
// A single-quoted JSON path with a quoted key member, so a column name with reserved
|
||||
// characters still resolves: $."col". The key's double quotes are backslash-escaped per
|
||||
// JSON path grammar.
|
||||
private static string JsonPath(string column) => "'$.\"" + column.Replace("\"", "\\\"") + "\"'";
|
||||
}
|
||||
|
||||
// Local copy of TriggerSqlGenerator's convention (that helper lives in the core assembly).
|
||||
private static string Ident(string name) => "\"" + name.Replace("\"", "\"\"") + "\"";
|
||||
|
||||
// A single-quoted JSON path with a quoted key member, so a column name with reserved characters
|
||||
// still resolves: $."col". The key's double quotes are backslash-escaped per JSON path grammar.
|
||||
private static string JsonPath(string column) => "'$.\"" + column.Replace("\"", "\\\"") + "\"'";
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user