perf(localdb): dereg API, O(1) backlog depth, byte-budget batching, prepare-once apply (0.2.0)
WP3.3 of the ScadaBridge architectural-review remediation. Six changes, all wire-compatible with 0.1.x peers — no proto change, no lib_schema_version bump. The on-disk bookkeeping schema goes to v2 and is upgraded in place when a file is opened. DeregisterReplicated(table). There was no way to stop capturing once a file had been registered, so a node that does not replicate kept paying three triggers inside every write transaction, appending to an oplog nothing would ever drain. Drops the triggers and prunes the table's oplog + row-version rows in one transaction; idempotent, and deliberately usable on a table this process never registered, since stale triggers left by an EARLIER build are the primary case. Deregistration must be symmetric across a pair, and already is enforced: the handshake compares registered-table digests fail-closed, so a node that drops a table its peer still replicates stops syncing loudly rather than diverging. RegisterReplicated(table, baselineExistingRows: true). Closes the late-opt-in hole: the snapshot streamer pages from __localdb_row_version, which only triggers populate, so a node that started replicating LATE never sent its pre-existing rows — silently, forever. Seeding alone would not have fixed it, because a node whose oplog starts at seq 1 presents no gap at the handshake and is told it needs no snapshot; the seed therefore also flags needs_snapshot, and the existing resync machinery carries the rows. The floor is HLC 0 stamped with this node's id: every genuine HLC is a UTC millisecond shifted left 16 bits and so is strictly greater, which makes a baselined row lose every comparison against a real remote write and win only where the peer has no version of that key at all. Two nodes baselining the same key both hold 0 and resolve by the node-id tie-break — convergent, and nothing better is available for two rows of unordered antiquity. Opt-in, so the documented CDC contract is unchanged unless asked. Pruning the ledger on deregister is safe precisely because re-registration can re-baseline; the two APIs are a pair. O(1) backlog depth. GetOplogDepth ran COUNT(*) over the entire unacked backlog, once a minute from maintenance AND on every metrics scrape — a full scan of a million rows exactly when a node is already behind. Now MAX(seq) - MAX(last_acked_seq, MIN(seq) - 1), floored at 0: three b-tree seeks, measured at 58 VM steps against 100,020 for the count on a 50k backlog. Exact, because seq is AUTOINCREMENT and every prune path removes a prefix (ack pruning by watermark, the row cap by count, the age cap by an HLC cutoff that rises with seq). The MIN clamp is what keeps it exact after a cap prune has moved the horizon above a stalled watermark — without it the caps would re-flag on every tick forever. Byte-budget batching. MaxBatchSize was the only cap on an outbound message, which made it a proxy for a byte budget that consumers had to tune against their widest row: ScadaBridge pinned it to 16 because 70 KB config_json times the 500 default is ~35 MB against a 4 MB gRPC limit. MaxBatchBytes (default 2 MB) now bounds delta and snapshot messages by summed serialized size, with the row count demoted to a secondary cap. A row larger than the budget is sent alone rather than stalling the stream on it. Since more, smaller messages would otherwise mean more apply transactions and more acks, inbound delta batches and acks ALREADY queued are coalesced — one transaction and one ack per burst, bounded by the sender's own caps, and a no-op under a trickle. LwwApplier prepares each statement once. It went through the generic transaction façade, which builds a fresh SqliteCommand per call: three SQL parses per applied row, on the path that drains a reconnect backlog. Statements are now held for the batch and rebound. Dropped __localdb_oplog_hlc. It indexed (table_name, pk_json, hlc), which no query has ever used — every oplog access path is by seq or a full hlc-predicate delete — while costing a b-tree insert inside every captured consumer write. 184/184 tests pass (was 149). New coverage: schema upgrade drops the index and preserves identity/watermarks/data, deregistration idempotency and trigger absence and cross-process cleanup, baseline seeding at the floor with LWW safety proved through the real applier, depth asserted equal to the counting query across every prune scenario plus a constant-time assertion over a 200k backlog, byte-budget chunking including the one-giant-row edge on both the delta and snapshot paths, prepare-once statement counting, and coalescing that must not swallow the message that ends its run. Version 0.2.0.
This commit is contained in:
Joseph Doherty
@@ -246,6 +246,17 @@ internal sealed class SyncSession
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Reads oplog deltas above the sent watermark and ships them, splitting each read into as many
|
||||
/// messages as <see cref="ReplicationOptions.MaxBatchBytes"/> requires.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// Row count alone cannot bound a message: rows are arbitrarily wide, so any row-count cap is
|
||||
/// really a guess about the widest row, and guessing wrong produces a message the transport
|
||||
/// rejects — replication then stops converging with a size error, not a data error. Budgeting by
|
||||
/// summed serialized bytes bounds the thing that is actually limited, and lets the row cap go
|
||||
/// back to meaning what it says.
|
||||
/// </remarks>
|
||||
private async Task PumpLoopAsync(ChannelWriter<SyncMessage> dataLane, CancellationToken ct)
|
||||
{
|
||||
while (true)
|
||||
@@ -254,12 +265,30 @@ internal sealed class SyncSession
|
||||
var batch = await _store.ReadBatchAboveAsync(_sentThruSeq, _options.MaxBatchSize, ct);
|
||||
if (batch.Count > 0)
|
||||
{
|
||||
var delta = new DeltaBatch();
|
||||
foreach (var entry in batch)
|
||||
delta.Entries.Add(ToProto(entry));
|
||||
// Blocking here on a full data lane IS the pump's backpressure.
|
||||
await dataLane.WriteAsync(new SyncMessage { DeltaBatch = delta }, ct);
|
||||
_sentThruSeq = batch[^1].Seq;
|
||||
var index = 0;
|
||||
while (index < batch.Count)
|
||||
{
|
||||
var delta = new DeltaBatch();
|
||||
var bytes = 0L;
|
||||
while (index < batch.Count)
|
||||
{
|
||||
var proto = ToProto(batch[index]);
|
||||
var size = EntrySize(proto);
|
||||
// The first entry of a message is taken however large it is: refusing an
|
||||
// oversized row would stall the stream on it permanently, whereas sending it
|
||||
// alone either succeeds or fails loudly at the transport with its real size.
|
||||
if (delta.Entries.Count > 0 && bytes + size > _options.MaxBatchBytes)
|
||||
break;
|
||||
delta.Entries.Add(proto);
|
||||
bytes += size;
|
||||
index++;
|
||||
}
|
||||
// Blocking here on a full data lane IS the pump's backpressure.
|
||||
await dataLane.WriteAsync(new SyncMessage { DeltaBatch = delta }, ct);
|
||||
// Advanced per MESSAGE, not per read: a fault mid-split must not re-send what
|
||||
// already went out, and must not skip what did not.
|
||||
_sentThruSeq = batch[index - 1].Seq;
|
||||
}
|
||||
continue;
|
||||
}
|
||||
|
||||
@@ -267,23 +296,43 @@ internal sealed class SyncSession
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Serialized bytes one entry contributes to its batch message: the entry's own encoding plus
|
||||
/// the repeated field's tag and length prefix, as protobuf itself computes them.
|
||||
/// </summary>
|
||||
private static long EntrySize(OplogEntry entry) =>
|
||||
1 + Google.Protobuf.CodedOutputStream.ComputeMessageSize(entry);
|
||||
|
||||
private async Task ReceiveLoopAsync(
|
||||
SyncDuplex duplex, ChannelWriter<SyncMessage> controlLane, CancellationToken ct)
|
||||
{
|
||||
while (await duplex.Inbox.WaitToReadAsync(ct))
|
||||
// One-message lookahead. Coalescing means reading ahead until a message turns out NOT to be
|
||||
// of the kind being coalesced, and a Channel cannot un-read — so that message is parked here
|
||||
// and processed first on the next pass. Its presence also keeps the outer loop from waiting
|
||||
// on the inbox when there is already work in hand.
|
||||
SyncMessage? lookahead = null;
|
||||
|
||||
while (lookahead is not null || await duplex.Inbox.WaitToReadAsync(ct))
|
||||
{
|
||||
while (duplex.Inbox.TryRead(out var msg))
|
||||
while (true)
|
||||
{
|
||||
var msg = lookahead;
|
||||
lookahead = null;
|
||||
if (msg is null)
|
||||
{
|
||||
if (!duplex.Inbox.TryRead(out var read))
|
||||
break;
|
||||
msg = read;
|
||||
}
|
||||
|
||||
switch (msg.MsgCase)
|
||||
{
|
||||
case SyncMessage.MsgOneofCase.DeltaBatch:
|
||||
// Deltas apply normally even mid-snapshot: LWW makes the interleave safe.
|
||||
await HandleDeltaBatchAsync(controlLane, msg.DeltaBatch, ct);
|
||||
lookahead = await HandleCoalescedDeltasAsync(duplex, controlLane, msg.DeltaBatch, ct);
|
||||
break;
|
||||
case SyncMessage.MsgOneofCase.DeltaAck:
|
||||
var acked = msg.DeltaAck.AppliedThruSeq;
|
||||
InterlockedMax(ref _peerAckedSeq, acked);
|
||||
await _store.RecordPeerAckAsync(acked, ct);
|
||||
lookahead = await HandleCoalescedAcksAsync(duplex, msg.DeltaAck, ct);
|
||||
break;
|
||||
case SyncMessage.MsgOneofCase.SnapshotBegin:
|
||||
if (SnapshotApplier is null)
|
||||
@@ -314,19 +363,98 @@ internal sealed class SyncSession
|
||||
}
|
||||
}
|
||||
|
||||
private async Task HandleDeltaBatchAsync(ChannelWriter<SyncMessage> controlLane, DeltaBatch batch, CancellationToken ct)
|
||||
/// <summary>
|
||||
/// Applies the given delta batch together with any delta batches ALREADY waiting in the inbox,
|
||||
/// as one transaction answered by one ack; returns the non-delta message that ended the run (to
|
||||
/// be processed next), or null if the inbox merely ran dry.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// Byte-budgeting the sender produces more, smaller messages, and one apply transaction plus one
|
||||
/// ack round-trip per message is the cost that would otherwise scale with that. Coalescing pays
|
||||
/// it once per burst instead — and a burst is exactly what a reconnect after an outage delivers.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// Only messages already queued are absorbed (<c>TryRead</c>, never a wait), so this adds no
|
||||
/// latency: a trickle of one batch at a time coalesces nothing and behaves exactly as before.
|
||||
/// The run stops at the sender's own caps, so the extra memory held is bounded by roughly one
|
||||
/// sender batch beyond the budget — the overshoot of finishing whichever batch crosses it.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
private async Task<SyncMessage?> HandleCoalescedDeltasAsync(
|
||||
SyncDuplex duplex, ChannelWriter<SyncMessage> controlLane, DeltaBatch first, CancellationToken ct)
|
||||
{
|
||||
if (batch.Entries.Count == 0)
|
||||
var entries = new List<OplogEntry>(first.Entries);
|
||||
var bytes = 0L;
|
||||
foreach (var entry in first.Entries)
|
||||
bytes += EntrySize(entry);
|
||||
|
||||
SyncMessage? lookahead = null;
|
||||
while (entries.Count < _options.MaxBatchSize && bytes < _options.MaxBatchBytes
|
||||
&& duplex.Inbox.TryRead(out var next))
|
||||
{
|
||||
if (next.MsgCase != SyncMessage.MsgOneofCase.DeltaBatch)
|
||||
{
|
||||
lookahead = next;
|
||||
break;
|
||||
}
|
||||
foreach (var entry in next.DeltaBatch.Entries)
|
||||
{
|
||||
entries.Add(entry);
|
||||
bytes += EntrySize(entry);
|
||||
}
|
||||
}
|
||||
|
||||
await ApplyDeltaEntriesAsync(controlLane, entries, ct);
|
||||
return lookahead;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Records the given ack together with any acks ALREADY waiting in the inbox, as a single
|
||||
/// watermark write; returns the non-ack message that ended the run, or null if the inbox ran dry.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// Exactly equivalent to recording each in turn, and strictly cheaper: <c>applied_thru_seq</c> is
|
||||
/// monotonic and <c>RecordPeerAckAsync</c> is MAX-guarded, so every superseded ack would have
|
||||
/// written a watermark that the final one overwrites and run a DELETE below its own horizon.
|
||||
/// Unbounded coalescing is safe here — acks collapse into one number, accumulating nothing.
|
||||
/// </remarks>
|
||||
private async Task<SyncMessage?> HandleCoalescedAcksAsync(
|
||||
SyncDuplex duplex, DeltaAck first, CancellationToken ct)
|
||||
{
|
||||
var acked = first.AppliedThruSeq;
|
||||
|
||||
SyncMessage? lookahead = null;
|
||||
while (duplex.Inbox.TryRead(out var next))
|
||||
{
|
||||
if (next.MsgCase != SyncMessage.MsgOneofCase.DeltaAck)
|
||||
{
|
||||
lookahead = next;
|
||||
break;
|
||||
}
|
||||
if (next.DeltaAck.AppliedThruSeq > acked)
|
||||
acked = next.DeltaAck.AppliedThruSeq;
|
||||
}
|
||||
|
||||
InterlockedMax(ref _peerAckedSeq, acked);
|
||||
await _store.RecordPeerAckAsync(acked, ct);
|
||||
return lookahead;
|
||||
}
|
||||
|
||||
private async Task ApplyDeltaEntriesAsync(
|
||||
ChannelWriter<SyncMessage> controlLane, IReadOnlyList<OplogEntry> batchEntries, CancellationToken ct)
|
||||
{
|
||||
if (batchEntries.Count == 0)
|
||||
return;
|
||||
|
||||
var now = _utcNow();
|
||||
var nowMs = now.ToUnixTimeMilliseconds();
|
||||
var driftLimitMs = nowMs + (long)_options.MaxHlcDriftAhead.TotalMilliseconds;
|
||||
var toApply = new List<OplogEntryRecord>(batch.Entries.Count);
|
||||
var toApply = new List<OplogEntryRecord>(batchEntries.Count);
|
||||
var maxSeq = 0L;
|
||||
var driftDeadLettered = 0;
|
||||
|
||||
foreach (var proto in batch.Entries)
|
||||
foreach (var proto in batchEntries)
|
||||
{
|
||||
if (proto.Seq > maxSeq) maxSeq = proto.Seq;
|
||||
var entry = FromProto(proto);
|
||||
|
||||
Reference in New Issue
Block a user