dohertj2/scadaproj
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(localdb): per-table needs_snapshot (0.3.0, on-disk schema v3)

Browse Source 
Baselining one table used to set a single per-DATABASE needs_snapshot bit, and a
snapshot streams every registered table in full — so opting one table into
replication late re-sent all of them, in both directions at once on a pair
enabling replication late. On a database holding a high-volume table (an event
log at its retention cap) that is the largest transfer this library performs.
0.2.1 documented the cost and deferred the fix as needing an on-disk change.

Schema v3 adds __localdb_snapshot_state (table_name PK, needs_snapshot), created
by the shared DDL with IF NOT EXISTS so an older file gains it on open; the v2->v3
transition is a version stamp only. RegisterReplicated(..., baselineExistingRows:
true) now flags that table alone. The database-wide flag survives unchanged for
the paths that genuinely reach every table — a cap prune or an impossible peer ack
loses delta continuity for the oplog as a whole. DeregisterReplicated drops a
table's pending flag with its ledger rows, since nothing would ever stream (and so
clear) it afterwards.

Wire compatibility is additive and negotiated by CAPABILITY, not version:
lib_schema_version stays 1 because ValidateHandshake compares it fail-closed for
equality, so bumping it would stop 0.1.x/0.2.x pairs syncing outright. Handshake
gains supports_partial_snapshot (field 5) and SnapshotBegin gains a tables scope
(field 2) where EMPTY means all tables — precisely what a pre-0.3.0 sender emits
and what a pre-0.3.0 receiver assumes. A peer that does not advertise the
capability is served a full snapshot: a scoped one would make it advance
last_applied_remote_seq to the snapshot's as-of and silently discard every delta
below it for the tables left out.

For the same reason a partial snapshot moves no watermark on either side. The
receiver keeps last_applied_remote_seq (carrying only the HLC across, which is
MAX-guarded), and the sender keeps _sentThruSeq, so the deltas the snapshot
overlaps are still pumped and merged by LWW.

Crash safety: snapshot debt now clears on the receiver's new SnapshotAck rather
than on enqueue, so an ungraceful exit part-way through re-streams the same scope
on reconnect. The ack rides the unbounded CONTROL lane like DeltaAck, so it
composes with 0.2.1's bounded 64-message inbox and its out-of-handshake snapshot.
Against a pre-0.3.0 peer the 0.2.x clear-on-enqueue behaviour is kept — there is
no ack to wait for — with the handshake's horizon gap check as the net.

Tests 199/199 (was 188): single-table baseline streams only that table, mixed-
version interop (peer without the capability gets a full snapshot), debt standing
until SnapshotAck, partial snapshot not advancing the receiver's watermark, no
SnapshotAck emitted to an old peer, end-to-end over two real databases, plus v2->v3
upgrade and per-table flag isolation.

README, ILocalDb remarks (which named this as the follow-up) and the umbrella
CLAUDE.md index updated.
This commit is contained in:
Joseph Doherty
2026-08-15 03:34:54 -04:00
parent f627cee9c0
commit cea3930b94
16 changed files with 791 additions and 96 deletions
Download Patch File Download Diff File Expand all files Collapse all files
ZB.MOM.WW.LocalDb/src/ZB.MOM.WW.LocalDb.Replication/Internal/SyncSession.cs
+139 -23
View File
@@ -47,6 +47,17 @@ internal sealed class SyncSession
// single receive-loop task, so no synchronization is needed.
private bool _snapshotReceiving;
// Whether the peer speaks the 0.3.0 snapshot extensions: scoped snapshots
// (SnapshotBegin.tables) and the SnapshotAck that answers a completed one. Written once during
// the handshake — which completes before either loop starts — then read-only.
private bool _peerSupportsPartialSnapshot;
// The scope of the snapshot this node is streaming, held until the peer's SnapshotAck discharges
// it; null on every session that owes none. Assigned on the pump task strictly BEFORE the
// snapshot's first message is enqueued and read on the receive loop when the ack lands, so the
// ordering is causal — volatile only to make the cross-thread publication explicit.
private SnapshotScope? _sentSnapshotScope;
public SyncSession(
SqliteLocalDb db, OplogStore store, LwwApplier applier, ReplicationOptions options, ILogger logger,
Func<DateTimeOffset>? utcNow = null, LocalDbMetrics? metrics = null, SyncStatus? status = null)
@@ -63,11 +74,12 @@ internal sealed class SyncSession
/// <summary>
/// Invoked when THIS node owes the peer a snapshot (its own snapshot-required computation, not
/// the peer's) before deltas resume. Receives a gated send delegate that enqueues on the bounded
/// data lane (never the raw transport); returns the as-of oplog seq the snapshot covers so the
/// pump can skip already-covered tail deltas. Wired to <see cref="SnapshotStreamer.SendAsync"/>.
/// the peer's) before deltas resume. Receives the negotiated <see cref="SnapshotScope"/> and a
/// gated send delegate that enqueues on the bounded data lane (never the raw transport); returns
/// the as-of oplog seq the snapshot covers so the pump can skip already-covered tail deltas.
/// Wired to <see cref="SnapshotStreamer.SendAsync"/>.
/// </summary>
public Func<Func<SyncMessage, CancellationToken, Task>, CancellationToken, Task<long>>? SnapshotSender { get; set; }
public Func<SnapshotScope, Func<SyncMessage, CancellationToken, Task>, CancellationToken, Task<long>>? SnapshotSender { get; set; }
/// <summary>
/// Consumes an inbound snapshot inline on the receive loop: SnapshotBegin opens the receive,
@@ -99,7 +111,7 @@ internal sealed class SyncSession
var writer = WriterLoopAsync(duplex, control.Reader, data.Reader, linked.Token);
try
{
var snapshotRequired = await HandshakeAsync(duplex, control.Writer, linked.Token);
var snapshotScope = await HandshakeAsync(duplex, control.Writer, linked.Token);
// The receive loop starts BEFORE any snapshot this node owes is streamed. That ordering
// is load-bearing, not incidental: a snapshot sent from inside the handshake left the
@@ -111,7 +123,7 @@ internal sealed class SyncSession
// ordinary flow control. Interleaving is already safe by design: snapshot rows and
// deltas both merge through LWW, and the snapshot's watermark write is MAX-guarded.
var receive = ReceiveLoopAsync(duplex, control.Writer, linked.Token);
var pump = SnapshotThenPumpLoopAsync(snapshotRequired, GatedDataSend, data.Writer, linked.Token);
var pump = SnapshotThenPumpLoopAsync(snapshotScope, GatedDataSend, data.Writer, linked.Token);
var loops = new[] { pump, receive, writer };
// The first COMPLETED task is the primary outcome. Cancel the siblings, observe every
@@ -135,12 +147,12 @@ internal sealed class SyncSession
}
/// <summary>
/// Runs the two-phase fail-closed handshake and returns whether THIS node owes the peer a full
/// snapshot (its own computation from its own state — the peer's answer drives the peer's send
/// direction, symmetrically). Sending that snapshot is deliberately NOT part of the handshake;
/// see <see cref="RunAsync"/>.
/// Runs the two-phase fail-closed handshake and returns the snapshot THIS node owes the peer —
/// null for none, otherwise the <see cref="SnapshotScope"/> to stream (its own computation from
/// its own state; the peer's answer drives the peer's send direction, symmetrically). Sending
/// that snapshot is deliberately NOT part of the handshake; see <see cref="RunAsync"/>.
/// </summary>
private async Task<bool> HandshakeAsync(
private async Task<SnapshotScope?> HandshakeAsync(
SyncDuplex duplex, ChannelWriter<SyncMessage> controlLane, CancellationToken ct)
{
var peerState = await _store.GetPeerStateAsync(ct);
@@ -158,6 +170,11 @@ internal sealed class SyncSession
ValidateHandshake(localHandshake, peerHandshake);
// Read before anything can act on it. A pre-0.3.0 peer leaves the field at its proto3
// default, so "false" covers both "explicitly unsupported" and "older build" without a
// version check — which is the point of advertising a capability rather than a version.
_peerSupportsPartialSnapshot = peerHandshake.SupportsPartialSnapshot;
// A different node id on the far end means the peer was replaced, not merely restarted — a
// rebuilt host, a restored-from-nothing volume. Its oplog numbers from 1 again, so the
// watermark we hold (in the OLD peer's seq space) would silently swallow its first N
@@ -191,16 +208,16 @@ internal sealed class SyncSession
_sentThruSeq = peerAppliedThruSeq;
Interlocked.Exchange(ref _peerAckedSeq, peerAppliedThruSeq);
var snapshotRequired = await ComputeSnapshotRequiredAsync(peerHandshake, peerState, ct);
var snapshotScope = await ComputeSnapshotScopeAsync(peerHandshake, peerState, ct);
await controlLane.WriteAsync(
new SyncMessage { HandshakeAck = new HandshakeAck { NodeId = _db.NodeId, SnapshotRequired = snapshotRequired } }, ct);
new SyncMessage { HandshakeAck = new HandshakeAck { NodeId = _db.NodeId, SnapshotRequired = snapshotScope is not null } }, ct);
var peerAckMsg = await ReceiveOneAsync(duplex, ct);
if (peerAckMsg.MsgCase != SyncMessage.MsgOneofCase.HandshakeAck)
throw new InvalidOperationException(
$"Replication protocol error: expected a HandshakeAck, got {peerAckMsg.MsgCase}.");
return snapshotRequired;
return snapshotScope;
}
/// <summary>
@@ -216,25 +233,55 @@ internal sealed class SyncSession
/// than racing the transport.
/// </remarks>
private async Task SnapshotThenPumpLoopAsync(
bool snapshotRequired, Func<SyncMessage, CancellationToken, Task> dataSend,
SnapshotScope? snapshotScope, Func<SyncMessage, CancellationToken, Task> dataSend,
ChannelWriter<SyncMessage> dataLane, CancellationToken ct)
{
if (snapshotRequired)
if (snapshotScope is not null)
{
if (SnapshotSender is null)
throw new NotSupportedException("snapshot required but no snapshot sender configured");
var snapshotAsOfSeq = await SnapshotSender(dataSend, ct);
// Published BEFORE the first frame is enqueued, so the peer's SnapshotAck can never
// arrive on the receive loop while this is still null.
Volatile.Write(ref _sentSnapshotScope, snapshotScope);
var snapshotAsOfSeq = await SnapshotSender(snapshotScope, dataSend, ct);
_metrics?.RecordSnapshotSent();
_status?.SetLastSyncUtc(_utcNow());
// The snapshot already carries every row state up to as-of; skipping the pump past it
// avoids re-sending tail deltas the peer would only discard via LWW.
if (snapshotAsOfSeq > _sentThruSeq)
// A FULL snapshot carries every row state up to as-of, so skipping the pump past it
// avoids re-sending tail deltas the peer would only discard via LWW. A PARTIAL one
// speaks only for its own tables, so the pump must still deliver every delta below
// as-of — the peer has not moved its watermark either (see SnapshotApplier.OnComplete).
if (!snapshotScope.IsPartial && snapshotAsOfSeq > _sentThruSeq)
_sentThruSeq = snapshotAsOfSeq;
// A peer that cannot acknowledge the snapshot leaves only the 0.2.x option: clear once
// the frames are on the data lane. The window that opens (enqueued, not yet delivered —
// a crash there loses the snapshot with the debt already discharged) is why 0.3.0 peers
// wait for the ack instead. The reconnect handshake's gap check re-detects it whenever
// the oplog is non-empty; an EMPTY oplog remains the undetectable corner.
if (!_peerSupportsPartialSnapshot)
await ClearSentSnapshotDebtAsync(ct);
}
await PumpLoopAsync(dataLane, ct);
}
/// <summary>
/// Discharges the snapshot debt this session's outbound snapshot covered, exactly once.
/// </summary>
/// <remarks>
/// Taking the scope out of the field before clearing makes this idempotent, which matters
/// because two paths reach it — the pre-0.3.0 clear-on-enqueue above and the peer's SnapshotAck
/// — and a peer that acknowledges anyway (a newer build than its handshake claimed) must not
/// clear a second time on top of flags a later baseline may already have re-raised.
/// </remarks>
private async Task ClearSentSnapshotDebtAsync(CancellationToken ct)
{
var scope = Interlocked.Exchange(ref _sentSnapshotScope, null);
if (scope is null)
return;
await _store.ClearSnapshotDebtAsync(scope.Tables, ct);
}
// The ONLY caller of duplex.Send. Control (handshake + acks) always preempts data so the peer's
// inbound processing is never starved of its acks behind a backlog of DeltaBatches.
private static async Task WriterLoopAsync(
@@ -384,6 +431,20 @@ internal sealed class SyncSession
_snapshotReceiving = false;
_metrics?.RecordSnapshotReceived();
_status?.SetLastSyncUtc(_utcNow());
// Acknowledge only to a peer that asked for it. Sent AFTER the applier's
// durable watermark write, so the ack means "I hold these rows", which is
// what lets the sender drop its per-table flags. On the unbounded control
// lane, like DeltaAck: an ack must never queue behind a data backlog.
if (_peerSupportsPartialSnapshot)
await controlLane.WriteAsync(
new SyncMessage { SnapshotAck = new SnapshotAck { AsOfSeq = msg.SnapshotComplete.AsOfSeq } }, ct);
break;
case SyncMessage.MsgOneofCase.SnapshotAck:
// The peer has durably applied the snapshot THIS node streamed, so the
// tables it covered no longer owe one. Nothing to reconcile against the
// ack's as-of seq: only one snapshot is ever sent per session, so the scope
// held in _sentSnapshotScope is unambiguously the one being acknowledged.
await ClearSentSnapshotDebtAsync(ct);
break;
default:
throw new InvalidOperationException(
@@ -565,6 +626,9 @@ internal sealed class SyncSession
NodeId = _db.NodeId,
LibSchemaVersion = LibSchemaVersion,
LastAppliedRemoteSeq = peerState.LastAppliedRemoteSeq,
// Unconditional: this build always understands a scoped snapshot and always answers a
// completed one. It is a statement about the code, not about configuration or state.
SupportsPartialSnapshot = true,
};
foreach (var table in _db.ReplicatedTables.Values.OrderBy(t => t.Name, StringComparer.Ordinal))
handshake.Tables.Add(new TableDigest { TableName = table.Name, Digest = table.Digest });
@@ -595,11 +659,63 @@ internal sealed class SyncSession
"Replication handshake rejected (fail-closed): " + string.Join("; ", parts) + ".");
}
private async Task<bool> ComputeSnapshotRequiredAsync(Handshake peerHandshake, PeerState peerState, CancellationToken ct)
/// <summary>
/// Decides what snapshot this node owes the peer: null for none, <see cref="SnapshotScope.Full"/>
/// for every registered table, or a scoped snapshot naming just the tables that need one.
/// </summary>
/// <remarks>
/// <para>
/// The two sources of snapshot debt are genuinely different in reach, which is why they are
/// stored separately. An <b>oplog-horizon</b> gap — the peer's watermark below the oldest delta
/// still streamable, which is what a cap prune or an impossible peer ack leaves behind — means
/// deltas have been lost for the oplog as a whole, so nothing short of every table is sound. A
/// <b>baseline</b> means one table's pre-existing rows were never captured, and says nothing at
/// all about the others.
/// </para>
/// <para>
/// A scope that happens to name every registered table is returned as Full rather than as a
/// list: it covers exactly what a full snapshot covers, so it should also get a full snapshot's
/// watermark advance on both sides, and it needs no promotion for a pre-0.3.0 peer.
/// </para>
/// <para>
/// Flags naming tables that are not registered are ignored (they can only stream nothing).
/// <c>DeregisterReplicated</c> drops them with the table's ledger rows, so the surviving way to
/// hold one is to drop the SQL table without deregistering — which the handshake's fail-closed
/// digest comparison rejects for its own reasons.
/// </para>
/// </remarks>
private async Task<SnapshotScope?> ComputeSnapshotScopeAsync(Handshake peerHandshake, PeerState peerState, CancellationToken ct)
{
if (peerState.NeedsSnapshot)
return true;
if (peerState.NeedsSnapshot || await HasHorizonGapAsync(peerHandshake, peerState, ct))
return SnapshotScope.Full;
var flagged = await _store.GetTablesNeedingSnapshotAsync(ct);
if (flagged.Count == 0)
return null;
var scoped = flagged.Where(_db.ReplicatedTables.ContainsKey).ToList();
if (scoped.Count == 0)
return null;
if (scoped.Count >= _db.ReplicatedTables.Count)
return SnapshotScope.Full;
// A peer that would read SnapshotBegin.tables as "all tables" and then advance its watermark
// past deltas this snapshot did not carry gets the full snapshot instead. Wider than asked
// for, but it is the only shape the older protocol can express correctly — and it is what
// 0.2.x did for every baseline.
if (!_peerSupportsPartialSnapshot)
{
_logger.LogInformation(
"Peer {PeerNodeId} does not support scoped snapshots; promoting the snapshot of {Tables} to a full one.",
PeerNodeId, string.Join(",", scoped));
return SnapshotScope.Full;
}
return new SnapshotScope(scoped);
}
private async Task<bool> HasHorizonGapAsync(Handshake peerHandshake, PeerState peerState, CancellationToken ct)
{
var minRows = await _db.QueryAsync(
"SELECT COALESCE(MIN(seq), 0) FROM __localdb_oplog", static r => r.GetInt64(0), null, ct);
var minSeq = minRows[0];