Commit Graph

7 Commits

Author SHA1 Message Date
Joseph Doherty 3e5a5fde81 fix(localdb): flush the control lane before a faulting session cancels its writer (0.3.1)
A session that faulted during the handshake cancelled its writer loop immediately,
which could kill its OWN Handshake while it still sat in the control lane: the
enqueue -> receive -> validate -> fault path has no yields, while the writer's
wakeup waits in the thread-pool queue, so with the peer's handshake already
buffered the cancel won the race almost every time (39/40 in the new pinning
test). Mismatch detection is symmetric — the peer computes its fail-closed
verdict FROM our Handshake — so the healthy peer never learned anything: in
tests it waited out the full 15 s receive cap (the register-row-41 flake on
SyncSessionTests.Handshake_DigestMismatch_FailsClosed), and in production it
would misclassify a permanent schema mismatch as a generic disconnect and
reconnect-loop at the fast backoff instead of taking the classified
FailedPrecondition -> LogError + max-backoff path.

RunAsync's fault teardown now completes both lanes so the writer drains what is
already enqueued and exits on its own, with a 2 s deferred cancel as the backstop
for a wedged transport. Caller-initiated cancellation is unchanged (the linked
token is already cancelled, so shutdown is not delayed), and nothing on the wire
changes — a 0.3.0 peer simply receives the Handshake it was always owed.

New SyncSessionTests.Handshake_FaultTeardown_StillTransmitsTheLocalHandshake pins
the race (40 concurrent preloaded-mismatch sessions; 39/40 red pre-fix, green
post-fix).

BatchBudgetTests hardened separately: the register's other flaky test
(Pump_RowLargerThanTheBudget_IsSentAloneRatherThanStallingTheStream) awaits the
scripted-peer channel without ever observing the session task, so ANY session
fault surfaced as an opaque 15 s timeout. A run-observing NextAsync overload now
races the session task at every read site in the class, so the next flake reports
the session's real exception instead. (The handshake-race root cause does not
structurally apply to this test — no handshake fault occurs there — so its
intermittent timeout is attributed to the fault-hiding wait, now removed.)

Suite 200/200 green. Closes ScadaBridge deferred-work register row 41 (cross-repo).
2026-08-18 07:11:14 -04:00
Joseph Doherty cea3930b94 feat(localdb): per-table needs_snapshot (0.3.0, on-disk schema v3)
Baselining one table used to set a single per-DATABASE needs_snapshot bit, and a
snapshot streams every registered table in full — so opting one table into
replication late re-sent all of them, in both directions at once on a pair
enabling replication late. On a database holding a high-volume table (an event
log at its retention cap) that is the largest transfer this library performs.
0.2.1 documented the cost and deferred the fix as needing an on-disk change.

Schema v3 adds __localdb_snapshot_state (table_name PK, needs_snapshot), created
by the shared DDL with IF NOT EXISTS so an older file gains it on open; the v2->v3
transition is a version stamp only. RegisterReplicated(..., baselineExistingRows:
true) now flags that table alone. The database-wide flag survives unchanged for
the paths that genuinely reach every table — a cap prune or an impossible peer ack
loses delta continuity for the oplog as a whole. DeregisterReplicated drops a
table's pending flag with its ledger rows, since nothing would ever stream (and so
clear) it afterwards.

Wire compatibility is additive and negotiated by CAPABILITY, not version:
lib_schema_version stays 1 because ValidateHandshake compares it fail-closed for
equality, so bumping it would stop 0.1.x/0.2.x pairs syncing outright. Handshake
gains supports_partial_snapshot (field 5) and SnapshotBegin gains a tables scope
(field 2) where EMPTY means all tables — precisely what a pre-0.3.0 sender emits
and what a pre-0.3.0 receiver assumes. A peer that does not advertise the
capability is served a full snapshot: a scoped one would make it advance
last_applied_remote_seq to the snapshot's as-of and silently discard every delta
below it for the tables left out.

For the same reason a partial snapshot moves no watermark on either side. The
receiver keeps last_applied_remote_seq (carrying only the HLC across, which is
MAX-guarded), and the sender keeps _sentThruSeq, so the deltas the snapshot
overlaps are still pumped and merged by LWW.

Crash safety: snapshot debt now clears on the receiver's new SnapshotAck rather
than on enqueue, so an ungraceful exit part-way through re-streams the same scope
on reconnect. The ack rides the unbounded CONTROL lane like DeltaAck, so it
composes with 0.2.1's bounded 64-message inbox and its out-of-handshake snapshot.
Against a pre-0.3.0 peer the 0.2.x clear-on-enqueue behaviour is kept — there is
no ack to wait for — with the handshake's horizon gap check as the net.

Tests 199/199 (was 188): single-table baseline streams only that table, mixed-
version interop (peer without the capability gets a full snapshot), debt standing
until SnapshotAck, partial snapshot not advancing the receiver's watermark, no
SnapshotAck emitted to an old peer, end-to-end over two real databases, plus v2->v3
upgrade and per-table flag isolation.

README, ILocalDb remarks (which named this as the follow-up) and the umbrella
CLAUDE.md index updated.
2026-08-15 03:34:54 -04:00
Joseph Doherty 8befebd476 fix(localdb): snapshot review fixes — MAX-guarded complete watermark, keyset pagination, phantom-row guard
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
2026-07-17 23:50:42 -04:00
Joseph Doherty 130e9c918f feat(localdb): snapshot resync (LWW-merging, tombstone-carrying, delta resume)
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
2026-07-17 23:35:50 -04:00
Joseph Doherty 183901e8ac test(localdb): pin drift-watermark branch; guard writer loop against completed-lane spin 2026-07-17 22:50:33 -04:00
Joseph Doherty 92ccbac8db fix(localdb): sync session concurrency fixes — two-lane writer, teardown fault fidelity, per-entry drift, gated snapshot seam 2026-07-17 22:44:42 -04:00
Joseph Doherty 925623b878 feat(localdb): SyncSession protocol state machine (handshake fail-closed, bidirectional delta pump) 2026-07-17 22:26:03 -04:00