fix(localdb): flush the control lane before a faulting session cancels its writer (0.3.1)

A session that faulted during the handshake cancelled its writer loop immediately,
which could kill its OWN Handshake while it still sat in the control lane: the
enqueue -> receive -> validate -> fault path has no yields, while the writer's
wakeup waits in the thread-pool queue, so with the peer's handshake already
buffered the cancel won the race almost every time (39/40 in the new pinning
test). Mismatch detection is symmetric — the peer computes its fail-closed
verdict FROM our Handshake — so the healthy peer never learned anything: in
tests it waited out the full 15 s receive cap (the register-row-41 flake on
SyncSessionTests.Handshake_DigestMismatch_FailsClosed), and in production it
would misclassify a permanent schema mismatch as a generic disconnect and
reconnect-loop at the fast backoff instead of taking the classified
FailedPrecondition -> LogError + max-backoff path.

RunAsync's fault teardown now completes both lanes so the writer drains what is
already enqueued and exits on its own, with a 2 s deferred cancel as the backstop
for a wedged transport. Caller-initiated cancellation is unchanged (the linked
token is already cancelled, so shutdown is not delayed), and nothing on the wire
changes — a 0.3.0 peer simply receives the Handshake it was always owed.

New SyncSessionTests.Handshake_FaultTeardown_StillTransmitsTheLocalHandshake pins
the race (40 concurrent preloaded-mismatch sessions; 39/40 red pre-fix, green
post-fix).

BatchBudgetTests hardened separately: the register's other flaky test
(Pump_RowLargerThanTheBudget_IsSentAloneRatherThanStallingTheStream) awaits the
scripted-peer channel without ever observing the session task, so ANY session
fault surfaced as an opaque 15 s timeout. A run-observing NextAsync overload now
races the session task at every read site in the class, so the next flake reports
the session's real exception instead. (The handshake-race root cause does not
structurally apply to this test — no handshake fault occurs there — so its
intermittent timeout is attributed to the fault-hiding wait, now removed.)

Suite 200/200 green. Closes ScadaBridge deferred-work register row 41 (cross-repo).
This commit is contained in:
Joseph Doherty
2026-08-18 07:11:14 -04:00
parent b62981ff80
commit 3e5a5fde81
4 changed files with 82 additions and 7 deletions
@@ -27,6 +27,10 @@ internal sealed class SyncSession
// inbound batch rate (the peer's flow control), so its lane can safely be unbounded.
private const int DataLaneCapacity = 16;
// How long a faulting session's teardown lets the writer keep flushing already-enqueued
// messages before the backstop cancel; only ever reached when the transport itself stalls.
private static readonly TimeSpan FaultFlushGrace = TimeSpan.FromSeconds(2);
private readonly SqliteLocalDb _db;
private readonly OplogStore _store;
private readonly LwwApplier _applier;
@@ -140,7 +144,20 @@ internal sealed class SyncSession
{
// Handshake-phase faults (and the primary rethrow above) land here: the writer may
// still be running, so stop and observe it before the primary exception surfaces.
linked.Cancel();
// It must be allowed to DRAIN, not be cancelled outright: the Handshake enqueued at
// the top of HandshakeAsync can still be sitting in the control lane (the enqueue →
// receive → validate → fault path has no yields, while the writer's wakeup waits in
// the thread-pool queue), and that message is the only thing that lets the peer
// compute its own fail-closed verdict — mismatch detection is symmetric, so this
// node's locally-thrown verdict reaches the peer only as the Handshake it transmits.
// Cancelling first killed it almost every time the peer's handshake was already
// buffered, leaving the healthy peer to misread a schema mismatch as a generic
// disconnect and reconnect-loop at the fast backoff. Completing the lanes lets the
// writer send what is already enqueued and exit on its own; the deferred cancel is
// the backstop that keeps a wedged transport from holding teardown hostage.
control.Writer.TryComplete();
data.Writer.TryComplete();
linked.CancelAfter(FaultFlushGrace);
await DrainSecondaryAsync(writer);
throw;
}