fix(localdb): flush the control lane before a faulting session cancels its writer (0.3.1)
A session that faulted during the handshake cancelled its writer loop immediately, which could kill its OWN Handshake while it still sat in the control lane: the enqueue -> receive -> validate -> fault path has no yields, while the writer's wakeup waits in the thread-pool queue, so with the peer's handshake already buffered the cancel won the race almost every time (39/40 in the new pinning test). Mismatch detection is symmetric — the peer computes its fail-closed verdict FROM our Handshake — so the healthy peer never learned anything: in tests it waited out the full 15 s receive cap (the register-row-41 flake on SyncSessionTests.Handshake_DigestMismatch_FailsClosed), and in production it would misclassify a permanent schema mismatch as a generic disconnect and reconnect-loop at the fast backoff instead of taking the classified FailedPrecondition -> LogError + max-backoff path. RunAsync's fault teardown now completes both lanes so the writer drains what is already enqueued and exits on its own, with a 2 s deferred cancel as the backstop for a wedged transport. Caller-initiated cancellation is unchanged (the linked token is already cancelled, so shutdown is not delayed), and nothing on the wire changes — a 0.3.0 peer simply receives the Handshake it was always owed. New SyncSessionTests.Handshake_FaultTeardown_StillTransmitsTheLocalHandshake pins the race (40 concurrent preloaded-mismatch sessions; 39/40 red pre-fix, green post-fix). BatchBudgetTests hardened separately: the register's other flaky test (Pump_RowLargerThanTheBudget_IsSentAloneRatherThanStallingTheStream) awaits the scripted-peer channel without ever observing the session task, so ANY session fault surfaced as an opaque 15 s timeout. A run-observing NextAsync overload now races the session task at every read site in the class, so the next flake reports the session's real exception instead. (The handshake-race root cause does not structurally apply to this test — no handshake fault occurs there — so its intermittent timeout is attributed to the fault-hiding wait, now removed.) Suite 200/200 green. Closes ScadaBridge deferred-work register row 41 (cross-repo).
This commit is contained in:
@@ -27,6 +27,10 @@ internal sealed class SyncSession
|
||||
// inbound batch rate (the peer's flow control), so its lane can safely be unbounded.
|
||||
private const int DataLaneCapacity = 16;
|
||||
|
||||
// How long a faulting session's teardown lets the writer keep flushing already-enqueued
|
||||
// messages before the backstop cancel; only ever reached when the transport itself stalls.
|
||||
private static readonly TimeSpan FaultFlushGrace = TimeSpan.FromSeconds(2);
|
||||
|
||||
private readonly SqliteLocalDb _db;
|
||||
private readonly OplogStore _store;
|
||||
private readonly LwwApplier _applier;
|
||||
@@ -140,7 +144,20 @@ internal sealed class SyncSession
|
||||
{
|
||||
// Handshake-phase faults (and the primary rethrow above) land here: the writer may
|
||||
// still be running, so stop and observe it before the primary exception surfaces.
|
||||
linked.Cancel();
|
||||
// It must be allowed to DRAIN, not be cancelled outright: the Handshake enqueued at
|
||||
// the top of HandshakeAsync can still be sitting in the control lane (the enqueue →
|
||||
// receive → validate → fault path has no yields, while the writer's wakeup waits in
|
||||
// the thread-pool queue), and that message is the only thing that lets the peer
|
||||
// compute its own fail-closed verdict — mismatch detection is symmetric, so this
|
||||
// node's locally-thrown verdict reaches the peer only as the Handshake it transmits.
|
||||
// Cancelling first killed it almost every time the peer's handshake was already
|
||||
// buffered, leaving the healthy peer to misread a schema mismatch as a generic
|
||||
// disconnect and reconnect-loop at the fast backoff. Completing the lanes lets the
|
||||
// writer send what is already enqueued and exit on its own; the deferred cancel is
|
||||
// the backstop that keeps a wedged transport from holding teardown hostage.
|
||||
control.Writer.TryComplete();
|
||||
data.Writer.TryComplete();
|
||||
linked.CancelAfter(FaultFlushGrace);
|
||||
await DrainSecondaryAsync(writer);
|
||||
throw;
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user