fix(localdb): flush the control lane before a faulting session cancels its writer (0.3.1)

A session that faulted during the handshake cancelled its writer loop immediately, which could kill its OWN Handshake while it still sat in the control lane: the enqueue -> receive -> validate -> fault path has no yields, while the writer's wakeup waits in the thread-pool queue, so with the peer's handshake already buffered the cancel won the race almost every time (39/40 in the new pinning test). Mismatch detection is symmetric — the peer computes its fail-closed verdict FROM our Handshake — so the healthy peer never learned anything: in tests it waited out the full 15 s receive cap (the register-row-41 flake on SyncSessionTests.Handshake_DigestMismatch_FailsClosed), and in production it would misclassify a permanent schema mismatch as a generic disconnect and reconnect-loop at the fast backoff instead of taking the classified FailedPrecondition -> LogError + max-backoff path. RunAsync's fault teardown now completes both lanes so the writer drains what is already enqueued and exits on its own, with a 2 s deferred cancel as the backstop for a wedged transport. Caller-initiated cancellation is unchanged (the linked token is already cancelled, so shutdown is not delayed), and nothing on the wire changes — a 0.3.0 peer simply receives the Handshake it was always owed. New SyncSessionTests.Handshake_FaultTeardown_StillTransmitsTheLocalHandshake pins the race (40 concurrent preloaded-mismatch sessions; 39/40 red pre-fix, green post-fix). BatchBudgetTests hardened separately: the register's other flaky test (Pump_RowLargerThanTheBudget_IsSentAloneRatherThanStallingTheStream) awaits the scripted-peer channel without ever observing the session task, so ANY session fault surfaced as an opaque 15 s timeout. A run-observing NextAsync overload now races the session task at every read site in the class, so the next flake reports the session's real exception instead. (The handshake-race root cause does not structurally apply to this test — no handshake fault occurs there — so its intermittent timeout is attributed to the fault-hiding wait, now removed.) Suite 200/200 green. Closes ScadaBridge deferred-work register row 41 (cross-repo).
2026-08-18 07:11:14 -04:00
parent b62981ff80
commit 3e5a5fde81
4 changed files with 82 additions and 7 deletions
@@ -208,6 +208,50 @@ public sealed class SyncSessionTests : IDisposable
        Assert.Empty(await ReadOrders(b.Db));
    }

+    [Fact]
+    public async Task Handshake_FaultTeardown_StillTransmitsTheLocalHandshake()
+    {
+        // Schema-mismatch detection is symmetric: each side computes its verdict FROM the other's
+        // Handshake. So a session that faults on the peer's handshake must still flush its own —
+        // otherwise the healthy peer learns nothing and waits out its receive cap (tests) or
+        // misclassifies the mismatch as a generic disconnect and reconnect-loops (production).
+        //
+        // The race being pinned: with the peer's handshake already buffered, the path from
+        // enqueueing our Handshake on the control lane to the validation throw has no yields,
+        // while the writer's wakeup sits in the thread-pool queue — a teardown that cancels the
+        // writer can therefore kill the Handshake before it ever transmits. Preloading makes the
+        // sprint as tight as possible, and running the scenarios concurrently starves the pool
+        // the way a busy test suite does.
+        const int Iterations = 40;
+        var sides = new List<Side>(Iterations);
+        for (var i = 0; i < Iterations; i++)
+            sides.Add(await NewSideAsync());
+
+        using var cts = new CancellationTokenSource(RunTimeout);
+        var runs = new List<(Task Run, ChannelReader<SyncMessage> FromSession)>(Iterations);
+        foreach (var side in sides)
+        {
+            var (duplex, toSession, fromSession) = ScriptedPeer();
+            var mismatched = MatchingHandshake(side.Db, "peer");
+            mismatched.Tables[0].Digest = "not-the-local-digest";
+            await toSession.WriteAsync(new SyncMessage { Handshake = mismatched }, cts.Token);
+            runs.Add((side.Session.RunAsync(duplex, cts.Token), fromSession));
+        }
+
+        var lost = 0;
+        foreach (var (run, fromSession) in runs)
+        {
+            await Assert.ThrowsAsync<LocalDbSchemaMismatchException>(() => run);
+            var sent = false;
+            while (fromSession.TryRead(out var msg))
+                sent |= msg.MsgCase == SyncMessage.MsgOneofCase.Handshake;
+            if (!sent)
+                lost++;
+        }
+
+        Assert.True(lost == 0, $"{lost}/{Iterations} sessions tore down without transmitting their own Handshake");
+    }
+
    [Fact]
    public async Task Handshake_SchemaVersionMismatch_FailsClosed()
    {