dohertj2/scadaproj
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix(localdb): flush the control lane before a faulting session cancels its writer (0.3.1)

Browse Source 
A session that faulted during the handshake cancelled its writer loop immediately,
which could kill its OWN Handshake while it still sat in the control lane: the
enqueue -> receive -> validate -> fault path has no yields, while the writer's
wakeup waits in the thread-pool queue, so with the peer's handshake already
buffered the cancel won the race almost every time (39/40 in the new pinning
test). Mismatch detection is symmetric — the peer computes its fail-closed
verdict FROM our Handshake — so the healthy peer never learned anything: in
tests it waited out the full 15 s receive cap (the register-row-41 flake on
SyncSessionTests.Handshake_DigestMismatch_FailsClosed), and in production it
would misclassify a permanent schema mismatch as a generic disconnect and
reconnect-loop at the fast backoff instead of taking the classified
FailedPrecondition -> LogError + max-backoff path.

RunAsync's fault teardown now completes both lanes so the writer drains what is
already enqueued and exits on its own, with a 2 s deferred cancel as the backstop
for a wedged transport. Caller-initiated cancellation is unchanged (the linked
token is already cancelled, so shutdown is not delayed), and nothing on the wire
changes — a 0.3.0 peer simply receives the Handshake it was always owed.

New SyncSessionTests.Handshake_FaultTeardown_StillTransmitsTheLocalHandshake pins
the race (40 concurrent preloaded-mismatch sessions; 39/40 red pre-fix, green
post-fix).

BatchBudgetTests hardened separately: the register's other flaky test
(Pump_RowLargerThanTheBudget_IsSentAloneRatherThanStallingTheStream) awaits the
scripted-peer channel without ever observing the session task, so ANY session
fault surfaced as an opaque 15 s timeout. A run-observing NextAsync overload now
races the session task at every read site in the class, so the next flake reports
the session's real exception instead. (The handshake-race root cause does not
structurally apply to this test — no handshake fault occurs there — so its
intermittent timeout is attributed to the fault-hiding wait, now removed.)

Suite 200/200 green. Closes ScadaBridge deferred-work register row 41 (cross-repo).
This commit is contained in:
Joseph Doherty
2026-08-18 07:11:14 -04:00
parent b62981ff80
commit 3e5a5fde81
4 changed files with 82 additions and 7 deletions
Download Patch File Download Diff File Expand all files Collapse all files
ZB.MOM.WW.LocalDb/tests/ZB.MOM.WW.LocalDb.Tests/BatchBudgetTests.cs
+19 -5
View File
@@ -70,6 +70,20 @@ public sealed class BatchBudgetTests : IDisposable
return msg!;
}
// Reads the next scripted-peer message while also observing the session task: a session that
// faults would otherwise leave this wait to run out the whole test timeout, reporting an
// opaque cancellation instead of the session's real exception.
private static async Task<SyncMessage> NextAsync(Task run, ChannelReader<SyncMessage> reader, CancellationToken ct)
{
var next = NextAsync(reader, ct);
if (ReferenceEquals(await Task.WhenAny(run, next), run))
{
await run;
throw new InvalidOperationException("The sync session ended before the expected message arrived.");
}
return await next;
}
private static async Task SwallowAsync(Task task)
{
try { await task; }
@@ -140,7 +154,7 @@ public sealed class BatchBudgetTests : IDisposable
var seen = new List<long>();
while (seen.Count < 20)
{
var msg = await NextAsync(fromSession, cts.Token);
var msg = await NextAsync(run, fromSession, cts.Token);
if (msg.MsgCase != SyncMessage.MsgOneofCase.DeltaBatch)
continue;
Assert.True(BatchBytes(msg.DeltaBatch) <= budget,
@@ -180,7 +194,7 @@ public sealed class BatchBudgetTests : IDisposable
var batches = new List<DeltaBatch>();
while (batches.Sum(b => b.Entries.Count) < 2)
{
var msg = await NextAsync(fromSession, cts.Token);
var msg = await NextAsync(run, fromSession, cts.Token);
if (msg.MsgCase == SyncMessage.MsgOneofCase.DeltaBatch)
batches.Add(msg.DeltaBatch);
}
@@ -220,7 +234,7 @@ public sealed class BatchBudgetTests : IDisposable
var seen = 0;
while (seen < 12)
{
var msg = await NextAsync(fromSession, cts.Token);
var msg = await NextAsync(run, fromSession, cts.Token);
if (msg.MsgCase != SyncMessage.MsgOneofCase.DeltaBatch)
continue;
Assert.True(msg.DeltaBatch.Entries.Count <= 5);
@@ -312,7 +326,7 @@ public sealed class BatchBudgetTests : IDisposable
}
SyncMessage ack;
do { ack = await NextAsync(fromSession, cts.Token); }
do { ack = await NextAsync(run, fromSession, cts.Token); }
while (ack.MsgCase != SyncMessage.MsgOneofCase.DeltaAck);
// The discriminator: uncoalesced, the FIRST ack could only cover seq 1.
@@ -348,7 +362,7 @@ public sealed class BatchBudgetTests : IDisposable
new SyncMessage { SnapshotBegin = new SnapshotBegin { AsOfSeq = 77 } }, cts.Token);
SyncMessage ack;
do { ack = await NextAsync(fromSession, cts.Token); }
do { ack = await NextAsync(run, fromSession, cts.Token); }
while (ack.MsgCase != SyncMessage.MsgOneofCase.DeltaAck);
Assert.Equal(1, ack.DeltaAck.AppliedThruSeq);