Compare commits

...

3 Commits

Author SHA1 Message Date
Joseph Doherty 1145df682e docs(index): OtOpcUa now registers LocalDb tables conditionally — baseline on enable, deregister self-heal on disable (row 39 closed) 2026-08-18 07:25:18 -04:00
Joseph Doherty 3e5a5fde81 fix(localdb): flush the control lane before a faulting session cancels its writer (0.3.1)
A session that faulted during the handshake cancelled its writer loop immediately,
which could kill its OWN Handshake while it still sat in the control lane: the
enqueue -> receive -> validate -> fault path has no yields, while the writer's
wakeup waits in the thread-pool queue, so with the peer's handshake already
buffered the cancel won the race almost every time (39/40 in the new pinning
test). Mismatch detection is symmetric — the peer computes its fail-closed
verdict FROM our Handshake — so the healthy peer never learned anything: in
tests it waited out the full 15 s receive cap (the register-row-41 flake on
SyncSessionTests.Handshake_DigestMismatch_FailsClosed), and in production it
would misclassify a permanent schema mismatch as a generic disconnect and
reconnect-loop at the fast backoff instead of taking the classified
FailedPrecondition -> LogError + max-backoff path.

RunAsync's fault teardown now completes both lanes so the writer drains what is
already enqueued and exits on its own, with a 2 s deferred cancel as the backstop
for a wedged transport. Caller-initiated cancellation is unchanged (the linked
token is already cancelled, so shutdown is not delayed), and nothing on the wire
changes — a 0.3.0 peer simply receives the Handshake it was always owed.

New SyncSessionTests.Handshake_FaultTeardown_StillTransmitsTheLocalHandshake pins
the race (40 concurrent preloaded-mismatch sessions; 39/40 red pre-fix, green
post-fix).

BatchBudgetTests hardened separately: the register's other flaky test
(Pump_RowLargerThanTheBudget_IsSentAloneRatherThanStallingTheStream) awaits the
scripted-peer channel without ever observing the session task, so ANY session
fault surfaced as an opaque 15 s timeout. A run-observing NextAsync overload now
races the session task at every read site in the class, so the next flake reports
the session's real exception instead. (The handshake-race root cause does not
structurally apply to this test — no handshake fault occurs there — so its
intermittent timeout is attributed to the fault-hiding wait, now removed.)

Suite 200/200 green. Closes ScadaBridge deferred-work register row 41 (cross-repo).
2026-08-18 07:11:14 -04:00
Joseph Doherty b62981ff80 fix(build): anchor pack/push scripts to their own repo root (0038d46 pattern, all seven)
Secrets/push.sh, Theme/pack.sh+push.sh, Audit/pack.sh+push.sh and Auth/pack.sh+push.sh
all carried the CWD-pack bug LocalDb's push.sh was cured of in 0038d46: a bare
'dotnet pack -c Release -o ./artifacts' packs whatever solution sits in the CALLER's
working directory, and the push scripts then shove the caller's ./artifacts/*.nupkg
onto the live Gitea feed (the incident that briefly published 25 unrelated packages).

Each script now cds to its own repo root, names its .slnx explicitly, writes to an
absolute artifacts path, and (push.sh) pushes an anchored glob. Verified by bash -n on
all seven plus a positive control: Audit's pack.sh run from a foreign CWD packed only
ZB.MOM.WW.Audit into its own artifacts dir and left the caller's directory untouched.
Push scripts deliberately NOT executed (they publish); pack half verified only.

Closes ScadaBridge deferred-work register row 42 (cross-repo).
2026-08-18 07:00:42 -04:00
12 changed files with 156 additions and 19 deletions
+1 -1
View File
File diff suppressed because one or more lines are too long
+7 -1
View File
@@ -1,4 +1,10 @@
#!/usr/bin/env bash
# pack.sh — produce the ZB.MOM.WW.Audit NuGet package into ./artifacts.
set -euo pipefail
dotnet pack -c Release -o ./artifacts
# cd to the repo root and name the solution + an absolute output path: `dotnet pack` with
# neither a project argument nor a fixed working directory packs whatever solution happens
# to sit in the CALLER's directory (the bug behind the LocalDb feed incident, 0038d46).
REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
cd "$REPO_ROOT"
dotnet pack ZB.MOM.WW.Audit.slnx -c Release -o "$REPO_ROOT/artifacts"
+13 -2
View File
@@ -16,9 +16,20 @@ set -euo pipefail
: "${GITEA_NUGET_SOURCE:?set GITEA_NUGET_SOURCE to your Gitea NuGet feed URL}"
: "${GITEA_NUGET_KEY:?set GITEA_NUGET_KEY to your Gitea access token}"
dotnet pack -c Release -o ./artifacts
# Named solution, named output, and a cd to the repo root, all for one reason: `dotnet pack`
# with neither a project argument nor a fixed working directory packs whatever solution
# happens to sit in the CALLER's directory — and then pushes it to a live feed under this
# script's name. Invoked by absolute path from another repo's checkout, the relative form
# packed and published 25 unrelated packages (LocalDb's copy of this script, fixed in
# 0038d46; recovered with `dotnet nuget delete`, but only because the token happened to
# carry delete rights). A publish script must be unambiguous about WHAT it publishes no
# matter where it is run.
REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
cd "$REPO_ROOT"
dotnet nuget push "./artifacts/*.nupkg" \
dotnet pack ZB.MOM.WW.Audit.slnx -c Release -o "$REPO_ROOT/artifacts"
dotnet nuget push "$REPO_ROOT/artifacts/*.nupkg" \
--source "$GITEA_NUGET_SOURCE" \
--api-key "$GITEA_NUGET_KEY" \
--skip-duplicate
+7 -1
View File
@@ -6,4 +6,10 @@
set -euo pipefail
dotnet pack -c Release -o ./artifacts
# cd to the repo root and name the solution + an absolute output path: `dotnet pack` with
# neither a project argument nor a fixed working directory packs whatever solution happens
# to sit in the CALLER's directory (the bug behind the LocalDb feed incident, 0038d46).
REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
cd "$REPO_ROOT"
dotnet pack ZB.MOM.WW.Auth.slnx -c Release -o "$REPO_ROOT/artifacts"
+13 -2
View File
@@ -16,9 +16,20 @@ set -euo pipefail
: "${GITEA_NUGET_SOURCE:?set GITEA_NUGET_SOURCE to your Gitea NuGet feed URL}"
: "${GITEA_NUGET_KEY:?set GITEA_NUGET_KEY to your Gitea access token}"
dotnet pack -c Release -o ./artifacts
# Named solution, named output, and a cd to the repo root, all for one reason: `dotnet pack`
# with neither a project argument nor a fixed working directory packs whatever solution
# happens to sit in the CALLER's directory — and then pushes it to a live feed under this
# script's name. Invoked by absolute path from another repo's checkout, the relative form
# packed and published 25 unrelated packages (LocalDb's copy of this script, fixed in
# 0038d46; recovered with `dotnet nuget delete`, but only because the token happened to
# carry delete rights). A publish script must be unambiguous about WHAT it publishes no
# matter where it is run.
REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
cd "$REPO_ROOT"
dotnet nuget push "./artifacts/*.nupkg" \
dotnet pack ZB.MOM.WW.Auth.slnx -c Release -o "$REPO_ROOT/artifacts"
dotnet nuget push "$REPO_ROOT/artifacts/*.nupkg" \
--source "$GITEA_NUGET_SOURCE" \
--api-key "$GITEA_NUGET_KEY" \
--skip-duplicate
+1 -1
View File
@@ -5,7 +5,7 @@
<Nullable>enable</Nullable>
<ImplicitUsings>enable</ImplicitUsings>
<LangVersion>latest</LangVersion>
<Version>0.3.0</Version>
<Version>0.3.1</Version>
<ManagePackageVersionsCentrally>true</ManagePackageVersionsCentrally>
<TreatWarningsAsErrors>true</TreatWarningsAsErrors>
<!-- NU1510 (package-pruning advice): the Replication and Tests projects carry a
@@ -27,6 +27,10 @@ internal sealed class SyncSession
// inbound batch rate (the peer's flow control), so its lane can safely be unbounded.
private const int DataLaneCapacity = 16;
// How long a faulting session's teardown lets the writer keep flushing already-enqueued
// messages before the backstop cancel; only ever reached when the transport itself stalls.
private static readonly TimeSpan FaultFlushGrace = TimeSpan.FromSeconds(2);
private readonly SqliteLocalDb _db;
private readonly OplogStore _store;
private readonly LwwApplier _applier;
@@ -140,7 +144,20 @@ internal sealed class SyncSession
{
// Handshake-phase faults (and the primary rethrow above) land here: the writer may
// still be running, so stop and observe it before the primary exception surfaces.
linked.Cancel();
// It must be allowed to DRAIN, not be cancelled outright: the Handshake enqueued at
// the top of HandshakeAsync can still be sitting in the control lane (the enqueue →
// receive → validate → fault path has no yields, while the writer's wakeup waits in
// the thread-pool queue), and that message is the only thing that lets the peer
// compute its own fail-closed verdict — mismatch detection is symmetric, so this
// node's locally-thrown verdict reaches the peer only as the Handshake it transmits.
// Cancelling first killed it almost every time the peer's handshake was already
// buffered, leaving the healthy peer to misread a schema mismatch as a generic
// disconnect and reconnect-loop at the fast backoff. Completing the lanes lets the
// writer send what is already enqueued and exit on its own; the deferred cancel is
// the backstop that keeps a wedged transport from holding teardown hostage.
control.Writer.TryComplete();
data.Writer.TryComplete();
linked.CancelAfter(FaultFlushGrace);
await DrainSecondaryAsync(writer);
throw;
}
@@ -70,6 +70,20 @@ public sealed class BatchBudgetTests : IDisposable
return msg!;
}
// Reads the next scripted-peer message while also observing the session task: a session that
// faults would otherwise leave this wait to run out the whole test timeout, reporting an
// opaque cancellation instead of the session's real exception.
private static async Task<SyncMessage> NextAsync(Task run, ChannelReader<SyncMessage> reader, CancellationToken ct)
{
var next = NextAsync(reader, ct);
if (ReferenceEquals(await Task.WhenAny(run, next), run))
{
await run;
throw new InvalidOperationException("The sync session ended before the expected message arrived.");
}
return await next;
}
private static async Task SwallowAsync(Task task)
{
try { await task; }
@@ -140,7 +154,7 @@ public sealed class BatchBudgetTests : IDisposable
var seen = new List<long>();
while (seen.Count < 20)
{
var msg = await NextAsync(fromSession, cts.Token);
var msg = await NextAsync(run, fromSession, cts.Token);
if (msg.MsgCase != SyncMessage.MsgOneofCase.DeltaBatch)
continue;
Assert.True(BatchBytes(msg.DeltaBatch) <= budget,
@@ -180,7 +194,7 @@ public sealed class BatchBudgetTests : IDisposable
var batches = new List<DeltaBatch>();
while (batches.Sum(b => b.Entries.Count) < 2)
{
var msg = await NextAsync(fromSession, cts.Token);
var msg = await NextAsync(run, fromSession, cts.Token);
if (msg.MsgCase == SyncMessage.MsgOneofCase.DeltaBatch)
batches.Add(msg.DeltaBatch);
}
@@ -220,7 +234,7 @@ public sealed class BatchBudgetTests : IDisposable
var seen = 0;
while (seen < 12)
{
var msg = await NextAsync(fromSession, cts.Token);
var msg = await NextAsync(run, fromSession, cts.Token);
if (msg.MsgCase != SyncMessage.MsgOneofCase.DeltaBatch)
continue;
Assert.True(msg.DeltaBatch.Entries.Count <= 5);
@@ -312,7 +326,7 @@ public sealed class BatchBudgetTests : IDisposable
}
SyncMessage ack;
do { ack = await NextAsync(fromSession, cts.Token); }
do { ack = await NextAsync(run, fromSession, cts.Token); }
while (ack.MsgCase != SyncMessage.MsgOneofCase.DeltaAck);
// The discriminator: uncoalesced, the FIRST ack could only cover seq 1.
@@ -348,7 +362,7 @@ public sealed class BatchBudgetTests : IDisposable
new SyncMessage { SnapshotBegin = new SnapshotBegin { AsOfSeq = 77 } }, cts.Token);
SyncMessage ack;
do { ack = await NextAsync(fromSession, cts.Token); }
do { ack = await NextAsync(run, fromSession, cts.Token); }
while (ack.MsgCase != SyncMessage.MsgOneofCase.DeltaAck);
Assert.Equal(1, ack.DeltaAck.AppliedThruSeq);
@@ -208,6 +208,50 @@ public sealed class SyncSessionTests : IDisposable
Assert.Empty(await ReadOrders(b.Db));
}
[Fact]
public async Task Handshake_FaultTeardown_StillTransmitsTheLocalHandshake()
{
// Schema-mismatch detection is symmetric: each side computes its verdict FROM the other's
// Handshake. So a session that faults on the peer's handshake must still flush its own —
// otherwise the healthy peer learns nothing and waits out its receive cap (tests) or
// misclassifies the mismatch as a generic disconnect and reconnect-loops (production).
//
// The race being pinned: with the peer's handshake already buffered, the path from
// enqueueing our Handshake on the control lane to the validation throw has no yields,
// while the writer's wakeup sits in the thread-pool queue — a teardown that cancels the
// writer can therefore kill the Handshake before it ever transmits. Preloading makes the
// sprint as tight as possible, and running the scenarios concurrently starves the pool
// the way a busy test suite does.
const int Iterations = 40;
var sides = new List<Side>(Iterations);
for (var i = 0; i < Iterations; i++)
sides.Add(await NewSideAsync());
using var cts = new CancellationTokenSource(RunTimeout);
var runs = new List<(Task Run, ChannelReader<SyncMessage> FromSession)>(Iterations);
foreach (var side in sides)
{
var (duplex, toSession, fromSession) = ScriptedPeer();
var mismatched = MatchingHandshake(side.Db, "peer");
mismatched.Tables[0].Digest = "not-the-local-digest";
await toSession.WriteAsync(new SyncMessage { Handshake = mismatched }, cts.Token);
runs.Add((side.Session.RunAsync(duplex, cts.Token), fromSession));
}
var lost = 0;
foreach (var (run, fromSession) in runs)
{
await Assert.ThrowsAsync<LocalDbSchemaMismatchException>(() => run);
var sent = false;
while (fromSession.TryRead(out var msg))
sent |= msg.MsgCase == SyncMessage.MsgOneofCase.Handshake;
if (!sent)
lost++;
}
Assert.True(lost == 0, $"{lost}/{Iterations} sessions tore down without transmitting their own Handshake");
}
[Fact]
public async Task Handshake_SchemaVersionMismatch_FailsClosed()
{
+13 -2
View File
@@ -16,9 +16,20 @@ set -euo pipefail
: "${GITEA_NUGET_SOURCE:?set GITEA_NUGET_SOURCE to your Gitea NuGet feed URL}"
: "${GITEA_NUGET_KEY:?set GITEA_NUGET_KEY to your Gitea access token}"
dotnet pack -c Release -o ./artifacts
# Named solution, named output, and a cd to the repo root, all for one reason: `dotnet pack`
# with neither a project argument nor a fixed working directory packs whatever solution
# happens to sit in the CALLER's directory — and then pushes it to a live feed under this
# script's name. Invoked by absolute path from another repo's checkout, the relative form
# packed and published 25 unrelated packages (LocalDb's copy of this script, fixed in
# 0038d46; recovered with `dotnet nuget delete`, but only because the token happened to
# carry delete rights). A publish script must be unambiguous about WHAT it publishes no
# matter where it is run.
REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
cd "$REPO_ROOT"
dotnet nuget push "./artifacts/*.nupkg" \
dotnet pack ZB.MOM.WW.Secrets.slnx -c Release -o "$REPO_ROOT/artifacts"
dotnet nuget push "$REPO_ROOT/artifacts/*.nupkg" \
--source "$GITEA_NUGET_SOURCE" \
--api-key "$GITEA_NUGET_KEY" \
--skip-duplicate
+7 -1
View File
@@ -6,4 +6,10 @@
set -euo pipefail
dotnet pack -c Release -o ./artifacts
# cd to the repo root and name the solution + an absolute output path: `dotnet pack` with
# neither a project argument nor a fixed working directory packs whatever solution happens
# to sit in the CALLER's directory (the bug behind the LocalDb feed incident, 0038d46).
REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
cd "$REPO_ROOT"
dotnet pack ZB.MOM.WW.Theme.slnx -c Release -o "$REPO_ROOT/artifacts"
+13 -2
View File
@@ -16,9 +16,20 @@ set -euo pipefail
: "${GITEA_NUGET_SOURCE:?set GITEA_NUGET_SOURCE to your Gitea NuGet feed URL}"
: "${GITEA_NUGET_KEY:?set GITEA_NUGET_KEY to your Gitea access token}"
dotnet pack -c Release -o ./artifacts
# Named solution, named output, and a cd to the repo root, all for one reason: `dotnet pack`
# with neither a project argument nor a fixed working directory packs whatever solution
# happens to sit in the CALLER's directory — and then pushes it to a live feed under this
# script's name. Invoked by absolute path from another repo's checkout, the relative form
# packed and published 25 unrelated packages (LocalDb's copy of this script, fixed in
# 0038d46; recovered with `dotnet nuget delete`, but only because the token happened to
# carry delete rights). A publish script must be unambiguous about WHAT it publishes no
# matter where it is run.
REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
cd "$REPO_ROOT"
dotnet nuget push "./artifacts/*.nupkg" \
dotnet pack ZB.MOM.WW.Theme.slnx -c Release -o "$REPO_ROOT/artifacts"
dotnet nuget push "$REPO_ROOT/artifacts/*.nupkg" \
--source "$GITEA_NUGET_SOURCE" \
--api-key "$GITEA_NUGET_KEY" \
--skip-duplicate