fix(localdb): root-cause the soak disk-I/O failure + ship the hardening follow-ups
The Phase 2 soak's "LocalDb fails under load" blocker is NOT a product defect. Root cause: host-side (macOS) sqlite3 reads of the live, bind-mounted WAL databases. POSIX advisory locks do not propagate across the virtiofs boundary, so the host reader believes it is the only connection, checkpoints on close and resets the WAL to 0 bytes under the container. The container's still-mapped WAL index then references frames that no longer exist and every subsequent statement fails SQLITE_IOERR_SHORT_READ (522) -> primary code 10, permanently until the process reopens the database. Reproduced on demand both on the rig (one sqlite3 SELECT reset a 4.6 MiB WAL and produced the first error one second later) and in a minimal python:3.12-alpine repro with no LocalDb or .NET involved. Refuted the converse: a freshly-reopened node sustained the full soak load 10+ minutes with zero errors. The original brief's isolation was confounded - both nodes had been poisoned by the same sampling pass, and a poisoned standby looks healthy only because it issues almost no statements. Corrections annotated in place. Hardening shipped alongside: - SqliteErrorCodes.Describe: log SQLite primary AND extended codes at the LocalDb-adjacent catch sites (the missing extended code is what made the original diagnosis so slow). - SiteAuditTelemetryActor: stop touching ActorContext across an await (NotSupportedException), with regression coverage. - infra/reseed.sh: apply the MSSQL init scripts explicitly. The official mssql/server image does not implement /docker-entrypoint-initdb.d, so a fresh volume hung the reseed forever; compose mounts annotated as informational. Deliberately NOT done: detect-and-reopen self-heal in SqliteLocalDb. It defends only against external interference, which is now prevented at the source, and same-kernel production readers see the locks correctly. Build 0 warnings; SiteAuditTelemetryActorTests 9/9, SiteEventLogging 70/70. Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
This commit is contained in:
+52
@@ -5,6 +5,7 @@ using Microsoft.Extensions.Logging.Abstractions;
|
||||
using Microsoft.Extensions.Options;
|
||||
using NSubstitute;
|
||||
using NSubstitute.ExceptionExtensions;
|
||||
using NSubstitute.ReceivedExtensions;
|
||||
using ZB.MOM.WW.ScadaBridge.AuditLog.Site.Telemetry;
|
||||
using ZB.MOM.WW.Audit;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Types.Audit;
|
||||
@@ -451,4 +452,55 @@ public class SiteAuditTelemetryActorTests : TestKit
|
||||
await _client.DidNotReceiveWithAnyArgs().IngestCachedTelemetryAsync(default!, default);
|
||||
await _queue.DidNotReceiveWithAnyArgs().ReadPendingCachedTelemetryAsync(default, default);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Regression for the 2026-07-20 rig finding (known-issues/2026-07-20-localdb-disk-io-error-under-load.md §8):
|
||||
/// both drain handlers await with <c>ConfigureAwait(false)</c>, so when a
|
||||
/// dependency call completes off the actor thread (as real SQLite/gRPC
|
||||
/// always do) the <c>finally</c>-block re-arm runs on a pool thread with no
|
||||
/// active ActorContext. Depending on what that thread's thread-static cell
|
||||
/// slot holds, <c>Context</c>/<c>Self</c> either throw
|
||||
/// <c>NotSupportedException: There is no active ActorContext</c> (crashing
|
||||
/// the actor once per drain — the rig's logged variant) or silently resolve
|
||||
/// to a STALE cell of some other actor, misrouting the tick so the drain
|
||||
/// loop stops. The two assertions below catch one variant each. Every other
|
||||
/// test in this class masks the bug by returning already-completed tasks
|
||||
/// from the mocks, which keeps the continuations on the actor thread.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public async Task Drains_Whose_Awaits_Complete_Off_The_Actor_Thread_Keep_Draining_Without_Crashing()
|
||||
{
|
||||
// Task.Delay completes on a timer thread; with ConfigureAwait(false)
|
||||
// everything after the await — including the finally-block re-arm —
|
||||
// stays off the actor context.
|
||||
_queue.ReadPendingAsync(Arg.Any<int>(), Arg.Any<CancellationToken>())
|
||||
.Returns(async _ =>
|
||||
{
|
||||
await Task.Delay(25).ConfigureAwait(false);
|
||||
return (IReadOnlyList<AuditEvent>)Array.Empty<AuditEvent>();
|
||||
});
|
||||
_queue.ReadPendingCachedTelemetryAsync(Arg.Any<int>(), Arg.Any<CancellationToken>())
|
||||
.Returns(async _ =>
|
||||
{
|
||||
await Task.Delay(25).ConfigureAwait(false);
|
||||
return (IReadOnlyList<AuditEvent>)Array.Empty<AuditEvent>();
|
||||
});
|
||||
|
||||
// Variant 1 (throw → restart storm): any NotSupportedException logged
|
||||
// during the run fails the filter. Variant 2 (silent misroute → the
|
||||
// drain loop stalls after the first tick): the sustained-drain
|
||||
// assertion inside fails because reads stop at 1.
|
||||
await EventFilter.Exception<NotSupportedException>().ExpectAsync(0, async () =>
|
||||
{
|
||||
CreateActorWithCachedDrain(Opts(busySeconds: 1, idleSeconds: 1));
|
||||
|
||||
// Three full cycles prove the finally-block re-arm works from an
|
||||
// off-context thread: tick → drain → re-arm → tick → …
|
||||
await AwaitAssertAsync(async () =>
|
||||
{
|
||||
await _queue.Received(Quantity.Within(3, int.MaxValue)).ReadPendingAsync(Arg.Any<int>(), Arg.Any<CancellationToken>());
|
||||
await _queue.Received(Quantity.Within(3, int.MaxValue)).ReadPendingCachedTelemetryAsync(Arg.Any<int>(), Arg.Any<CancellationToken>());
|
||||
}, TimeSpan.FromSeconds(10));
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user