fix(central): review findings — no client-side audit truncation, insert-first upsert, QI-safe scripts, honest operator replies
Six adversarial-review findings in the central SQL/ingest layer. F1 (AuditLogRepository.InsertChunkAsync) — the set-based ingest declared each string parameter at its COLUMN width (Actor/Target 256, Action 64, Outcome 16, Category 32, SourceNode 64), so SqlClient truncated an over-long value at bind time and committed the mutilated row — silent, in an append-only store, with no PayloadTruncated flag — while the per-row and reconciliation paths sent the same value in full and let the server reject it with 2628. Bind at the value's own length instead; explicit SqlDbType is kept (it fixes the VALUES constructor's derived column types and datetime2 precision). Design: reject everywhere, truncate nowhere — matching today's per-row behaviour. F2 (SiteCallAuditRepository.UpsertAsync) — the single-statement upsert ran the monotonic UPDATE first and INSERTed only if nothing matched. Two writers racing the first packet of one TrackedOperationId (the cached dual-write and the reconciliation pull carry DIFFERENT lifecycle states) both matched nothing, and the loser then skipped its INSERT or swallowed a 2627 — dropping its Status/RetryCount/HttpStatus/TerminalAtUtc. Legs swapped to `IF NOT EXISTS … INSERT; UPDATE <monotonic>` — still one round trip, and the loser's UPDATE now lands on the winner's row. The duplicate-key catch re-runs the monotonic UPDATE for the same reason. Moved to raw SQL with explicitly-typed parameters so the intricate rank predicate exists in exactly one place (an untyped DateTime would bind as `datetime` and round the freshness tiebreaker). F3 (docs/plans/sql/*.sql) — filtered-index DDL failed with error 1934 under the documented `docker exec … sqlcmd` path, which defaults QUOTED_IDENTIFIER OFF; once IX_Notifications_Delivered exists, QI-OFF DML on Notifications fails too. All four scripts now open with `SET QUOTED_IDENTIFIER ON; SET ANSI_NULLS ON; GO` (own batch, so it is in force when the next batch parses), and the migration convention in Component-ConfigurationDatabase.md documents `sqlcmd -I`. Verified live: the pre-fix script fails 1934 without -I, the fixed one applies. F4 (SiteCallAuditActor) — the off-mailbox reconciliation/purge passes reuse the injected repository, so tests drove one DbContext from the pass and a mailbox handler concurrently. Serialized at the CALL via a private SerializedRepository wrapper applied only by the test constructors, rather than running the pass on-mailbox: production keeps its PipeTo shape untouched, and the existing "a blocked drain does not stall ingest/query/KPI" regression tests stay meaningful (they would have been invalidated by suspending the mailbox). F5 (AuditLogIngestActor) — when the batch failed because the 20 s IngestBudget expired, the per-row fallback reused the same expired token: N instant failures, N counter bumps, zero accepted. The fallback now gets a fresh 5 s budget (inside the 30 s outer Ask), and a blown budget bumps the failure counter ONCE for the batch instead of once per row. F6 (NotificationOutboxRepository.UpdateAsync) — ExecuteUpdate's row count was discarded, so an operator Retry/Discard of a notification the retention purge had already deleted reported success (the pre-ExecuteUpdate code threw DbUpdateConcurrencyException). UpdateAsync now returns whether a row matched; the operator one-shots answer "notification not found" and emit no audit row for the action that did not happen, while the dispatcher logs a warning (its delivery already happened; nothing to retry). GetByIdAsync switched to AsNoTracking since the write is out-of-band. Tests: 5 new SQL-backed regressions (over-long Target rejected on both paths + boundary round-trip; concurrent first-write and already-created-by-another-writer upserts; vanished-row UpdateAsync), a token-identity pin on the ingest fallback, a repository-concurrency detector for the SiteCallAudit passes, and vanished-row operator-path tests. The F1/F2/F4 regressions were each confirmed failing against the pre-fix code. Suites: ConfigurationDatabase 369, AuditLog 378, SiteCallAudit 66, NotificationOutbox 152 — all green, solution builds with 0 warnings.
This commit is contained in:
Joseph Doherty
+82
-1
@@ -707,6 +707,85 @@ public class AuditLogRepositoryTests : IClassFixture<MsSqlMigrationFixture>
|
||||
Assert.Equal(250, count);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// An over-long value must be REJECTED by the server on every write path, not
|
||||
/// clipped to fit by the client. The set-based path used to declare each
|
||||
/// string parameter at its COLUMN width (Target/Actor 256, Action 64,
|
||||
/// Outcome 16, Category 32, SourceNode 64), which makes SqlClient truncate the
|
||||
/// value at bind time and commit the shortened row — in an APPEND-ONLY audit
|
||||
/// store, with no PayloadTruncated flag to admit it happened — while the
|
||||
/// per-row and reconciliation paths sent the same value in full and let the
|
||||
/// server raise 2628/8152. Reject-everywhere is the contract; this test pins
|
||||
/// both halves of it.
|
||||
/// </summary>
|
||||
[SkippableFact]
|
||||
public async Task InsertManyIfNotExistsAsync_TargetLongerThanColumn_IsRejected_NotTruncated()
|
||||
{
|
||||
Skip.IfNot(_fixture.Available, _fixture.SkipReason);
|
||||
|
||||
var siteId = NewSiteId();
|
||||
|
||||
// dbo.AuditLog.Target is nvarchar(256).
|
||||
var overlongTarget = new string('T', 300);
|
||||
var evt = NewEvent(
|
||||
siteId,
|
||||
occurredAtUtc: new DateTime(2026, 5, 22, 12, 0, 0, DateTimeKind.Utc),
|
||||
target: overlongTarget);
|
||||
|
||||
await using var batchContext = CreateContext();
|
||||
var batchRepo = new AuditLogRepository(batchContext);
|
||||
|
||||
// 2628 = "String or binary data would be truncated in table …" (SQL 2019+),
|
||||
// 8152 = its pre-2019 predecessor. Either proves the server saw the full value.
|
||||
var batchEx = await Assert.ThrowsAsync<SqlException>(
|
||||
() => batchRepo.InsertManyIfNotExistsAsync(new[] { evt }));
|
||||
Assert.Contains(batchEx.Number, new[] { 2628, 8152 });
|
||||
|
||||
// The per-row path — which the batch path falls back to — rejects it too.
|
||||
await using var rowContext = CreateContext();
|
||||
var rowRepo = new AuditLogRepository(rowContext);
|
||||
var rowEx = await Assert.ThrowsAsync<SqlException>(
|
||||
() => rowRepo.InsertIfNotExistsAsync(evt));
|
||||
Assert.Contains(rowEx.Number, new[] { 2628, 8152 });
|
||||
|
||||
// Nothing landed — in particular, no 256-character mutilated copy.
|
||||
await using var readContext = CreateContext();
|
||||
var rows = await readContext.Set<AuditLogRow>()
|
||||
.Where(e => e.SourceSiteId == siteId)
|
||||
.ToListAsync();
|
||||
|
||||
Assert.Empty(rows);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// The companion to the rejection test: a value that FITS must still round-trip
|
||||
/// through the set-based path byte for byte, at the exact column boundary.
|
||||
/// </summary>
|
||||
[SkippableFact]
|
||||
public async Task InsertManyIfNotExistsAsync_TargetAtColumnLimit_RoundTripsIntact()
|
||||
{
|
||||
Skip.IfNot(_fixture.Available, _fixture.SkipReason);
|
||||
|
||||
var siteId = NewSiteId();
|
||||
var boundaryTarget = new string('T', 256);
|
||||
var evt = NewEvent(
|
||||
siteId,
|
||||
occurredAtUtc: new DateTime(2026, 5, 22, 12, 30, 0, DateTimeKind.Utc),
|
||||
target: boundaryTarget);
|
||||
|
||||
await using var context = CreateContext();
|
||||
var repo = new AuditLogRepository(context);
|
||||
Assert.Equal(1, await repo.InsertManyIfNotExistsAsync(new[] { evt }));
|
||||
|
||||
await using var readContext = CreateContext();
|
||||
var loaded = await readContext.Set<AuditLogRow>()
|
||||
.Where(e => e.SourceSiteId == siteId)
|
||||
.ToListAsync();
|
||||
|
||||
Assert.Single(loaded);
|
||||
Assert.Equal(boundaryTarget, loaded[0].Target);
|
||||
}
|
||||
|
||||
[SkippableFact]
|
||||
public async Task InsertManyIfNotExistsAsync_EmptyBatch_IsANoOp()
|
||||
{
|
||||
@@ -1346,12 +1425,14 @@ public class AuditLogRepositoryTests : IClassFixture<MsSqlMigrationFixture>
|
||||
string? errorMessage = null,
|
||||
Guid? executionId = null,
|
||||
Guid? parentExecutionId = null,
|
||||
string? sourceNode = null) =>
|
||||
string? sourceNode = null,
|
||||
string? target = null) =>
|
||||
ScadaBridgeAuditEventFactory.Create(
|
||||
channel: channel,
|
||||
kind: kind,
|
||||
status: status,
|
||||
occurredAtUtc: occurredAtUtc,
|
||||
target: target,
|
||||
sourceNode: sourceNode,
|
||||
sourceSiteId: siteId,
|
||||
executionId: executionId,
|
||||
|
||||
+109
-9
@@ -101,15 +101,12 @@ public class SiteCallAuditRepositoryTests : IClassFixture<MsSqlMigrationFixture>
|
||||
{
|
||||
Skip.IfNot(_fixture.Available, _fixture.SkipReason);
|
||||
|
||||
// WP2.2 collapsed the two-round-trip "insert-if-absent then monotonic
|
||||
// update" into ONE batch that updates first and inserts only when nothing
|
||||
// matched. That makes @@ROWCOUNT = 0 ambiguous: it means "no such row"
|
||||
// AND "the monotonic guard rejected this packet". Guarding the insert on
|
||||
// @@ROWCOUNT alone would therefore let every stale/regressive packet
|
||||
// append a SECOND row for an id that already exists — silently forking
|
||||
// the mirror. The re-check for the row's existence is what prevents that,
|
||||
// and this pins it: after a rejected regressive upsert there is still
|
||||
// exactly ONE row, still carrying the advanced state.
|
||||
// A rejected packet must leave the mirror untouched — one row, still
|
||||
// carrying the advanced state. The hazard this pins is the insert leg
|
||||
// firing for a row that already exists and forking the mirror into two
|
||||
// rows for one id: the insert is gated on IF NOT EXISTS alone (never on
|
||||
// "the update matched nothing", which is ALSO what a monotonic rejection
|
||||
// looks like), so a stale or regressive packet is a pure no-op.
|
||||
var id = TrackedOperationId.New();
|
||||
await using var context = CreateContext();
|
||||
var repo = new SiteCallAuditRepository(context);
|
||||
@@ -137,6 +134,109 @@ public class SiteCallAuditRepositoryTests : IClassFixture<MsSqlMigrationFixture>
|
||||
Assert.Null(loaded[0].LastError);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// The deterministic half of the concurrent-first-write regression: the OTHER
|
||||
/// writer already committed the row, so this call's insert leg is skipped
|
||||
/// entirely and only the monotonic UPDATE can carry its state in. Under the
|
||||
/// UPDATE-first shape the loser of a first-write race ran its UPDATE BEFORE
|
||||
/// any row existed and then skipped its INSERT, dropping the packet's
|
||||
/// Status/RetryCount/HttpStatus/TerminalAtUtc on the floor.
|
||||
/// </summary>
|
||||
[SkippableFact]
|
||||
public async Task UpsertAsync_RowAlreadyCreatedByAnotherWriter_StillAppliesThisPacketsState()
|
||||
{
|
||||
Skip.IfNot(_fixture.Available, _fixture.SkipReason);
|
||||
|
||||
var id = TrackedOperationId.New();
|
||||
var t0 = new DateTime(2026, 5, 22, 8, 0, 0, DateTimeKind.Utc);
|
||||
|
||||
// Writer 1 (say the reconciliation pull) wins the insert with an early state.
|
||||
await using var winnerContext = CreateContext();
|
||||
await new SiteCallAuditRepository(winnerContext).UpsertAsync(
|
||||
NewRow(id, status: "Submitted", createdAtUtc: t0, updatedAtUtc: t0));
|
||||
|
||||
// Writer 2 (the cached dual-write) carries a LATER lifecycle state for the
|
||||
// same id, on its own connection.
|
||||
await using var loserContext = CreateContext();
|
||||
var loserRepo = new SiteCallAuditRepository(loserContext);
|
||||
await loserRepo.UpsertAsync(NewRow(
|
||||
id,
|
||||
status: "Parked",
|
||||
retryCount: 4,
|
||||
lastError: "gave up",
|
||||
httpStatus: 503,
|
||||
createdAtUtc: t0,
|
||||
updatedAtUtc: t0.AddSeconds(30),
|
||||
terminal: true));
|
||||
|
||||
await using var readContext = CreateContext();
|
||||
var loaded = await readContext.Set<SiteCall>()
|
||||
.Where(s => s.TrackedOperationId == id)
|
||||
.ToListAsync();
|
||||
|
||||
Assert.Single(loaded);
|
||||
Assert.Equal("Parked", loaded[0].Status);
|
||||
Assert.Equal(4, loaded[0].RetryCount);
|
||||
Assert.Equal(503, loaded[0].HttpStatus);
|
||||
Assert.Equal("gave up", loaded[0].LastError);
|
||||
Assert.NotNull(loaded[0].TerminalAtUtc);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// The real-concurrency half: two writers racing the FIRST packet of the same
|
||||
/// id, on separate connections, carrying DIFFERENT lifecycle states. However
|
||||
/// the interleaving falls — either insert winning, or the loser eating a 2627
|
||||
/// on its insert leg — the NEWER state must be the one on the row afterwards,
|
||||
/// and exactly one row must exist.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// Repeated over several ids because the interleaving is not controllable; one
|
||||
/// id would make the test a coin flip on whether it visits the racing path at
|
||||
/// all. Every iteration must hold regardless of which way it raced, so the
|
||||
/// test itself is not flaky — only its coverage of the rare branch is
|
||||
/// probabilistic.
|
||||
/// </remarks>
|
||||
[SkippableFact]
|
||||
public async Task UpsertAsync_TwoConcurrentFirstWrites_NewerStateSurvives()
|
||||
{
|
||||
Skip.IfNot(_fixture.Available, _fixture.SkipReason);
|
||||
|
||||
var t0 = new DateTime(2026, 5, 22, 9, 0, 0, DateTimeKind.Utc);
|
||||
|
||||
for (var i = 0; i < 12; i++)
|
||||
{
|
||||
var id = TrackedOperationId.New();
|
||||
|
||||
await using var contextA = CreateContext();
|
||||
await using var contextB = CreateContext();
|
||||
var repoA = new SiteCallAuditRepository(contextA);
|
||||
var repoB = new SiteCallAuditRepository(contextB);
|
||||
|
||||
var early = NewRow(id, status: "Submitted", createdAtUtc: t0, updatedAtUtc: t0);
|
||||
var late = NewRow(
|
||||
id,
|
||||
status: "Delivered",
|
||||
retryCount: 2,
|
||||
createdAtUtc: t0,
|
||||
updatedAtUtc: t0.AddSeconds(10),
|
||||
terminal: true);
|
||||
|
||||
await Task.WhenAll(
|
||||
Task.Run(() => repoA.UpsertAsync(early)),
|
||||
Task.Run(() => repoB.UpsertAsync(late)));
|
||||
|
||||
await using var readContext = CreateContext();
|
||||
var loaded = await readContext.Set<SiteCall>()
|
||||
.Where(s => s.TrackedOperationId == id)
|
||||
.ToListAsync();
|
||||
|
||||
Assert.Single(loaded);
|
||||
Assert.Equal("Delivered", loaded[0].Status);
|
||||
Assert.Equal(2, loaded[0].RetryCount);
|
||||
Assert.NotNull(loaded[0].TerminalAtUtc);
|
||||
}
|
||||
}
|
||||
|
||||
[SkippableFact]
|
||||
public async Task UpsertAsync_SameStatus_EqualUpdatedAt_IsNoOp()
|
||||
{
|
||||
|
||||
@@ -537,7 +537,7 @@ public class NotificationOutboxRepositoryTests : IDisposable
|
||||
var loaded = await _repository.GetByIdAsync(id);
|
||||
loaded!.Status = NotificationStatus.Delivered;
|
||||
loaded.DeliveredAt = new DateTimeOffset(2026, 5, 19, 9, 0, 0, TimeSpan.Zero);
|
||||
await _repository.UpdateAsync(loaded);
|
||||
Assert.True(await _repository.UpdateAsync(loaded));
|
||||
|
||||
_context.ChangeTracker.Clear();
|
||||
var reloaded = await _context.Notifications.FindAsync(id);
|
||||
@@ -545,6 +545,33 @@ public class NotificationOutboxRepositoryTests : IDisposable
|
||||
Assert.Equal(new DateTimeOffset(2026, 5, 19, 9, 0, 0, TimeSpan.Zero), reloaded.DeliveredAt);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// The write is a targeted UPDATE that bypasses the change tracker, so a row
|
||||
/// deleted between the read and the write simply matches nothing. The row
|
||||
/// count is the ONLY not-found signal available, and callers (the operator
|
||||
/// retry/discard one-shots) depend on it to avoid reporting success against a
|
||||
/// purged notification.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public async Task UpdateAsync_RowPurgedBeforeWrite_ReturnsFalse()
|
||||
{
|
||||
var id = Guid.NewGuid().ToString();
|
||||
_context.Notifications.Add(MakeNotification(id, NotificationStatus.Parked));
|
||||
await _context.SaveChangesAsync();
|
||||
_context.ChangeTracker.Clear();
|
||||
|
||||
var loaded = await _repository.GetByIdAsync(id);
|
||||
Assert.NotNull(loaded);
|
||||
|
||||
// The daily retention purge removes the row while the operator's request
|
||||
// is in flight.
|
||||
await _context.Notifications.Where(n => n.NotificationId == id).ExecuteDeleteAsync();
|
||||
_context.ChangeTracker.Clear();
|
||||
|
||||
loaded!.Status = NotificationStatus.Pending;
|
||||
Assert.False(await _repository.UpdateAsync(loaded));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task QueryAsync_AppliesFilters_OrdersByCreatedAtDescending_AndPaginates()
|
||||
{
|
||||
|
||||
Reference in New Issue
Block a user