fix(auditlog)+fix(siteeventlog): complete audit write tasks only after commit; run event-log retention on standby (arch-review 2026-08-19 P4-2 + P4-4)
P4-2: SqliteAuditWriter.FlushBatch completed each pending row's Task (TrySetResult) inside the per-row insert loop, before transaction.Commit() ran. A later failure in the same batch (another row's insert throwing, or the commit itself failing) rolled the whole transaction back, but the earlier rows' Tasks had already completed successfully -- the caller and FallbackAuditWriter (whose ring-buffer fallback exists precisely for this outage) both saw false success while the rows were actually gone from the database; the outer catch's TrySetException became a silent no-op against an already-completed TaskCompletionSource. Every row's Completion is now deferred and only resolved (TrySetResult for the whole batch, or TrySetException for the whole batch) after transaction.Commit() succeeds or the transaction rolls back, mirroring SiteEventLogger.FlushBatch. P4-4: EventLogPurgeService gated the daily site_events purge on the active-node check, early-exiting on the standby, on the stale premise that "the standby receives no writes". DataConnectionActor runs on both site nodes and writes site_events regardless of active/standby status (register rows 89/90/97), so an unreplicated standby (the rig's site-b/site-c posture) grew its site_events file forever with no retention and no cap. The gate is removed; every node now purges its own file every tick. Safe on a replicated pair because the retention/cap deletes are CDC-captured and converge under LWW regardless of which node issues them; required on an unreplicated pair because each node's file is independent. The active-node check is retained only to annotate the purge tick's log line. Tests: new SqliteAuditWriterCommitFailureTests forces a genuine non-swallowed mid-batch INSERT failure via a real SQLite trigger + custom scalar function on the writer's own connection (reflection seam), proving (a) a row whose own insert succeeded still faults when a later row in the same batch fails, and (b) FallbackAuditWriter rings every faulted row and increments the failure counter for each -- both assertions fail against the pre-fix code (proof-of-failure verified by temporarily reverting the fix). EventLogPurgeServiceTests' standby case is inverted to assert the purge now runs (not skips) on a standby node.
This commit is contained in:
@@ -484,7 +484,16 @@ public class SqliteAuditWriter : IAuditWriter, ISiteAuditQueue, IAsyncDisposable
|
||||
{
|
||||
eventCmd.ExecuteNonQuery();
|
||||
fwdCmd.ExecuteNonQuery();
|
||||
pending.Completion.TrySetResult();
|
||||
// Do NOT complete pending.Completion here (arch-review 2026-08-19
|
||||
// P4-2). A row's INSERTs succeeding is not durable until the
|
||||
// transaction actually commits below — completing the caller's Task
|
||||
// this early means a later commit failure (SQLITE_FULL, IOERR, or a
|
||||
// later row in this same batch throwing) reports false success: the
|
||||
// row is rolled back out of the database but the caller — and
|
||||
// FallbackAuditWriter, whose ring-buffer fallback exists precisely
|
||||
// for this outage — already saw it as delivered. Every row in this
|
||||
// batch is completed with TrySetResult() together, AFTER
|
||||
// transaction.Commit() succeeds, mirroring SiteEventLogger.FlushBatch.
|
||||
}
|
||||
catch (SqliteException ex) when (ex.SqliteErrorCode == SqliteErrorConstraint)
|
||||
{
|
||||
@@ -492,21 +501,35 @@ public class SqliteAuditWriter : IAuditWriter, ISiteAuditQueue, IAsyncDisposable
|
||||
// audit_event PRIMARY KEY throws before the sidecar insert
|
||||
// runs, so neither table gains a second row. Treat as
|
||||
// success: the lifecycle event is durably recorded under
|
||||
// the first writer's payload.
|
||||
// the first writer's payload. Still deferred to after Commit()
|
||||
// below — the surrounding transaction (this row's canonical/sidecar
|
||||
// insert never ran, but earlier rows in the batch did) must still
|
||||
// commit for that to be true.
|
||||
_logger.LogDebug(ex,
|
||||
"Duplicate EventId {EventId} swallowed by SqliteAuditWriter",
|
||||
evt.EventId);
|
||||
pending.Completion.TrySetResult();
|
||||
}
|
||||
}
|
||||
|
||||
transaction.Commit();
|
||||
_flushCountForTests++;
|
||||
|
||||
// Only NOW — after the whole batch is durably committed — tell every
|
||||
// caller (and FallbackAuditWriter) that their row landed.
|
||||
foreach (var pending in batch)
|
||||
{
|
||||
pending.Completion.TrySetResult();
|
||||
}
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
transaction.Rollback();
|
||||
_logger.LogError(ex, "SqliteAuditWriter batch insert failed; faulting {Count} pending events", batch.Count);
|
||||
// Fault EVERY row in the batch, including any whose own INSERTs succeeded
|
||||
// above — none of that is true anymore, since the whole transaction (this
|
||||
// row's insert included) just rolled back. No TrySetResult has run yet for
|
||||
// any row in this batch (see above), so every TrySetException here is a
|
||||
// real transition, not a no-op racing an already-completed Task.
|
||||
foreach (var pending in batch)
|
||||
{
|
||||
pending.Completion.TrySetException(ex);
|
||||
|
||||
Reference in New Issue
Block a user