fix(auditlog)+fix(siteeventlog): complete audit write tasks only after commit; run event-log retention on standby (arch-review 2026-08-19 P4-2 + P4-4)

P4-2: SqliteAuditWriter.FlushBatch completed each pending row's Task
(TrySetResult) inside the per-row insert loop, before transaction.Commit()
ran. A later failure in the same batch (another row's insert throwing, or
the commit itself failing) rolled the whole transaction back, but the
earlier rows' Tasks had already completed successfully -- the caller and
FallbackAuditWriter (whose ring-buffer fallback exists precisely for this
outage) both saw false success while the rows were actually gone from the
database; the outer catch's TrySetException became a silent no-op against
an already-completed TaskCompletionSource. Every row's Completion is now
deferred and only resolved (TrySetResult for the whole batch, or
TrySetException for the whole batch) after transaction.Commit() succeeds
or the transaction rolls back, mirroring SiteEventLogger.FlushBatch.

P4-4: EventLogPurgeService gated the daily site_events purge on the
active-node check, early-exiting on the standby, on the stale premise that
"the standby receives no writes". DataConnectionActor runs on both site
nodes and writes site_events regardless of active/standby status (register
rows 89/90/97), so an unreplicated standby (the rig's site-b/site-c
posture) grew its site_events file forever with no retention and no cap.
The gate is removed; every node now purges its own file every tick. Safe
on a replicated pair because the retention/cap deletes are CDC-captured
and converge under LWW regardless of which node issues them; required on
an unreplicated pair because each node's file is independent. The
active-node check is retained only to annotate the purge tick's log line.

Tests: new SqliteAuditWriterCommitFailureTests forces a genuine
non-swallowed mid-batch INSERT failure via a real SQLite trigger + custom
scalar function on the writer's own connection (reflection seam), proving
(a) a row whose own insert succeeded still faults when a later row in the
same batch fails, and (b) FallbackAuditWriter rings every faulted row and
increments the failure counter for each -- both assertions fail against
the pre-fix code (proof-of-failure verified by temporarily reverting the
fix). EventLogPurgeServiceTests' standby case is inverted to assert the
purge now runs (not skips) on a standby node.
This commit is contained in:
Joseph Doherty
2026-08-19 14:44:37 -04:00
parent 26ab6f8f09
commit 6c11362d9f
8 changed files with 290 additions and 67 deletions
@@ -410,32 +410,32 @@ public class EventLogPurgeServiceTests : IDisposable
Assert.Empty(exceptions);
}
// ── SiteEventLogging-019: purge runs only on the active node ──
// ── arch-review 2026-08-19 P4-4: purge runs on every node, not just the active one ──
// (supersedes the old SiteEventLogging-019 "active-node-only gate" — DataConnectionActor
// writes site_events on both site nodes, so a standby that skipped its own purge would
// grow its file unbounded on an unreplicated site.)
[Fact]
public void RunPurge_OnStandbyNode_SkipsAllWork()
public void RunPurge_OnStandbyNode_StillRunsThePurge()
{
// SiteEventLogging-019: per design, the daily purge runs on the active
// node only. The standby's local SQLite receives no writes, so purging
// there is unnecessary; we gate the purge tick on the injected
// active-node check and early-exit when it returns false. The row
// inserted here is well past retention, so a real purge would delete
// it — the standby gate must leave it intact.
// P4-4 regression: before the fix, a standby (isActiveNode returning false) early-exited
// the purge tick entirely, leaving this well-past-retention row in place forever on an
// unreplicated standby. The gate is gone — the standby purges its own file exactly like
// the active node.
InsertEventWithTimestamp(DateTimeOffset.UtcNow.AddDays(-31));
Assert.Equal(1, GetEventCount());
var purge = CreatePurgeService(isActiveNode: () => false);
purge.RunPurge();
Assert.Equal(1, GetEventCount());
Assert.Equal(0, GetEventCount());
}
[Fact]
public void RunPurge_OnActiveNode_RunsTheRetentionPurge()
{
// SiteEventLogging-019: when the active-node check returns true the
// service runs the purge as before. Pinned alongside the standby case
// so a future regression that inverts the gate is caught.
// Pinned alongside the standby case above so a future regression that reintroduces
// asymmetric active/standby purge behaviour is caught from either side.
InsertEventWithTimestamp(DateTimeOffset.UtcNow.AddDays(-31));
InsertEventWithTimestamp(DateTimeOffset.UtcNow);
@@ -446,12 +446,11 @@ public class EventLogPurgeServiceTests : IDisposable
}
[Fact]
public void RunPurge_WithNullCheck_FallsBackToRunning()
public void RunPurge_WithNullCheck_StillRuns()
{
// SiteEventLogging-019: when no active-node check is supplied (the
// default for non-clustered hosts and pre-existing tests), the service
// preserves the pre-fix "run on every tick" behaviour rather than
// silently skipping every tick. Backward compatibility guard.
// When no active-node check is supplied (the default for non-clustered hosts and
// pre-existing tests), the service runs the purge — unaffected by P4-4 since the
// check was never gating anything for this caller either.
InsertEventWithTimestamp(DateTimeOffset.UtcNow.AddDays(-31));
var purge = CreatePurgeService(isActiveNode: null);