fix(auditlog)+fix(siteeventlog): complete audit write tasks only after commit; run event-log retention on standby (arch-review 2026-08-19 P4-2 + P4-4)

P4-2: SqliteAuditWriter.FlushBatch completed each pending row's Task
(TrySetResult) inside the per-row insert loop, before transaction.Commit()
ran. A later failure in the same batch (another row's insert throwing, or
the commit itself failing) rolled the whole transaction back, but the
earlier rows' Tasks had already completed successfully -- the caller and
FallbackAuditWriter (whose ring-buffer fallback exists precisely for this
outage) both saw false success while the rows were actually gone from the
database; the outer catch's TrySetException became a silent no-op against
an already-completed TaskCompletionSource. Every row's Completion is now
deferred and only resolved (TrySetResult for the whole batch, or
TrySetException for the whole batch) after transaction.Commit() succeeds
or the transaction rolls back, mirroring SiteEventLogger.FlushBatch.

P4-4: EventLogPurgeService gated the daily site_events purge on the
active-node check, early-exiting on the standby, on the stale premise that
"the standby receives no writes". DataConnectionActor runs on both site
nodes and writes site_events regardless of active/standby status (register
rows 89/90/97), so an unreplicated standby (the rig's site-b/site-c
posture) grew its site_events file forever with no retention and no cap.
The gate is removed; every node now purges its own file every tick. Safe
on a replicated pair because the retention/cap deletes are CDC-captured
and converge under LWW regardless of which node issues them; required on
an unreplicated pair because each node's file is independent. The
active-node check is retained only to annotate the purge tick's log line.

Tests: new SqliteAuditWriterCommitFailureTests forces a genuine
non-swallowed mid-batch INSERT failure via a real SQLite trigger + custom
scalar function on the writer's own connection (reflection seam), proving
(a) a row whose own insert succeeded still faults when a later row in the
same batch fails, and (b) FallbackAuditWriter rings every faulted row and
increments the failure counter for each -- both assertions fail against
the pre-fix code (proof-of-failure verified by temporarily reverting the
fix). EventLogPurgeServiceTests' standby case is inverted to assert the
purge now runs (not skips) on a standby node.
This commit is contained in:
Joseph Doherty
2026-08-19 14:44:37 -04:00
parent 26ab6f8f09
commit 6c11362d9f
8 changed files with 290 additions and 67 deletions
@@ -210,20 +210,23 @@ public static class SiteServiceRegistration
return new AkkaClusterNodeProvider(akkaService, siteRole);
});
// The EventLogPurgeService runs on every
// site host node but consults this optional gate each tick and early-exits on
// the standby. Register it to delegate to IClusterNodeProvider.SelfIsPrimary
// (the canonical "this node is the oldest Up member (singleton host)" check) so purge
// runs ONLY on the active node — no duplicated cluster logic. Non-clustered test hosts that
// never call SiteServiceRegistration leave it unregistered, so the purge defaults
// to always-run (the pre-fix behaviour, preserved).
// The EventLogPurgeService runs on every site host node and, as of arch-review
// 2026-08-19 P4-4, purges its own site_events file on EVERY node — the old
// early-exit-on-standby gate this delegate drove is gone (DataConnectionActor
// writes site_events on both site nodes, so a standby that never purged would grow
// unbounded on an unreplicated site). Register it to delegate to
// IClusterNodeProvider.SelfIsPrimary (the canonical "this node is the oldest Up
// member (singleton host)" check) purely so the purge tick's log line can say
// whether this node is active — no duplicated cluster logic, no gating. Non-clustered
// test hosts that never call SiteServiceRegistration leave it unregistered, so the
// purge just logs every tick as active.
services.AddSingleton<SiteEventLogActiveNodeCheck>(sp =>
{
var nodeProvider = sp.GetRequiredService<IClusterNodeProvider>();
return () => nodeProvider.SelfIsPrimary;
});
// Wording-only companion to the gate above: when a purge on a replication-enabled node
// Wording-only companion to the delegate above: when a purge on a replication-enabled node
// actually deletes rows, it appends an operator note explaining the LocalDbOplogBacklog
// spike that follows. site_events is a replicated table and a retention/cap DELETE is
// CDC-captured like any other write (LocalDb Phase 2 — no purge-exemption path), so the