dohertj2/ScadaBridge
1
0
Fork 0
Code Issues 2 Pull Requests Actions Packages Projects Releases Wiki Activity

fix(auditlog): P5-6 — deterministic synthetic ReconciliationAbandoned EventId

Browse Source 
SiteAuditReconciliationActor minted a fresh random EventId for the synthetic
ReconciliationAbandoned row every time a permanently-abandoned row was
re-abandoned. Because the reconciliation cursor is held at `since` by ANY
co-batch unresolved row (see PullSiteAsync's hasUnresolvedFailure gate), a
poisoned row's per-EventId attempt counter restarts every time it reappears
in a stuck batch, so it gets re-abandoned roughly every
MaxPermanentInsertAttempts ticks — minting an unbounded number of duplicate
central audit rows for the SAME lost event.

Fix: DeriveAbandonmentEventId(Guid lostEventId) derives the synthetic
EventId deterministically (SHA-256 over a fixed namespace GUID + the lost
row's bytes, RFC4122-v5-shaped). Re-abandonment of the same lost row now
always produces the same synthetic EventId, so
IAuditLogRepository.InsertIfNotExistsAsync's existing idempotency collapses
every repeat into the first insert. Also adds an in-memory _abandonedEventIds
set as a CHEAP SHORT-CIRCUIT ONLY (skips the wasted ~25 min retry-then-abandon
cycle for a row already known abandoned this process's lifetime) — it is not
the correctness mechanism; it resets on singleton restart/failover, at which
point the deterministic id alone still keeps re-abandonment idempotent.

Tests (SiteAuditReconciliationActorTests):
- ReAbandonment_SameLostEventId_YieldsExactlyOneCentralRow: end-to-end repro
  of the defect trigger (poison row + a rotating-EventId co-batch row that
  never itself reaches the threshold, holding the cursor forever) across two
  full abandon cycles. Proof of failure: run against the unfixed source
  (git stash of the source file only, pure-function tests temporarily
  #if false'd to keep the file compiling), the test failed with exactly the
  predicted defect shape — 2 distinct synthetic rows for one lost EventId.
  Passes post-fix (1 row).
- DeriveAbandonmentEventId_DifferentLostRows_YieldDistinctSyntheticIds and
  DeriveAbandonmentEventId_IsPureAndStableAcrossIndependentInvocations:
  pure-function coverage of the derivation directly, including a pinned
  golden-value regression guard proving it is a function of the input GUID's
  value alone (no clock/randomness/process state).
This commit is contained in:
Joseph Doherty
2026-08-19 16:20:05 -04:00
parent abbbcd1a8d
commit 88fbc9e351
2 changed files with 331 additions and 41 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/ZB.MOM.WW.ScadaBridge.AuditLog/Central/SiteAuditReconciliationActor.cs
+149 -41
View File
@@ -1,3 +1,4 @@
using System.Security.Cryptography;
using System.Text.Json;
using Akka.Actor;
using Microsoft.Extensions.DependencyInjection;
@@ -125,6 +126,34 @@ public class SiteAuditReconciliationActor : ReceiveActor
/// </summary>
private readonly Dictionary<Guid, int> _failedInsertAttempts = new();
/// <summary>
/// EventIds already permanently abandoned earlier in this process's
/// lifetime — a cheap short-circuit ONLY, not a correctness mechanism.
/// </summary>
/// <remarks>
/// <para>
/// The cursor can be held at <c>since</c> indefinitely by an unrelated
/// co-batch row that keeps failing (see <c>hasUnresolvedFailure</c> in
/// <see cref="PullSiteAsync"/>), which means an already-abandoned row can
/// keep reappearing in the pulled batch tick after tick. Without this
/// set, each reappearance would re-run the full
/// <see cref="MaxPermanentInsertAttempts"/>-attempt retry cycle (since
/// <see cref="_failedInsertAttempts"/> was cleared when the row was first
/// abandoned) purely to re-derive the same conclusion. Checking this set
/// first skips straight to "already resolved" instead of burning another
/// ~25 minutes per reappearance.
/// </para>
/// <para>
/// This is deliberately NOT what makes re-abandonment safe —
/// <see cref="DeriveAbandonmentEventId"/>'s determinism is. This set is
/// in-memory only and is empty again after a singleton restart/failover,
/// at which point the full retry-then-abandon cycle simply runs again;
/// the deterministic EventId still makes that re-run's synthetic insert
/// collapse into the SAME row instead of minting a duplicate.
/// </para>
/// </remarks>
private readonly HashSet<Guid> _abandonedEventIds = new();
/// <summary>
/// Number of consecutive central-insert failures before a row
/// is permanently abandoned with a Critical log entry and the cursor is
@@ -133,6 +162,15 @@ public class SiteAuditReconciliationActor : ReceiveActor
/// </summary>
private const int MaxPermanentInsertAttempts = 5;
/// <summary>
/// Fixed namespace GUID mixed into <see cref="DeriveAbandonmentEventId"/>'s
/// hash input. Any fixed value works — it exists only to separate this
/// derivation's ID space from other GUID-from-GUID derivations that might
/// ever reuse the same technique elsewhere in the codebase.
/// </summary>
private static readonly Guid AbandonmentEventIdNamespace =
new("f2b6a6d1-6e2b-4b8b-9b3e-9a6b6a2c1d4e");
private ICancelable? _timer;
/// <summary>
@@ -285,54 +323,70 @@ public class SiteAuditReconciliationActor : ReceiveActor
foreach (var evt in response.Events)
{
var advanceForThisRow = false;
try
if (_abandonedEventIds.Contains(evt.EventId))
{
// Idempotent repository write: duplicate EventIds (from a
// concurrent push, or a retry of this very pull) collapse to
// a no-op on InsertIfNotExistsAsync.
// IngestedAtUtc is a DetailsJson field on the canonical record —
// stamp it via the projection helper.
var ingested = AuditRowProjection.WithIngestedAtUtc(evt, nowUtc);
await repository.InsertIfNotExistsAsync(ingested).ConfigureAwait(false);
_failedInsertAttempts.Remove(evt.EventId);
// Cheap short-circuit ONLY (see the field's doc comment): this row was
// already permanently abandoned earlier in this process's lifetime and
// keeps reappearing because an unrelated co-batch row is holding the
// cursor at `since`. Skip straight to "resolved" instead of re-running
// the whole MaxPermanentInsertAttempts retry cycle — the insert is known
// to fail, and DeriveAbandonmentEventId would make a re-run's synthetic
// row collapse into the existing one anyway.
advanceForThisRow = true;
}
catch (Exception ex)
else
{
// Per-row catch so one bad event does not abandon
// the rest of the batch. Track the failure count per EventId —
// below MaxPermanentInsertAttempts the cursor is HELD BACK so
// the next tick re-pulls and retries; at the threshold the row
// is permanently abandoned (LogCritical + cursor advances past)
// to keep a truly broken row from blocking all subsequent
// progress for the site.
var attempts = _failedInsertAttempts.GetValueOrDefault(evt.EventId) + 1;
_failedInsertAttempts[evt.EventId] = attempts;
if (attempts >= MaxPermanentInsertAttempts)
try
{
_logger.LogCritical(
ex,
"Permanently abandoning AuditEvent {EventId} from site {SiteId} after {Attempts} consecutive insert failures; cursor will advance past it.",
evt.EventId,
site.SiteId,
attempts);
// Idempotent repository write: duplicate EventIds (from a
// concurrent push, or a retry of this very pull) collapse to
// a no-op on InsertIfNotExistsAsync.
// IngestedAtUtc is a DetailsJson field on the canonical record —
// stamp it via the projection helper.
var ingested = AuditRowProjection.WithIngestedAtUtc(evt, nowUtc);
await repository.InsertIfNotExistsAsync(ingested).ConfigureAwait(false);
_failedInsertAttempts.Remove(evt.EventId);
advanceForThisRow = true;
await EmitAbandonmentRecordAsync(repository, evt, site.SiteId, ex, nowUtc)
.ConfigureAwait(false);
}
else
catch (Exception ex)
{
_logger.LogError(
ex,
"Reconciliation ingest failed for AuditEvent {EventId} from site {SiteId} (attempt {Attempts}/{Max}); cursor held back for retry.",
evt.EventId,
site.SiteId,
attempts,
MaxPermanentInsertAttempts);
hasUnresolvedFailure = true;
// Per-row catch so one bad event does not abandon
// the rest of the batch. Track the failure count per EventId —
// below MaxPermanentInsertAttempts the cursor is HELD BACK so
// the next tick re-pulls and retries; at the threshold the row
// is permanently abandoned (LogCritical + cursor advances past)
// to keep a truly broken row from blocking all subsequent
// progress for the site.
var attempts = _failedInsertAttempts.GetValueOrDefault(evt.EventId) + 1;
_failedInsertAttempts[evt.EventId] = attempts;
if (attempts >= MaxPermanentInsertAttempts)
{
_logger.LogCritical(
ex,
"Permanently abandoning AuditEvent {EventId} from site {SiteId} after {Attempts} consecutive insert failures; cursor will advance past it.",
evt.EventId,
site.SiteId,
attempts);
_failedInsertAttempts.Remove(evt.EventId);
_abandonedEventIds.Add(evt.EventId);
advanceForThisRow = true;
await EmitAbandonmentRecordAsync(repository, evt, site.SiteId, ex, nowUtc)
.ConfigureAwait(false);
}
else
{
_logger.LogError(
ex,
"Reconciliation ingest failed for AuditEvent {EventId} from site {SiteId} (attempt {Attempts}/{Max}); cursor held back for retry.",
evt.EventId,
site.SiteId,
attempts,
MaxPermanentInsertAttempts);
hasUnresolvedFailure = true;
}
}
}
@@ -372,14 +426,27 @@ public class SiteAuditReconciliationActor : ReceiveActor
/// row when a pulled event is permanently abandoned — so the loss is
/// queryable in the Audit Log rather than living only in the Critical log line.
/// The synthetic row is built through the same
/// <see cref="ScadaBridgeAuditEventFactory"/> the ingest path uses (fresh
/// <c>EventId</c>, <c>Actor="system"</c>, channel preserved from the lost row's
/// <see cref="ScadaBridgeAuditEventFactory"/> the ingest path uses (a
/// <see cref="DeriveAbandonmentEventId"/>-derived <c>EventId</c>,
/// <c>Actor="system"</c>, channel preserved from the lost row's
/// <c>Category</c> where parseable) and carries the abandoned <c>EventId</c>,
/// the source site, and the final error in <c>Extra</c>. It runs in its own
/// try/catch: the synthetic row is small and well-formed, so the row-specific
/// fault that killed the original will not recur; if it somehow does, log and
/// continue — the abandonment cursor must never be blocked a second time.
/// </summary>
/// <remarks>
/// The synthetic <c>EventId</c> is deterministic (derived from
/// <c>lost.EventId</c>), not fresh, specifically so this call is idempotent:
/// a co-batch row that never resolves can hold the reconciliation cursor at
/// <c>since</c> indefinitely, causing the SAME lost row to be re-pulled and
/// re-abandoned on a later tick (its <see cref="_failedInsertAttempts"/> entry
/// was cleared on the first abandonment, so it restarts). Every re-abandonment
/// of the same lost row now derives the same synthetic id, so
/// <see cref="IAuditLogRepository.InsertIfNotExistsAsync"/> collapses every
/// repeat into the first insert instead of minting an unbounded number of
/// duplicate rows for one lost event.
/// </remarks>
private async Task EmitAbandonmentRecordAsync(
IAuditLogRepository repository,
ZB.MOM.WW.Audit.AuditEvent lost,
@@ -403,6 +470,7 @@ public class SiteAuditReconciliationActor : ReceiveActor
});
var synthetic = ScadaBridgeAuditEventFactory.Create(
eventId: DeriveAbandonmentEventId(lost.EventId),
channel: channel,
kind: AuditKind.ReconciliationAbandoned,
status: AuditStatus.Failed,
@@ -428,6 +496,46 @@ public class SiteAuditReconciliationActor : ReceiveActor
}
}
/// <summary>
/// Derives a deterministic synthetic <see cref="ZB.MOM.WW.Audit.AuditEvent.EventId"/>
/// for the <see cref="AuditKind.ReconciliationAbandoned"/> row minted when
/// <paramref name="lostEventId"/> is permanently abandoned.
/// </summary>
/// <remarks>
/// <para>
/// Pure function of <paramref name="lostEventId"/> alone — no clock, no
/// randomness, no actor state. Same input always produces the same output,
/// including across a process restart, which is what makes re-abandonment
/// idempotent under <see cref="IAuditLogRepository.InsertIfNotExistsAsync"/>
/// even after a singleton failover wipes every in-memory tracking
/// dictionary this actor keeps (<see cref="_failedInsertAttempts"/>,
/// <see cref="_abandonedEventIds"/>, the cursors themselves).
/// </para>
/// <para>
/// Built as a name-based (RFC 4122 "version 5"-shaped) GUID: SHA-256 over
/// a fixed namespace GUID concatenated with <paramref name="lostEventId"/>'s
/// bytes, truncated to 16 bytes with the version/variant bits set so the
/// result is a well-formed GUID. This is not a cryptographic use of
/// SHA-256 — collision resistance against an adversarial input is not the
/// goal, only a stable, well-distributed GUID-to-GUID mapping so distinct
/// lost rows get distinct synthetic ids.
/// </para>
/// </remarks>
internal static Guid DeriveAbandonmentEventId(Guid lostEventId)
{
Span<byte> input = stackalloc byte[32];
AbandonmentEventIdNamespace.TryWriteBytes(input[..16]);
lostEventId.TryWriteBytes(input[16..]);
Span<byte> hash = stackalloc byte[32];
SHA256.HashData(input, hash);
var guidBytes = hash[..16].ToArray();
guidBytes[6] = (byte)((guidBytes[6] & 0x0F) | 0x50); // version 5
guidBytes[8] = (byte)((guidBytes[8] & 0x3F) | 0x80); // RFC 4122 variant
return new Guid(guidBytes);
}
/// <summary>
/// Flips the per-site stalled flag based on whether this tick drained the
/// queue. A "draining" cycle is one where the server reported no more rows