feat(sitecallaudit): periodic reconciliation pull back-fills lost telemetry

Add a periodic reconciliation tick to SiteCallAuditActor that, per site,
pulls changed SiteCall rows since a per-site UpdatedAtUtc cursor and upserts
them idempotently (monotonic UpsertAsync) — the documented self-heal for lost
best-effort gRPC telemetry. Mirrors SiteAuditReconciliationActor's structure
(per-site cursor, per-site try/catch failure isolation, advance cursor by max
observed UpdatedAtUtc) minus the stalled-detection EventStream machinery.

Dependency wiring: add an acyclic SiteCallAudit -> AuditLog project reference
and resolve IPullSiteCallsClient + ISiteEnumerator (central-only singletons
registered by AddAuditLogCentralReconciliationClient) from the IServiceProvider
the production ctor already holds — no Host Props.Create change needed. The
repo-only test ctor injects neither collaborator, so the tick is gated off
there. A new public test ctor injects fake client + enumerator + repo so the
tick is unit-testable in-memory (public, not internal: Akka's ActivatorProducer
uses public-only reflection binding).

Options: ReconciliationInterval (default 5 min, clamped >= 1s so a zero config
value can't spin the scheduler) + ReconciliationBatchSize (default 500), plus a
test-only override that bypasses the clamp for millisecond cadences.

Tests (all in-memory, no live MSSQL): absent row is upserted on a tick; second
tick advances the cursor past already-pulled rows; one failing site does not
sink other sites; repo-only ctor does not start the tick.
This commit is contained in:
Joseph Doherty
2026-06-15 12:01:22 -04:00
parent 6b0140dd62
commit e427b38fb3
4 changed files with 623 additions and 12 deletions
@@ -1,11 +1,11 @@
namespace ZB.MOM.WW.ScadaBridge.SiteCallAudit;
/// <summary>
/// Configuration options for the Site Call Audit (#22) read-side: stuck-call
/// detection and KPI windowing. Mirrors the KPI-relevant subset of
/// <c>NotificationOutboxOptions</c> — the reconciliation, purge and dispatch
/// cadence options the Notification Outbox carries are not part of the Site
/// Call Audit read-side backend and are deliberately omitted here.
/// Configuration options for the Site Call Audit (#22): stuck-call detection +
/// KPI windowing for the read-side, plus the cadence knobs for the periodic
/// per-site reconciliation pull (self-heal for lost telemetry). Mirrors the
/// KPI-relevant subset of <c>NotificationOutboxOptions</c> and the
/// scheduler-cadence shape of <c>SiteAuditReconciliationOptions</c>.
/// </summary>
public class SiteCallAuditOptions
{
@@ -44,4 +44,52 @@ public class SiteCallAuditOptions
/// </para>
/// </summary>
public TimeSpan RelayTimeout { get; set; } = TimeSpan.FromSeconds(10);
// ── Reconciliation tick (#22): periodic per-site self-heal pull ──
/// <summary>
/// Period of the reconciliation tick. Each tick visits every known site
/// once, pulls changed <c>SiteCall</c> rows since a per-site cursor, and
/// upserts them idempotently — the documented self-heal when best-effort
/// push telemetry is lost. Default 5 minutes, matching the sibling
/// <c>SiteAuditReconciliationOptions</c> (#23) cadence. Clamped to at least
/// <see cref="MinReconciliationInterval"/> via <see cref="ReconciliationInterval"/>.
/// </summary>
public TimeSpan ReconciliationInterval { get; set; } = TimeSpan.FromMinutes(5);
/// <summary>
/// Test-only override for the reconciliation tick cadence — bypasses the
/// <see cref="MinReconciliationInterval"/> clamp so unit tests can drop the
/// cadence to milliseconds. Production config never sets this; leave null.
/// </summary>
public TimeSpan? ReconciliationIntervalOverride { get; set; }
/// <summary>
/// Maximum number of <c>SiteCall</c> rows requested per <c>PullSiteCalls</c>
/// RPC. Default 500. A <c>MoreAvailable=true</c> response signals the cursor
/// advanced and the next tick should keep draining the backlog.
/// </summary>
public int ReconciliationBatchSize { get; set; } = 500;
/// <summary>
/// Minimum interval the config-bound <see cref="ReconciliationInterval"/> can
/// resolve to. Clamps a misconfigured <c>0</c> (or negative) value away from
/// <see cref="TimeSpan.Zero"/>, which would make Akka's
/// <c>ScheduleTellRepeatedlyCancelable</c> spin — the exact footgun flagged in
/// a prior review of the sibling reconciliation options.
/// </summary>
private static readonly TimeSpan MinReconciliationInterval = TimeSpan.FromSeconds(1);
/// <summary>
/// Resolves the effective reconciliation tick interval: the test override
/// when set (bypassing the clamp), otherwise <see cref="ReconciliationInterval"/>
/// clamped to at least <see cref="MinReconciliationInterval"/> so a
/// zero/negative config value can never yield <see cref="TimeSpan.Zero"/>.
/// </summary>
public TimeSpan ResolvedReconciliationInterval =>
ReconciliationIntervalOverride is { } o
? o
: ReconciliationInterval < MinReconciliationInterval
? MinReconciliationInterval
: ReconciliationInterval;
}