fix(config-db): EnableRetryOnFailure fleet-wide + execution-strategy wrap of the combined-telemetry dual-write

Installs SqlServerRetryingExecutionStrategy (5 retries, 30s cap) so transient SQL
faults retry transparently on every read-side path. Manual transactions aren't
auto-retried, so AuditLogIngestActor's idempotent dual-write is wrapped in an
explicit CreateExecutionStrategy().ExecuteAsync to become retriable. Verified
empirically that existing manual transactions (BundleImporter) are unaffected —
EF skips the strategy while a transaction is already active.
This commit is contained in:
Joseph Doherty
2026-07-09 08:03:30 -04:00
parent 6b06d1efcb
commit 1dd89d523b
4 changed files with 90 additions and 26 deletions
@@ -229,6 +229,13 @@ VALUES
/// <inheritdoc />
public async Task<long> SwitchOutPartitionAsync(DateTime monthBoundary, TimeSpan? commandTimeout = null, CancellationToken ct = default)
{
// Task 12 (arch-review 04 S5) note: the drop-and-rebuild batch below runs via
// ExecuteSqlRaw with NO EF user-transaction — it carries its own server-side
// BEGIN TRANSACTION / TRY-CATCH / ROLLBACK — so the DbContext's retrying
// execution strategy (EnableRetryOnFailure) MAY auto-replay the whole batch on
// a transient fault. That replay is safe: every step is IF-EXISTS / IF-NOT-EXISTS
// guarded and the staging table is GUID-suffixed, so a re-run is idempotent.
//
// Maintenance timeout in whole seconds (ADO.NET CommandTimeout unit). Null leaves the
// provider default in place. See AuditLogPurgeOptions.MaintenanceCommandTimeoutMinutes /
// arch-review 04 S2 for why the ~30s default is unsafe for the switch-out dance.
@@ -28,7 +28,22 @@ public static class ServiceCollectionExtensions
// CreateProtector during model building.
services.AddDbContext<ScadaBridgeDbContext>((serviceProvider, options) =>
{
options.UseSqlServer(connectionString)
options.UseSqlServer(
connectionString,
// Task 12 (arch-review 04 S5): connection resiliency. A transient
// SQL fault (failover, throttling, dropped connection) previously
// surfaced raw to every read-side caller — KPI asks, outbox
// queries, Tracking.Status, execution-tree walks. EnableRetryOnFailure
// installs the SqlServerRetryingExecutionStrategy so those retry
// transparently. Note: manual (user-initiated) transactions are NOT
// auto-retried — EF executes them directly while a transaction is
// active — so the combined-telemetry dual-write is wrapped in an
// explicit CreateExecutionStrategy() to become retriable (see
// AuditLogIngestActor.OnCachedTelemetryAsync).
sql => sql.EnableRetryOnFailure(
maxRetryCount: 5,
maxRetryDelay: TimeSpan.FromSeconds(30),
errorNumbersToAdd: null))
.ConfigureWarnings(w => w.Ignore(
Microsoft.EntityFrameworkCore.Diagnostics.RelationalEventId.PendingModelChangesWarning));
});