677263af57
The per-IP unlock throttle's counters were a process-local ConcurrentDictionary on each central node, so the effective fleet cap was N x MaxUnlockAttemptsPerIpPerHour per client (failing over to the other node handed the client a fresh budget) and a node restart cleared the window outright. The window now lives in the central MS SQL BundleUnlockAttempt table — the HA store both central nodes already share, so no new infrastructure, no new port, no consensus protocol: - Commons: BundleUnlockAttempt entity + IBundleUnlockAttemptRepository. - ConfigurationDatabase: entity config (one index, IX_BundleUnlockAttempt_Client_Attempted, serving both the windowed count and the prune), repository (AsNoTracking count, ExecuteDeleteAsync prune), DbSet, DI registration, migration AddBundleUnlockAttemptTable + the hand-written idempotent production script under docs/plans/sql/. - Transport: IBundleUnlockAttemptWindow seam + SqlBundleUnlockAttemptWindow (singleton; resolves the scoped repository from its own scope per call, like the central purge singletons). BundleUnlockRateLimiter keeps the policy and gains TryRegisterAttemptAsync; BundleImporter.LoadAsync calls it. Behaviour preserved from row 44: locked bundles spend no hourly budget (the lockout check still runs first), throttle denials burn no per-bundle strike and are NOT recorded so they cannot roll the window forward, unattributable clients share the stricter <unattributed> bucket, and the budget counts attempts rather than failures. Config keys and their semantics are unchanged, so the rig's 500 override stays valid. Three deliberate trades, documented in both Transport docs: - FAILS OPEN. A store error logs a Warning naming the client and cap, and permits the attempt. This is a best-effort rate cap over an already authenticated, Administrator-gated surface — the availability of legitimate imports outranks the cap, and a DB blip must not make the import wizard unusable. Falling back to the process-local window was considered and rejected: it would still block a legitimate operator's 11th import. - Approximate. Count-then-insert is not atomic across nodes, so simultaneous attempts can overshoot by the number in flight — bounded, self-correcting within the window, and not worth provider-specific raw SQL. - Pruned opportunistically on the unlock path (CAS-claimed at most every 15 min per process, bounded to 1000 rows, retaining 2x the window, best-effort try/catch) rather than by a new background service — Transport owns no central scheduler. Same shape as DeploymentService's terminal-record sweep. Tests: Transport.IntegrationTests 116 passed (+10 — two limiter instances over one store enforcing a single combined window, restart durability, denials burning no budget, sliding release, bucket isolation, store-failure fail-open, caller-cancellation not swallowed, no-shared-window fallback, prune and its negative control); ConfigurationDatabase 395 passed (+5).
168 lines
7.4 KiB
C#
168 lines
7.4 KiB
C#
using Microsoft.Extensions.DependencyInjection;
|
|
using Microsoft.Extensions.Logging;
|
|
using Microsoft.Extensions.Logging.Abstractions;
|
|
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Repositories;
|
|
|
|
namespace ZB.MOM.WW.ScadaBridge.Transport.Import;
|
|
|
|
/// <summary>
|
|
/// Register row 52: <see cref="IBundleUnlockAttemptWindow"/> backed by the central MS SQL
|
|
/// <c>BundleUnlockAttempt</c> table — the HA store both central nodes already share, so
|
|
/// the fleet-wide cap needs no new infrastructure, no new port and no consensus protocol.
|
|
/// <para>
|
|
/// Registered as a singleton (the limiter that consumes it is one) and therefore resolves
|
|
/// a scoped <see cref="IBundleUnlockAttemptRepository"/> from its own
|
|
/// <see cref="IServiceScopeFactory"/> scope per call, the same way the central purge
|
|
/// singletons reach the database.
|
|
/// </para>
|
|
/// <para>
|
|
/// <b>Accuracy is deliberately approximate.</b> The count-then-insert pair is not atomic
|
|
/// across the two central nodes, so genuinely simultaneous attempts can overshoot the cap
|
|
/// by up to the number in flight. A conditional single-statement insert would close that,
|
|
/// at the cost of provider-specific SQL for a best-effort rate cap layered over an
|
|
/// already-authenticated, role-gated surface. Overshoot is bounded and self-corrects
|
|
/// within the window; it is not worth raw SQL.
|
|
/// </para>
|
|
/// </summary>
|
|
public sealed class SqlBundleUnlockAttemptWindow : IBundleUnlockAttemptWindow
|
|
{
|
|
/// <summary>
|
|
/// Minimum wall-clock gap between opportunistic prunes. The prune runs on the unlock
|
|
/// path (Transport owns no central scheduler, and the design constraint was "no new
|
|
/// background service"), so it is claimed at most this often per process — the same
|
|
/// rate-limited opportunistic-sweep shape <c>DeploymentService</c> uses for terminal
|
|
/// deployment records.
|
|
/// </summary>
|
|
internal static readonly TimeSpan PruneMinInterval = TimeSpan.FromMinutes(15);
|
|
|
|
/// <summary>
|
|
/// Upper bound on rows deleted by one opportunistic prune pass. Bounded so a sweep on
|
|
/// a request path can never become a long-running delete, even if the table grew while
|
|
/// prunes were failing.
|
|
/// </summary>
|
|
internal const int PruneBatchSize = 1000;
|
|
|
|
/// <summary>
|
|
/// Multiple of the trailing window that rows are retained for before becoming
|
|
/// prunable. Kept above 1 so a prune racing a count can never delete a row the count
|
|
/// still needs.
|
|
/// </summary>
|
|
internal const int PruneRetentionWindows = 2;
|
|
|
|
private readonly IServiceScopeFactory _scopeFactory;
|
|
private readonly TimeProvider _timeProvider;
|
|
private readonly ILogger<SqlBundleUnlockAttemptWindow> _logger;
|
|
|
|
/// <summary>Ticks of the last claimed prune, as a CAS-guarded process-local claim.</summary>
|
|
private long _lastPruneTicks;
|
|
|
|
/// <summary>
|
|
/// Initializes a new <see cref="SqlBundleUnlockAttemptWindow"/>.
|
|
/// </summary>
|
|
/// <param name="scopeFactory">Scope factory used to resolve the scoped attempt repository per call.</param>
|
|
/// <param name="timeProvider">Clock used for the window boundary and the prune cadence.</param>
|
|
/// <param name="logger">Optional logger for prune diagnostics.</param>
|
|
public SqlBundleUnlockAttemptWindow(
|
|
IServiceScopeFactory scopeFactory,
|
|
TimeProvider timeProvider,
|
|
ILogger<SqlBundleUnlockAttemptWindow>? logger = null)
|
|
{
|
|
_scopeFactory = scopeFactory ?? throw new ArgumentNullException(nameof(scopeFactory));
|
|
_timeProvider = timeProvider ?? throw new ArgumentNullException(nameof(timeProvider));
|
|
_logger = logger ?? NullLogger<SqlBundleUnlockAttemptWindow>.Instance;
|
|
|
|
// Seed the prune claim to "now" so a freshly started node does not prune on its
|
|
// very first unlock — the table is almost certainly already tidy, and a node that
|
|
// restarts under load would otherwise prune on every restart.
|
|
_lastPruneTicks = _timeProvider.GetUtcNow().UtcTicks;
|
|
}
|
|
|
|
/// <inheritdoc />
|
|
public async Task<bool> TryRegisterAttemptAsync(
|
|
string clientKey, int maxAttemptsPerWindow, TimeSpan window, CancellationToken ct = default)
|
|
{
|
|
ArgumentException.ThrowIfNullOrWhiteSpace(clientKey);
|
|
if (maxAttemptsPerWindow < 1)
|
|
{
|
|
throw new ArgumentOutOfRangeException(
|
|
nameof(maxAttemptsPerWindow), "Limit must be at least 1.");
|
|
}
|
|
|
|
if (window <= TimeSpan.Zero)
|
|
{
|
|
throw new ArgumentOutOfRangeException(nameof(window), "Window must be positive.");
|
|
}
|
|
|
|
var now = _timeProvider.GetUtcNow().UtcDateTime;
|
|
var key = clientKey.Trim();
|
|
|
|
using var scope = _scopeFactory.CreateScope();
|
|
var repo = scope.ServiceProvider.GetRequiredService<IBundleUnlockAttemptRepository>();
|
|
|
|
// Two round trips rather than one conditional insert — see the class remarks on
|
|
// why the resulting overshoot is accepted. Exceptions propagate: the fail-open
|
|
// decision lives in BundleUnlockRateLimiter.
|
|
var used = await repo.CountAttemptsSinceAsync(key, now - window, ct).ConfigureAwait(false);
|
|
if (used >= maxAttemptsPerWindow)
|
|
{
|
|
// Rejected attempts are NOT recorded — a denial burns no budget, so a client
|
|
// hammering the endpoint cannot roll its own window forward indefinitely.
|
|
return false;
|
|
}
|
|
|
|
await repo.RecordAttemptAsync(key, now, ct).ConfigureAwait(false);
|
|
|
|
await TryPruneAsync(repo, now, window, ct).ConfigureAwait(false);
|
|
return true;
|
|
}
|
|
|
|
/// <summary>
|
|
/// Opportunistic, rate-limited, bounded prune of aged-out attempt rows. Best-effort:
|
|
/// any failure is logged and swallowed, because retention maintenance must never fail
|
|
/// the operator's actual unlock.
|
|
/// </summary>
|
|
/// <param name="repo">Repository resolved for the current call's scope.</param>
|
|
/// <param name="nowUtc">Current UTC instant.</param>
|
|
/// <param name="window">The trailing window; rows older than a multiple of it are prunable.</param>
|
|
/// <param name="ct">Cancellation token.</param>
|
|
/// <returns>A task that completes when the prune attempt has finished (or been skipped).</returns>
|
|
private async Task TryPruneAsync(
|
|
IBundleUnlockAttemptRepository repo, DateTime nowUtc, TimeSpan window, CancellationToken ct)
|
|
{
|
|
var previous = Interlocked.Read(ref _lastPruneTicks);
|
|
if (nowUtc.Ticks - previous < PruneMinInterval.Ticks)
|
|
{
|
|
return;
|
|
}
|
|
|
|
// Claim the sweep so concurrent unlocks on this node don't all prune at once.
|
|
if (Interlocked.CompareExchange(ref _lastPruneTicks, nowUtc.Ticks, previous) != previous)
|
|
{
|
|
return;
|
|
}
|
|
|
|
try
|
|
{
|
|
var cutoff = nowUtc - (window * PruneRetentionWindows);
|
|
var deleted = await repo
|
|
.PruneAttemptsOlderThanAsync(cutoff, PruneBatchSize, ct)
|
|
.ConfigureAwait(false);
|
|
|
|
if (deleted > 0)
|
|
{
|
|
_logger.LogDebug(
|
|
"Pruned {Deleted} bundle-unlock attempt row(s) older than {Cutoff:o}.", deleted, cutoff);
|
|
}
|
|
}
|
|
catch (OperationCanceledException) when (ct.IsCancellationRequested)
|
|
{
|
|
throw;
|
|
}
|
|
catch (Exception ex)
|
|
{
|
|
// Never fail the unlock for a retention sweep.
|
|
_logger.LogWarning(ex, "Opportunistic bundle-unlock attempt prune failed; will retry on a later attempt.");
|
|
}
|
|
}
|
|
}
|