fix(transport)+test: close deferred-work register rows 58, 60, 65 (Low-tail sweep)
Row 65 — trigger FIRED, fixed. BundleUnlockAttempt rows were pruned only by
nodes actively serving unlocks, so a cluster that stopped importing entirely
left its last rows in place forever. The row's own trigger was "Transport gains
a central scheduler a prune tick could ride" — it has one now:
BundleSessionEvictionService, the central-only 1-minute session sweep added by
the transport-session fixes. It now also ticks the new
IBundleUnlockAttemptWindow.PruneAsync. Both drivers share the ONE 15-minute CAS
cadence claim, so ticking on a 1-minute sweep costs at most one bounded DELETE
per quarter hour and the two can never prune concurrently. PruneAsync is
contractually non-throwing (unlike TryRegisterAttemptAsync, which throws and
lets the limiter own the fail-open decision) — a background sweep has no caller
to report to and must survive a database blip. Both Transport docs updated;
three tests in FleetWideUnlockThrottleTests, incl. the retention sweep running
with no unlock traffic anywhere in the test, which is the whole point.
Row 58 — closed with a pin, and the OLD pin was found blind. The existing
Deployments_Dispose_UnsubscribesFromNotifier is behavioural: it is satisfied by
the _disposed guard alone, so deleting the `-=` from Dispose leaves it green
while the singleton notifier keeps a delegate pinning the circuit-scoped
component. Verified by mutation — with the `-=` removed the old test still
passed and only the two new pins failed. Added:
* a structural retention pin asserting the notifier's invocation list actually
shrinks to 0 on Dispose;
* a source scan pinning the row's stated TRIGGER ("a second page subscribes,
or any subscriber lands without a matching Dispose unsubscribe") — every
CentralUI file that subscribes must also unsubscribe and be IDisposable,
with a non-empty guard so the scan cannot pass vacuously.
Row 60 — closed with a pin, and the register's premise corrected.
GlobalArtifactSnapshot does NOT hold EF-materialised entities: it is projected
into pure Commons DTOs before the fan-out. The EF state that genuinely crosses
into the parallel send is the tracked Site entities the lambda closes over, and
that is safe only while lazy loading is off. Pinned in ConfigurationDatabase
.Tests two ways — no proxies extension on the built DbContextOptions, and no
EntityFrameworkCore.Proxies package reference anywhere in the repo (the
structurally stronger half: UseLazyLoadingProxies() is not a compilable call
without it). The real invariant is now spelled out at the fan-out itself.
Row 59 — no product change; added the missing tripwire remark stating the
constraint the safety argument rests on (serial by design, one scoped DbContext,
do not Task.WhenAll; if it gets slow the fix is a scope PER SITE, not a shared
scope under fan-out). Line numbers in the register have drifted to :528-545.
Rows 36, 37, 64: verification only, no change — see the sweep report.
This commit is contained in:
@@ -291,7 +291,22 @@ public class ArtifactDeploymentService
|
||||
site.Id, globals, cancellationToken, deploymentId);
|
||||
}
|
||||
|
||||
// Deploy to each site in parallel with per-site timeout
|
||||
// Deploy to each site in parallel with per-site timeout.
|
||||
//
|
||||
// Deferred-work register row 60 — what makes this fan-out safe, stated precisely
|
||||
// because the invariant is easy to break from a distance:
|
||||
// * `globals` is NOT EF state. FetchGlobalArtifactsAsync projects the entities
|
||||
// into pure Commons message DTOs and drops the entities, so the snapshot the
|
||||
// lambda closes over cannot touch a DbContext at all.
|
||||
// * `site` IS a tracked EF entity, and it is the one piece of EF state that
|
||||
// genuinely crosses into the parallel phase. Only already-materialised scalar
|
||||
// properties are read below (Id, SiteIdentifier, Name) and NO navigation
|
||||
// property is traversed — with lazy loading off (pinned by
|
||||
// ConfigurationDatabase.Tests.AddConfigurationDatabase_DoesNotEnableLazyLoading)
|
||||
// those are plain field reads that never re-enter the shared context.
|
||||
// So: do not traverse a navigation property on `site` in here, and do not move
|
||||
// per-site command building (which DOES query) inside this lambda — either one
|
||||
// puts concurrent load on one non-thread-safe DbContext.
|
||||
var tasks = sites.Select(async site =>
|
||||
{
|
||||
try
|
||||
|
||||
@@ -522,6 +522,20 @@ public class SiteCallAuditActor : ReceiveActor
|
||||
/// pass (the failing site's cursor is left at its previous value so the next
|
||||
/// tick retries the same window).
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <b>SERIAL BY DESIGN — do NOT fan this loop out (register row 59).</b> The
|
||||
/// <paramref name="repository"/> is a single EF Core <c>DbContext</c>-backed instance
|
||||
/// resolved once per tick by <see cref="OnReconciliationTickAsync"/>, and a DbContext
|
||||
/// is not thread-safe. The `await`-in-`foreach` below is the ONLY thing keeping this
|
||||
/// tick's scope — which spans every site x up to
|
||||
/// <see cref="MaxReconciliationPagesPerTick"/> pages of upserts — to one concurrent
|
||||
/// user. Replacing it with <c>Task.WhenAll</c> to overlap the per-site network pulls
|
||||
/// would reintroduce register row 45 (concurrent use of one scoped DbContext), which
|
||||
/// surfaces as sporadic
|
||||
/// "A second operation was started on this context instance" failures rather than a
|
||||
/// clean break. If the serial pass ever becomes too slow, the fix is a scope PER SITE
|
||||
/// (resolve a repository inside the loop body), not a shared scope under fan-out.
|
||||
/// </remarks>
|
||||
private async Task ReconcileSitesAsync(
|
||||
IReadOnlyList<SiteEntry> sites, IPullSiteCallsClient client, ISiteCallAuditRepository repository)
|
||||
{
|
||||
|
||||
@@ -14,12 +14,22 @@ namespace ZB.MOM.WW.ScadaBridge.Transport.Import;
|
||||
/// design contract is "bundles are not retained server-side after ApplyAsync
|
||||
/// commits". This service keeps abandoned / failed sessions from pinning that
|
||||
/// plaintext for the full 30-minute TTL when no other traffic flows.
|
||||
/// <para>
|
||||
/// <b>Register row 65: it also ticks the bundle-unlock attempt prune.</b> That retention
|
||||
/// sweep used to ride the unlock path alone, on the reasoning that Transport owned no
|
||||
/// central scheduler — this service is that scheduler, so a cluster that stops importing
|
||||
/// entirely no longer leaves its last <c>BundleUnlockAttempt</c> rows in place
|
||||
/// indefinitely. The window keeps its own 15-minute cadence claim, so ticking it on the
|
||||
/// 1-minute session sweep costs at most one bounded DELETE per quarter hour, and the two
|
||||
/// drivers can never prune concurrently.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
internal sealed class BundleSessionEvictionService : BackgroundService
|
||||
{
|
||||
private static readonly TimeSpan SweepInterval = TimeSpan.FromMinutes(1);
|
||||
|
||||
private readonly IBundleSessionStore _sessionStore;
|
||||
private readonly IBundleUnlockAttemptWindow _unlockAttemptWindow;
|
||||
private readonly ILogger<BundleSessionEvictionService> _logger;
|
||||
|
||||
/// <summary>
|
||||
@@ -35,12 +45,15 @@ internal sealed class BundleSessionEvictionService : BackgroundService
|
||||
/// Initializes a new instance of <see cref="BundleSessionEvictionService"/>.
|
||||
/// </summary>
|
||||
/// <param name="sessionStore">The bundle session store to sweep for expired sessions.</param>
|
||||
/// <param name="unlockAttemptWindow">Shared unlock-attempt window whose retention prune this sweep also ticks (register row 65).</param>
|
||||
/// <param name="logger">Logger for sweep diagnostics.</param>
|
||||
public BundleSessionEvictionService(
|
||||
IBundleSessionStore sessionStore,
|
||||
IBundleUnlockAttemptWindow unlockAttemptWindow,
|
||||
ILogger<BundleSessionEvictionService> logger)
|
||||
{
|
||||
_sessionStore = sessionStore ?? throw new ArgumentNullException(nameof(sessionStore));
|
||||
_unlockAttemptWindow = unlockAttemptWindow ?? throw new ArgumentNullException(nameof(unlockAttemptWindow));
|
||||
_logger = logger ?? throw new ArgumentNullException(nameof(logger));
|
||||
_openSessionCountProvider = () => _sessionStore.OpenSessionCount;
|
||||
}
|
||||
@@ -95,6 +108,26 @@ internal sealed class BundleSessionEvictionService : BackgroundService
|
||||
{
|
||||
_logger.LogWarning(ex, "Bundle session sweep failed; will retry on next interval.");
|
||||
}
|
||||
|
||||
// Register row 65: retention tick for the fleet-wide unlock-attempt window.
|
||||
// PruneAsync is contractually non-throwing and self-limits to its own
|
||||
// 15-minute cadence, so this is a cheap no-op on most sweeps. Kept separate
|
||||
// from the session sweep's try so neither maintenance duty can mask the other.
|
||||
try
|
||||
{
|
||||
await _unlockAttemptWindow
|
||||
.PruneAsync(BundleUnlockRateLimiter.DefaultWindow, stoppingToken)
|
||||
.ConfigureAwait(false);
|
||||
}
|
||||
catch (OperationCanceledException) when (stoppingToken.IsCancellationRequested)
|
||||
{
|
||||
return;
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
_logger.LogWarning(
|
||||
ex, "Bundle-unlock attempt prune tick failed; will retry on next interval.");
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -25,4 +25,23 @@ public interface IBundleUnlockAttemptWindow
|
||||
/// <returns><c>true</c> when the attempt was recorded (within budget); <c>false</c> when the bucket is exhausted.</returns>
|
||||
Task<bool> TryRegisterAttemptAsync(
|
||||
string clientKey, int maxAttemptsPerWindow, TimeSpan window, CancellationToken ct = default);
|
||||
|
||||
/// <summary>
|
||||
/// Register row 65: drives one retention sweep of aged-out attempt rows independently
|
||||
/// of the unlock path. The prune used to ride <see cref="TryRegisterAttemptAsync"/>
|
||||
/// only, so a cluster that stopped importing entirely left its last rows in place
|
||||
/// indefinitely; <c>BundleSessionEvictionService</c> — Transport's periodic
|
||||
/// central-only sweep — now also ticks this, which closes that.
|
||||
/// <para>
|
||||
/// Unlike <see cref="TryRegisterAttemptAsync"/>, implementations must NOT throw on
|
||||
/// store failure: this is pure retention maintenance with no caller to report to, and
|
||||
/// the sweep that drives it must never be taken down by a database blip. Implementations
|
||||
/// are also expected to keep their own cadence claim, so calling this more often than
|
||||
/// the intended prune interval is harmless.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
/// <param name="window">Trailing window the attempts are counted over (one hour in production); rows are retained for a multiple of it.</param>
|
||||
/// <param name="ct">Cancellation token.</param>
|
||||
/// <returns>A task that completes when the prune attempt has finished (or been skipped for cadence).</returns>
|
||||
Task PruneAsync(TimeSpan window, CancellationToken ct = default);
|
||||
}
|
||||
|
||||
@@ -27,11 +27,12 @@ namespace ZB.MOM.WW.ScadaBridge.Transport.Import;
|
||||
public sealed class SqlBundleUnlockAttemptWindow : IBundleUnlockAttemptWindow
|
||||
{
|
||||
/// <summary>
|
||||
/// Minimum wall-clock gap between opportunistic prunes. The prune runs on the unlock
|
||||
/// path (Transport owns no central scheduler, and the design constraint was "no new
|
||||
/// background service"), so it is claimed at most this often per process — the same
|
||||
/// rate-limited opportunistic-sweep shape <c>DeploymentService</c> uses for terminal
|
||||
/// deployment records.
|
||||
/// Minimum wall-clock gap between prunes, claimed at most this often per process — the
|
||||
/// same rate-limited sweep shape <c>DeploymentService</c> uses for terminal deployment
|
||||
/// records. Both prune drivers share the one claim: the opportunistic sweep on the
|
||||
/// unlock path, and (register row 65) <c>BundleSessionEvictionService</c>'s periodic
|
||||
/// tick, which is what keeps the table converging on a cluster that has stopped
|
||||
/// importing altogether.
|
||||
/// </summary>
|
||||
internal static readonly TimeSpan PruneMinInterval = TimeSpan.FromMinutes(15);
|
||||
|
||||
@@ -116,6 +117,39 @@ public sealed class SqlBundleUnlockAttemptWindow : IBundleUnlockAttemptWindow
|
||||
return true;
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
public async Task PruneAsync(TimeSpan window, CancellationToken ct = default)
|
||||
{
|
||||
if (window <= TimeSpan.Zero)
|
||||
{
|
||||
throw new ArgumentOutOfRangeException(nameof(window), "Window must be positive.");
|
||||
}
|
||||
|
||||
// Register row 65: the same rate-limited, bounded, best-effort sweep the unlock
|
||||
// path runs — deliberately sharing the ONE prune claim, so a background tick and a
|
||||
// concurrent unlock cannot both prune, and the effective cadence stays
|
||||
// PruneMinInterval no matter how often this is called.
|
||||
try
|
||||
{
|
||||
using var scope = _scopeFactory.CreateScope();
|
||||
var repo = scope.ServiceProvider.GetRequiredService<IBundleUnlockAttemptRepository>();
|
||||
await TryPruneAsync(repo, _timeProvider.GetUtcNow().UtcDateTime, window, ct)
|
||||
.ConfigureAwait(false);
|
||||
}
|
||||
catch (OperationCanceledException) when (ct.IsCancellationRequested)
|
||||
{
|
||||
throw;
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
// The interface contract is "must not throw": there is no caller to report to,
|
||||
// and the periodic sweep driving this must survive a database blip. Resolving
|
||||
// the scope/repository is outside TryPruneAsync's own try, so it is caught here.
|
||||
_logger.LogWarning(
|
||||
ex, "Scheduled bundle-unlock attempt prune failed; will retry on the next sweep.");
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Opportunistic, rate-limited, bounded prune of aged-out attempt rows. Best-effort:
|
||||
/// any failure is logged and swallowed, because retention maintenance must never fail
|
||||
|
||||
Reference in New Issue
Block a user