Files
ScadaBridge/src/ZB.MOM.WW.ScadaBridge.AuditLog/Central/AuditLogPartitionMaintenanceService.cs
T
Joseph Doherty 9110a4eb01 fix(auditlog,health): harden hosted-service shutdown against disposed CTS
The host does not guarantee IHostedService.StopAsync is driven before the DI
container is disposed — WebApplicationFactory's teardown reaches Dispose first
— so cancelling the internal CTS from StopAsync threw ObjectDisposedException
and aborted the host's whole shutdown sequence. Four services shared the same
copy-pasted lifecycle and the same two races: StopAsync cancelling an already-
disposed CTS, and StartAsync reading _cts.Token lazily inside the Task.Run
lambda, which faults the loop task the host awaits when Dispose wins that race.

Each service now captures the token on the caller's thread, tolerates a
disposed CTS, and cancels-before-disposing so the loop is always signalled and
its pending Task.Delay sees a cancelled token rather than a dead source.
SiteAuditBacklogReporter also gains the outer OperationCanceledException guard
its sibling SiteAuditRetentionService already carried (arch-review 04 R2, R7),
without which a shutdown landing mid-probe threw TaskCanceledException out of
Host.StopAsync.

Surfaced while verifying the Gitea #15 test-harness fix: in Host.Tests the
aborted teardown skipped the fixture's env-var restore, contaminating every
later test in the run.

Refs: Gitea #15
2026-07-16 23:31:19 -04:00

199 lines
8.6 KiB
C#

using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;
using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Options;
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces;
namespace ZB.MOM.WW.ScadaBridge.AuditLog.Central;
/// <summary>
/// Central <see cref="IHostedService"/> that rolls
/// <c>pf_AuditLog_Month</c> forward once a day. Each tick opens a fresh DI
/// scope, resolves <see cref="IPartitionMaintenance"/>, and calls
/// <see cref="IPartitionMaintenance.EnsureLookaheadAsync"/> to SPLIT any
/// missing future boundaries — the partition function must always cover at
/// least <see cref="AuditLogPartitionMaintenanceOptions.LookaheadMonths"/>
/// future months, otherwise inserts past the highest boundary accumulate in
/// a single unbounded tail partition that <c>SwitchOutPartitionAsync</c>
/// cannot purge cleanly.
/// </summary>
/// <remarks>
/// <para>
/// <b>Why a hosted service, not an actor.</b>
/// <see cref="AuditLogPurgeActor"/> sits inside the central singleton
/// because it needs supervised lifecycle alongside the rest of the
/// reconciliation / ingest pipeline. Roll-forward is genuinely a once-a-day
/// chore with no cross-actor coordination, so we use the much simpler
/// hosted-service pattern: <c>Task.Run</c> on start, <c>Task.Delay</c>
/// between ticks, cancellation on stop. Reusing
/// <see cref="IPartitionMaintenance"/> from the central node-only DI graph
/// keeps the contract testable without any actor framework involvement.
/// </para>
/// <para>
/// <b>Failure containment.</b> The tick body wraps the maintenance call in
/// a try/catch so a transient SQL Server error never tears down the hosted
/// service — the next tick simply retries. The exception is logged with
/// the original stack trace at <c>Error</c> level; ops surfaces can subscribe to the logger to alert on
/// repeated failures.
/// </para>
/// <para>
/// <b>Startup ordering.</b> A first tick fires immediately at
/// <see cref="StartAsync"/> so a fresh deployment doesn't need to wait
/// <see cref="AuditLogPartitionMaintenanceOptions.IntervalSeconds"/> for
/// the partition function to come up to spec. This is also what the brief
/// asks for ("Run once on startup").
/// </para>
/// <para>
/// <b>DI scope per tick.</b> <see cref="IPartitionMaintenance"/> is scoped
/// (alongside the rest of the EF repositories) because the implementation
/// reuses the per-scope <c>ScadaBridgeDbContext</c>. A hosted service is a
/// singleton, so it must open and dispose a scope around each tick — the
/// same pattern <see cref="AuditLogPurgeActor"/> uses.
/// </para>
/// </remarks>
public sealed class AuditLogPartitionMaintenanceService : IHostedService, IDisposable
{
private readonly IServiceScopeFactory _scopeFactory;
private readonly IOptions<AuditLogPartitionMaintenanceOptions> _options;
private readonly ILogger<AuditLogPartitionMaintenanceService> _logger;
private CancellationTokenSource? _cts;
private Task? _loop;
/// <summary>
/// Initializes the maintenance service with its required dependencies.
/// </summary>
/// <param name="scopeFactory">Scope factory used to open DI scopes for each maintenance run.</param>
/// <param name="options">Partition maintenance options (retention period, purge interval, etc.).</param>
/// <param name="logger">Logger for this service.</param>
public AuditLogPartitionMaintenanceService(
IServiceScopeFactory scopeFactory,
IOptions<AuditLogPartitionMaintenanceOptions> options,
ILogger<AuditLogPartitionMaintenanceService> logger)
{
_scopeFactory = scopeFactory ?? throw new ArgumentNullException(nameof(scopeFactory));
_options = options ?? throw new ArgumentNullException(nameof(options));
_logger = logger ?? throw new ArgumentNullException(nameof(logger));
}
/// <summary>
/// Starts the background maintenance loop, firing an immediate first tick and then
/// repeating every <see cref="AuditLogPartitionMaintenanceOptions.IntervalSeconds"/>.
/// </summary>
/// <param name="ct">Cancellation token provided by the host.</param>
/// <returns>A completed task; the loop runs independently on a background thread.</returns>
public Task StartAsync(CancellationToken ct)
{
// Linked CTS lets StopAsync's cancellation AND the host's shutdown
// token both terminate the loop; either side firing aborts the
// pending Task.Delay.
var cts = CancellationTokenSource.CreateLinkedTokenSource(ct);
_cts = cts;
// Read Token here, on the caller's thread, rather than inside the lambda:
// the lambda runs whenever the thread pool gets to it, so a Dispose landing
// first would make the deferred _cts.Token read throw ObjectDisposedException
// and fault the loop task — which StopAsync hands to the host to await. The
// token struct stays usable once captured.
var token = cts.Token;
_loop = Task.Run(() => RunLoopAsync(token), CancellationToken.None);
return Task.CompletedTask;
}
private async Task RunLoopAsync(CancellationToken ct)
{
// Run once on startup so a fresh deployment isn't gated on the
// IntervalSeconds initial wait — the brief calls this out explicitly.
await SafeMaintainAsync(ct).ConfigureAwait(false);
while (!ct.IsCancellationRequested)
{
try
{
await Task.Delay(TimeSpan.FromSeconds(_options.Value.IntervalSeconds), ct)
.ConfigureAwait(false);
}
catch (OperationCanceledException)
{
break;
}
await SafeMaintainAsync(ct).ConfigureAwait(false);
}
}
private async Task SafeMaintainAsync(CancellationToken ct)
{
try
{
await using var scope = _scopeFactory.CreateAsyncScope();
var maintenance = scope.ServiceProvider.GetRequiredService<IPartitionMaintenance>();
var added = await maintenance
.EnsureLookaheadAsync(_options.Value.LookaheadMonths, ct)
.ConfigureAwait(false);
if (added.Count > 0)
{
_logger.LogInformation(
"AuditLogPartitionMaintenance added {Count} boundaries: {Boundaries}",
added.Count,
string.Join(", ", added.Select(b => b.ToString("yyyy-MM-dd"))));
}
}
catch (Exception ex)
{
// Catch-all is deliberate: the hosted service must survive every
// class of tick failure (transient SQL, DI resolution, etc.) so
// the next tick gets a chance. The brief's contract is
// "exception logged, not propagated".
_logger.LogError(ex, "AuditLogPartitionMaintenance tick failed");
}
}
/// <summary>
/// Signals the maintenance loop to stop by cancelling its linked token,
/// then returns the loop task so the host can await its completion.
/// </summary>
/// <param name="ct">Cancellation token provided by the host (unused — the internal CTS is cancelled directly).</param>
/// <returns>The background loop task, or a completed task if the loop was never started.</returns>
public Task StopAsync(CancellationToken ct)
{
try
{
_cts?.Cancel();
}
catch (ObjectDisposedException)
{
// Stop-after-Dispose is a legal ordering — WebApplicationFactory's
// teardown disposes the container before driving StopAsync. Dispose
// already cancelled the loop, so there is nothing left to signal, and
// letting this escape would abort the host's whole shutdown sequence.
}
return _loop ?? Task.CompletedTask;
}
/// <summary>
/// Cancels and disposes the internal <see cref="CancellationTokenSource"/> used to
/// stop the maintenance loop.
/// </summary>
/// <remarks>
/// Cancels before disposing so the loop is always signalled, even when the host
/// disposes the container without having driven <see cref="StopAsync"/> first.
/// It also keeps the loop's pending <c>Task.Delay(interval, token)</c> safe: an
/// already-cancelled token makes Delay complete as cancelled rather than register
/// a callback against a dead source and throw.
/// </remarks>
public void Dispose()
{
try
{
_cts?.Cancel();
}
catch (ObjectDisposedException)
{
// Already disposed — Dispose is idempotent.
}
_cts?.Dispose();
}
}