fix(deploymentmanager): bulk site deploy skips up-to-date instances and un-poisons the shared DbContext (arch-review Low P7-5 + P7-6)

P7-6 (behaviour fix, plan decision 1): DeploySiteAsync redeployed EVERY
deployable instance unconditionally. A redeploy is not idempotent at the site —
it resets static attribute overrides and clears native alarm state by design — so
one site-wide push destroyed live operational state on every unchanged instance
in the batch, which is normally most of them. Bulk deploy now pre-filters with
IStaleInstanceProbe (the same seam the bundle importer and Deployments page use;
its memo timing was fixed by the Medium wave's P7-4): an instance whose freshly
computed revision hash still matches its DeployedConfigSnapshot is reported in the
existing additive SkippedInstances list with reason "up-to-date" and never
flattened, staged or sent. The filter is deliberately conservative — every
uncertainty deploys: only an Enabled instance qualifies, a missing snapshot
disqualifies, a prior record that ShouldQuerySiteBeforeRedeploy flags goes through
prepare so the reconcile that HEALS it still runs, and a null/faulting probe means
"indeterminate". New additive MgmtDeploySiteCommand.Force (default false = the new
filtered behaviour) plus CLI `deploy site --force` restores the unconditional push
for the case central cannot see (restored volume, rebuilt node). Composes with the
Medium wave's P7-2: Disabled wins — force suppresses the up-to-date filter only,
never the Disabled skip.

P7-5 (verified, then fixed): the cascade REPRODUCES. All three bulk phases share
one scoped DbContext, and a failed SaveChanges does not untrack the offending
entity — EF re-attempts it on the next save, including MarkDeploymentFailedAsync's
cleanup write. Measured on a real SQLite-backed ScadaBridgeDbContext: a permanent
save failure on instance 1's post-success side effects failed instances 2, 3 and 4
and left all three deployment records stranded InProgress. MarkDeploymentFailedAsync
now detaches the pending changes (new IDeploymentManagerRepository.DetachPendingChanges)
when the cleanup save fails and retries the Failed write once on the clean tracker,
so later instances save cleanly and the record reaches a terminal status. The
DeployFailed audit row moved outside the status-write try (best-effort, must not be
lost because the status write failed).

Tests: 5 new DeploySiteAsyncTests (skip + reason + no site round-trip; --force
deploys; Disabled skipped under both force values; the three not-enough-to-skip
guards), 2 new IntegrationTests over a real EF context (the poisoned-context
cascade, and the narrow EF tracking fact the fix rests on), CLI --force flag
parse test, and a MgmtDeploySite wire-default test. Proof of failure: both P7-6
behaviour tests and both P7-5 assertions fail against the unfixed code.
This commit is contained in:
Joseph Doherty
2026-08-19 16:37:42 -04:00
parent 6d0d8b6258
commit b34ec10f8b
14 changed files with 942 additions and 29 deletions
@@ -355,4 +355,26 @@ public interface IDeploymentManagerRepository
/// <param name="cancellationToken">A cancellation token that can be used to cancel the operation.</param>
/// <returns>A task representing the number of entities saved.</returns>
Task<int> SaveChangesAsync(CancellationToken cancellationToken = default);
/// <summary>
/// Abandons every pending (added / modified / deleted) change on the unit of
/// work, leaving the underlying store untouched, and returns how many entries
/// were dropped.
///
/// <para>
/// <b>Recovery only.</b> A <see cref="SaveChangesAsync"/> failure does NOT
/// untrack the entity that caused it: it stays in the change set and is
/// re-attempted by the NEXT save, so one unsavable entity poisons every
/// later save made through the same unit of work. That matters most on the
/// bulk site-deploy path, where all three phases share ONE scoped context
/// across every instance in the batch — a persistent failure on instance 1
/// would otherwise fail instance 2..N's status writes as well, including the
/// cleanup write that exists precisely to stop records being stranded
/// <c>InProgress</c>. Call this only after a save has already failed and its
/// pending work has been given up on; anything not yet committed is lost by
/// design.
/// </para>
/// </summary>
/// <returns>The number of tracked entries detached (0 when nothing was pending).</returns>
int DetachPendingChanges();
}
@@ -21,7 +21,16 @@ public record MgmtDeployArtifactsCommand(int? SiteId = null);
/// </para>
/// </summary>
/// <param name="SiteId">Database id of the target site. Required — this command is never fleet-wide.</param>
public record MgmtDeploySiteCommand(int SiteId);
/// <param name="Force">
/// Deploy every deployable instance even when its flattened config already matches
/// the revision the site is running. Default <see langword="false"/>: an unchanged
/// instance is skipped and reported in the summary's <c>SkippedInstances</c>,
/// because a redeploy resets static attribute overrides and native alarm state at
/// the site. Set it when central's recorded state cannot be trusted — a site whose
/// data volume was restored or whose node was rebuilt looks up-to-date from central
/// and needs the push anyway. Never overrides the Disabled skip.
/// </param>
public record MgmtDeploySiteCommand(int SiteId, bool Force = false);
public record QueryDeploymentsCommand(int? InstanceId = null, string? Status = null, int Page = 1, int PageSize = 50);
/// <summary>