dohertj2/ScadaBridge
1
0
Fork 0
Code Issues 2 Pull Requests Actions Packages Projects Releases Wiki Activity

fix(deploy+cli): review findings — honest CLI timeouts, watermark-complete staleness, phase-2 staging, lock-safe cancellation

Browse Source 
Six adversarial-review findings, each verified against the code first.

F1 (HIGH) CLI HttpClient capped every call at min(30s, caller timeout),
silently truncating deploy site's 5-minute BulkDeployTimeout and the
5-minute bundle export/preview/import calls — which printed a fake
"504 Request timed out" while the server kept working. HttpClient.Timeout
is now Timeout.InfiniteTimeSpan (the per-call CTS is the single overall
deadline, connect included) with the connect phase bounded separately on
SocketsHttpHandler.ConnectTimeout. The env override is renamed to
SCADABRIDGE_HTTP_CONNECT_TIMEOUT_SECONDS to match its new meaning.

F2 (HIGH) StaleInstanceProbe's process-static memo served stale hashes
because nothing bumped the watermark on three paths:
  (a) BundleImporter commits through the raw DbContext, so no import ever
      moved the watermark — a second import overwriting the same template
      could be OMITTED from ImportResult.StaleInstanceIds. It now bumps
      once per apply ATTEMPT: after the commit, and after the rollback too
      (the probe runs pre-commit, so a rolled-back attempt leaves memos for
      state that never landed; bumping on both paths is the simplest
      correct shape, versus threading transaction awareness through a
      process-static cache).
  (b) CollectWatermarkBumps' default: arm silently no-op'd, contradicting
      its own doc. It now sets unattributed=true — an over-broad bump costs
      extra work, a missed one produces stale work.
  (c) DataConnection edits route through SiteRepository.SaveChangesAsync,
      which had no watermark at all, yet Protocol/Primary+Backup config/
      FailoverRetryCount are revision-hash inputs. It now bumps (BumpAll —
      a connection has no owning template) after a commit that touched one.

F3 (MED) CLI TemplateTableProjection read child ARRAYS, but ListTemplates
now returns database-projected TemplateSummary rows, so template list
printed all zeros. It now prefers the *Count scalars and falls back to
array length (template get still returns full entities). --detail help
text and README corrected: a listing cannot yield definitions, so --detail
renders the raw summary payload and template get --id is the full dump.

F4 (MED) DeploySiteAsync staged every PendingDeployment in phase 1 against
a 5-min TTL while phase 2 reached them one batch at a time, so tail
instances' fetch tokens could expire before their command was sent.
Staging moved into phase 2, immediately before each send; prepare keeps
its flatten/validate/record work. The staging write is the phase's only
repository touch and is serialised behind a 1-permit semaphore, so the
non-thread-safe DbContext constraint holds and the sends stay concurrent.

F5 (MED, latent) DeploySiteAsync leaked every held operation lock if
cancelled — a wedged per-instance semaphore is permanent for the process.
Phase 2 no longer throws (cancellation is recorded as a per-instance
outcome so phase 3 still runs), and an escape from phase 1 or 3 now
unwinds every unfinalised entry: Failed status + lock release.

F6 (LOW) ScriptCompileVerdictCache's promotion wrote hot directly,
bypassing SegmentCapacity (true ceiling 3x against a documented 2x).
Promotion now goes through Store, keeping generational semantics; _hot
and _cold are volatile.

Tests: CLI 396, DeploymentManager 133, ManagementService 494,
TemplateEngine 478, ScriptAnalysis 60, Transport 157, Transport
integration 106, ConfigurationDatabase 366 — all green, 0 build warnings.
The F2/F4/F5 regression tests were each confirmed to FAIL with their fix
reverted.
This commit is contained in:
Joseph Doherty
2026-08-14 23:51:04 -04:00
parent b1de9dfdd4
commit e0e4b24679
16 changed files with 1172 additions and 174 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/ZB.MOM.WW.ScadaBridge.DeploymentManager/DeploymentService.cs
+301 -89
View File
@@ -193,7 +193,8 @@ public class DeploymentService
using (prepared.LockHandle)
{
var outcome = await SendDeploymentAsync(prepared, cancellationToken);
var outcome = await StageAndSendDeploymentAsync(
prepared, user, dbGate: null, cancellationToken);
return await FinalizeDeploymentAsync(prepared, outcome, user, cancellationToken);
}
}
@@ -201,9 +202,18 @@ public class DeploymentService
/// <summary>
/// Phase 1 of a deployment (serial, database-bound): validate the state
/// transition, take the per-instance operation lock, mint the deployment id,
/// flatten + validate, run query-before-redeploy reconciliation, stage the
/// <c>PendingDeployment</c> row and insert the <c>InProgress</c>
/// <see cref="DeploymentRecord"/>.
/// flatten + validate, run query-before-redeploy reconciliation and insert the
/// <c>InProgress</c> <see cref="DeploymentRecord"/>.
///
/// <para>
/// Note what is NOT here: the <c>PendingDeployment</c> row. Its fetch token
/// expires <see cref="CommunicationOptions.PendingDeploymentTtl"/> after it is
/// STAGED, so staging in phase 1 started the clock on every instance in a bulk
/// deploy at once while phase 2 reached them one batch at a time — with enough
/// instances the tail's tokens expired before their command was ever sent, and
/// the site's fetch 404'd. Staging therefore moved to
/// <see cref="StageAndSendDeploymentAsync"/>, immediately before each send.
/// </para>
///
/// <para>
/// Everything here touches the scoped, non-thread-safe <c>DbContext</c>, so it
@@ -367,48 +377,135 @@ public class DeploymentService
try
{
// Notify-and-fetch: instead of shipping the (potentially oversized,
// silently-dropped >128 KB) flattened config inline in a
// DeployInstanceCommand, stage it in a PendingDeployment row and send
// a small RefreshDeploymentCommand. The site fetches the config from
// CentralFetchBaseUrl over HTTP using the per-deployment fetch token.
// Site routing is resolved here (a repository read) so phase 2 needs
// nothing but the staging write itself.
var siteId = await ResolveSiteIdentifierAsync(instance.SiteId, cancellationToken);
var token = DeploymentFetchToken.Generate();
var stagedAt = DateTimeOffset.UtcNow;
await _repository.AddPendingDeploymentAsync(new PendingDeployment(
deploymentId, instanceId, revisionHash, configJson, token,
stagedAt, stagedAt + _commOptions.PendingDeploymentTtl), cancellationToken);
await _repository.SaveChangesAsync(cancellationToken);
var command = new RefreshDeploymentCommand(
deploymentId, instance.UniqueName, revisionHash, user, stagedAt,
_commOptions.CentralFetchBaseUrl, token);
// Cleanup of the staged PendingDeployment is TTL-based ONLY — the row
// is deliberately NOT deleted on success or on failure. On a
// central-side Ask timeout the site may have applied AND told the
// standby node to fetch; deleting now would 404 that in-flight
// standby fetch and break failover. Supersession bounds pending rows
// to ≤1 per instance and the fetch endpoint enforces the TTL, so
// leaving rows for TTL purge is safe. Expired rows are swept by the
// central PendingDeploymentPurgeActor singleton on its
// CommunicationOptions.PendingDeploymentPurgeInterval cadence.
return new PreparedDeployment(
instance, deploymentId, record, revisionHash, configJson,
siteId, command, lockHandle, EarlyResult: null);
siteId, lockHandle, EarlyResult: null);
}
catch (Exception ex)
{
// Staging failed before anything was sent. Record the failure exactly
// as the post-send path does (never leave the record InProgress) and
// release the lock — there is no phase 2 or 3 to run.
// Preparation failed before anything was staged or sent. Record the
// failure exactly as the post-send path does (never leave the record
// InProgress) and release the lock — there is no phase 2 or 3 to run.
await MarkDeploymentFailedAsync(record, instance, deploymentId, user, ex);
lockHandle.Dispose();
return PreparedDeployment.Resolved(FailureResultFor(ex));
}
}
/// <summary>
/// Stages the deployment's <c>PendingDeployment</c> row and immediately sends
/// its <c>RefreshDeploymentCommand</c> — phase 2.
///
/// <para>
/// Notify-and-fetch: instead of shipping the (potentially oversized,
/// silently-dropped &gt;128 KB) flattened config inline in a
/// <c>DeployInstanceCommand</c>, the config is staged in a
/// <c>PendingDeployment</c> row and a small <c>RefreshDeploymentCommand</c> is
/// sent; the site then fetches the config from <c>CentralFetchBaseUrl</c> over
/// HTTP using the per-deployment fetch token.
/// </para>
///
/// <para>
/// <b>Staging is immediately before the send, on purpose.</b> The row's
/// <c>ExpiresAt</c> is <c>stagedAt + PendingDeploymentTtl</c> (5 min by
/// default) and the fetch endpoint enforces it, so the gap between staging and
/// sending is dead time burned off the token's life. Staging the whole batch up
/// front in phase 1 made that gap grow with batch size — for a large site the
/// last instances' tokens could expire before their command was even sent.
/// </para>
///
/// <para>
/// <b>Serialisation.</b> Staging is the ONE repository touch in this otherwise
/// network-only phase, and the scoped <c>DbContext</c> is not thread-safe, so a
/// bulk deploy passes <paramref name="dbGate"/> — a 1-permit semaphore that lets
/// exactly one in-flight send do its staging write at a time. The sends
/// themselves stay fully concurrent (they touch no repository), so this costs no
/// parallelism, and the single-instance path passes <see langword="null"/>
/// because there is nothing to serialise against.
/// </para>
///
/// <para>
/// A staging failure is returned as the phase-2 fault rather than thrown, so
/// phase 3 marks the record Failed on exactly the same path as a send fault —
/// the record must never be left InProgress.
/// </para>
/// </summary>
/// <param name="prepared">The prepared deployment to stage and send.</param>
/// <param name="user">User attributed on the refresh command.</param>
/// <param name="dbGate">Serialises the staging write across a concurrent batch; <see langword="null"/> for a batch of one.</param>
/// <param name="cancellationToken">Cancellation token, already carrying any per-instance deadline.</param>
/// <returns>The site's response, or the exception that prevented one.</returns>
private async Task<SendOutcome> StageAndSendDeploymentAsync(
PreparedDeployment prepared,
string user,
SemaphoreSlim? dbGate,
CancellationToken cancellationToken)
{
RefreshDeploymentCommand command;
try
{
if (dbGate is not null)
await dbGate.WaitAsync(cancellationToken);
try
{
command = await StagePendingDeploymentAsync(prepared, user, cancellationToken);
}
finally
{
dbGate?.Release();
}
}
catch (Exception ex)
{
return new SendOutcome(null, ex);
}
return await SendDeploymentAsync(prepared, command, cancellationToken);
}
/// <summary>
/// Writes the <c>PendingDeployment</c> row for a prepared deployment and builds
/// the matching <c>RefreshDeploymentCommand</c>, stamping <c>stagedAt</c> at the
/// moment of the write.
///
/// <para>
/// Cleanup of the staged row is TTL-based ONLY — it is deliberately NOT deleted
/// on success or on failure. On a central-side Ask timeout the site may have
/// applied AND told the standby node to fetch; deleting now would 404 that
/// in-flight standby fetch and break failover. Supersession bounds pending rows
/// to ≤1 per instance and the fetch endpoint enforces the TTL, so leaving rows
/// for TTL purge is safe. Expired rows are swept by the central
/// <c>PendingDeploymentPurgeActor</c> singleton on its
/// <see cref="CommunicationOptions.PendingDeploymentPurgeInterval"/> cadence.
/// </para>
/// </summary>
/// <param name="prepared">The prepared deployment being staged.</param>
/// <param name="user">User attributed on the refresh command.</param>
/// <param name="cancellationToken">Cancellation token.</param>
/// <returns>The refresh command to send to the site.</returns>
private async Task<RefreshDeploymentCommand> StagePendingDeploymentAsync(
PreparedDeployment prepared,
string user,
CancellationToken cancellationToken)
{
var token = DeploymentFetchToken.Generate();
var stagedAt = DateTimeOffset.UtcNow;
await _repository.AddPendingDeploymentAsync(new PendingDeployment(
prepared.DeploymentId, prepared.Instance.Id, prepared.RevisionHash,
prepared.ConfigJson, token,
stagedAt, stagedAt + _commOptions.PendingDeploymentTtl), cancellationToken);
await _repository.SaveChangesAsync(cancellationToken);
return new RefreshDeploymentCommand(
prepared.DeploymentId, prepared.Instance.UniqueName, prepared.RevisionHash,
user, stagedAt, _commOptions.CentralFetchBaseUrl, token);
}
/// <summary>
/// Phase 2 of a deployment (parallelisable, network-bound): the
/// <c>RefreshDeploymentCommand</c> round-trip to the site. Touches no
@@ -416,10 +513,12 @@ public class DeploymentService
/// out across a batch while phases 1 and 3 stay serial.
/// </summary>
/// <param name="prepared">The prepared deployment to send.</param>
/// <param name="command">The refresh command built by the staging step immediately before this call.</param>
/// <param name="cancellationToken">Cancellation token, already carrying any per-instance deadline.</param>
/// <returns>The site's response, or the exception that prevented one.</returns>
private async Task<SendOutcome> SendDeploymentAsync(
PreparedDeployment prepared,
RefreshDeploymentCommand command,
CancellationToken cancellationToken)
{
try
@@ -429,7 +528,7 @@ public class DeploymentService
prepared.DeploymentId, prepared.Instance.UniqueName, prepared.SiteIdentifier);
var response = await _communicationService.RefreshDeploymentAsync(
prepared.SiteIdentifier, prepared.Command, cancellationToken);
prepared.SiteIdentifier, command, cancellationToken);
return new SendOutcome(response, null);
}
catch (Exception ex)
@@ -630,7 +729,6 @@ public class DeploymentService
string RevisionHash,
string ConfigJson,
string SiteIdentifier,
RefreshDeploymentCommand Command,
IDisposable? LockHandle,
Result<DeploymentRecord>? EarlyResult)
{
@@ -638,7 +736,7 @@ public class DeploymentService
/// <param name="result">The result to hand back to the caller.</param>
/// <returns>A prepared deployment whose <see cref="EarlyResult"/> is set.</returns>
public static PreparedDeployment Resolved(Result<DeploymentRecord> result) =>
new(null!, string.Empty, null!, string.Empty, string.Empty, string.Empty, null!, null, result);
new(null!, string.Empty, null!, string.Empty, string.Empty, string.Empty, null, result);
}
/// <summary>
@@ -655,21 +753,23 @@ public class DeploymentService
/// </para>
/// <list type="number">
/// <item>
/// <b>Prepare (serial).</b> Every instance is flattened, validated, staged
/// and given an <c>InProgress</c> record on the caller's single scoped
/// <b>Prepare (serial).</b> Every instance is flattened, validated and given
/// an <c>InProgress</c> record on the caller's single scoped
/// <c>DbContext</c>. All instances share ONE <see cref="FlattenSession"/>,
/// so a template chain common to N instances is walked once, and the
/// session-global queries (shared scripts, schema library, the site's data
/// connections) run once for the whole batch instead of once per instance.
/// </item>
/// <item>
/// <b>Send (bounded parallel).</b> Site round-trips run concurrently up to
/// <see cref="DeploymentManagerOptions.SiteDeploymentMaxParallelism"/>,
/// <b>Stage + send (bounded parallel).</b> Site round-trips run concurrently
/// up to <see cref="DeploymentManagerOptions.SiteDeploymentMaxParallelism"/>,
/// each under its own
/// <see cref="DeploymentManagerOptions.SiteDeploymentTimeoutPerInstance"/>
/// deadline, so one wedged instance cannot stall the batch. This phase
/// touches no repository — that is precisely why it is the only phase that
/// may run in parallel against a non-thread-safe <c>DbContext</c>.
/// deadline, so one wedged instance cannot stall the batch. Each instance's
/// <c>PendingDeployment</c> row is staged here, immediately before its own
/// send, so a fetch token's TTL is not burned waiting for the batch ahead of
/// it; that single repository touch is serialised behind a 1-permit
/// semaphore, since the <c>DbContext</c> is not thread-safe.
/// </item>
/// <item>
/// <b>Finalize (serial).</b> Terminal statuses, post-success side effects
@@ -710,54 +810,81 @@ public class DeploymentService
var prepared = new List<PreparedDeployment>(instances.Count);
var results = new List<InstanceDeploymentResult>(instances.Count);
foreach (var instance in instances)
// Index of the first prepared deployment phase 3 has NOT yet finalised.
// Everything from here to the end of `prepared` still owns an operation
// lock and an InProgress record, so the cleanup handler below knows exactly
// what it must unwind if the operation escapes early.
var finalizedUpTo = 0;
try
{
cancellationToken.ThrowIfCancellationRequested();
foreach (var instance in instances)
{
cancellationToken.ThrowIfCancellationRequested();
PreparedDeployment step;
try
{
step = await PrepareDeploymentAsync(instance.Id, user, session, cancellationToken);
}
catch (Exception ex)
{
// A prepare fault (most commonly a TimeoutException from the
// operation lock because another operation holds this instance)
// fails only this instance. Recording it and moving on is what
// makes a bulk deploy usable while individual instances are busy.
_logger.LogWarning(ex,
"Preparing instance {Instance} for bulk deployment of site {SiteId} failed",
instance.UniqueName, site.SiteIdentifier);
results.Add(new InstanceDeploymentResult(
instance.Id, instance.UniqueName, null, false, ex.Message));
continue;
PreparedDeployment step;
try
{
step = await PrepareDeploymentAsync(instance.Id, user, session, cancellationToken);
}
catch (Exception ex)
{
// A prepare fault (most commonly a TimeoutException from the
// operation lock because another operation holds this instance)
// fails only this instance. Recording it and moving on is what
// makes a bulk deploy usable while individual instances are busy.
_logger.LogWarning(ex,
"Preparing instance {Instance} for bulk deployment of site {SiteId} failed",
instance.UniqueName, site.SiteIdentifier);
results.Add(new InstanceDeploymentResult(
instance.Id, instance.UniqueName, null, false, ex.Message));
continue;
}
if (step.EarlyResult is { } early)
{
results.Add(ToInstanceResult(instance.Id, instance.UniqueName, early));
continue;
}
prepared.Add(step);
}
if (step.EarlyResult is { } early)
{
results.Add(ToInstanceResult(instance.Id, instance.UniqueName, early));
continue;
}
// ---- Phase 2: stage + bounded-parallel site round-trips. ----
// Never throws: faults (cancellation included) come back as per-instance
// outcomes so phase 3 still runs for every prepared deployment.
var outcomes = await SendPreparedAsync(prepared, user, cancellationToken);
prepared.Add(step);
// ---- Phase 3: finalize, serially, releasing each lock as we go. ----
for (; finalizedUpTo < prepared.Count; finalizedUpTo++)
{
var step = prepared[finalizedUpTo];
try
{
var result = await FinalizeDeploymentAsync(step, outcomes[finalizedUpTo], user, cancellationToken);
results.Add(ToInstanceResult(step.Instance.Id, step.Instance.UniqueName, result, step.DeploymentId));
}
finally
{
step.LockHandle?.Dispose();
}
}
}
// ---- Phase 2: bounded-parallel site round-trips. ----
var outcomes = await SendPreparedAsync(prepared, cancellationToken);
// ---- Phase 3: finalize, serially, releasing each lock as we go. ----
for (var i = 0; i < prepared.Count; i++)
catch (Exception ex)
{
var step = prepared[i];
try
{
var result = await FinalizeDeploymentAsync(step, outcomes[i], user, cancellationToken);
results.Add(ToInstanceResult(step.Instance.Id, step.Instance.UniqueName, result, step.DeploymentId));
}
finally
{
step.LockHandle?.Dispose();
}
// Nothing above may escape while a prepared deployment still holds its
// per-instance operation lock: OperationLockManager hands out real
// semaphores, so an undisposed handle wedges that instance against
// EVERY future mutating command for the life of the process — a
// permanent, restart-only outage for one instance. The two live escape
// routes are phase 1's ThrowIfCancellationRequested and a fault from
// phase 3's own persistence; both land here.
//
// Records are finalised as Failed for the same reason the send/persist
// paths do it: a deployment must never be left InProgress. Uses
// CancellationToken.None inside (see MarkDeploymentFailedAsync) so the
// already-cancelled token cannot abort the cleanup writes themselves.
await ReleasePreparedAsync(prepared, finalizedUpTo, user, ex);
throw;
}
var summary = new SiteDeploymentSummary(
@@ -776,15 +903,76 @@ public class DeploymentService
}
/// <summary>
/// Runs phase 2 for a whole batch: every prepared deployment's site round-trip,
/// concurrent up to <see cref="DeploymentManagerOptions.SiteDeploymentMaxParallelism"/>
/// and each bounded by
/// Unwinds the prepared deployments a bulk deploy never finalised, after the
/// operation escaped early (cancellation, or a fault in phase 3's own
/// persistence). For each one it writes the terminal Failed status and releases
/// the per-instance operation lock.
///
/// <para>
/// Best-effort throughout: a fault while unwinding one entry must not stop the
/// remaining locks from being released, and must not mask the original
/// exception the caller is about to rethrow. The lock release in particular is
/// in a <c>finally</c>, because leaking it is the unrecoverable outcome — the
/// Failed status can be reconciled by an operator, a wedged semaphore cannot.
/// </para>
/// </summary>
/// <param name="prepared">The batch's prepared deployments.</param>
/// <param name="from">Index of the first entry phase 3 did not finalise.</param>
/// <param name="user">User attributed on the failure audit rows.</param>
/// <param name="cause">The exception that aborted the operation.</param>
/// <returns>A task that completes once every outstanding entry has been unwound.</returns>
private async Task ReleasePreparedAsync(
IReadOnlyList<PreparedDeployment> prepared,
int from,
string user,
Exception cause)
{
for (var i = from; i < prepared.Count; i++)
{
var step = prepared[i];
try
{
await MarkDeploymentFailedAsync(
step.Record, step.Instance, step.DeploymentId, user, cause);
}
catch (Exception ex)
{
_logger.LogError(ex,
"Failed to finalize deployment {DeploymentId} for instance {Instance} while unwinding " +
"an aborted bulk deployment",
step.DeploymentId, step.Instance.UniqueName);
}
finally
{
step.LockHandle?.Dispose();
}
}
}
/// <summary>
/// Runs phase 2 for a whole batch: every prepared deployment's staging write
/// plus site round-trip, concurrent up to
/// <see cref="DeploymentManagerOptions.SiteDeploymentMaxParallelism"/> and each
/// bounded by
/// <see cref="DeploymentManagerOptions.SiteDeploymentTimeoutPerInstance"/>.
/// Outcomes are returned positionally so phase 3 can pair them back with their
/// prepared deployment.
///
/// <para>
/// Never throws on cancellation: the whole point of returning outcomes
/// positionally is that phase 3 must run for EVERY prepared deployment, whatever
/// happened in phase 2 — otherwise records stay InProgress and their operation
/// locks stay held. A cancelled instance is recorded as its own
/// <see cref="SendOutcome"/> fault and finalised as Failed like any other.
/// </para>
/// </summary>
/// <param name="prepared">The prepared deployments to stage and send.</param>
/// <param name="user">User attributed on each refresh command.</param>
/// <param name="cancellationToken">Cancellation token for the batch.</param>
/// <returns>One outcome per prepared deployment, positionally aligned.</returns>
private async Task<SendOutcome[]> SendPreparedAsync(
IReadOnlyList<PreparedDeployment> prepared,
string user,
CancellationToken cancellationToken)
{
var outcomes = new SendOutcome[prepared.Count];
@@ -792,15 +980,39 @@ public class DeploymentService
return outcomes;
using var gate = new SemaphoreSlim(_options.SiteDeploymentMaxParallelism);
// Serialises the ONE repository touch left in this phase (the staging
// write) across the concurrent sends — the scoped DbContext is not
// thread-safe. The sends themselves stay fully concurrent.
using var dbGate = new SemaphoreSlim(1, 1);
var sends = prepared.Select(async (step, index) =>
{
await gate.WaitAsync(cancellationToken);
// Cancellation must not escape as an exception: it would abandon phase 3
// for every instance, leaking their operation locks (a wedged
// per-instance semaphore is permanent for the life of the process) and
// stranding their records InProgress.
try
{
await gate.WaitAsync(cancellationToken);
}
catch (OperationCanceledException ex)
{
outcomes[index] = new SendOutcome(null, ex);
return;
}
try
{
using var cts = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken);
cts.CancelAfter(_options.SiteDeploymentTimeoutPerInstance);
outcomes[index] = await SendDeploymentAsync(step, cts.Token);
outcomes[index] = await StageAndSendDeploymentAsync(step, user, dbGate, cts.Token);
}
catch (Exception ex)
{
// StageAndSendDeploymentAsync already converts its own faults into
// outcomes; this is the belt-and-braces net so no send task can
// fault Task.WhenAll and skip phase 3.
outcomes[index] = new SendOutcome(null, ex);
}
finally
{