fix(deploy+cli): review findings — honest CLI timeouts, watermark-complete staleness, phase-2 staging, lock-safe cancellation
Six adversarial-review findings, each verified against the code first.
F1 (HIGH) CLI HttpClient capped every call at min(30s, caller timeout),
silently truncating deploy site's 5-minute BulkDeployTimeout and the
5-minute bundle export/preview/import calls — which printed a fake
"504 Request timed out" while the server kept working. HttpClient.Timeout
is now Timeout.InfiniteTimeSpan (the per-call CTS is the single overall
deadline, connect included) with the connect phase bounded separately on
SocketsHttpHandler.ConnectTimeout. The env override is renamed to
SCADABRIDGE_HTTP_CONNECT_TIMEOUT_SECONDS to match its new meaning.
F2 (HIGH) StaleInstanceProbe's process-static memo served stale hashes
because nothing bumped the watermark on three paths:
(a) BundleImporter commits through the raw DbContext, so no import ever
moved the watermark — a second import overwriting the same template
could be OMITTED from ImportResult.StaleInstanceIds. It now bumps
once per apply ATTEMPT: after the commit, and after the rollback too
(the probe runs pre-commit, so a rolled-back attempt leaves memos for
state that never landed; bumping on both paths is the simplest
correct shape, versus threading transaction awareness through a
process-static cache).
(b) CollectWatermarkBumps' default: arm silently no-op'd, contradicting
its own doc. It now sets unattributed=true — an over-broad bump costs
extra work, a missed one produces stale work.
(c) DataConnection edits route through SiteRepository.SaveChangesAsync,
which had no watermark at all, yet Protocol/Primary+Backup config/
FailoverRetryCount are revision-hash inputs. It now bumps (BumpAll —
a connection has no owning template) after a commit that touched one.
F3 (MED) CLI TemplateTableProjection read child ARRAYS, but ListTemplates
now returns database-projected TemplateSummary rows, so template list
printed all zeros. It now prefers the *Count scalars and falls back to
array length (template get still returns full entities). --detail help
text and README corrected: a listing cannot yield definitions, so --detail
renders the raw summary payload and template get --id is the full dump.
F4 (MED) DeploySiteAsync staged every PendingDeployment in phase 1 against
a 5-min TTL while phase 2 reached them one batch at a time, so tail
instances' fetch tokens could expire before their command was sent.
Staging moved into phase 2, immediately before each send; prepare keeps
its flatten/validate/record work. The staging write is the phase's only
repository touch and is serialised behind a 1-permit semaphore, so the
non-thread-safe DbContext constraint holds and the sends stay concurrent.
F5 (MED, latent) DeploySiteAsync leaked every held operation lock if
cancelled — a wedged per-instance semaphore is permanent for the process.
Phase 2 no longer throws (cancellation is recorded as a per-instance
outcome so phase 3 still runs), and an escape from phase 1 or 3 now
unwinds every unfinalised entry: Failed status + lock release.
F6 (LOW) ScriptCompileVerdictCache's promotion wrote hot directly,
bypassing SegmentCapacity (true ceiling 3x against a documented 2x).
Promotion now goes through Store, keeping generational semantics; _hot
and _cold are volatile.
Tests: CLI 396, DeploymentManager 133, ManagementService 494,
TemplateEngine 478, ScriptAnalysis 60, Transport 157, Transport
integration 106, ConfigurationDatabase 366 — all green, 0 build warnings.
The F2/F4/F5 regression tests were each confirmed to FAIL with their fix
reverted.
This commit is contained in:
@@ -87,6 +87,12 @@ public sealed class BundleImporter : IBundleImporter
|
||||
// then best-effort empty (informational only, never gates the import).
|
||||
private readonly IStaleInstanceProbe? _staleInstanceProbe;
|
||||
private readonly IScriptArtifactChangeBus? _scriptArtifactChangeBus;
|
||||
// Optional. Bumped once per apply ATTEMPT (see ApplyAsync) because this
|
||||
// importer commits through the raw DbContext, bypassing
|
||||
// TemplateEngineRepository.SaveChangesAsync — the only place that otherwise
|
||||
// derives watermark bumps from the change tracker. Null in hosts that don't
|
||||
// register it; the caches then simply never get this invalidation signal.
|
||||
private readonly ITemplateGraphWatermark? _watermark;
|
||||
private readonly ILogger<BundleImporter>? _logger;
|
||||
|
||||
/// <summary>
|
||||
@@ -112,6 +118,7 @@ public sealed class BundleImporter : IBundleImporter
|
||||
/// <param name="staleInstanceProbe">Optional: recomputes a deployed instance's current revision hash so the importer can enumerate instances stale-ed by a template overwrite. Null when no flattening pipeline is registered (Transport-without-DeploymentManager hosts) — staleness is then skipped.</param>
|
||||
/// <param name="scriptArtifactChangeBus">Optional: node-local bus notified AFTER a successful commit that script-bearing artifacts (ApiMethod / SharedScript / Template) were overwritten or renamed, so compiled-handler caches can invalidate. Null on hosts that don't register it (staleness/invalidation then relies on the consumer's own self-heal).</param>
|
||||
/// <param name="logger">Optional logger.</param>
|
||||
/// <param name="watermark">Optional: template-graph version watermark, bumped once per apply attempt so the flatten-session caches and the process-wide staleness memo cannot serve pre-import (or rolled-back) readings. Null on hosts that don't register it.</param>
|
||||
public BundleImporter(
|
||||
BundleSerializer bundleSerializer,
|
||||
ManifestValidator manifestValidator,
|
||||
@@ -132,7 +139,8 @@ public sealed class BundleImporter : IBundleImporter
|
||||
SemanticValidator semanticValidator,
|
||||
IStaleInstanceProbe? staleInstanceProbe = null,
|
||||
IScriptArtifactChangeBus? scriptArtifactChangeBus = null,
|
||||
ILogger<BundleImporter>? logger = null)
|
||||
ILogger<BundleImporter>? logger = null,
|
||||
ITemplateGraphWatermark? watermark = null)
|
||||
{
|
||||
_bundleSerializer = bundleSerializer ?? throw new ArgumentNullException(nameof(bundleSerializer));
|
||||
_manifestValidator = manifestValidator ?? throw new ArgumentNullException(nameof(manifestValidator));
|
||||
@@ -153,6 +161,7 @@ public sealed class BundleImporter : IBundleImporter
|
||||
_semanticValidator = semanticValidator ?? throw new ArgumentNullException(nameof(semanticValidator));
|
||||
_staleInstanceProbe = staleInstanceProbe;
|
||||
_scriptArtifactChangeBus = scriptArtifactChangeBus;
|
||||
_watermark = watermark;
|
||||
_logger = logger;
|
||||
}
|
||||
|
||||
@@ -1327,6 +1336,7 @@ public sealed class BundleImporter : IBundleImporter
|
||||
content, resolutions, nameMap, session, bundleImportId, user, ct).ConfigureAwait(false);
|
||||
await tx.CommitAsync(ct).ConfigureAwait(false);
|
||||
await tx.DisposeAsync().ConfigureAwait(false);
|
||||
BumpGraphWatermark();
|
||||
return applied;
|
||||
}
|
||||
catch
|
||||
@@ -1351,6 +1361,14 @@ public sealed class BundleImporter : IBundleImporter
|
||||
// persist on the next SaveChangesAsync (a retry or the
|
||||
// failure-row write below).
|
||||
_dbContext.ChangeTracker.Clear();
|
||||
|
||||
// Bump on the ROLLBACK path too. ComputeStaleInstanceIdsAsync
|
||||
// runs PRE-commit (it reads the staged change tracker), so a
|
||||
// rolled-back attempt has already written StaleInstanceProbe
|
||||
// memos describing state that never landed. Leaving the
|
||||
// watermark untouched would keep those memos "current"
|
||||
// forever — including for the strategy's own next retry.
|
||||
BumpGraphWatermark();
|
||||
throw;
|
||||
}
|
||||
}).ConfigureAwait(false);
|
||||
@@ -1651,6 +1669,38 @@ public sealed class BundleImporter : IBundleImporter
|
||||
Warnings: validationWarnings);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Invalidates every template-graph cache keyed on the
|
||||
/// <see cref="ITemplateGraphWatermark"/>, called once per apply ATTEMPT — after
|
||||
/// the commit on the success path, and after the rollback on the failure path.
|
||||
///
|
||||
/// <para>
|
||||
/// <b>Why the importer must do this itself.</b> Watermark bumps are normally
|
||||
/// derived from the change tracker by
|
||||
/// <c>TemplateEngineRepository.SaveChangesAsync</c>, but this importer commits
|
||||
/// through the raw <c>ScadaBridgeDbContext</c> (a single deferred
|
||||
/// <c>SaveChangesAsync</c> plus a handful of intra-transaction flushes), so not
|
||||
/// one of its writes went through that path. The observable bug: a second
|
||||
/// import overwriting the same template could be OMITTED from
|
||||
/// <c>ImportResult.StaleInstanceIds</c>, because <c>StaleInstanceProbe</c>'s
|
||||
/// process-static memo — validated purely against watermark readings that never
|
||||
/// moved — served the FIRST import's revision hash.
|
||||
/// </para>
|
||||
///
|
||||
/// <para>
|
||||
/// <b>Why both paths, and why <see cref="ITemplateGraphWatermark.BumpAll"/>.</b>
|
||||
/// The probe runs pre-commit, so a rolled-back attempt leaves memos for state
|
||||
/// that never landed; bumping on rollback is what makes the memo non-transactional
|
||||
/// yet still sound (the simplest correct shape — a transactional memo would mean
|
||||
/// threading commit/rollback awareness through a process-static cache). And an
|
||||
/// import touches an arbitrary, unattributed set of templates, instances and data
|
||||
/// connections, so the unattributed global bump is both the correct granularity
|
||||
/// and cheap: an import is a rare, heavyweight operation, and the watermark can
|
||||
/// only ever cause EXTRA flattening work, never stale work.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
private void BumpGraphWatermark() => _watermark?.BumpAll();
|
||||
|
||||
/// <summary>
|
||||
/// T-007: zeros the session's <see cref="BundleSession.DecryptedContent"/>
|
||||
/// buffer in place so any caller still holding a reference observes the
|
||||
|
||||
Reference in New Issue
Block a user