dohertj2/ScadaBridge
1
0
Fork 0
Code Issues 2 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

fix(deploy+cli): review findings — honest CLI timeouts, watermark-complete staleness, phase-2 staging, lock-safe cancellation

Browse Source 
Six adversarial-review findings, each verified against the code first.

F1 (HIGH) CLI HttpClient capped every call at min(30s, caller timeout),
silently truncating deploy site's 5-minute BulkDeployTimeout and the
5-minute bundle export/preview/import calls — which printed a fake
"504 Request timed out" while the server kept working. HttpClient.Timeout
is now Timeout.InfiniteTimeSpan (the per-call CTS is the single overall
deadline, connect included) with the connect phase bounded separately on
SocketsHttpHandler.ConnectTimeout. The env override is renamed to
SCADABRIDGE_HTTP_CONNECT_TIMEOUT_SECONDS to match its new meaning.

F2 (HIGH) StaleInstanceProbe's process-static memo served stale hashes
because nothing bumped the watermark on three paths:
  (a) BundleImporter commits through the raw DbContext, so no import ever
      moved the watermark — a second import overwriting the same template
      could be OMITTED from ImportResult.StaleInstanceIds. It now bumps
      once per apply ATTEMPT: after the commit, and after the rollback too
      (the probe runs pre-commit, so a rolled-back attempt leaves memos for
      state that never landed; bumping on both paths is the simplest
      correct shape, versus threading transaction awareness through a
      process-static cache).
  (b) CollectWatermarkBumps' default: arm silently no-op'd, contradicting
      its own doc. It now sets unattributed=true — an over-broad bump costs
      extra work, a missed one produces stale work.
  (c) DataConnection edits route through SiteRepository.SaveChangesAsync,
      which had no watermark at all, yet Protocol/Primary+Backup config/
      FailoverRetryCount are revision-hash inputs. It now bumps (BumpAll —
      a connection has no owning template) after a commit that touched one.

F3 (MED) CLI TemplateTableProjection read child ARRAYS, but ListTemplates
now returns database-projected TemplateSummary rows, so template list
printed all zeros. It now prefers the *Count scalars and falls back to
array length (template get still returns full entities). --detail help
text and README corrected: a listing cannot yield definitions, so --detail
renders the raw summary payload and template get --id is the full dump.

F4 (MED) DeploySiteAsync staged every PendingDeployment in phase 1 against
a 5-min TTL while phase 2 reached them one batch at a time, so tail
instances' fetch tokens could expire before their command was sent.
Staging moved into phase 2, immediately before each send; prepare keeps
its flatten/validate/record work. The staging write is the phase's only
repository touch and is serialised behind a 1-permit semaphore, so the
non-thread-safe DbContext constraint holds and the sends stay concurrent.

F5 (MED, latent) DeploySiteAsync leaked every held operation lock if
cancelled — a wedged per-instance semaphore is permanent for the process.
Phase 2 no longer throws (cancellation is recorded as a per-instance
outcome so phase 3 still runs), and an escape from phase 1 or 3 now
unwinds every unfinalised entry: Failed status + lock release.

F6 (LOW) ScriptCompileVerdictCache's promotion wrote hot directly,
bypassing SegmentCapacity (true ceiling 3x against a documented 2x).
Promotion now goes through Store, keeping generational semantics; _hot
and _cold are volatile.

Tests: CLI 396, DeploymentManager 133, ManagementService 494,
TemplateEngine 478, ScriptAnalysis 60, Transport 157, Transport
integration 106, ConfigurationDatabase 366 — all green, 0 build warnings.
The F2/F4/F5 regression tests were each confirmed to FAIL with their fix
reverted.
This commit is contained in:
Joseph Doherty
2026-08-14 23:51:04 -04:00
parent b1de9dfdd4
commit e0e4b24679
16 changed files with 1172 additions and 174 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/ZB.MOM.WW.ScadaBridge.CLI/ManagementHttpClient.cs
+51 -25
View File
@@ -9,33 +9,52 @@ public class ManagementHttpClient : IDisposable
private readonly HttpClient _httpClient;
/// <summary>
/// WP2.6e (arch-review misc — CLI HttpClient timeout): default overall
/// <see cref="HttpClient.Timeout"/> for the shared client construction (30 s). This
/// bounds a hung/black-holed connection — before this, the public constructor left
/// <see cref="HttpClient.Timeout"/> at its framework default (100 s), silently longer
/// than most CLI callers' own per-request <c>TimeSpan timeout</c> argument
/// (<see cref="SendCommandAsync"/>/<see cref="SendGetAsync"/>/<see cref="SendPostAsync"/>
/// already bound each call via their own <see cref="CancellationTokenSource"/>, but a
/// connection attempt that never completes at all — no response headers, ever — is
/// bounded by <see cref="HttpClient.Timeout"/> instead, since that governs the whole
/// request/response including connect). Config-overridable via the
/// <c>SCADABRIDGE_HTTP_TIMEOUT_SECONDS</c> environment variable, consistent with how
/// every other CLI setting is overridden (see <see cref="CliConfig"/>) — kept
/// self-contained here (no <see cref="CliConfig"/>/command-file plumbing) since CLI
/// commands are owned by a separate work package this phase.
/// Default bound on the CONNECT phase only (30 s) — how long a TCP/TLS
/// connection attempt to a black-holed management endpoint may hang before the
/// call fails.
///
/// <para>
/// It is deliberately NOT an overall request timeout.
/// <see cref="HttpClient.Timeout"/> caps the whole request/response, so setting
/// it to any fixed value silently truncates every caller whose own per-call
/// <c>TimeSpan timeout</c> argument is longer: the effective deadline becomes
/// <c>min(HttpClient.Timeout, caller timeout)</c>. That is exactly what a 30 s
/// client timeout did to <c>deploy site</c>'s 5-minute bulk deploy and to the
/// five-minute <c>bundle</c> export/preview/import calls — each printed a fake
/// <c>504 Request timed out</c> at 30 s while the server carried on working.
/// <see cref="HttpClient.Timeout"/> is therefore
/// <see cref="Timeout.InfiniteTimeSpan"/> and the per-call
/// <see cref="CancellationTokenSource"/> in
/// <see cref="SendCommandAsync"/>/<see cref="SendGetAsync"/>/<see cref="SendPostAsync"/>
/// is the SINGLE overall deadline — it bounds connect too, since the token is
/// passed into the send itself.
/// </para>
///
/// <para>
/// The connect bound lives on <see cref="SocketsHttpHandler.ConnectTimeout"/>
/// instead, which is connect-scoped and so cannot truncate a long-running
/// request that has already reached the server. Overridable via the
/// <c>SCADABRIDGE_HTTP_CONNECT_TIMEOUT_SECONDS</c> environment variable,
/// consistent with how every other CLI setting is overridden (see
/// <see cref="CliConfig"/>) — kept self-contained here (no
/// <see cref="CliConfig"/>/command-file plumbing) since CLI commands are owned
/// by a separate work package this phase.
/// </para>
/// </summary>
public static readonly TimeSpan DefaultTimeout = TimeSpan.FromSeconds(30);
public static readonly TimeSpan DefaultConnectTimeout = TimeSpan.FromSeconds(30);
/// <summary>Test seam (WP2.6e) — the effective <see cref="HttpClient.Timeout"/> this instance was constructed with.</summary>
/// <summary>Test seam — the effective <see cref="HttpClient.Timeout"/> this instance was constructed with.</summary>
internal TimeSpan EffectiveTimeout { get; }
/// <summary>
/// Resolves the effective default timeout: the <c>SCADABRIDGE_HTTP_TIMEOUT_SECONDS</c>
/// environment variable when set to a positive integer, otherwise <see cref="DefaultTimeout"/>.
/// Resolves the effective connect timeout: the
/// <c>SCADABRIDGE_HTTP_CONNECT_TIMEOUT_SECONDS</c> environment variable when set
/// to a positive integer, otherwise <see cref="DefaultConnectTimeout"/>.
/// </summary>
private static TimeSpan ResolveDefaultTimeout()
/// <returns>The connect timeout to apply to the socket handler.</returns>
internal static TimeSpan ResolveConnectTimeout()
{
var env = Environment.GetEnvironmentVariable("SCADABRIDGE_HTTP_TIMEOUT_SECONDS");
var env = Environment.GetEnvironmentVariable("SCADABRIDGE_HTTP_CONNECT_TIMEOUT_SECONDS");
if (!string.IsNullOrWhiteSpace(env)
&& int.TryParse(env, out var seconds)
&& seconds > 0)
@@ -43,19 +62,26 @@ public class ManagementHttpClient : IDisposable
return TimeSpan.FromSeconds(seconds);
}
return DefaultTimeout;
return DefaultConnectTimeout;
}
/// <summary>
/// Initializes a new instance of the <see cref="ManagementHttpClient"/> class, with
/// <see cref="HttpClient.Timeout"/> set to <see cref="ResolveDefaultTimeout"/>
/// (30 s, or the <c>SCADABRIDGE_HTTP_TIMEOUT_SECONDS</c> override).
/// Initializes a new instance of the <see cref="ManagementHttpClient"/> class with
/// an INFINITE <see cref="HttpClient.Timeout"/> (each call supplies its own
/// deadline) over a <see cref="SocketsHttpHandler"/> whose
/// <see cref="SocketsHttpHandler.ConnectTimeout"/> is
/// <see cref="ResolveConnectTimeout"/>.
/// </summary>
/// <param name="baseUrl">The base URL for the management API.</param>
/// <param name="username">The username for HTTP Basic authentication.</param>
/// <param name="password">The password for HTTP Basic authentication.</param>
public ManagementHttpClient(string baseUrl, string username, string password)
: this(new HttpClient { Timeout = ResolveDefaultTimeout() }, baseUrl, username, password)
: this(
new HttpClient(new SocketsHttpHandler { ConnectTimeout = ResolveConnectTimeout() })
{
Timeout = Timeout.InfiniteTimeSpan
},
baseUrl, username, password)
{
}