fix(deploy+cli): review findings — honest CLI timeouts, watermark-complete staleness, phase-2 staging, lock-safe cancellation

Six adversarial-review findings, each verified against the code first.

F1 (HIGH) CLI HttpClient capped every call at min(30s, caller timeout),
silently truncating deploy site's 5-minute BulkDeployTimeout and the
5-minute bundle export/preview/import calls — which printed a fake
"504 Request timed out" while the server kept working. HttpClient.Timeout
is now Timeout.InfiniteTimeSpan (the per-call CTS is the single overall
deadline, connect included) with the connect phase bounded separately on
SocketsHttpHandler.ConnectTimeout. The env override is renamed to
SCADABRIDGE_HTTP_CONNECT_TIMEOUT_SECONDS to match its new meaning.

F2 (HIGH) StaleInstanceProbe's process-static memo served stale hashes
because nothing bumped the watermark on three paths:
  (a) BundleImporter commits through the raw DbContext, so no import ever
      moved the watermark — a second import overwriting the same template
      could be OMITTED from ImportResult.StaleInstanceIds. It now bumps
      once per apply ATTEMPT: after the commit, and after the rollback too
      (the probe runs pre-commit, so a rolled-back attempt leaves memos for
      state that never landed; bumping on both paths is the simplest
      correct shape, versus threading transaction awareness through a
      process-static cache).
  (b) CollectWatermarkBumps' default: arm silently no-op'd, contradicting
      its own doc. It now sets unattributed=true — an over-broad bump costs
      extra work, a missed one produces stale work.
  (c) DataConnection edits route through SiteRepository.SaveChangesAsync,
      which had no watermark at all, yet Protocol/Primary+Backup config/
      FailoverRetryCount are revision-hash inputs. It now bumps (BumpAll —
      a connection has no owning template) after a commit that touched one.

F3 (MED) CLI TemplateTableProjection read child ARRAYS, but ListTemplates
now returns database-projected TemplateSummary rows, so template list
printed all zeros. It now prefers the *Count scalars and falls back to
array length (template get still returns full entities). --detail help
text and README corrected: a listing cannot yield definitions, so --detail
renders the raw summary payload and template get --id is the full dump.

F4 (MED) DeploySiteAsync staged every PendingDeployment in phase 1 against
a 5-min TTL while phase 2 reached them one batch at a time, so tail
instances' fetch tokens could expire before their command was sent.
Staging moved into phase 2, immediately before each send; prepare keeps
its flatten/validate/record work. The staging write is the phase's only
repository touch and is serialised behind a 1-permit semaphore, so the
non-thread-safe DbContext constraint holds and the sends stay concurrent.

F5 (MED, latent) DeploySiteAsync leaked every held operation lock if
cancelled — a wedged per-instance semaphore is permanent for the process.
Phase 2 no longer throws (cancellation is recorded as a per-instance
outcome so phase 3 still runs), and an escape from phase 1 or 3 now
unwinds every unfinalised entry: Failed status + lock release.

F6 (LOW) ScriptCompileVerdictCache's promotion wrote hot directly,
bypassing SegmentCapacity (true ceiling 3x against a documented 2x).
Promotion now goes through Store, keeping generational semantics; _hot
and _cold are volatile.

Tests: CLI 396, DeploymentManager 133, ManagementService 494,
TemplateEngine 478, ScriptAnalysis 60, Transport 157, Transport
integration 106, ConfigurationDatabase 366 — all green, 0 build warnings.
The F2/F4/F5 regression tests were each confirmed to FAIL with their fix
reverted.

src/ZB.MOM.WW.ScadaBridge.CLI/Commands/TemplateCommands.cs

@@ -159,14 +159,20 @@ public static class TemplateCommands
 
     private static Command BuildList(Option<string> urlOption, Option<string> formatOption, Option<string> usernameOption, Option<string> passwordOption)
     {
+        // The server projects this listing in the database and returns
+        // TemplateSummary rows (counts, no child collections), so --detail cannot
+        // produce full definitions here — it only skips the CLI-side column
+        // projection and renders the summary payload verbatim. Full definitions
+        // come from `template get --id`.
         var detailOption = new Option<bool>("--detail")
         {
-            Description = "Include full template definitions (all attributes/alarms/scripts) in table output. "
-                + "Without it, table output is a compact summary (counts only). JSON output is always full."
+            Description = "Render the raw list payload in table output instead of the compact column projection. "
+                + "The server returns summary rows (member counts only) for a listing, so this does NOT include "
+                + "attribute/alarm/script definitions — use 'template get --id <int>' for those. No effect on JSON output."
         };
         var skipOption = new Option<int>("--skip") { Description = "Offset paging: number of items to skip (default 0)" };
         var takeOption = new Option<int?>("--take") { Description = "Offset paging: max items to return (1..1000; omit for unlimited)" };
-        var cmd = new Command("list") { Description = "List all templates (compact table summary; use --detail for the full dump)" };
+        var cmd = new Command("list") { Description = "List all templates (compact table summary with member counts; definitions via 'template get')" };
         cmd.Add(detailOption);
         cmd.Add(skipOption);
         cmd.Add(takeOption);

src/ZB.MOM.WW.ScadaBridge.CLI/Commands/TemplateTableProjection.cs

@@ -5,12 +5,30 @@ namespace ZB.MOM.WW.ScadaBridge.CLI.Commands;
 
 /// <summary>
 /// Compact table projection for <c>template list</c> / <c>template get</c>.
-/// The management API returns full <c>Template</c> entities — every attribute, alarm,
+///
+/// <para>
+/// <c>template get</c> returns a full <c>Template</c> entity — every attribute, alarm,
 /// script, and composition inline — which the generic table renderer dumps as one giant
-/// cell per template (~171 KB for a real catalogue, unusable in a terminal). This
-/// projector reduces each template to id / name / description / parent / derived plus
-/// member <em>counts</em>, leaving JSON output untouched (callers pass this only on the
-/// table path) and the full dump available via the command's <c>--detail</c> flag.
+/// cell per template (~171 KB for a real catalogue, unusable in a terminal).
+/// <c>template list</c> no longer returns entities at all: the management
+/// <c>ListTemplates</c> handler projects in the DATABASE and returns
+/// <c>TemplateSummary</c> rows whose children are already reduced to
+/// <c>attributeCount</c>/<c>alarmCount</c>/… scalars.
+/// </para>
+///
+/// <para>
+/// This projector handles BOTH shapes: it prefers the pre-computed <c>*Count</c>
+/// scalar and falls back to the length of the matching child array, so the same
+/// column set renders for a summary row and for a full entity. Reading only the
+/// arrays (as it originally did) made <c>template list</c> print a wall of zeros
+/// once the server switched to summaries.
+/// </para>
+///
+/// <para>
+/// JSON output is left untouched (callers pass this only on the table path), and
+/// the command's <c>--detail</c> flag skips the projection to render the raw
+/// payload as-is.
+/// </para>
 /// </summary>
 internal static class TemplateTableProjection
 {
@@ -67,11 +85,11 @@ internal static class TemplateTableProjection
             ["description"] = Str(element, "description"),
             ["parentTemplateId"] = Int(element, "parentTemplateId"),
             ["isDerived"] = Bool(element, "isDerived"),
-            ["#attrs"] = Count(element, "attributes"),
-            ["#alarms"] = Count(element, "alarms"),
-            ["#scripts"] = Count(element, "scripts"),
-            ["#comps"] = Count(element, "compositions"),
-            ["#nativeAlarms"] = Count(element, "nativeAlarmSources"),
+            ["#attrs"] = Count(element, "attributeCount", "attributes"),
+            ["#alarms"] = Count(element, "alarmCount", "alarms"),
+            ["#scripts"] = Count(element, "scriptCount", "scripts"),
+            ["#comps"] = Count(element, "compositionCount", "compositions"),
+            ["#nativeAlarms"] = Count(element, "nativeAlarmSourceCount", "nativeAlarmSources"),
         };
     }
 
@@ -104,9 +122,28 @@ internal static class TemplateTableProjection
             ? JsonValue.Create(v.GetBoolean())
             : null;
 
-    private static JsonNode Count(JsonElement obj, string name)
-        => JsonValue.Create(
-            TryGetPropertyCI(obj, name, out var v) && v.ValueKind == JsonValueKind.Array
+    /// <summary>
+    /// Member count for a column, preferring the summary payload's pre-computed
+    /// scalar (<paramref name="countName"/>) and falling back to the length of the
+    /// full entity's child array (<paramref name="arrayName"/>). Zero when neither
+    /// is present.
+    /// </summary>
+    /// <param name="obj">The template object being projected.</param>
+    /// <param name="countName">Scalar count property on a <c>TemplateSummary</c> row.</param>
+    /// <param name="arrayName">Child-collection property on a full <c>Template</c> entity.</param>
+    /// <returns>The member count as a JSON number.</returns>
+    private static JsonNode Count(JsonElement obj, string countName, string arrayName)
+    {
+        if (TryGetPropertyCI(obj, countName, out var scalar)
+            && scalar.ValueKind == JsonValueKind.Number
+            && scalar.TryGetInt32(out var n))
+        {
+            return JsonValue.Create(n);
+        }
+
+        return JsonValue.Create(
+            TryGetPropertyCI(obj, arrayName, out var v) && v.ValueKind == JsonValueKind.Array
                 ? v.GetArrayLength()
                 : 0);
+    }
 }

src/ZB.MOM.WW.ScadaBridge.CLI/ManagementHttpClient.cs

@@ -9,33 +9,52 @@ public class ManagementHttpClient : IDisposable
     private readonly HttpClient _httpClient;
 
     /// <summary>
-    /// WP2.6e (arch-review misc — CLI HttpClient timeout): default overall
-    /// <see cref="HttpClient.Timeout"/> for the shared client construction (30 s). This
-    /// bounds a hung/black-holed connection — before this, the public constructor left
-    /// <see cref="HttpClient.Timeout"/> at its framework default (100 s), silently longer
-    /// than most CLI callers' own per-request <c>TimeSpan timeout</c> argument
-    /// (<see cref="SendCommandAsync"/>/<see cref="SendGetAsync"/>/<see cref="SendPostAsync"/>
-    /// already bound each call via their own <see cref="CancellationTokenSource"/>, but a
-    /// connection attempt that never completes at all — no response headers, ever — is
-    /// bounded by <see cref="HttpClient.Timeout"/> instead, since that governs the whole
-    /// request/response including connect). Config-overridable via the
-    /// <c>SCADABRIDGE_HTTP_TIMEOUT_SECONDS</c> environment variable, consistent with how
-    /// every other CLI setting is overridden (see <see cref="CliConfig"/>) — kept
-    /// self-contained here (no <see cref="CliConfig"/>/command-file plumbing) since CLI
-    /// commands are owned by a separate work package this phase.
+    /// Default bound on the CONNECT phase only (30 s) — how long a TCP/TLS
+    /// connection attempt to a black-holed management endpoint may hang before the
+    /// call fails.
+    ///
+    /// <para>
+    /// It is deliberately NOT an overall request timeout.
+    /// <see cref="HttpClient.Timeout"/> caps the whole request/response, so setting
+    /// it to any fixed value silently truncates every caller whose own per-call
+    /// <c>TimeSpan timeout</c> argument is longer: the effective deadline becomes
+    /// <c>min(HttpClient.Timeout, caller timeout)</c>. That is exactly what a 30 s
+    /// client timeout did to <c>deploy site</c>'s 5-minute bulk deploy and to the
+    /// five-minute <c>bundle</c> export/preview/import calls — each printed a fake
+    /// <c>504 Request timed out</c> at 30 s while the server carried on working.
+    /// <see cref="HttpClient.Timeout"/> is therefore
+    /// <see cref="Timeout.InfiniteTimeSpan"/> and the per-call
+    /// <see cref="CancellationTokenSource"/> in
+    /// <see cref="SendCommandAsync"/>/<see cref="SendGetAsync"/>/<see cref="SendPostAsync"/>
+    /// is the SINGLE overall deadline — it bounds connect too, since the token is
+    /// passed into the send itself.
+    /// </para>
+    ///
+    /// <para>
+    /// The connect bound lives on <see cref="SocketsHttpHandler.ConnectTimeout"/>
+    /// instead, which is connect-scoped and so cannot truncate a long-running
+    /// request that has already reached the server. Overridable via the
+    /// <c>SCADABRIDGE_HTTP_CONNECT_TIMEOUT_SECONDS</c> environment variable,
+    /// consistent with how every other CLI setting is overridden (see
+    /// <see cref="CliConfig"/>) — kept self-contained here (no
+    /// <see cref="CliConfig"/>/command-file plumbing) since CLI commands are owned
+    /// by a separate work package this phase.
+    /// </para>
     /// </summary>
-    public static readonly TimeSpan DefaultTimeout = TimeSpan.FromSeconds(30);
+    public static readonly TimeSpan DefaultConnectTimeout = TimeSpan.FromSeconds(30);
 
-    /// <summary>Test seam (WP2.6e) — the effective <see cref="HttpClient.Timeout"/> this instance was constructed with.</summary>
+    /// <summary>Test seam — the effective <see cref="HttpClient.Timeout"/> this instance was constructed with.</summary>
     internal TimeSpan EffectiveTimeout { get; }
 
     /// <summary>
-    /// Resolves the effective default timeout: the <c>SCADABRIDGE_HTTP_TIMEOUT_SECONDS</c>
-    /// environment variable when set to a positive integer, otherwise <see cref="DefaultTimeout"/>.
+    /// Resolves the effective connect timeout: the
+    /// <c>SCADABRIDGE_HTTP_CONNECT_TIMEOUT_SECONDS</c> environment variable when set
+    /// to a positive integer, otherwise <see cref="DefaultConnectTimeout"/>.
     /// </summary>
-    private static TimeSpan ResolveDefaultTimeout()
+    /// <returns>The connect timeout to apply to the socket handler.</returns>
+    internal static TimeSpan ResolveConnectTimeout()
     {
-        var env = Environment.GetEnvironmentVariable("SCADABRIDGE_HTTP_TIMEOUT_SECONDS");
+        var env = Environment.GetEnvironmentVariable("SCADABRIDGE_HTTP_CONNECT_TIMEOUT_SECONDS");
         if (!string.IsNullOrWhiteSpace(env)
             && int.TryParse(env, out var seconds)
             && seconds > 0)
@@ -43,19 +62,26 @@ public class ManagementHttpClient : IDisposable
             return TimeSpan.FromSeconds(seconds);
         }
 
-        return DefaultTimeout;
+        return DefaultConnectTimeout;
     }
 
     /// <summary>
-    /// Initializes a new instance of the <see cref="ManagementHttpClient"/> class, with
-    /// <see cref="HttpClient.Timeout"/> set to <see cref="ResolveDefaultTimeout"/>
-    /// (30 s, or the <c>SCADABRIDGE_HTTP_TIMEOUT_SECONDS</c> override).
+    /// Initializes a new instance of the <see cref="ManagementHttpClient"/> class with
+    /// an INFINITE <see cref="HttpClient.Timeout"/> (each call supplies its own
+    /// deadline) over a <see cref="SocketsHttpHandler"/> whose
+    /// <see cref="SocketsHttpHandler.ConnectTimeout"/> is
+    /// <see cref="ResolveConnectTimeout"/>.
     /// </summary>
     /// <param name="baseUrl">The base URL for the management API.</param>
     /// <param name="username">The username for HTTP Basic authentication.</param>
     /// <param name="password">The password for HTTP Basic authentication.</param>
     public ManagementHttpClient(string baseUrl, string username, string password)
-        : this(new HttpClient { Timeout = ResolveDefaultTimeout() }, baseUrl, username, password)
+        : this(
+            new HttpClient(new SocketsHttpHandler { ConnectTimeout = ResolveConnectTimeout() })
+            {
+                Timeout = Timeout.InfiniteTimeSpan
+            },
+            baseUrl, username, password)
     {
     }
 

src/ZB.MOM.WW.ScadaBridge.CLI/README.md

@@ -86,28 +86,33 @@ Exit codes:
 
 #### `template list`
 
-List all templates. **Table** output (`--format table`) shows a compact summary — id,
-name, description, parent, and member **counts** (`#attrs`, `#alarms`, `#scripts`,
-`#comps`, `#nativeAlarms`) — so it stays readable in a terminal. Add `--detail` to dump
-the full attribute/alarm/script/composition definitions in the table. **JSON** output
-(`--format json`) is always the full, unmodified payload regardless of `--detail`.
+List all templates. The server projects this listing in the database and returns
+**summary rows** — no child collections, member counts only. **Table** output
+(`--format table`) renders them as id, name, description, parent, and member **counts**
+(`#attrs`, `#alarms`, `#scripts`, `#comps`, `#nativeAlarms`), so it stays readable in a
+terminal. `--detail` skips that column projection and renders the summary payload
+verbatim; it does **not** add attribute/alarm/script definitions — a listing never
+carries them. Use `template get --id <int>` for a template's full definition. **JSON**
+output (`--format json`) is always the unmodified payload regardless of `--detail`.
 
 ```sh
-scadabridge --url <url> template list                       # compact table
-scadabridge --url <url> --format table template list --detail   # full table dump
-scadabridge --url <url> --format json template list        # full JSON (always)
+scadabridge --url <url> template list                            # compact table
+scadabridge --url <url> --format table template list --detail    # raw summary rows
+scadabridge --url <url> --format json template list              # unmodified JSON (always)
 ```
 
 | Option | Required | Description |
 |--------|----------|-------------|
-| `--detail` | no | Include full definitions in table output (no effect on JSON) |
+| `--detail` | no | Render the raw summary payload instead of the compact column projection (no effect on JSON; does not add definitions) |
 | `--skip` | no | Offset paging: number of items to skip (default 0) |
 | `--take` | no | Offset paging: max items to return (clamped 1..1000; omit for unlimited) |
 
 #### `template get`
 
-Get a single template by ID. Like `template list`, table output is a compact summary
-unless `--detail` is supplied; JSON output is always full.
+Get a single template by ID. Unlike `template list`, this returns the **full** template
+entity with every attribute, alarm, script and composition inline. Table output is a
+compact summary (the same counts as `template list`) unless `--detail` is supplied, which
+dumps the full definitions; JSON output is always full.
 
 ```sh
 scadabridge --url <url> template get --id <int> [--detail]