fix(deploy+cli): review findings — honest CLI timeouts, watermark-complete staleness, phase-2 staging, lock-safe cancellation

Six adversarial-review findings, each verified against the code first.

F1 (HIGH) CLI HttpClient capped every call at min(30s, caller timeout),
silently truncating deploy site's 5-minute BulkDeployTimeout and the
5-minute bundle export/preview/import calls — which printed a fake
"504 Request timed out" while the server kept working. HttpClient.Timeout
is now Timeout.InfiniteTimeSpan (the per-call CTS is the single overall
deadline, connect included) with the connect phase bounded separately on
SocketsHttpHandler.ConnectTimeout. The env override is renamed to
SCADABRIDGE_HTTP_CONNECT_TIMEOUT_SECONDS to match its new meaning.

F2 (HIGH) StaleInstanceProbe's process-static memo served stale hashes
because nothing bumped the watermark on three paths:
  (a) BundleImporter commits through the raw DbContext, so no import ever
      moved the watermark — a second import overwriting the same template
      could be OMITTED from ImportResult.StaleInstanceIds. It now bumps
      once per apply ATTEMPT: after the commit, and after the rollback too
      (the probe runs pre-commit, so a rolled-back attempt leaves memos for
      state that never landed; bumping on both paths is the simplest
      correct shape, versus threading transaction awareness through a
      process-static cache).
  (b) CollectWatermarkBumps' default: arm silently no-op'd, contradicting
      its own doc. It now sets unattributed=true — an over-broad bump costs
      extra work, a missed one produces stale work.
  (c) DataConnection edits route through SiteRepository.SaveChangesAsync,
      which had no watermark at all, yet Protocol/Primary+Backup config/
      FailoverRetryCount are revision-hash inputs. It now bumps (BumpAll —
      a connection has no owning template) after a commit that touched one.

F3 (MED) CLI TemplateTableProjection read child ARRAYS, but ListTemplates
now returns database-projected TemplateSummary rows, so template list
printed all zeros. It now prefers the *Count scalars and falls back to
array length (template get still returns full entities). --detail help
text and README corrected: a listing cannot yield definitions, so --detail
renders the raw summary payload and template get --id is the full dump.

F4 (MED) DeploySiteAsync staged every PendingDeployment in phase 1 against
a 5-min TTL while phase 2 reached them one batch at a time, so tail
instances' fetch tokens could expire before their command was sent.
Staging moved into phase 2, immediately before each send; prepare keeps
its flatten/validate/record work. The staging write is the phase's only
repository touch and is serialised behind a 1-permit semaphore, so the
non-thread-safe DbContext constraint holds and the sends stay concurrent.

F5 (MED, latent) DeploySiteAsync leaked every held operation lock if
cancelled — a wedged per-instance semaphore is permanent for the process.
Phase 2 no longer throws (cancellation is recorded as a per-instance
outcome so phase 3 still runs), and an escape from phase 1 or 3 now
unwinds every unfinalised entry: Failed status + lock release.

F6 (LOW) ScriptCompileVerdictCache's promotion wrote hot directly,
bypassing SegmentCapacity (true ceiling 3x against a documented 2x).
Promotion now goes through Store, keeping generational semantics; _hot
and _cold are volatile.

Tests: CLI 396, DeploymentManager 133, ManagementService 494,
TemplateEngine 478, ScriptAnalysis 60, Transport 157, Transport
integration 106, ConfigurationDatabase 366 — all green, 0 build warnings.
The F2/F4/F5 regression tests were each confirmed to FAIL with their fix
reverted.

tests/ZB.MOM.WW.ScadaBridge.CLI.Tests/ManagementHttpClientTests.cs

@@ -1,4 +1,6 @@
+using System.Diagnostics;
 using System.Net;
+using System.Net.Sockets;
 using System.Text;
 using ZB.MOM.WW.ScadaBridge.CLI;
 
@@ -106,32 +108,41 @@ public class ManagementHttpClientTests
 }
 
 /// <summary>
-/// WP2.6e (arch-review misc — CLI HttpClient timeout): the public
-/// <see cref="ManagementHttpClient"/> constructor must bound its underlying
-/// <see cref="HttpClient.Timeout"/> explicitly (30 s default) rather than leaving the
-/// framework's 100 s default in place, and must honor the
-/// <c>SCADABRIDGE_HTTP_TIMEOUT_SECONDS</c> override — consistent with how every other
-/// CLI setting is environment-overridable (<see cref="CliConfig"/>). Runs in the shared
-/// "Environment" collection (see <see cref="TestCollections"/>) so it never races another
-/// test mutating process-wide environment variables.
+/// The public <see cref="ManagementHttpClient"/> constructor must leave
+/// <see cref="HttpClient.Timeout"/> INFINITE so the per-call
+/// <see cref="CancellationTokenSource"/> is the single overall deadline — a fixed
+/// client timeout silently truncated every caller with a longer per-call timeout
+/// (<c>deploy site</c>'s 5-minute bulk deploy, the 5-minute <c>bundle</c> calls),
+/// which then printed a fake 504 while the server kept working. The connect phase
+/// is bounded separately on <see cref="SocketsHttpHandler.ConnectTimeout"/>, honoring
+/// the <c>SCADABRIDGE_HTTP_CONNECT_TIMEOUT_SECONDS</c> override — consistent with how
+/// every other CLI setting is environment-overridable (<see cref="CliConfig"/>). Runs
+/// in the shared "Environment" collection (see <see cref="TestCollections"/>) so it
+/// never races another test mutating process-wide environment variables.
 /// </summary>
 [Collection("Environment")]
 public class ManagementHttpClientTimeoutTests
 {
-    private const string EnvVar = "SCADABRIDGE_HTTP_TIMEOUT_SECONDS";
+    private const string EnvVar = "SCADABRIDGE_HTTP_CONNECT_TIMEOUT_SECONDS";
 
     [Fact]
-    public void DefaultConstructor_SetsThirtySecondTimeout_WhenEnvVarUnset()
+    public void DefaultConstructor_LeavesClientTimeoutInfinite()
+    {
+        using var client = new ManagementHttpClient("http://localhost:9001", "user", "pass");
+
+        Assert.Equal(Timeout.InfiniteTimeSpan, client.EffectiveTimeout);
+    }
+
+    [Fact]
+    public void ConnectTimeout_DefaultsToThirtySeconds_WhenEnvVarUnset()
     {
         var original = Environment.GetEnvironmentVariable(EnvVar);
         try
         {
             Environment.SetEnvironmentVariable(EnvVar, null);
 
-            using var client = new ManagementHttpClient("http://localhost:9001", "user", "pass");
-
-            Assert.Equal(TimeSpan.FromSeconds(30), ManagementHttpClient.DefaultTimeout);
-            Assert.Equal(TimeSpan.FromSeconds(30), client.EffectiveTimeout);
+            Assert.Equal(TimeSpan.FromSeconds(30), ManagementHttpClient.DefaultConnectTimeout);
+            Assert.Equal(TimeSpan.FromSeconds(30), ManagementHttpClient.ResolveConnectTimeout());
         }
         finally
         {
@@ -144,16 +155,14 @@ public class ManagementHttpClientTimeoutTests
     [InlineData("-5")]
     [InlineData("not-a-number")]
     [InlineData("")]
-    public void InvalidOrNonPositiveEnvValue_FallsBackToDefault(string value)
+    public void InvalidOrNonPositiveEnvValue_FallsBackToDefaultConnectTimeout(string value)
     {
         var original = Environment.GetEnvironmentVariable(EnvVar);
         try
         {
             Environment.SetEnvironmentVariable(EnvVar, value);
 
-            using var client = new ManagementHttpClient("http://localhost:9001", "user", "pass");
-
-            Assert.Equal(TimeSpan.FromSeconds(30), client.EffectiveTimeout);
+            Assert.Equal(TimeSpan.FromSeconds(30), ManagementHttpClient.ResolveConnectTimeout());
         }
         finally
         {
@@ -162,20 +171,79 @@ public class ManagementHttpClientTimeoutTests
     }
 
     [Fact]
-    public void PositiveEnvValue_OverridesDefaultTimeout()
+    public void PositiveEnvValue_OverridesDefaultConnectTimeout()
     {
         var original = Environment.GetEnvironmentVariable(EnvVar);
         try
         {
             Environment.SetEnvironmentVariable(EnvVar, "5");
 
-            using var client = new ManagementHttpClient("http://localhost:9001", "user", "pass");
-
-            Assert.Equal(TimeSpan.FromSeconds(5), client.EffectiveTimeout);
+            Assert.Equal(TimeSpan.FromSeconds(5), ManagementHttpClient.ResolveConnectTimeout());
         }
         finally
         {
             Environment.SetEnvironmentVariable(EnvVar, original);
         }
     }
+
+    /// <summary>
+    /// The regression that matters: a per-call timeout LONGER than the old 30 s
+    /// client cap must actually be honored. Two calls against the same hanging
+    /// local listener — one with a short deadline, one with a longer one — must
+    /// time out in that order and at their own deadlines, which is only possible
+    /// if <see cref="HttpClient.Timeout"/> is not silently capping both. Uses a
+    /// real socket (not the stub handler) so the connect + send path is exercised
+    /// end to end, and sub-second deadlines so the test stays fast.
+    /// </summary>
+    [Fact]
+    public async Task PerCallTimeoutLongerThanTheOldClientCap_IsHonored()
+    {
+        // A listener that accepts connections and then never answers: every
+        // request hangs until the caller's own deadline fires.
+        var listener = new TcpListener(IPAddress.Loopback, 0);
+        listener.Start();
+        var port = ((IPEndPoint)listener.LocalEndpoint).Port;
+        var accepted = new List<TcpClient>();
+        var acceptLoop = Task.Run(async () =>
+        {
+            try
+            {
+                while (true)
+                    accepted.Add(await listener.AcceptTcpClientAsync());
+            }
+            catch (ObjectDisposedException) { /* listener stopped — expected */ }
+            catch (SocketException) { /* listener stopped — expected */ }
+        });
+
+        try
+        {
+            using var client = new ManagementHttpClient($"http://127.0.0.1:{port}", "user", "pass");
+
+            var shortSw = Stopwatch.StartNew();
+            var shortResponse = await client.SendCommandAsync("ListSites", new { }, TimeSpan.FromMilliseconds(300));
+            shortSw.Stop();
+
+            var longSw = Stopwatch.StartNew();
+            var longResponse = await client.SendCommandAsync("ListSites", new { }, TimeSpan.FromMilliseconds(1500));
+            longSw.Stop();
+
+            Assert.Equal("TIMEOUT", shortResponse.ErrorCode);
+            Assert.Equal("TIMEOUT", longResponse.ErrorCode);
+
+            // The longer deadline must genuinely outlast the shorter one rather
+            // than both being clipped to a single client-wide cap.
+            Assert.True(
+                longSw.Elapsed > TimeSpan.FromMilliseconds(1000),
+                $"1.5 s per-call timeout returned after only {longSw.ElapsedMilliseconds} ms — the client cap truncated it.");
+            Assert.True(
+                shortSw.Elapsed < TimeSpan.FromMilliseconds(1000),
+                $"300 ms per-call timeout took {shortSw.ElapsedMilliseconds} ms.");
+        }
+        finally
+        {
+            listener.Stop();
+            foreach (var c in accepted) c.Dispose();
+            await acceptLoop;
+        }
+    }
 }