fix(deploy+cli): review findings — honest CLI timeouts, watermark-complete staleness, phase-2 staging, lock-safe cancellation

Six adversarial-review findings, each verified against the code first. F1 (HIGH) CLI HttpClient capped every call at min(30s, caller timeout), silently truncating deploy site's 5-minute BulkDeployTimeout and the 5-minute bundle export/preview/import calls — which printed a fake "504 Request timed out" while the server kept working. HttpClient.Timeout is now Timeout.InfiniteTimeSpan (the per-call CTS is the single overall deadline, connect included) with the connect phase bounded separately on SocketsHttpHandler.ConnectTimeout. The env override is renamed to SCADABRIDGE_HTTP_CONNECT_TIMEOUT_SECONDS to match its new meaning. F2 (HIGH) StaleInstanceProbe's process-static memo served stale hashes because nothing bumped the watermark on three paths: (a) BundleImporter commits through the raw DbContext, so no import ever moved the watermark — a second import overwriting the same template could be OMITTED from ImportResult.StaleInstanceIds. It now bumps once per apply ATTEMPT: after the commit, and after the rollback too (the probe runs pre-commit, so a rolled-back attempt leaves memos for state that never landed; bumping on both paths is the simplest correct shape, versus threading transaction awareness through a process-static cache). (b) CollectWatermarkBumps' default: arm silently no-op'd, contradicting its own doc. It now sets unattributed=true — an over-broad bump costs extra work, a missed one produces stale work. (c) DataConnection edits route through SiteRepository.SaveChangesAsync, which had no watermark at all, yet Protocol/Primary+Backup config/ FailoverRetryCount are revision-hash inputs. It now bumps (BumpAll — a connection has no owning template) after a commit that touched one. F3 (MED) CLI TemplateTableProjection read child ARRAYS, but ListTemplates now returns database-projected TemplateSummary rows, so template list printed all zeros. It now prefers the *Count scalars and falls back to array length (template get still returns full entities). --detail help text and README corrected: a listing cannot yield definitions, so --detail renders the raw summary payload and template get --id is the full dump. F4 (MED) DeploySiteAsync staged every PendingDeployment in phase 1 against a 5-min TTL while phase 2 reached them one batch at a time, so tail instances' fetch tokens could expire before their command was sent. Staging moved into phase 2, immediately before each send; prepare keeps its flatten/validate/record work. The staging write is the phase's only repository touch and is serialised behind a 1-permit semaphore, so the non-thread-safe DbContext constraint holds and the sends stay concurrent. F5 (MED, latent) DeploySiteAsync leaked every held operation lock if cancelled — a wedged per-instance semaphore is permanent for the process. Phase 2 no longer throws (cancellation is recorded as a per-instance outcome so phase 3 still runs), and an escape from phase 1 or 3 now unwinds every unfinalised entry: Failed status + lock release. F6 (LOW) ScriptCompileVerdictCache's promotion wrote hot directly, bypassing SegmentCapacity (true ceiling 3x against a documented 2x). Promotion now goes through Store, keeping generational semantics; _hot and _cold are volatile. Tests: CLI 396, DeploymentManager 133, ManagementService 494, TemplateEngine 478, ScriptAnalysis 60, Transport 157, Transport integration 106, ConfigurationDatabase 366 — all green, 0 build warnings. The F2/F4/F5 regression tests were each confirmed to FAIL with their fix reverted.
2026-08-14 23:51:04 -04:00
parent b1de9dfdd4
commit e0e4b24679
16 changed files with 1172 additions and 174 deletions
@@ -1,4 +1,6 @@
+using System.Diagnostics;
 using System.Net;
+using System.Net.Sockets;
 using System.Text;
 using ZB.MOM.WW.ScadaBridge.CLI;

@@ -106,32 +108,41 @@ public class ManagementHttpClientTests
 }

 /// <summary>
-/// WP2.6e (arch-review misc — CLI HttpClient timeout): the public
-/// <see cref="ManagementHttpClient"/> constructor must bound its underlying
-/// <see cref="HttpClient.Timeout"/> explicitly (30 s default) rather than leaving the
-/// framework's 100 s default in place, and must honor the
-/// <c>SCADABRIDGE_HTTP_TIMEOUT_SECONDS</c> override — consistent with how every other
-/// CLI setting is environment-overridable (<see cref="CliConfig"/>). Runs in the shared
-/// "Environment" collection (see <see cref="TestCollections"/>) so it never races another
-/// test mutating process-wide environment variables.
+/// The public <see cref="ManagementHttpClient"/> constructor must leave
+/// <see cref="HttpClient.Timeout"/> INFINITE so the per-call
+/// <see cref="CancellationTokenSource"/> is the single overall deadline — a fixed
+/// client timeout silently truncated every caller with a longer per-call timeout
+/// (<c>deploy site</c>'s 5-minute bulk deploy, the 5-minute <c>bundle</c> calls),
+/// which then printed a fake 504 while the server kept working. The connect phase
+/// is bounded separately on <see cref="SocketsHttpHandler.ConnectTimeout"/>, honoring
+/// the <c>SCADABRIDGE_HTTP_CONNECT_TIMEOUT_SECONDS</c> override — consistent with how
+/// every other CLI setting is environment-overridable (<see cref="CliConfig"/>). Runs
+/// in the shared "Environment" collection (see <see cref="TestCollections"/>) so it
+/// never races another test mutating process-wide environment variables.
 /// </summary>
 [Collection("Environment")]
 public class ManagementHttpClientTimeoutTests
 {
-    private const string EnvVar = "SCADABRIDGE_HTTP_TIMEOUT_SECONDS";
+    private const string EnvVar = "SCADABRIDGE_HTTP_CONNECT_TIMEOUT_SECONDS";

    [Fact]
-    public void DefaultConstructor_SetsThirtySecondTimeout_WhenEnvVarUnset()
+    public void DefaultConstructor_LeavesClientTimeoutInfinite()
+    {
+        using var client = new ManagementHttpClient("http://localhost:9001", "user", "pass");
+
+        Assert.Equal(Timeout.InfiniteTimeSpan, client.EffectiveTimeout);
+    }
+
+    [Fact]
+    public void ConnectTimeout_DefaultsToThirtySeconds_WhenEnvVarUnset()
    {
        var original = Environment.GetEnvironmentVariable(EnvVar);
        try
        {
            Environment.SetEnvironmentVariable(EnvVar, null);

-            using var client = new ManagementHttpClient("http://localhost:9001", "user", "pass");
-
-            Assert.Equal(TimeSpan.FromSeconds(30), ManagementHttpClient.DefaultTimeout);
-            Assert.Equal(TimeSpan.FromSeconds(30), client.EffectiveTimeout);
+            Assert.Equal(TimeSpan.FromSeconds(30), ManagementHttpClient.DefaultConnectTimeout);
+            Assert.Equal(TimeSpan.FromSeconds(30), ManagementHttpClient.ResolveConnectTimeout());
        }
        finally
        {
@@ -144,16 +155,14 @@ public class ManagementHttpClientTimeoutTests
    [InlineData("-5")]
    [InlineData("not-a-number")]
    [InlineData("")]
-    public void InvalidOrNonPositiveEnvValue_FallsBackToDefault(string value)
+    public void InvalidOrNonPositiveEnvValue_FallsBackToDefaultConnectTimeout(string value)
    {
        var original = Environment.GetEnvironmentVariable(EnvVar);
        try
        {
            Environment.SetEnvironmentVariable(EnvVar, value);

-            using var client = new ManagementHttpClient("http://localhost:9001", "user", "pass");
-
-            Assert.Equal(TimeSpan.FromSeconds(30), client.EffectiveTimeout);
+            Assert.Equal(TimeSpan.FromSeconds(30), ManagementHttpClient.ResolveConnectTimeout());
        }
        finally
        {
@@ -162,20 +171,79 @@ public class ManagementHttpClientTimeoutTests
    }

    [Fact]
-    public void PositiveEnvValue_OverridesDefaultTimeout()
+    public void PositiveEnvValue_OverridesDefaultConnectTimeout()
    {
        var original = Environment.GetEnvironmentVariable(EnvVar);
        try
        {
            Environment.SetEnvironmentVariable(EnvVar, "5");

-            using var client = new ManagementHttpClient("http://localhost:9001", "user", "pass");
-
-            Assert.Equal(TimeSpan.FromSeconds(5), client.EffectiveTimeout);
+            Assert.Equal(TimeSpan.FromSeconds(5), ManagementHttpClient.ResolveConnectTimeout());
        }
        finally
        {
            Environment.SetEnvironmentVariable(EnvVar, original);
        }
    }
+
+    /// <summary>
+    /// The regression that matters: a per-call timeout LONGER than the old 30 s
+    /// client cap must actually be honored. Two calls against the same hanging
+    /// local listener — one with a short deadline, one with a longer one — must
+    /// time out in that order and at their own deadlines, which is only possible
+    /// if <see cref="HttpClient.Timeout"/> is not silently capping both. Uses a
+    /// real socket (not the stub handler) so the connect + send path is exercised
+    /// end to end, and sub-second deadlines so the test stays fast.
+    /// </summary>
+    [Fact]
+    public async Task PerCallTimeoutLongerThanTheOldClientCap_IsHonored()
+    {
+        // A listener that accepts connections and then never answers: every
+        // request hangs until the caller's own deadline fires.
+        var listener = new TcpListener(IPAddress.Loopback, 0);
+        listener.Start();
+        var port = ((IPEndPoint)listener.LocalEndpoint).Port;
+        var accepted = new List<TcpClient>();
+        var acceptLoop = Task.Run(async () =>
+        {
+            try
+            {
+                while (true)
+                    accepted.Add(await listener.AcceptTcpClientAsync());
+            }
+            catch (ObjectDisposedException) { /* listener stopped — expected */ }
+            catch (SocketException) { /* listener stopped — expected */ }
+        });
+
+        try
+        {
+            using var client = new ManagementHttpClient($"http://127.0.0.1:{port}", "user", "pass");
+
+            var shortSw = Stopwatch.StartNew();
+            var shortResponse = await client.SendCommandAsync("ListSites", new { }, TimeSpan.FromMilliseconds(300));
+            shortSw.Stop();
+
+            var longSw = Stopwatch.StartNew();
+            var longResponse = await client.SendCommandAsync("ListSites", new { }, TimeSpan.FromMilliseconds(1500));
+            longSw.Stop();
+
+            Assert.Equal("TIMEOUT", shortResponse.ErrorCode);
+            Assert.Equal("TIMEOUT", longResponse.ErrorCode);
+
+            // The longer deadline must genuinely outlast the shorter one rather
+            // than both being clipped to a single client-wide cap.
+            Assert.True(
+                longSw.Elapsed > TimeSpan.FromMilliseconds(1000),
+                $"1.5 s per-call timeout returned after only {longSw.ElapsedMilliseconds} ms — the client cap truncated it.");
+            Assert.True(
+                shortSw.Elapsed < TimeSpan.FromMilliseconds(1000),
+                $"300 ms per-call timeout took {shortSw.ElapsedMilliseconds} ms.");
+        }
+        finally
+        {
+            listener.Stop();
+            foreach (var c in accepted) c.Dispose();
+            await acceptLoop;
+        }
+    }
 }
@@ -1,5 +1,6 @@
 using System.Text.Json;
 using ZB.MOM.WW.ScadaBridge.CLI.Commands;
+using ZB.MOM.WW.ScadaBridge.Commons.Types.Templates;

 namespace ZB.MOM.WW.ScadaBridge.CLI.Tests;

@@ -7,6 +8,14 @@ namespace ZB.MOM.WW.ScadaBridge.CLI.Tests;
 /// Tests for the compact <c>template list</c>/<c>get</c> table projection (followup #6):
 /// the full per-template attribute/alarm/script dumps are collapsed to counts so table
 /// output is usable in a terminal, while the array/object shape is preserved.
+///
+/// <para>
+/// Two server shapes must both project correctly: <c>template get</c> still returns a
+/// full <c>Template</c> entity with child ARRAYS, while <c>template list</c> returns
+/// database-projected <see cref="TemplateSummary"/> rows carrying pre-computed COUNT
+/// scalars and no arrays at all. Reading only the arrays made every listed template
+/// render as zeros.
+/// </para>
 /// </summary>
 public class TemplateTableProjectionTests
 {
@@ -92,6 +101,72 @@ public class TemplateTableProjectionTests
        Assert.False(root.TryGetProperty("attributes", out _));
    }

+    /// <summary>
+    /// The <c>template list</c> shape. Serialised from the REAL
+    /// <see cref="TemplateSummary"/> record rather than hand-written JSON, so a
+    /// rename of one of its count properties fails this test instead of silently
+    /// putting zeros back on every row.
+    /// </summary>
+    [Fact]
+    public void ProjectSummary_SummaryRows_ReadsCountScalars()
+    {
+        var rows = new[]
+        {
+            new TemplateSummary(
+                Id: 3, Name: "MESReceiver", Description: "base", ParentTemplateId: null,
+                FolderId: null, IsDerived: false, OwnerCompositionId: null,
+                AttributeCount: 3, AlarmCount: 1, ScriptCount: 2,
+                CompositionCount: 0, NativeAlarmSourceCount: 0),
+            new TemplateSummary(
+                Id: 5, Name: "LeftMESReceiver", Description: null, ParentTemplateId: 3,
+                FolderId: 2, IsDerived: false, OwnerCompositionId: null,
+                AttributeCount: 1, AlarmCount: 0, ScriptCount: 0,
+                CompositionCount: 1, NativeAlarmSourceCount: 1),
+        };
+        var json = JsonSerializer.Serialize(rows,
+            new JsonSerializerOptions { PropertyNamingPolicy = JsonNamingPolicy.CamelCase });
+
+        var compact = TemplateTableProjection.ProjectSummary(json);
+
+        using var doc = JsonDocument.Parse(compact);
+        var root = doc.RootElement;
+        Assert.Equal(2, root.GetArrayLength());
+
+        var first = root[0];
+        Assert.Equal(3, first.GetProperty("id").GetInt32());
+        Assert.Equal("MESReceiver", first.GetProperty("name").GetString());
+        Assert.Equal(3, first.GetProperty("#attrs").GetInt32());
+        Assert.Equal(1, first.GetProperty("#alarms").GetInt32());
+        Assert.Equal(2, first.GetProperty("#scripts").GetInt32());
+        Assert.Equal(0, first.GetProperty("#comps").GetInt32());
+        Assert.Equal(0, first.GetProperty("#nativeAlarms").GetInt32());
+
+        var second = root[1];
+        Assert.Equal(5, second.GetProperty("id").GetInt32());
+        Assert.Equal(3, second.GetProperty("parentTemplateId").GetInt32());
+        Assert.Equal(1, second.GetProperty("#attrs").GetInt32());
+        Assert.Equal(1, second.GetProperty("#comps").GetInt32());
+        Assert.Equal(1, second.GetProperty("#nativeAlarms").GetInt32());
+    }
+
+    /// <summary>
+    /// A count scalar wins over a child array when (hypothetically) both are
+    /// present, so a payload that gains summary fields never regresses to array
+    /// counting.
+    /// </summary>
+    [Fact]
+    public void ProjectSummary_PrefersCountScalarOverArray()
+    {
+        const string bothJson = """
+        { "id": 1, "name": "T", "attributeCount": 42, "attributes": [ {"id":1} ] }
+        """;
+
+        var compact = TemplateTableProjection.ProjectSummary(bothJson);
+
+        using var doc = JsonDocument.Parse(compact);
+        Assert.Equal(42, doc.RootElement.GetProperty("#attrs").GetInt32());
+    }
+
    [Fact]
    public void ProjectSummary_NonJson_ReturnedVerbatim()
    {