fix(deploy+cli): review findings — honest CLI timeouts, watermark-complete staleness, phase-2 staging, lock-safe cancellation

Six adversarial-review findings, each verified against the code first. F1 (HIGH) CLI HttpClient capped every call at min(30s, caller timeout), silently truncating deploy site's 5-minute BulkDeployTimeout and the 5-minute bundle export/preview/import calls — which printed a fake "504 Request timed out" while the server kept working. HttpClient.Timeout is now Timeout.InfiniteTimeSpan (the per-call CTS is the single overall deadline, connect included) with the connect phase bounded separately on SocketsHttpHandler.ConnectTimeout. The env override is renamed to SCADABRIDGE_HTTP_CONNECT_TIMEOUT_SECONDS to match its new meaning. F2 (HIGH) StaleInstanceProbe's process-static memo served stale hashes because nothing bumped the watermark on three paths: (a) BundleImporter commits through the raw DbContext, so no import ever moved the watermark — a second import overwriting the same template could be OMITTED from ImportResult.StaleInstanceIds. It now bumps once per apply ATTEMPT: after the commit, and after the rollback too (the probe runs pre-commit, so a rolled-back attempt leaves memos for state that never landed; bumping on both paths is the simplest correct shape, versus threading transaction awareness through a process-static cache). (b) CollectWatermarkBumps' default: arm silently no-op'd, contradicting its own doc. It now sets unattributed=true — an over-broad bump costs extra work, a missed one produces stale work. (c) DataConnection edits route through SiteRepository.SaveChangesAsync, which had no watermark at all, yet Protocol/Primary+Backup config/ FailoverRetryCount are revision-hash inputs. It now bumps (BumpAll — a connection has no owning template) after a commit that touched one. F3 (MED) CLI TemplateTableProjection read child ARRAYS, but ListTemplates now returns database-projected TemplateSummary rows, so template list printed all zeros. It now prefers the *Count scalars and falls back to array length (template get still returns full entities). --detail help text and README corrected: a listing cannot yield definitions, so --detail renders the raw summary payload and template get --id is the full dump. F4 (MED) DeploySiteAsync staged every PendingDeployment in phase 1 against a 5-min TTL while phase 2 reached them one batch at a time, so tail instances' fetch tokens could expire before their command was sent. Staging moved into phase 2, immediately before each send; prepare keeps its flatten/validate/record work. The staging write is the phase's only repository touch and is serialised behind a 1-permit semaphore, so the non-thread-safe DbContext constraint holds and the sends stay concurrent. F5 (MED, latent) DeploySiteAsync leaked every held operation lock if cancelled — a wedged per-instance semaphore is permanent for the process. Phase 2 no longer throws (cancellation is recorded as a per-instance outcome so phase 3 still runs), and an escape from phase 1 or 3 now unwinds every unfinalised entry: Failed status + lock release. F6 (LOW) ScriptCompileVerdictCache's promotion wrote hot directly, bypassing SegmentCapacity (true ceiling 3x against a documented 2x). Promotion now goes through Store, keeping generational semantics; _hot and _cold are volatile. Tests: CLI 396, DeploymentManager 133, ManagementService 494, TemplateEngine 478, ScriptAnalysis 60, Transport 157, Transport integration 106, ConfigurationDatabase 366 — all green, 0 build warnings. The F2/F4/F5 regression tests were each confirmed to FAIL with their fix reverted.
2026-08-14 23:51:04 -04:00
parent b1de9dfdd4
commit e0e4b24679
16 changed files with 1172 additions and 174 deletions
@@ -1335,6 +1335,122 @@ public sealed class BundleImporterApplyTests : IDisposable
        Assert.DoesNotContain(notDeployedInstanceId, result.StaleInstanceIds);
    }

+    /// <summary>
+    /// Two consecutive imports overwriting the SAME template must both report the
+    /// deployed instance as stale.
+    ///
+    /// <para>
+    /// <c>StaleInstanceProbe</c> memoises its computed revision hash against
+    /// <c>ITemplateGraphWatermark</c> readings, in a PROCESS-STATIC dictionary. The
+    /// importer commits through the raw <c>ScadaBridgeDbContext</c>, bypassing
+    /// <c>TemplateEngineRepository.SaveChangesAsync</c> — the only place that
+    /// otherwise derives watermark bumps from the change tracker — so before the
+    /// fix nothing an import wrote ever moved the watermark. The second import then
+    /// re-served the FIRST import's hash and the instance silently dropped out of
+    /// <c>ImportResult.StaleInstanceIds</c>: the operator is told nothing needs
+    /// redeploying while the site runs a config that has drifted twice.
+    /// </para>
+    ///
+    /// <para>
+    /// The snapshot is refreshed between the two imports (simulating the redeploy
+    /// the first stale report prompts), so the second import's staleness verdict
+    /// depends entirely on a FRESH hash rather than on the leftover drift.
+    /// </para>
+    /// </summary>
+    [Fact]
+    public async Task ApplyAsync_second_overwrite_of_same_template_still_marks_instance_stale()
+    {
+        // The memo is process-static and keyed by instance id alone; start clean so
+        // this test does not inherit another class's entry.
+        StaleInstanceProbe.ClearMemos();
+
+        await SeedSiteAsync();
+
+        // Shape A → bundle A.
+        await using (var scope = _provider.CreateAsyncScope())
+        {
+            var ctx = scope.ServiceProvider.GetRequiredService<ScadaBridgeDbContext>();
+            var t = new Template("Pump") { Description = "shape-a" };
+            t.Attributes.Add(new TemplateAttribute("Flow") { DataType = DataType.Float, Value = "1.0" });
+            ctx.Templates.Add(t);
+            await ctx.SaveChangesAsync();
+        }
+        var sessionA = await ExportAndLoadAsync();
+
+        // Shape B → bundle B (a DIFFERENT flattened output from shape A).
+        await using (var scope = _provider.CreateAsyncScope())
+        {
+            var ctx = scope.ServiceProvider.GetRequiredService<ScadaBridgeDbContext>();
+            var t = await ctx.Templates.Include(x => x.Attributes).SingleAsync(x => x.Name == "Pump");
+            t.Description = "shape-b";
+            t.Attributes.Single(a => a.Name == "Flow").Value = "2.0";
+            await ctx.SaveChangesAsync();
+        }
+        var sessionB = await ExportAndLoadAsync();
+
+        // Target starts on a third shape, and the deployed snapshot captures it.
+        await using (var scope = _provider.CreateAsyncScope())
+        {
+            var ctx = scope.ServiceProvider.GetRequiredService<ScadaBridgeDbContext>();
+            var t = await ctx.Templates.Include(x => x.Attributes).SingleAsync(x => x.Name == "Pump");
+            t.Description = "shape-target";
+            t.Attributes.Single(a => a.Name == "Flow").Value = "0.0";
+            await ctx.SaveChangesAsync();
+        }
+        var instanceId = await SeedDeployedInstanceWithRealSnapshotAsync("Pump", "Pump-Deployed");
+
+        // Import #1 — Overwrite to shape A. Drifts the instance off its snapshot.
+        ImportResult first;
+        await using (var scope = _provider.CreateAsyncScope())
+        {
+            var importer = scope.ServiceProvider.GetRequiredService<IBundleImporter>();
+            first = await importer.ApplyAsync(sessionA,
+                new List<ImportResolution> { new("Template", "Pump", ResolutionAction.Overwrite, null) },
+                user: "bob");
+        }
+        Assert.Contains(instanceId, first.StaleInstanceIds);
+
+        // The operator redeploys: the snapshot now matches the post-import-#1 config,
+        // so the instance is genuinely up to date going into import #2.
+        await RefreshDeployedSnapshotAsync(instanceId);
+
+        // Import #2 — Overwrite the SAME template to shape B. The instance drifts
+        // again and must be reported again.
+        ImportResult second;
+        await using (var scope = _provider.CreateAsyncScope())
+        {
+            var importer = scope.ServiceProvider.GetRequiredService<IBundleImporter>();
+            second = await importer.ApplyAsync(sessionB,
+                new List<ImportResolution> { new("Template", "Pump", ResolutionAction.Overwrite, null) },
+                user: "bob");
+        }
+
+        Assert.Equal(1, second.Overwritten);
+        Assert.Contains(instanceId, second.StaleInstanceIds);
+    }
+
+    /// <summary>
+    /// Re-points the instance's <see cref="DeployedConfigSnapshot"/> at its CURRENT
+    /// flattened revision hash — the state a successful redeploy would leave behind.
+    /// </summary>
+    /// <param name="instanceId">Instance whose snapshot is refreshed.</param>
+    /// <returns>A task that completes once the snapshot is updated.</returns>
+    private async Task RefreshDeployedSnapshotAsync(int instanceId)
+    {
+        await using var scope = _provider.CreateAsyncScope();
+        var ctx = scope.ServiceProvider.GetRequiredService<ScadaBridgeDbContext>();
+        var pipeline = scope.ServiceProvider.GetRequiredService<IFlatteningPipeline>();
+
+        var flattened = await pipeline.FlattenAndValidateAsync(instanceId);
+        Assert.True(flattened.IsSuccess,
+            $"Snapshot refresh flatten failed: {(flattened.IsFailure ? flattened.Error : "(success)")}");
+
+        var snapshot = await ctx.DeployedConfigSnapshots.SingleAsync(s => s.InstanceId == instanceId);
+        snapshot.RevisionHash = flattened.Value.RevisionHash;
+        snapshot.ConfigurationJson = System.Text.Json.JsonSerializer.Serialize(flattened.Value.Configuration);
+        await ctx.SaveChangesAsync();
+    }
+
    // ============ #05-T14: post-commit ScriptArtifactsChanged publish ============

    [Fact]