fix(deploy+cli): review findings — honest CLI timeouts, watermark-complete staleness, phase-2 staging, lock-safe cancellation

Six adversarial-review findings, each verified against the code first. F1 (HIGH) CLI HttpClient capped every call at min(30s, caller timeout), silently truncating deploy site's 5-minute BulkDeployTimeout and the 5-minute bundle export/preview/import calls — which printed a fake "504 Request timed out" while the server kept working. HttpClient.Timeout is now Timeout.InfiniteTimeSpan (the per-call CTS is the single overall deadline, connect included) with the connect phase bounded separately on SocketsHttpHandler.ConnectTimeout. The env override is renamed to SCADABRIDGE_HTTP_CONNECT_TIMEOUT_SECONDS to match its new meaning. F2 (HIGH) StaleInstanceProbe's process-static memo served stale hashes because nothing bumped the watermark on three paths: (a) BundleImporter commits through the raw DbContext, so no import ever moved the watermark — a second import overwriting the same template could be OMITTED from ImportResult.StaleInstanceIds. It now bumps once per apply ATTEMPT: after the commit, and after the rollback too (the probe runs pre-commit, so a rolled-back attempt leaves memos for state that never landed; bumping on both paths is the simplest correct shape, versus threading transaction awareness through a process-static cache). (b) CollectWatermarkBumps' default: arm silently no-op'd, contradicting its own doc. It now sets unattributed=true — an over-broad bump costs extra work, a missed one produces stale work. (c) DataConnection edits route through SiteRepository.SaveChangesAsync, which had no watermark at all, yet Protocol/Primary+Backup config/ FailoverRetryCount are revision-hash inputs. It now bumps (BumpAll — a connection has no owning template) after a commit that touched one. F3 (MED) CLI TemplateTableProjection read child ARRAYS, but ListTemplates now returns database-projected TemplateSummary rows, so template list printed all zeros. It now prefers the *Count scalars and falls back to array length (template get still returns full entities). --detail help text and README corrected: a listing cannot yield definitions, so --detail renders the raw summary payload and template get --id is the full dump. F4 (MED) DeploySiteAsync staged every PendingDeployment in phase 1 against a 5-min TTL while phase 2 reached them one batch at a time, so tail instances' fetch tokens could expire before their command was sent. Staging moved into phase 2, immediately before each send; prepare keeps its flatten/validate/record work. The staging write is the phase's only repository touch and is serialised behind a 1-permit semaphore, so the non-thread-safe DbContext constraint holds and the sends stay concurrent. F5 (MED, latent) DeploySiteAsync leaked every held operation lock if cancelled — a wedged per-instance semaphore is permanent for the process. Phase 2 no longer throws (cancellation is recorded as a per-instance outcome so phase 3 still runs), and an escape from phase 1 or 3 now unwinds every unfinalised entry: Failed status + lock release. F6 (LOW) ScriptCompileVerdictCache's promotion wrote hot directly, bypassing SegmentCapacity (true ceiling 3x against a documented 2x). Promotion now goes through Store, keeping generational semantics; _hot and _cold are volatile. Tests: CLI 396, DeploymentManager 133, ManagementService 494, TemplateEngine 478, ScriptAnalysis 60, Transport 157, Transport integration 106, ConfigurationDatabase 366 — all green, 0 build warnings. The F2/F4/F5 regression tests were each confirmed to FAIL with their fix reverted.
2026-08-14 23:51:04 -04:00
parent b1de9dfdd4
commit e0e4b24679
16 changed files with 1172 additions and 174 deletions
@@ -1,4 +1,6 @@
+using System.Diagnostics;
 using System.Net;
+using System.Net.Sockets;
 using System.Text;
 using ZB.MOM.WW.ScadaBridge.CLI;

@@ -106,32 +108,41 @@ public class ManagementHttpClientTests
 }

 /// <summary>
-/// WP2.6e (arch-review misc — CLI HttpClient timeout): the public
-/// <see cref="ManagementHttpClient"/> constructor must bound its underlying
-/// <see cref="HttpClient.Timeout"/> explicitly (30 s default) rather than leaving the
-/// framework's 100 s default in place, and must honor the
-/// <c>SCADABRIDGE_HTTP_TIMEOUT_SECONDS</c> override — consistent with how every other
-/// CLI setting is environment-overridable (<see cref="CliConfig"/>). Runs in the shared
-/// "Environment" collection (see <see cref="TestCollections"/>) so it never races another
-/// test mutating process-wide environment variables.
+/// The public <see cref="ManagementHttpClient"/> constructor must leave
+/// <see cref="HttpClient.Timeout"/> INFINITE so the per-call
+/// <see cref="CancellationTokenSource"/> is the single overall deadline — a fixed
+/// client timeout silently truncated every caller with a longer per-call timeout
+/// (<c>deploy site</c>'s 5-minute bulk deploy, the 5-minute <c>bundle</c> calls),
+/// which then printed a fake 504 while the server kept working. The connect phase
+/// is bounded separately on <see cref="SocketsHttpHandler.ConnectTimeout"/>, honoring
+/// the <c>SCADABRIDGE_HTTP_CONNECT_TIMEOUT_SECONDS</c> override — consistent with how
+/// every other CLI setting is environment-overridable (<see cref="CliConfig"/>). Runs
+/// in the shared "Environment" collection (see <see cref="TestCollections"/>) so it
+/// never races another test mutating process-wide environment variables.
 /// </summary>
 [Collection("Environment")]
 public class ManagementHttpClientTimeoutTests
 {
-    private const string EnvVar = "SCADABRIDGE_HTTP_TIMEOUT_SECONDS";
+    private const string EnvVar = "SCADABRIDGE_HTTP_CONNECT_TIMEOUT_SECONDS";

    [Fact]
-    public void DefaultConstructor_SetsThirtySecondTimeout_WhenEnvVarUnset()
+    public void DefaultConstructor_LeavesClientTimeoutInfinite()
+    {
+        using var client = new ManagementHttpClient("http://localhost:9001", "user", "pass");
+
+        Assert.Equal(Timeout.InfiniteTimeSpan, client.EffectiveTimeout);
+    }
+
+    [Fact]
+    public void ConnectTimeout_DefaultsToThirtySeconds_WhenEnvVarUnset()
    {
        var original = Environment.GetEnvironmentVariable(EnvVar);
        try
        {
            Environment.SetEnvironmentVariable(EnvVar, null);

-            using var client = new ManagementHttpClient("http://localhost:9001", "user", "pass");
-
-            Assert.Equal(TimeSpan.FromSeconds(30), ManagementHttpClient.DefaultTimeout);
-            Assert.Equal(TimeSpan.FromSeconds(30), client.EffectiveTimeout);
+            Assert.Equal(TimeSpan.FromSeconds(30), ManagementHttpClient.DefaultConnectTimeout);
+            Assert.Equal(TimeSpan.FromSeconds(30), ManagementHttpClient.ResolveConnectTimeout());
        }
        finally
        {
@@ -144,16 +155,14 @@ public class ManagementHttpClientTimeoutTests
    [InlineData("-5")]
    [InlineData("not-a-number")]
    [InlineData("")]
-    public void InvalidOrNonPositiveEnvValue_FallsBackToDefault(string value)
+    public void InvalidOrNonPositiveEnvValue_FallsBackToDefaultConnectTimeout(string value)
    {
        var original = Environment.GetEnvironmentVariable(EnvVar);
        try
        {
            Environment.SetEnvironmentVariable(EnvVar, value);

-            using var client = new ManagementHttpClient("http://localhost:9001", "user", "pass");
-
-            Assert.Equal(TimeSpan.FromSeconds(30), client.EffectiveTimeout);
+            Assert.Equal(TimeSpan.FromSeconds(30), ManagementHttpClient.ResolveConnectTimeout());
        }
        finally
        {
@@ -162,20 +171,79 @@ public class ManagementHttpClientTimeoutTests
    }

    [Fact]
-    public void PositiveEnvValue_OverridesDefaultTimeout()
+    public void PositiveEnvValue_OverridesDefaultConnectTimeout()
    {
        var original = Environment.GetEnvironmentVariable(EnvVar);
        try
        {
            Environment.SetEnvironmentVariable(EnvVar, "5");

-            using var client = new ManagementHttpClient("http://localhost:9001", "user", "pass");
-
-            Assert.Equal(TimeSpan.FromSeconds(5), client.EffectiveTimeout);
+            Assert.Equal(TimeSpan.FromSeconds(5), ManagementHttpClient.ResolveConnectTimeout());
        }
        finally
        {
            Environment.SetEnvironmentVariable(EnvVar, original);
        }
    }
+
+    /// <summary>
+    /// The regression that matters: a per-call timeout LONGER than the old 30 s
+    /// client cap must actually be honored. Two calls against the same hanging
+    /// local listener — one with a short deadline, one with a longer one — must
+    /// time out in that order and at their own deadlines, which is only possible
+    /// if <see cref="HttpClient.Timeout"/> is not silently capping both. Uses a
+    /// real socket (not the stub handler) so the connect + send path is exercised
+    /// end to end, and sub-second deadlines so the test stays fast.
+    /// </summary>
+    [Fact]
+    public async Task PerCallTimeoutLongerThanTheOldClientCap_IsHonored()
+    {
+        // A listener that accepts connections and then never answers: every
+        // request hangs until the caller's own deadline fires.
+        var listener = new TcpListener(IPAddress.Loopback, 0);
+        listener.Start();
+        var port = ((IPEndPoint)listener.LocalEndpoint).Port;
+        var accepted = new List<TcpClient>();
+        var acceptLoop = Task.Run(async () =>
+        {
+            try
+            {
+                while (true)
+                    accepted.Add(await listener.AcceptTcpClientAsync());
+            }
+            catch (ObjectDisposedException) { /* listener stopped — expected */ }
+            catch (SocketException) { /* listener stopped — expected */ }
+        });
+
+        try
+        {
+            using var client = new ManagementHttpClient($"http://127.0.0.1:{port}", "user", "pass");
+
+            var shortSw = Stopwatch.StartNew();
+            var shortResponse = await client.SendCommandAsync("ListSites", new { }, TimeSpan.FromMilliseconds(300));
+            shortSw.Stop();
+
+            var longSw = Stopwatch.StartNew();
+            var longResponse = await client.SendCommandAsync("ListSites", new { }, TimeSpan.FromMilliseconds(1500));
+            longSw.Stop();
+
+            Assert.Equal("TIMEOUT", shortResponse.ErrorCode);
+            Assert.Equal("TIMEOUT", longResponse.ErrorCode);
+
+            // The longer deadline must genuinely outlast the shorter one rather
+            // than both being clipped to a single client-wide cap.
+            Assert.True(
+                longSw.Elapsed > TimeSpan.FromMilliseconds(1000),
+                $"1.5 s per-call timeout returned after only {longSw.ElapsedMilliseconds} ms — the client cap truncated it.");
+            Assert.True(
+                shortSw.Elapsed < TimeSpan.FromMilliseconds(1000),
+                $"300 ms per-call timeout took {shortSw.ElapsedMilliseconds} ms.");
+        }
+        finally
+        {
+            listener.Stop();
+            foreach (var c in accepted) c.Dispose();
+            await acceptLoop;
+        }
+    }
 }
@@ -1,5 +1,6 @@
 using System.Text.Json;
 using ZB.MOM.WW.ScadaBridge.CLI.Commands;
+using ZB.MOM.WW.ScadaBridge.Commons.Types.Templates;

 namespace ZB.MOM.WW.ScadaBridge.CLI.Tests;

@@ -7,6 +8,14 @@ namespace ZB.MOM.WW.ScadaBridge.CLI.Tests;
 /// Tests for the compact <c>template list</c>/<c>get</c> table projection (followup #6):
 /// the full per-template attribute/alarm/script dumps are collapsed to counts so table
 /// output is usable in a terminal, while the array/object shape is preserved.
+///
+/// <para>
+/// Two server shapes must both project correctly: <c>template get</c> still returns a
+/// full <c>Template</c> entity with child ARRAYS, while <c>template list</c> returns
+/// database-projected <see cref="TemplateSummary"/> rows carrying pre-computed COUNT
+/// scalars and no arrays at all. Reading only the arrays made every listed template
+/// render as zeros.
+/// </para>
 /// </summary>
 public class TemplateTableProjectionTests
 {
@@ -92,6 +101,72 @@ public class TemplateTableProjectionTests
        Assert.False(root.TryGetProperty("attributes", out _));
    }

+    /// <summary>
+    /// The <c>template list</c> shape. Serialised from the REAL
+    /// <see cref="TemplateSummary"/> record rather than hand-written JSON, so a
+    /// rename of one of its count properties fails this test instead of silently
+    /// putting zeros back on every row.
+    /// </summary>
+    [Fact]
+    public void ProjectSummary_SummaryRows_ReadsCountScalars()
+    {
+        var rows = new[]
+        {
+            new TemplateSummary(
+                Id: 3, Name: "MESReceiver", Description: "base", ParentTemplateId: null,
+                FolderId: null, IsDerived: false, OwnerCompositionId: null,
+                AttributeCount: 3, AlarmCount: 1, ScriptCount: 2,
+                CompositionCount: 0, NativeAlarmSourceCount: 0),
+            new TemplateSummary(
+                Id: 5, Name: "LeftMESReceiver", Description: null, ParentTemplateId: 3,
+                FolderId: 2, IsDerived: false, OwnerCompositionId: null,
+                AttributeCount: 1, AlarmCount: 0, ScriptCount: 0,
+                CompositionCount: 1, NativeAlarmSourceCount: 1),
+        };
+        var json = JsonSerializer.Serialize(rows,
+            new JsonSerializerOptions { PropertyNamingPolicy = JsonNamingPolicy.CamelCase });
+
+        var compact = TemplateTableProjection.ProjectSummary(json);
+
+        using var doc = JsonDocument.Parse(compact);
+        var root = doc.RootElement;
+        Assert.Equal(2, root.GetArrayLength());
+
+        var first = root[0];
+        Assert.Equal(3, first.GetProperty("id").GetInt32());
+        Assert.Equal("MESReceiver", first.GetProperty("name").GetString());
+        Assert.Equal(3, first.GetProperty("#attrs").GetInt32());
+        Assert.Equal(1, first.GetProperty("#alarms").GetInt32());
+        Assert.Equal(2, first.GetProperty("#scripts").GetInt32());
+        Assert.Equal(0, first.GetProperty("#comps").GetInt32());
+        Assert.Equal(0, first.GetProperty("#nativeAlarms").GetInt32());
+
+        var second = root[1];
+        Assert.Equal(5, second.GetProperty("id").GetInt32());
+        Assert.Equal(3, second.GetProperty("parentTemplateId").GetInt32());
+        Assert.Equal(1, second.GetProperty("#attrs").GetInt32());
+        Assert.Equal(1, second.GetProperty("#comps").GetInt32());
+        Assert.Equal(1, second.GetProperty("#nativeAlarms").GetInt32());
+    }
+
+    /// <summary>
+    /// A count scalar wins over a child array when (hypothetically) both are
+    /// present, so a payload that gains summary fields never regresses to array
+    /// counting.
+    /// </summary>
+    [Fact]
+    public void ProjectSummary_PrefersCountScalarOverArray()
+    {
+        const string bothJson = """
+        { "id": 1, "name": "T", "attributeCount": 42, "attributes": [ {"id":1} ] }
+        """;
+
+        var compact = TemplateTableProjection.ProjectSummary(bothJson);
+
+        using var doc = JsonDocument.Parse(compact);
+        Assert.Equal(42, doc.RootElement.GetProperty("#attrs").GetInt32());
+    }
+
    [Fact]
    public void ProjectSummary_NonJson_ReturnedVerbatim()
    {
@@ -7,6 +7,7 @@ using ZB.MOM.WW.ScadaBridge.Commons.Entities.Sites;
 using ZB.MOM.WW.ScadaBridge.Commons.Entities.Templates;
 using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums;
 using ZB.MOM.WW.ScadaBridge.Commons.Types.Notifications;
+using ZB.MOM.WW.ScadaBridge.Commons.Types.Templates;
 using ZB.MOM.WW.ScadaBridge.ConfigurationDatabase;
 using ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Repositories;
 using ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Services;
@@ -906,6 +907,71 @@ public class SiteRepositoryTests : IDisposable
    {
        Assert.Throws<ArgumentNullException>(() => new SiteRepository(null!));
    }
+
+    /// <summary>
+    /// A data connection's protocol and primary/backup configuration are flattening
+    /// inputs — <c>FlatteningService</c> packages them into the flattened config's
+    /// <c>Connections</c> map and <c>RevisionHashService</c> folds them into the
+    /// revision hash. Without a watermark bump on save, the process-wide
+    /// <c>StaleInstanceProbe</c> memo and the flatten-session caches keep serving the
+    /// pre-edit hash, so an instance whose deployed config genuinely drifted reads as
+    /// up to date. (The bump is unattributed — a connection has no owning template —
+    /// so both the global and structure counters must move.)
+    /// </summary>
+    [Fact]
+    public async Task SaveChanges_DataConnectionEdit_BumpsWatermark()
+    {
+        var watermark = new TemplateGraphWatermark();
+        var repository = new SiteRepository(_context, watermark);
+
+        var site = new Site("Site1", "S-001");
+        await repository.AddSiteAsync(site);
+        await repository.SaveChangesAsync();
+
+        var afterSiteOnly = (watermark.Global, watermark.StructureVersion);
+
+        var conn = new DataConnection("Conn1", "OpcUa", site.Id);
+        await repository.AddDataConnectionAsync(conn);
+        await repository.SaveChangesAsync();
+
+        Assert.True(watermark.Global > afterSiteOnly.Global, "adding a data connection did not bump the global watermark");
+        Assert.True(watermark.StructureVersion > afterSiteOnly.StructureVersion,
+            "adding a data connection did not bump the structure watermark");
+
+        var afterAdd = (watermark.Global, watermark.StructureVersion);
+
+        conn.Protocol = "MxGateway";
+        await repository.UpdateDataConnectionAsync(conn);
+        await repository.SaveChangesAsync();
+
+        Assert.True(watermark.Global > afterAdd.Global, "editing a data connection did not bump the watermark");
+
+        var afterUpdate = watermark.Global;
+
+        await repository.DeleteDataConnectionAsync(conn.Id);
+        await repository.SaveChangesAsync();
+
+        Assert.True(watermark.Global > afterUpdate, "deleting a data connection did not bump the watermark");
+    }
+
+    /// <summary>
+    /// The bump is scoped to the entity that actually feeds the flattener: a save
+    /// that touches no data connection must leave the watermark alone, or every
+    /// site/area edit would needlessly invalidate the whole flatten cache.
+    /// </summary>
+    [Fact]
+    public async Task SaveChanges_WithoutDataConnectionChange_DoesNotBumpWatermark()
+    {
+        var watermark = new TemplateGraphWatermark();
+        var repository = new SiteRepository(_context, watermark);
+
+        var site = new Site("Site1", "S-001");
+        await repository.AddSiteAsync(site);
+        await repository.SaveChangesAsync();
+
+        Assert.Equal(0, watermark.Global);
+        Assert.Equal(0, watermark.StructureVersion);
+    }
 }

 public class DeploymentManagerRepositoryTests : IDisposable
@@ -1,3 +1,5 @@
+using System.Collections.Concurrent;
+using System.Diagnostics;
 using Akka.Actor;
 using Akka.TestKit.Xunit2;
 using Microsoft.Extensions.Logging.Abstractions;
@@ -209,6 +211,124 @@ public class DeploySiteAsyncTests : TestKit
        Assert.Contains("not found", result.Error);
    }

+    /// <summary>
+    /// A <c>PendingDeployment</c>'s fetch token expires
+    /// <c>PendingDeploymentTtl</c> after the row is STAGED, so any delay between
+    /// staging and sending is dead time burned off the token's life. Staging the
+    /// whole batch up front in phase 1 made that delay grow with batch size — the
+    /// tail instances' tokens could expire before their command was ever sent, and
+    /// the site's fetch then 404s.
+    ///
+    /// <para>
+    /// This pins the fix: every instance's <c>RefreshDeploymentCommand</c> must
+    /// arrive at the site carrying a FRESH <c>Timestamp</c> (the staging instant),
+    /// regardless of how long the batch ahead of it took. Run serially with a slow
+    /// site so the batch takes far longer than the freshness bound being asserted —
+    /// under the old shape the last instance's token would already be ~2 s old.
+    /// </para>
+    /// </summary>
+    [Fact]
+    public async Task DeploySiteAsync_StagesEachTokenImmediatelyBeforeItsOwnSend()
+    {
+        const int instanceCount = 20;
+        var perSendDelay = TimeSpan.FromMilliseconds(100);
+        ArrangeInstances(instanceCount);
+
+        var ages = new ConcurrentBag<TimeSpan>();
+        var commActor = Sys.ActorOf(Props.Create(() => new TokenAgeRecordingSiteActor(ages, perSendDelay)));
+
+        // Parallelism 1 makes the batch strictly serial, so the accumulated lag a
+        // front-loaded staging phase would produce is at its largest.
+        var service = CreateService(commActor, maxParallelism: 1);
+
+        var sw = Stopwatch.StartNew();
+        var result = await service.DeploySiteAsync(SiteId, "admin");
+        sw.Stop();
+
+        Assert.True(result.IsSuccess);
+        Assert.Equal(instanceCount, result.Value.SuccessCount);
+        Assert.Equal(instanceCount, ages.Count);
+
+        // The batch really did take a long time — otherwise the freshness assertion
+        // below would pass vacuously.
+        Assert.True(sw.Elapsed > TimeSpan.FromMilliseconds(1500),
+            $"the batch completed in {sw.ElapsedMilliseconds} ms; too fast to prove token freshness");
+
+        // ...yet no token was stale when its command reached the site.
+        var oldest = ages.Max();
+        Assert.True(oldest < TimeSpan.FromMilliseconds(500),
+            $"a fetch token was already {oldest.TotalMilliseconds:F0} ms old on arrival — " +
+            "staging is not tracking the send.");
+    }
+
+    /// <summary>
+    /// Cancelling mid-batch must not leak operation locks. Phase 1's
+    /// <c>ThrowIfCancellationRequested</c> escapes <c>DeploySiteAsync</c> while every
+    /// already-prepared deployment still holds its per-instance lock — and
+    /// <c>OperationLockManager</c> hands out real semaphores, so an undisposed handle
+    /// wedges that instance against every future mutating command for the life of
+    /// the process. Their records must also be finalised as Failed rather than left
+    /// InProgress.
+    /// </summary>
+    [Fact]
+    public async Task DeploySiteAsync_CancelledMidPrepare_ReleasesEveryLock_AndFailsPreparedRecords()
+    {
+        ArrangeInstances(6);
+
+        using var cts = new CancellationTokenSource();
+
+        // Cancel while preparing the third instance: the first two are fully
+        // prepared and holding locks when the loop's next cancellation check throws.
+        _pipeline
+            .FlattenAndValidateAsync(3, Arg.Any<CancellationToken>(), Arg.Any<bool>(), Arg.Any<FlattenSession?>())
+            .Returns(_ =>
+            {
+                cts.Cancel();
+                var config = new FlattenedConfiguration { InstanceUniqueName = "Inst-03" };
+                return Result<FlatteningPipelineResult>.Success(
+                    new FlatteningPipelineResult(config, "sha256:3", ValidationResult.Success()));
+            });
+
+        var commActor = Sys.ActorOf(Props.Create(() =>
+            new ThrottledSiteActor(new ConcurrencyTracker(), slowInstanceName: null, slowDelay: TimeSpan.Zero)));
+
+        var service = CreateService(commActor, maxParallelism: 2);
+
+        await Assert.ThrowsAnyAsync<OperationCanceledException>(
+            () => service.DeploySiteAsync(SiteId, "admin", cts.Token));
+
+        Assert.Equal(0, _lockManager.TrackedLockCount);
+
+        // No prepared deployment may be left InProgress.
+        await _repo.Received().UpdateDeploymentRecordAsync(
+            Arg.Is<DeploymentRecord>(r => r.Status == DeploymentStatus.Failed),
+            Arg.Any<CancellationToken>());
+    }
+
+    /// <summary>
+    /// Answers every <c>RefreshDeploymentCommand</c> after a fixed delay, recording
+    /// how old each command's staging <c>Timestamp</c> already was on arrival.
+    /// </summary>
+    private sealed class TokenAgeRecordingSiteActor : ReceiveActor
+    {
+        public TokenAgeRecordingSiteActor(ConcurrentBag<TimeSpan> ages, TimeSpan delay)
+        {
+            Receive<SiteEnvelope>(env =>
+            {
+                if (env.Message is not RefreshDeploymentCommand cmd)
+                    return;
+
+                ages.Add(DateTimeOffset.UtcNow - cmd.Timestamp);
+
+                var replyTo = Sender;
+                Context.System.Scheduler.Advanced.ScheduleOnce(delay, () =>
+                    replyTo.Tell(new DeploymentStatusResponse(
+                        cmd.DeploymentId, cmd.InstanceUniqueName,
+                        DeploymentStatus.Success, null, DateTimeOffset.UtcNow)));
+            });
+        }
+    }
+
    /// <summary>Records the peak number of simultaneously in-flight site round-trips.</summary>
    private sealed class ConcurrencyTracker
    {
@@ -0,0 +1,99 @@
+using NSubstitute;
+using ZB.MOM.WW.ScadaBridge.Commons.Types;
+using ZB.MOM.WW.ScadaBridge.Commons.Types.Flattening;
+using ZB.MOM.WW.ScadaBridge.Commons.Types.Templates;
+using ZB.MOM.WW.ScadaBridge.TemplateEngine.Flattening;
+
+namespace ZB.MOM.WW.ScadaBridge.DeploymentManager.Tests;
+
+/// <summary>
+/// The staleness fast path: <see cref="StaleInstanceProbe"/> memoises a computed
+/// revision hash against the <see cref="ITemplateGraphWatermark"/> readings it was
+/// computed under, and the memo is PROCESS-STATIC. That makes the watermark the
+/// only thing standing between a cached hash and a wrong answer — so these tests
+/// pin that the memo is served only while the watermark has not moved, and is
+/// dropped the moment it has.
+///
+/// <para>
+/// The failure this guards against is a real one: a data-connection edit, or a
+/// bundle import committing through the raw <c>DbContext</c>, changes the flattened
+/// output without going through the repository path that derives bumps from the
+/// change tracker. If that write does not bump, the memo below keeps answering with
+/// the old hash forever.
+/// </para>
+/// </summary>
+public class StaleInstanceProbeTests
+{
+    private const int InstanceId = 1;
+
+    private readonly TemplateGraphWatermark _watermark = new();
+    private readonly IFlatteningPipeline _pipeline = Substitute.For<IFlatteningPipeline>();
+    private string _currentHash = "sha256:first";
+    private int _flattenCount;
+
+    public StaleInstanceProbeTests()
+    {
+        // Process-static memo: start every test from a clean slate.
+        StaleInstanceProbe.ClearMemos();
+
+        _pipeline.CreateSession().Returns(_ => new FlattenSession(_watermark));
+        _pipeline
+            .FlattenAndValidateAsync(InstanceId, Arg.Any<CancellationToken>(), Arg.Any<bool>(), Arg.Any<FlattenSession?>())
+            .Returns(_ =>
+            {
+                _flattenCount++;
+                var config = new FlattenedConfiguration { InstanceUniqueName = "Inst-01" };
+                return Result<FlatteningPipelineResult>.Success(
+                    new FlatteningPipelineResult(config, _currentHash, ValidationResult.Success()));
+            });
+    }
+
+    private StaleInstanceProbe CreateProbe() => new(_pipeline, _watermark);
+
+    [Fact]
+    public async Task UnchangedWatermark_ServesMemoisedHash_WithoutReflattening()
+    {
+        var probe = CreateProbe();
+
+        Assert.Equal("sha256:first", await probe.GetCurrentRevisionHashAsync(InstanceId));
+        Assert.Equal(1, _flattenCount);
+
+        Assert.Equal("sha256:first", await probe.GetCurrentRevisionHashAsync(InstanceId));
+        Assert.Equal(1, _flattenCount);
+    }
+
+    /// <summary>
+    /// <c>BumpAll</c> is the unattributed fallback used by everything that changes a
+    /// flattening input without an owning template id — data-connection saves and
+    /// bundle imports both rely on it. It must invalidate the memo.
+    /// </summary>
+    [Fact]
+    public async Task BumpAll_InvalidatesTheMemo()
+    {
+        var probe = CreateProbe();
+        await probe.GetCurrentRevisionHashAsync(InstanceId);
+
+        // The underlying config drifted (e.g. a data connection was repointed).
+        _currentHash = "sha256:second";
+
+        // Without a bump the stale hash would still be served...
+        Assert.Equal("sha256:first", await probe.GetCurrentRevisionHashAsync(InstanceId));
+
+        _watermark.BumpAll();
+
+        Assert.Equal("sha256:second", await probe.GetCurrentRevisionHashAsync(InstanceId));
+        Assert.Equal(2, _flattenCount);
+    }
+
+    [Fact]
+    public async Task BumpInstance_InvalidatesTheMemoForThatInstance()
+    {
+        var probe = CreateProbe();
+        await probe.GetCurrentRevisionHashAsync(InstanceId);
+
+        _currentHash = "sha256:second";
+        _watermark.BumpInstance(InstanceId);
+
+        Assert.Equal("sha256:second", await probe.GetCurrentRevisionHashAsync(InstanceId));
+    }
+}
@@ -105,6 +105,36 @@ public class ScriptCompileVerdictCacheEvictionTests
            $"cache grew to {ScriptCompileVerdictCache.Count} entries, exceeding its two-generation bound");
    }

+    /// <summary>
+    /// The bound must hold under a PROMOTION-heavy workload too, not just a
+    /// pure-insert one. Promotion used to write straight into hot
+    /// (<c>_hot[key] = verdict</c>) with no capacity check, so re-reading a working
+    /// set bigger than one segment pulled the whole cold generation back into hot
+    /// on top of what hot already held — hot alone reached 2 × SegmentCapacity and
+    /// the total 3 ×, against a documented 2 ×. Pure-insert overflow never hits
+    /// that path because every key is distinct.
+    /// </summary>
+    [Fact]
+    public void PromotionOverflow_KeepsCacheBounded()
+    {
+        ScriptCompileVerdictCache.Clear();
+
+        // Fill past one rotation so a large working set is sitting in cold with
+        // hot already partly full.
+        const int WorkingSet = 3000;
+        for (var i = 0; i < WorkingSet; i++)
+            Lookup($"// promo {i}", static () => (true, null));
+
+        Assert.True(ScriptCompileVerdictCache.Evictions > 0);
+
+        // Re-read the whole working set: every entry still in cold promotes.
+        for (var i = 0; i < WorkingSet; i++)
+            Lookup($"// promo {i}", static () => (true, null));
+
+        Assert.True(ScriptCompileVerdictCache.Count <= 4096,
+            $"cache grew to {ScriptCompileVerdictCache.Count} entries under promotion, exceeding its two-generation bound");
+    }
+
    [Fact]
    public void SurfaceIsPartOfTheKey_AcrossEviction()
    {
@@ -1335,6 +1335,122 @@ public sealed class BundleImporterApplyTests : IDisposable
        Assert.DoesNotContain(notDeployedInstanceId, result.StaleInstanceIds);
    }

+    /// <summary>
+    /// Two consecutive imports overwriting the SAME template must both report the
+    /// deployed instance as stale.
+    ///
+    /// <para>
+    /// <c>StaleInstanceProbe</c> memoises its computed revision hash against
+    /// <c>ITemplateGraphWatermark</c> readings, in a PROCESS-STATIC dictionary. The
+    /// importer commits through the raw <c>ScadaBridgeDbContext</c>, bypassing
+    /// <c>TemplateEngineRepository.SaveChangesAsync</c> — the only place that
+    /// otherwise derives watermark bumps from the change tracker — so before the
+    /// fix nothing an import wrote ever moved the watermark. The second import then
+    /// re-served the FIRST import's hash and the instance silently dropped out of
+    /// <c>ImportResult.StaleInstanceIds</c>: the operator is told nothing needs
+    /// redeploying while the site runs a config that has drifted twice.
+    /// </para>
+    ///
+    /// <para>
+    /// The snapshot is refreshed between the two imports (simulating the redeploy
+    /// the first stale report prompts), so the second import's staleness verdict
+    /// depends entirely on a FRESH hash rather than on the leftover drift.
+    /// </para>
+    /// </summary>
+    [Fact]
+    public async Task ApplyAsync_second_overwrite_of_same_template_still_marks_instance_stale()
+    {
+        // The memo is process-static and keyed by instance id alone; start clean so
+        // this test does not inherit another class's entry.
+        StaleInstanceProbe.ClearMemos();
+
+        await SeedSiteAsync();
+
+        // Shape A → bundle A.
+        await using (var scope = _provider.CreateAsyncScope())
+        {
+            var ctx = scope.ServiceProvider.GetRequiredService<ScadaBridgeDbContext>();
+            var t = new Template("Pump") { Description = "shape-a" };
+            t.Attributes.Add(new TemplateAttribute("Flow") { DataType = DataType.Float, Value = "1.0" });
+            ctx.Templates.Add(t);
+            await ctx.SaveChangesAsync();
+        }
+        var sessionA = await ExportAndLoadAsync();
+
+        // Shape B → bundle B (a DIFFERENT flattened output from shape A).
+        await using (var scope = _provider.CreateAsyncScope())
+        {
+            var ctx = scope.ServiceProvider.GetRequiredService<ScadaBridgeDbContext>();
+            var t = await ctx.Templates.Include(x => x.Attributes).SingleAsync(x => x.Name == "Pump");
+            t.Description = "shape-b";
+            t.Attributes.Single(a => a.Name == "Flow").Value = "2.0";
+            await ctx.SaveChangesAsync();
+        }
+        var sessionB = await ExportAndLoadAsync();
+
+        // Target starts on a third shape, and the deployed snapshot captures it.
+        await using (var scope = _provider.CreateAsyncScope())
+        {
+            var ctx = scope.ServiceProvider.GetRequiredService<ScadaBridgeDbContext>();
+            var t = await ctx.Templates.Include(x => x.Attributes).SingleAsync(x => x.Name == "Pump");
+            t.Description = "shape-target";
+            t.Attributes.Single(a => a.Name == "Flow").Value = "0.0";
+            await ctx.SaveChangesAsync();
+        }
+        var instanceId = await SeedDeployedInstanceWithRealSnapshotAsync("Pump", "Pump-Deployed");
+
+        // Import #1 — Overwrite to shape A. Drifts the instance off its snapshot.
+        ImportResult first;
+        await using (var scope = _provider.CreateAsyncScope())
+        {
+            var importer = scope.ServiceProvider.GetRequiredService<IBundleImporter>();
+            first = await importer.ApplyAsync(sessionA,
+                new List<ImportResolution> { new("Template", "Pump", ResolutionAction.Overwrite, null) },
+                user: "bob");
+        }
+        Assert.Contains(instanceId, first.StaleInstanceIds);
+
+        // The operator redeploys: the snapshot now matches the post-import-#1 config,
+        // so the instance is genuinely up to date going into import #2.
+        await RefreshDeployedSnapshotAsync(instanceId);
+
+        // Import #2 — Overwrite the SAME template to shape B. The instance drifts
+        // again and must be reported again.
+        ImportResult second;
+        await using (var scope = _provider.CreateAsyncScope())
+        {
+            var importer = scope.ServiceProvider.GetRequiredService<IBundleImporter>();
+            second = await importer.ApplyAsync(sessionB,
+                new List<ImportResolution> { new("Template", "Pump", ResolutionAction.Overwrite, null) },
+                user: "bob");
+        }
+
+        Assert.Equal(1, second.Overwritten);
+        Assert.Contains(instanceId, second.StaleInstanceIds);
+    }
+
+    /// <summary>
+    /// Re-points the instance's <see cref="DeployedConfigSnapshot"/> at its CURRENT
+    /// flattened revision hash — the state a successful redeploy would leave behind.
+    /// </summary>
+    /// <param name="instanceId">Instance whose snapshot is refreshed.</param>
+    /// <returns>A task that completes once the snapshot is updated.</returns>
+    private async Task RefreshDeployedSnapshotAsync(int instanceId)
+    {
+        await using var scope = _provider.CreateAsyncScope();
+        var ctx = scope.ServiceProvider.GetRequiredService<ScadaBridgeDbContext>();
+        var pipeline = scope.ServiceProvider.GetRequiredService<IFlatteningPipeline>();
+
+        var flattened = await pipeline.FlattenAndValidateAsync(instanceId);
+        Assert.True(flattened.IsSuccess,
+            $"Snapshot refresh flatten failed: {(flattened.IsFailure ? flattened.Error : "(success)")}");
+
+        var snapshot = await ctx.DeployedConfigSnapshots.SingleAsync(s => s.InstanceId == instanceId);
+        snapshot.RevisionHash = flattened.Value.RevisionHash;
+        snapshot.ConfigurationJson = System.Text.Json.JsonSerializer.Serialize(flattened.Value.Configuration);
+        await ctx.SaveChangesAsync();
+    }
+
    // ============ #05-T14: post-commit ScriptArtifactsChanged publish ============

    [Fact]