fix(runtime): a removed driver's late writes no longer resurrect its resilience/telemetry state (#609)

DriverHostActor.StopChild released the tracker (Invalidate) and evicted the hub synchronously, but Context.Stop is asynchronous: the stopping child's in-flight SubscribeBulk completed through its capability invoker AFTER the release, and its RecordCallStart/RecordSuccess/RecordCallComplete re-created the tracker entry, which DriverResilienceStatusPublisherService then re-emitted every tick forever. Confirmed with a temporary stack capture in the tracker (removed): every post-Release write came from DriverInstanceActor.HandleSubscribeAsync -> CapabilityInvoker.ExecuteAsync; the test failed whenever it landed before the second publisher tick. - DriverResilienceStatusTracker.Retire (Release) drops the id's snapshots and ignores every later Record* for it; one guarded write path re-checks after the upsert so a write racing Retire cannot survive it. Reinstate (called by DriverCapabilityInvokerFactory.Create) lifts it, so the same id re-added in a later deployment records again. - TelemetryLocalHub.Evict now also holds the id (late Health/Resilience dropped, neither cached nor fanned out); ITelemetryLocalHub.Readmit (default no-op), called at the top of SpawnChild, lifts it. - The removal test is now deterministic: an invoker decorator holds the child's subscribe inside the real invoker's bracket across the removal and signals when its late writes are done (positive control) — red 5/5 without the fix, green 20/20 with it. New unit tests for the tracker guard (incl. a Retire race), the factory Release/Create pair, the hub hold/readmit, and a re-add-same-id host test. Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-08 09:07:11 -04:00
parent 5c0623ad2b
commit 4a10ea1e1c
10 changed files with 418 additions and 42 deletions
@@ -45,6 +45,47 @@ public sealed class DriverCapabilityInvokerFactoryTests
        tracker.TryGet("d2", "d2")!.IsBreakerOpen.ShouldBeTrue("other instances must be untouched");
    }

+    /// <summary>
+    ///     #609 — a call the removed instance still had in flight finishes through its (now released) invoker
+    ///     AFTER <c>Release</c>; its RecordCallStart/RecordSuccess/RecordCallComplete must not re-create the
+    ///     tracker entry, or the resilience publisher re-emits the removed driver on every tick. The neighbour's
+    ///     call through its own invoker is the positive control — the same path still records.
+    /// </summary>
+    [Fact]
+    public async Task A_call_through_a_released_invoker_records_nothing()
+    {
+        var tracker = new DriverResilienceStatusTracker();
+        var factory = new DriverCapabilityInvokerFactory(
+            new DriverResiliencePipelineBuilder(statusTracker: tracker), tracker, _ => DriverTier.A);
+        var released = factory.Create("gone", "Modbus");
+        var live = factory.Create("live", "Modbus");
+
+        factory.Release("gone");
+        await released.ExecuteAsync(DriverCapability.Subscribe, "gone", _ => ValueTask.FromResult(1), CancellationToken.None);
+        await live.ExecuteAsync(DriverCapability.Subscribe, "live", _ => ValueTask.FromResult(1), CancellationToken.None);
+
+        tracker.Snapshot().Select(e => e.DriverInstanceId).ShouldBe(new[] { "live" });
+    }
+
+    /// <summary>
+    ///     #609 — re-adding the same instance id in a later deployment (a fresh <c>Create</c> after
+    ///     <c>Release</c>) records again: the release must not outlive the removal.
+    /// </summary>
+    [Fact]
+    public async Task Create_after_Release_records_again_for_the_same_instance()
+    {
+        var tracker = new DriverResilienceStatusTracker();
+        var factory = new DriverCapabilityInvokerFactory(
+            new DriverResiliencePipelineBuilder(statusTracker: tracker), tracker, _ => DriverTier.A);
+        factory.Create("d1", "Modbus");
+        factory.Release("d1");
+
+        var readded = factory.Create("d1", "Modbus");
+        await readded.ExecuteAsync(DriverCapability.Subscribe, "d1", _ => ValueTask.FromResult(1), CancellationToken.None);
+
+        tracker.TryGet("d1", "d1").ShouldNotBeNull();
+    }
+
    /// <summary>
    ///     End-to-end proof the per-instance override REACHES execution: a ResilienceConfig that sets
    ///     Read's retryCount to 0 must suppress the tier-A Read retry — a failing Read is attempted once,
@@ -172,4 +172,81 @@ public sealed class DriverResilienceStatusTrackerTests

        tracker.TryGet("drv", "host")!.ConsecutiveFailures.ShouldBe(500);
    }
+
+    /// <summary>
+    ///     #609 — <c>Retire</c> (a removed instance) drops the instance's snapshots AND ignores every write
+    ///     that arrives after it. A removed driver's child stops asynchronously, so a capability call it still
+    ///     had in flight finishes through its invoker after the host released it; each of those writes used to
+    ///     re-create the entry, and the resilience publisher re-emitted the removed driver every tick, forever.
+    ///     The neighbour is the positive control: the same writes against a live instance still land.
+    /// </summary>
+    [Fact]
+    public void Retire_drops_the_instance_and_ignores_every_late_write_for_it()
+    {
+        var tracker = new DriverResilienceStatusTracker();
+        tracker.RecordBreakerOpen("gone", "host-a", Now);
+        tracker.RecordFailure("gone", "host-b", Now);
+
+        tracker.Retire("gone").ShouldBe(2);
+
+        foreach (var id in new[] { "gone", "live" })
+        {
+            tracker.RecordCallStart(id, "host-a");
+            tracker.RecordSuccess(id, "host-a", Now);
+            tracker.RecordCallComplete(id, "host-a");
+            tracker.RecordFailure(id, "host-a", Now);
+            tracker.RecordBreakerOpen(id, "host-a", Now);
+            tracker.RecordBreakerClose(id, "host-a", Now);
+            tracker.RecordRecycle(id, "host-a", Now);
+            tracker.RecordFootprint(id, "host-a", 1, 2, Now);
+        }
+
+        tracker.Snapshot().Select(e => e.DriverInstanceId).ShouldBe(new[] { "live" });
+        tracker.TryGet("gone", "host-a").ShouldBeNull();
+        tracker.TryGet("gone", "host-b").ShouldBeNull();
+    }
+
+    /// <summary>
+    ///     #609 — a retired id is not retired forever: the same instance id re-added in a later deployment is
+    ///     <c>Reinstate</c>d (by the invoker factory's <c>Create</c>) and records again.
+    /// </summary>
+    [Fact]
+    public void Reinstate_lets_a_retired_instance_record_again()
+    {
+        var tracker = new DriverResilienceStatusTracker();
+        tracker.Retire("drv");
+        tracker.RecordFailure("drv", "host", Now);
+        tracker.TryGet("drv", "host").ShouldBeNull(); // control: still retired
+
+        tracker.Reinstate("drv");
+        tracker.RecordFailure("drv", "host", Now);
+
+        tracker.TryGet("drv", "host")!.ConsecutiveFailures.ShouldBe(1);
+    }
+
+    /// <summary>
+    ///     #609 — a write racing <c>Retire</c> never survives it. Writers hammer the instance while it is
+    ///     retired; whatever interleaving occurs, nothing is left once both sides finish.
+    /// </summary>
+    [Fact]
+    public async Task A_write_racing_Retire_never_resurrects_the_instance()
+    {
+        for (var round = 0; round < 50; round++)
+        {
+            var tracker = new DriverResilienceStatusTracker();
+            tracker.RecordFailure("drv", "host", Now);
+            using var go = new ManualResetEventSlim();
+            var writers = Enumerable.Range(0, 4).Select(_ => Task.Run(() =>
+            {
+                go.Wait();
+                for (var i = 0; i < 200; i++) tracker.RecordFailure("drv", "host", Now);
+            }, TestContext.Current.CancellationToken)).ToArray();
+
+            go.Set();
+            tracker.Retire("drv");
+            await Task.WhenAll(writers);
+
+            tracker.TryGet("drv", "host").ShouldBeNull($"round {round}");
+        }
+    }
 }