fix(runtime): a removed driver's late writes no longer resurrect its resilience/telemetry state (#609)
DriverHostActor.StopChild released the tracker (Invalidate) and evicted the hub synchronously, but Context.Stop is asynchronous: the stopping child's in-flight SubscribeBulk completed through its capability invoker AFTER the release, and its RecordCallStart/RecordSuccess/RecordCallComplete re-created the tracker entry, which DriverResilienceStatusPublisherService then re-emitted every tick forever. Confirmed with a temporary stack capture in the tracker (removed): every post-Release write came from DriverInstanceActor.HandleSubscribeAsync -> CapabilityInvoker.ExecuteAsync; the test failed whenever it landed before the second publisher tick. - DriverResilienceStatusTracker.Retire (Release) drops the id's snapshots and ignores every later Record* for it; one guarded write path re-checks after the upsert so a write racing Retire cannot survive it. Reinstate (called by DriverCapabilityInvokerFactory.Create) lifts it, so the same id re-added in a later deployment records again. - TelemetryLocalHub.Evict now also holds the id (late Health/Resilience dropped, neither cached nor fanned out); ITelemetryLocalHub.Readmit (default no-op), called at the top of SpawnChild, lifts it. - The removal test is now deterministic: an invoker decorator holds the child's subscribe inside the real invoker's bracket across the removal and signals when its late writes are done (positive control) — red 5/5 without the fix, green 20/20 with it. New unit tests for the tracker guard (incl. a Retire race), the factory Release/Create pair, the hub hold/readmit, and a re-add-same-id host test. Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
This commit is contained in:
@@ -58,6 +58,55 @@ public sealed class TelemetryLocalHubTests
|
||||
items.OfType<TelemetryItem.Resilience>().Single().E.DriverInstanceId.ShouldBe("drv-b");
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// #609 — the host evicts a removed instance while its child is still stopping, and that child can
|
||||
/// still publish (e.g. a health snapshot when an in-flight connect/subscribe completes). A snapshot for
|
||||
/// an evicted instance must not re-prime the cache — nor reach live subscribers, where it would
|
||||
/// re-populate central's view of a driver that no longer exists. A neighbour emitted at the same time is
|
||||
/// the positive control.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public void A_late_snapshot_for_an_evicted_instance_is_neither_cached_nor_delivered()
|
||||
{
|
||||
var hub = new TelemetryLocalHub();
|
||||
hub.Emit(new TelemetryItem.Health(Health("drv-a")));
|
||||
hub.Evict("drv-a");
|
||||
using var live = hub.Subscribe(boundedCapacity: 16);
|
||||
|
||||
hub.Emit(new TelemetryItem.Health(Health("drv-a")));
|
||||
hub.Emit(new TelemetryItem.Resilience(Resilience("drv-a", "host-1")));
|
||||
hub.Emit(new TelemetryItem.Health(Health("drv-b")));
|
||||
|
||||
Drain(live).Select(Id).ShouldBe(new[] { "drv-b" });
|
||||
using var primed = hub.Subscribe(boundedCapacity: 16);
|
||||
Drain(primed).Select(Id).ShouldBe(new[] { "drv-b" });
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// #609 — eviction does not outlive the removal: the same instance id re-spawned in a later deployment
|
||||
/// is <c>Readmit</c>ted and its snapshots are cached again.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public void Readmit_lets_an_evicted_instance_publish_again()
|
||||
{
|
||||
var hub = new TelemetryLocalHub();
|
||||
hub.Evict("drv-a");
|
||||
hub.Readmit("drv-a");
|
||||
|
||||
hub.Emit(new TelemetryItem.Health(Health("drv-a")));
|
||||
hub.Emit(new TelemetryItem.Resilience(Resilience("drv-a", "host-1")));
|
||||
|
||||
using var sub = hub.Subscribe(boundedCapacity: 16);
|
||||
Drain(sub).Select(Id).ShouldBe(new[] { "drv-a", "drv-a" });
|
||||
}
|
||||
|
||||
private static string Id(TelemetryItem item) => item switch
|
||||
{
|
||||
TelemetryItem.Health h => h.E.DriverInstanceId,
|
||||
TelemetryItem.Resilience r => r.E.DriverInstanceId,
|
||||
_ => "<other>",
|
||||
};
|
||||
|
||||
[Fact]
|
||||
public void New_subscriber_is_primed_with_cached_Health_snapshot()
|
||||
{
|
||||
|
||||
Reference in New Issue
Block a user