fix(runtime): a removed driver's late writes no longer resurrect its resilience/telemetry state (#609)
DriverHostActor.StopChild released the tracker (Invalidate) and evicted the hub synchronously, but Context.Stop is asynchronous: the stopping child's in-flight SubscribeBulk completed through its capability invoker AFTER the release, and its RecordCallStart/RecordSuccess/RecordCallComplete re-created the tracker entry, which DriverResilienceStatusPublisherService then re-emitted every tick forever. Confirmed with a temporary stack capture in the tracker (removed): every post-Release write came from DriverInstanceActor.HandleSubscribeAsync -> CapabilityInvoker.ExecuteAsync; the test failed whenever it landed before the second publisher tick. - DriverResilienceStatusTracker.Retire (Release) drops the id's snapshots and ignores every later Record* for it; one guarded write path re-checks after the upsert so a write racing Retire cannot survive it. Reinstate (called by DriverCapabilityInvokerFactory.Create) lifts it, so the same id re-added in a later deployment records again. - TelemetryLocalHub.Evict now also holds the id (late Health/Resilience dropped, neither cached nor fanned out); ITelemetryLocalHub.Readmit (default no-op), called at the top of SpawnChild, lifts it. - The removal test is now deterministic: an invoker decorator holds the child's subscribe inside the real invoker's bracket across the removal and signals when its late writes are done (positive control) — red 5/5 without the fix, green 20/20 with it. New unit tests for the tracker guard (incl. a Retire race), the factory Release/Create pair, the hub hold/readmit, and a re-add-same-id host test. Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
This commit is contained in:
@@ -3198,6 +3198,10 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers, IWithUnboundedS
|
||||
|
||||
private void SpawnChild(DriverInstanceSpec spec)
|
||||
{
|
||||
// #609: an instance id removed by an earlier deployment (StopChild ⇒ Evict, which also holds back late
|
||||
// items) may come back; lift the hold BEFORE the child exists, since it publishes health as it starts.
|
||||
// The resilience tracker's equivalent is lifted by _invokerFactory.Create below.
|
||||
_telemetryHub?.Readmit(spec.DriverInstanceId);
|
||||
var stub = DriverInstanceActor.ShouldStub(spec.DriverType, _localRoles);
|
||||
IDriver? driver = null;
|
||||
if (!stub)
|
||||
@@ -3388,7 +3392,10 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers, IWithUnboundedS
|
||||
// #609: the instance left the plan (StopChild is only reached via plan.ToStop), so its cached
|
||||
// Health/Resilience snapshots must not replay to new telemetry subscribers. Release the shared
|
||||
// resilience state FIRST: the resilience publisher re-emits the tracker's snapshots every tick, so
|
||||
// evicting the hub alone was undone within one interval.
|
||||
// evicting the hub alone was undone within one interval. Context.Stop is asynchronous, so the child can
|
||||
// still finish an in-flight capability call (observed: its SubscribeBulk) and publish health AFTER these
|
||||
// two calls; both therefore HOLD the id — late writes are dropped — until a later SpawnChild of the same
|
||||
// id lifts it (the tracker via _invokerFactory.Create, the hub via Readmit).
|
||||
_invokerFactory.Release(driverInstanceId);
|
||||
_telemetryHub?.Evict(driverInstanceId);
|
||||
_log.Info("DriverHost {Node}: stopped driver child {Id}", _localNode, driverInstanceId);
|
||||
|
||||
Reference in New Issue
Block a user