fix(runtime): a removed driver's late writes no longer resurrect its resilience/telemetry state (#609)
DriverHostActor.StopChild released the tracker (Invalidate) and evicted the hub synchronously, but Context.Stop is asynchronous: the stopping child's in-flight SubscribeBulk completed through its capability invoker AFTER the release, and its RecordCallStart/RecordSuccess/RecordCallComplete re-created the tracker entry, which DriverResilienceStatusPublisherService then re-emitted every tick forever. Confirmed with a temporary stack capture in the tracker (removed): every post-Release write came from DriverInstanceActor.HandleSubscribeAsync -> CapabilityInvoker.ExecuteAsync; the test failed whenever it landed before the second publisher tick. - DriverResilienceStatusTracker.Retire (Release) drops the id's snapshots and ignores every later Record* for it; one guarded write path re-checks after the upsert so a write racing Retire cannot survive it. Reinstate (called by DriverCapabilityInvokerFactory.Create) lifts it, so the same id re-added in a later deployment records again. - TelemetryLocalHub.Evict now also holds the id (late Health/Resilience dropped, neither cached nor fanned out); ITelemetryLocalHub.Readmit (default no-op), called at the top of SpawnChild, lifts it. - The removal test is now deterministic: an invoker decorator holds the child's subscribe inside the real invoker's bracket across the removal and signals when its late writes are done (positive control) — red 5/5 without the fix, green 20/20 with it. New unit tests for the tracker guard (incl. a Retire race), the factory Release/Create pair, the hub hold/readmit, and a re-add-same-id host test. Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
This commit is contained in:
+10
-3
@@ -256,10 +256,17 @@ driver node**:
|
||||
The cache is evicted per instance (`ITelemetryLocalHub.Evict`) when `DriverHostActor.StopChild` removes a
|
||||
driver from the plan (#609), so a decommissioned instance's last state is not replayed to new subscribers.
|
||||
`StopChild` first releases the instance's resilience state (`IDriverCapabilityInvokerFactory.Release` →
|
||||
pipeline-builder + `DriverResilienceStatusTracker.Invalidate`): `DriverResilienceStatusPublisherService`
|
||||
pipeline-builder + `DriverResilienceStatusTracker.Retire`): `DriverResilienceStatusPublisherService`
|
||||
re-emits the whole tracker every 5 s, so evicting only the hub was undone within one tick (review R1 M1).
|
||||
Residual: a Health emit already in flight from the stopping child can land after the eviction and
|
||||
re-cache that one snapshot — a narrow race, not a recurring replay.
|
||||
⚠️ **Both releases HOLD the id, not just clear it.** `Context.Stop` is asynchronous, so the stopping child
|
||||
can still finish a capability call it had in flight — observed: its `SubscribeBulk`, whose
|
||||
`RecordCallStart`/`RecordSuccess`/`RecordCallComplete` ran after the release and re-created the tracker
|
||||
entry, which the publisher then re-emitted every tick for the life of the process — and can still publish a
|
||||
Health snapshot. So the tracker ignores every `Record*` for a retired id, and the hub drops every later
|
||||
Health/Resilience item for an evicted id (neither cached nor fanned out), until the same id is spawned
|
||||
again: `IDriverCapabilityInvokerFactory.Create` reinstates it in the tracker and `SpawnChild` calls
|
||||
`ITelemetryLocalHub.Readmit`. Residual: a late write from the *old* child that lands after a re-add of the
|
||||
same id is indistinguishable from the new child's and is kept — the same window a respawn always had.
|
||||
- **Connection indicator** — the `/alerts` and `/script-log` live/disconnected pill is driven by
|
||||
**aggregate** stream health across all dialers: connected when **at least one** node stream is
|
||||
up, disconnected when all are down. This matches today's DPS `SubscribeAck`/`PostStop` pill
|
||||
|
||||
Reference in New Issue
Block a user