fix(runtime): a removed driver's late writes no longer resurrect its resilience/telemetry state (#609)

DriverHostActor.StopChild released the tracker (Invalidate) and evicted the hub synchronously, but
Context.Stop is asynchronous: the stopping child's in-flight SubscribeBulk completed through its
capability invoker AFTER the release, and its RecordCallStart/RecordSuccess/RecordCallComplete
re-created the tracker entry, which DriverResilienceStatusPublisherService then re-emitted every tick
forever. Confirmed with a temporary stack capture in the tracker (removed): every post-Release write
came from DriverInstanceActor.HandleSubscribeAsync -> CapabilityInvoker.ExecuteAsync; the test failed
whenever it landed before the second publisher tick.

- DriverResilienceStatusTracker.Retire (Release) drops the id's snapshots and ignores every later
  Record* for it; one guarded write path re-checks after the upsert so a write racing Retire cannot
  survive it. Reinstate (called by DriverCapabilityInvokerFactory.Create) lifts it, so the same id
  re-added in a later deployment records again.
- TelemetryLocalHub.Evict now also holds the id (late Health/Resilience dropped, neither cached nor
  fanned out); ITelemetryLocalHub.Readmit (default no-op), called at the top of SpawnChild, lifts it.
- The removal test is now deterministic: an invoker decorator holds the child's subscribe inside the
  real invoker's bracket across the removal and signals when its late writes are done (positive
  control) — red 5/5 without the fix, green 20/20 with it. New unit tests for the tracker guard
  (incl. a Retire race), the factory Release/Create pair, the hub hold/readmit, and a re-add-same-id
  host test.

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
This commit is contained in:
Joseph Doherty
2026-10-08 09:07:11 -04:00
parent 5c0623ad2b
commit 4a10ea1e1c
10 changed files with 418 additions and 42 deletions
+10 -3
View File
@@ -256,10 +256,17 @@ driver node**:
The cache is evicted per instance (`ITelemetryLocalHub.Evict`) when `DriverHostActor.StopChild` removes a
driver from the plan (#609), so a decommissioned instance's last state is not replayed to new subscribers.
`StopChild` first releases the instance's resilience state (`IDriverCapabilityInvokerFactory.Release` →
pipeline-builder + `DriverResilienceStatusTracker.Invalidate`): `DriverResilienceStatusPublisherService`
pipeline-builder + `DriverResilienceStatusTracker.Retire`): `DriverResilienceStatusPublisherService`
re-emits the whole tracker every 5 s, so evicting only the hub was undone within one tick (review R1 M1).
Residual: a Health emit already in flight from the stopping child can land after the eviction and
re-cache that one snapshot — a narrow race, not a recurring replay.
⚠️ **Both releases HOLD the id, not just clear it.** `Context.Stop` is asynchronous, so the stopping child
can still finish a capability call it had in flight — observed: its `SubscribeBulk`, whose
`RecordCallStart`/`RecordSuccess`/`RecordCallComplete` ran after the release and re-created the tracker
entry, which the publisher then re-emitted every tick for the life of the process — and can still publish a
Health snapshot. So the tracker ignores every `Record*` for a retired id, and the hub drops every later
Health/Resilience item for an evicted id (neither cached nor fanned out), until the same id is spawned
again: `IDriverCapabilityInvokerFactory.Create` reinstates it in the tracker and `SpawnChild` calls
`ITelemetryLocalHub.Readmit`. Residual: a late write from the *old* child that lands after a re-add of the
same id is indistinguishable from the new child's and is kept — the same window a respawn always had.
- **Connection indicator** — the `/alerts` and `/script-log` live/disconnected pill is driven by
**aggregate** stream health across all dialers: connected when **at least one** node stream is
up, disconnected when all are down. This matches today's DPS `SubscribeAck`/`PostStop` pill