fix(runtime): evict a stopped driver's retained values from the dependency mux

DriverHostActor now tells the mux SourceRemoved(driverInstanceId)
wherever a driver object ends: StopChild (deploy removal and the stop
half of a config-change respawn) and an operator Restart. It is not
sent on a supervision restart, which re-uses the same driver. Before
this nothing sent it, so a removed driver's last Good stayed in the
mux and replayed to the next replay-opting subscriber as current.

DriverHostActorSourceRemovedTests drives a real mux with two real
driver children: after a deploy drops DI-1 only DI-2's value replays
(positive control: both replay before the removal), and a supervision
restart of DI-1 leaves its value replayable.

Residual (documented on EvictRetainedValues): a burst the old child
enqueued just before it stopped can still be forwarded after the
eviction; the next publish, a comms-loss fan or the grace sweep
bounds it.

Refs #731 #734
Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
This commit is contained in:
Joseph Doherty
2026-10-22 14:36:54 -04:00
parent a33145fe71
commit 5de1d661bb
3 changed files with 145 additions and 5 deletions
@@ -3840,6 +3840,7 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers, IWithUnboundedS
StopDependencyAdapter(driverInstanceId);
Context.Stop(entry.Actor);
_children.Remove(driverInstanceId);
EvictRetainedValues(driverInstanceId);
// The instance left the plan (StopChild is only reached via plan.ToStop), so its cached
// Health/Resilience snapshots must not replay to new telemetry subscribers. Release the shared
// resilience state FIRST: the resilience publisher re-emits the tracker's snapshots every tick, so
@@ -3858,6 +3859,25 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers, IWithUnboundedS
_log.Info("DriverHost {Node}: stopped driver child {Id}", _localNode, driverInstanceId);
}
/// <summary>
/// Tells the dependency mux to drop every retained value this driver instance produced, so a stopped
/// driver's last Good is never replayed to a later replay-opting subscriber as if it were current. Sent
/// wherever a driver OBJECT ends: a deploy removal, the stop half of a config-change respawn
/// (<see cref="StopChild"/>) and an operator Restart (a new driver object). Deliberately NOT sent on a
/// supervision restart (<see cref="HandleReprovisionRequested"/>): that re-uses the same driver, whose last value is
/// still its current one. Sent before any respawn, on the same host→mux pair the new incarnation's
/// values travel, so the eviction is ordered ahead of them.
/// </summary>
/// <remarks>
/// Residual: a burst the old child enqueued on this host just before it stopped can still be forwarded
/// after the eviction and re-retain a value. The window is one mailbox drain; a comms-loss fan or the
/// next real publish overwrites it, and the mux's grace sweep bounds it once no replay subscriber holds
/// the ref.
/// </remarks>
/// <param name="driverInstanceId">The driver instance that ended.</param>
private void EvictRetainedValues(string driverInstanceId) =>
_dependencyMux?.Tell(new DependencyMuxActor.SourceRemoved(driverInstanceId));
private static string ActorNameFor(string driverInstanceId, long generation)
{
// Akka actor names cannot contain '/', ':', or whitespace. Mangle defensively. The monotonic
@@ -3905,6 +3925,9 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers, IWithUnboundedS
StopDependencyAdapter(msg.DriverInstanceId);
Context.Stop(entry.Actor);
_children.Remove(msg.DriverInstanceId);
// An operator Restart builds a NEW driver object, so the old one's retained values are evicted like a
// removal's; the respawned driver republishes its own.
EvictRetainedValues(msg.DriverInstanceId);
// The old incarnation's health must not read as the new one's until it publishes its own.
_healthPublisher.Forget(msg.DriverInstanceId);
@@ -52,11 +52,9 @@ namespace ZB.MOM.WW.OtOpcUa.Runtime.VirtualTags;
/// <b>Source removal.</b> The mux records which driver instance last published each retained ref;
/// <see cref="SourceRemoved"/> evicts every value that driver produced, so a removed driver's last
/// Good is never replayed as current. A comms-loss fan (a Bad batch) already overwrites the retained
/// value with Bad. ⚠️ As of fix round 1 no production actor sends <see cref="SourceRemoved"/> yet —
/// <c>DriverHostActor</c> (outside the scripting lane) must Tell it when it stops / removes a driver
/// child. Until then a removed driver's last value can still be replayed to a replay-opting
/// subscriber that re-registers for that ref while another replay-opting subscriber holds it, or
/// within the grace window.
/// value with Bad. <c>DriverHostActor</c> sends <see cref="SourceRemoved"/> wherever a driver object
/// ends (a deploy removal, the stop half of a config-change respawn, an operator Restart), and never
/// on a supervision restart, which re-uses the same driver.
/// </para>
/// </summary>
public sealed class DependencyMuxActor : ReceiveActor, IWithTimers