feat(drivers): periodic reconcile of desired vs actual subscriptions (#488)
Option A from the issue — the cheap consistency check, no interface change. The desired ref set was re-applied on a Connected TRANSITION (`ResubscribeDesired`) and on a deploy's `SetDesiredSubscriptions`, and nowhere else. A subscription lost while the driver STAYED Connected — a subscribe that failed and was never retried, a handle dropped down an error path — left the actor subscribed to nothing while still reporting Healthy, indefinitely. `ReconcileSubscription` now rides the existing 30 s `health-poll` tick: while Connected, a driver holding a desired set but no live handle is re-subscribed. Gated on `ISubscribable`. A driver that cannot subscribe at all can still be handed a desired set, and without the guard it would be told `Subscribe` every tick and fail every tick, forever. This is a state-machine consistency check, NOT a probe — it can only see that this actor holds no handle, never that a handle is present but dead server-side, because `ISubscriptionHandle` is opaque. That is option B, and it stays deferred until a real occurrence shows the handle outliving the subscription; it would need a liveness member implemented across all eight drivers with different subscription mechanics. A redundant `Subscribe` is possible (a tick queued ahead of an already-queued one observes the same no-handle state) and is harmless: `HandleSubscribeAsync` is a `ReceiveAsync`, so no tick is processed mid-subscribe, and it drops any prior subscription before establishing the new one. Suppressing it would need a pending-subscribe flag threaded through every self-tell site — more state than the race is worth. `healthPollInterval` becomes an optional Props parameter, mirroring the existing `rediscoverInterval` seam, so a test can drive the reconcile without waiting 30 s. Three tests. The reconcile itself is proven by a stub whose FIRST subscribe throws: nothing else would ever retry it, so a second attempt can only come from the reconcile — disabling `ReconcileSubscription` turns it red. The two guard tests are absence assertions and are bounded by a positive control rather than a sleep: a counting health publisher makes ticks PROCESSED observable, so "still one subscribe" means the reconcile ran and declined, not that the timer had not fired. Explicitly not deploy-time drift (#486) — re-asserting an already-correct desired set would not have helped there.
This commit is contained in:
@@ -114,6 +114,18 @@ internal sealed class SubscribableStubDriver : StubDriver, ISubscribable
|
||||
/// synchronously-completed stub task hides (the continuation otherwise runs inline).</summary>
|
||||
public bool UnsubscribeYields { get; set; }
|
||||
|
||||
/// <summary>
|
||||
/// How many of the first <see cref="SubscribeAsync"/> calls throw before one succeeds. Default 0
|
||||
/// (always succeed) leaves existing behaviour untouched.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// This is how a test reaches the state the #488 reconcile exists for: the actor is Connected and
|
||||
/// holds a desired ref set, but the subscribe that should have established the handle failed, so
|
||||
/// <c>_subscriptionHandle</c> is null and nothing else will ever re-assert it — there is no
|
||||
/// connect transition coming, and no deploy.
|
||||
/// </remarks>
|
||||
public int SubscribeFailuresBeforeSuccess { get; set; }
|
||||
|
||||
/// <summary>Subscribes to the specified full references.</summary>
|
||||
/// <param name="fullReferences">The full references to subscribe to.</param>
|
||||
/// <param name="publishingInterval">The publishing interval.</param>
|
||||
@@ -121,8 +133,13 @@ internal sealed class SubscribableStubDriver : StubDriver, ISubscribable
|
||||
public Task<ISubscriptionHandle> SubscribeAsync(
|
||||
IReadOnlyList<string> fullReferences, TimeSpan publishingInterval, CancellationToken cancellationToken)
|
||||
{
|
||||
Interlocked.Increment(ref SubscribeCount);
|
||||
var attempt = Interlocked.Increment(ref SubscribeCount);
|
||||
LastSubscribedRefs = fullReferences;
|
||||
if (attempt <= SubscribeFailuresBeforeSuccess)
|
||||
{
|
||||
throw new InvalidOperationException($"stub subscribe failure #{attempt}");
|
||||
}
|
||||
|
||||
return Task.FromResult<ISubscriptionHandle>(_handle);
|
||||
}
|
||||
|
||||
|
||||
Reference in New Issue
Block a user