fix(mtconnect): close the silent-refusal + teardown-wedge defects in the pump (Task 11 review)
C1 (critical): a latched "this endpoint cannot stream" verdict could end with a GREEN driver holding a permanently silent subscription. DriverInstanceActor handles a tag-set change as Unsubscribe-then-Subscribe with no re-initialize; the unsubscribe cleared the stream-degraded flag, the resubscribe was refused by the latch without a word, and one successful read then reported Healthy. StartSampleStreamCore now re-asserts the degradation and logs a Warning naming the remedy, and teardown no longer clears the flag while the latch is set. NOT promoted to Faulted: DriverHealthReport maps any Faulted driver to a /readyz 503, so that would de-ready the whole node over one misconfigured Agent whose reads are perfectly healthy. Faulted stays for the case that earns it. I1: teardown waits are bounded (5 s) and honour the caller's token. They were unbounded and ignored it, so DriverInstanceActor's 5 s budget was inert and PostStop — which blocks a dispatcher thread on ShutdownAsync while holding the lifecycle semaphore — could be wedged forever by one blocking subscriber. The cancellation source is disposed only when the loop is provably gone. Applied to StopProbeLoopAsync too: Task 13 reproduced the identical shape, and closing the class in one method while leaving the other as the pattern to copy is worse than not closing it. I2: the reconnect ladder was monotonic for the process lifetime, so ~9 unrelated drops pinned the driver at MaxBackoffMs forever. A stream that delivered before dropping now starts a fresh ladder (at attempt 1, not 0, so MinBackoffMs is still honoured). I3: MaxBackoffMs <= 0 clamped every delay to zero — an operator-authorable reconnect spin. Treated as unset, like a multiplier that cannot grow. I4: the session published instanceId without NextSequence, so after a restart it held the new agent's id beside the dead one's cursor, and a gap/OUT_OF_RANGE re-baseline (id unchanged) never published the cursor at all. Both move in one CAS, and the pump publishes its cursor as it exits. I5: OnDataChange was a multicast Invoke — the first throwing handler aborted the rest of the list, starving every later subscriber. Now walked by hand with the catch inside the loop. The test that claimed to cover this registered the recorder BEFORE the thrower, so it passed regardless; swapped, it was red. Faults are latched to one Warning per stream generation (Debug thereafter) so a consistently-throwing consumer cannot flood the log. Also: the pump restarts after an unexpected fault (IsCompleted, not null); a device-scope that matches nothing is a Warning, not a Debug tally; a device or component with neither name nor id is skipped rather than emitting a blank path segment; DiscoverAsync's remark no longer claims DriverInstanceActor retries (it does not for a Once driver, and injection is dormant in v3); and two flake seeds in the fake are gone (Dispose re-read its own counter; _disposeCts was disposed under a racing SampleAsync). 439/439. Every changed behaviour falsified by mutation.
This commit is contained in:
@@ -37,6 +37,9 @@ internal sealed class CannedAgentClient : IMTConnectAgentClient, IDisposable
|
||||
{
|
||||
private readonly CancellationTokenSource _disposeCts = new();
|
||||
|
||||
private readonly Lock _sampleWaitersLock = new();
|
||||
private readonly List<(int Threshold, TaskCompletionSource Waiter)> _sampleWaiters = [];
|
||||
|
||||
/// <summary>Chunks a test has scripted but not yet pumped — see <see cref="ScriptChunks"/>.</summary>
|
||||
private readonly ConcurrentQueue<MTConnectStreamsResult> _script = new();
|
||||
|
||||
@@ -201,6 +204,47 @@ internal sealed class CannedAgentClient : IMTConnectAgentClient, IDisposable
|
||||
/// <summary>The <c>from</c> sequence the most recent <c>/sample</c> enumeration was opened at.</summary>
|
||||
public long? LastSampleFrom { get; private set; }
|
||||
|
||||
/// <summary>
|
||||
/// A task completing once <see cref="SampleCallCount"/> has reached
|
||||
/// <paramref name="count"/> — the deterministic "the pump has now opened its Nth stream"
|
||||
/// barrier, for the reconnect paths where no chunk is ever delivered and
|
||||
/// <see cref="PumpAsync"/> therefore cannot be the barrier.
|
||||
/// </summary>
|
||||
/// <param name="count">The enumeration count to wait for.</param>
|
||||
public Task WaitForSampleCallsAsync(int count)
|
||||
{
|
||||
lock (_sampleWaitersLock)
|
||||
{
|
||||
if (Volatile.Read(ref _sampleCallCount) >= count)
|
||||
{
|
||||
return Task.CompletedTask;
|
||||
}
|
||||
|
||||
var waiter = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously);
|
||||
_sampleWaiters.Add((count, waiter));
|
||||
|
||||
return waiter.Task;
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>Releases any <see cref="WaitForSampleCallsAsync"/> waiter the new call count satisfies.</summary>
|
||||
private void ReleaseSampleWaiters(int reached)
|
||||
{
|
||||
lock (_sampleWaitersLock)
|
||||
{
|
||||
for (var i = _sampleWaiters.Count - 1; i >= 0; i--)
|
||||
{
|
||||
if (_sampleWaiters[i].Threshold > reached)
|
||||
{
|
||||
continue;
|
||||
}
|
||||
|
||||
_sampleWaiters[i].Waiter.TrySetResult();
|
||||
_sampleWaiters.RemoveAt(i);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// <inheritdoc/>
|
||||
public async Task<MTConnectProbeModel> ProbeAsync(CancellationToken ct)
|
||||
{
|
||||
@@ -258,7 +302,7 @@ internal sealed class CannedAgentClient : IMTConnectAgentClient, IDisposable
|
||||
long from, [EnumeratorCancellation] CancellationToken ct)
|
||||
{
|
||||
ObjectDisposedException.ThrowIf(IsDisposed, this);
|
||||
Interlocked.Increment(ref _sampleCallCount);
|
||||
ReleaseSampleWaiters(Interlocked.Increment(ref _sampleCallCount));
|
||||
LastSampleFrom = from;
|
||||
|
||||
// Thrown before the first chunk, exactly like a real client that fails its /sample request:
|
||||
@@ -372,18 +416,33 @@ internal sealed class CannedAgentClient : IMTConnectAgentClient, IDisposable
|
||||
}
|
||||
|
||||
/// <inheritdoc/>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// <b>The first-disposer test uses the value <see cref="Interlocked.Increment(ref int)"/>
|
||||
/// returned</b>, not a re-read of the counter. Re-reading is a race: two concurrent
|
||||
/// disposes can both increment and then both read 2, so BOTH take the early return and
|
||||
/// the stream is never closed — a subscription test would then hang waiting for a
|
||||
/// teardown that silently did nothing.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b><see cref="_disposeCts"/> is cancelled but deliberately NOT disposed.</b> A
|
||||
/// <c>/sample</c> enumeration that is starting concurrently reaches
|
||||
/// <c>CreateLinkedTokenSource(ct, _disposeCts.Token)</c>, and reading <c>.Token</c> on a
|
||||
/// disposed source throws <see cref="ObjectDisposedException"/> — a flake that would
|
||||
/// surface as a random unrelated failure in whichever test happened to lose the race.
|
||||
/// Leaking one cancelled source per fake is free; a cancelled source needs no disposal
|
||||
/// to release anything a test cares about.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
public void Dispose()
|
||||
{
|
||||
Interlocked.Increment(ref _disposeCount);
|
||||
|
||||
if (DisposeCount > 1)
|
||||
if (Interlocked.Increment(ref _disposeCount) > 1)
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
_disposeCts.Cancel();
|
||||
Volatile.Read(ref _chunks).Writer.TryComplete();
|
||||
_disposeCts.Dispose();
|
||||
}
|
||||
|
||||
private sealed record ScriptedChunk(MTConnectStreamsResult Result, TaskCompletionSource Consumed);
|
||||
|
||||
Reference in New Issue
Block a user