fix(mtconnect): close the silent-refusal + teardown-wedge defects in the pump (Task 11 review)

C1 (critical): a latched "this endpoint cannot stream" verdict could end with a GREEN driver holding a permanently silent subscription. DriverInstanceActor handles a tag-set change as Unsubscribe-then-Subscribe with no re-initialize; the unsubscribe cleared the stream-degraded flag, the resubscribe was refused by the latch without a word, and one successful read then reported Healthy. StartSampleStreamCore now re-asserts the degradation and logs a Warning naming the remedy, and teardown no longer clears the flag while the latch is set. NOT promoted to Faulted: DriverHealthReport maps any Faulted driver to a /readyz 503, so that would de-ready the whole node over one misconfigured Agent whose reads are perfectly healthy. Faulted stays for the case that earns it. I1: teardown waits are bounded (5 s) and honour the caller's token. They were unbounded and ignored it, so DriverInstanceActor's 5 s budget was inert and PostStop — which blocks a dispatcher thread on ShutdownAsync while holding the lifecycle semaphore — could be wedged forever by one blocking subscriber. The cancellation source is disposed only when the loop is provably gone. Applied to StopProbeLoopAsync too: Task 13 reproduced the identical shape, and closing the class in one method while leaving the other as the pattern to copy is worse than not closing it. I2: the reconnect ladder was monotonic for the process lifetime, so ~9 unrelated drops pinned the driver at MaxBackoffMs forever. A stream that delivered before dropping now starts a fresh ladder (at attempt 1, not 0, so MinBackoffMs is still honoured). I3: MaxBackoffMs <= 0 clamped every delay to zero — an operator-authorable reconnect spin. Treated as unset, like a multiplier that cannot grow. I4: the session published instanceId without NextSequence, so after a restart it held the new agent's id beside the dead one's cursor, and a gap/OUT_OF_RANGE re-baseline (id unchanged) never published the cursor at all. Both move in one CAS, and the pump publishes its cursor as it exits. I5: OnDataChange was a multicast Invoke — the first throwing handler aborted the rest of the list, starving every later subscriber. Now walked by hand with the catch inside the loop. The test that claimed to cover this registered the recorder BEFORE the thrower, so it passed regardless; swapped, it was red. Faults are latched to one Warning per stream generation (Debug thereafter) so a consistently-throwing consumer cannot flood the log. Also: the pump restarts after an unexpected fault (IsCompleted, not null); a device-scope that matches nothing is a Warning, not a Debug tally; a device or component with neither name nor id is skipped rather than emitting a blank path segment; DiscoverAsync's remark no longer claims DriverInstanceActor retries (it does not for a Once driver, and injection is dormant in v3); and two flake seeds in the fake are gone (Dispose re-read its own counter; _disposeCts was disposed under a racing SampleAsync). 439/439. Every changed behaviour falsified by mutation.
2026-07-24 17:16:13 -04:00
parent a92584c45e
commit a3a0cfe9bb
6 changed files with 926 additions and 78 deletions
@@ -121,6 +121,53 @@ public sealed class MTConnectHostAndRediscoverTests
        await ticker.ParkedAsync().WaitAsync(Watchdog, Ct);
    }

+    /// <summary>
+    ///     <b>Teardown symmetry with the <c>/sample</c> pump (remediation I1).</b> The probe loop is
+    ///     the second long-lived background task in this driver, and it is stopped the same way and
+    ///     from the same place: while the lifecycle semaphore is held, on a path
+    ///     <c>DriverInstanceActor.PostStop</c> blocks an Akka dispatcher thread on. So its wait is
+    ///     bounded and honours the caller's token too — a blocking
+    ///     <see cref="IHostConnectivityProbe.OnHostStatusChanged"/> handler must not be able to wedge
+    ///     shutdown, any more than a blocking data-change handler can.
+    /// </summary>
+    /// <remarks>
+    ///     Its blast radius is genuinely smaller than the pump's (each iteration is bounded by the
+    ///     validated-positive <c>Probe.Timeout</c>, where the pump's is an unbounded long poll), but
+    ///     leaving the two divergent would make the un-bounded one the pattern a future reader
+    ///     copies — the defect class would be closed in one place and re-opened in the other, in the
+    ///     same file.
+    /// </remarks>
+    [Fact]
+    public async Task Shutdown_honours_the_callers_deadline_when_a_host_status_subscriber_blocks()
+    {
+        var (driver, _, ticker) = await ProbingDriverAsync();
+        var blocking = new ManualResetEventSlim(false);
+        var entered = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously);
+
+        ((IHostConnectivityProbe)driver).OnHostStatusChanged += (_, _) =>
+        {
+            entered.TrySetResult();
+            blocking.Wait();
+        };
+
+        try
+        {
+            // The first successful probe transitions Unknown -> Running and raises the event, which
+            // now blocks inside the loop.
+            ticker.Tick();
+            await entered.Task.WaitAsync(Watchdog, Ct);
+
+            using var deadline = new CancellationTokenSource(TimeSpan.FromMilliseconds(100));
+            await driver.ShutdownAsync(deadline.Token).WaitAsync(Watchdog, Ct);
+
+            driver.ProbeLoopTask.ShouldBeNull();
+        }
+        finally
+        {
+            blocking.Set();
+        }
+    }
+
    private static ConcurrentQueue<RediscoveryEventArgs> RecordRediscovery(MTConnectDriver driver)
    {
        var seen = new ConcurrentQueue<RediscoveryEventArgs>();