feat(resilience): route IDriver.InitializeAsync through the capability invoker (R2-10 T10 closed)

The connect path was the one capability call the resilience pipeline never saw, so a driver that could not connect contributed zero breaker input and the /hosts resilience panel read 'All targets healthy' against a dead endpoint. InitializeAsync now routes through the invoker as DriverCapability.Initialize, keyed on the driver-instance id. - No pipeline retry: the actor's RetryConnect timer is the retry loop (parser forces retryCount overrides to 0, same funnel as write-shaped). - Breaker threshold 2, not the Read-sized 5: the breaker samples 30 s and the reconnect timer fires every 10 s, so at most three attempts fit one window — threshold 5 could never open. Pinned by its own test. - An open breaker fast-fails into the same InitializeFailed path as a real connect failure; the 15 s break window costs at most one extra tick. The best-effort teardown ShutdownAsync stays unwrapped. - Companion fix: DriverCapabilityInvokerFactory.Create now invalidates the instance's tracker snapshots with its pipelines — a respawn during an outage discarded the only breaker whose OnClosed could clear the OPEN flag, stranding a false 'Breaker OPEN' chip forever. Live gate PASSED on docker-dev (image symbol-verified per container): rtu-gate against dead 10.100.0.35:5021 opened its Initialize breaker on both centrals with the red 'BREAKER OPEN' chip on /hosts; a socat 5021→5020 forwarder recovered it (both breakers closed within 1 s, panel back to 'All targets healthy'); healthy MAIN-modbus recorded zero breaker events. Full suite 50/50 green. Evidence: docs/plans/2026-08-07-adminui-followups-and-live-gates.md §7.
2026-08-08 05:45:08 -04:00
parent c9f182b2d7
commit dac7e1885d
18 changed files with 445 additions and 8 deletions
@@ -80,6 +80,100 @@ public sealed class DriverInstanceActorResilienceWiringTests : RuntimeActorTestB
        invoker.WriteCalls.ShouldContain(c => c.Host == "plc-7" && c.IsIdempotent == false);
    }

+    /// <summary>
+    ///     R2-10 T10 closure: <c>IDriver.InitializeAsync</c> routes through the invoker as
+    ///     <see cref="DriverCapability.Initialize"/> keyed on the driver-instance id (init spans the
+    ///     whole driver, like a bulk subscribe). Before this wiring the connect path was the ONE
+    ///     capability call the invoker never saw — a driver that could not connect contributed zero
+    ///     breaker input, so the resilience panel stayed quiet for a dead endpoint.
+    /// </summary>
+    [Fact]
+    public void Initialize_routes_through_the_capability_invoker()
+    {
+        var invoker = new RecordingInvoker();
+        var driver = new HostResolvingWritableDriver();
+        var actor = Sys.ActorOf(DriverInstanceActor.Props(driver, invoker: invoker));
+
+        actor.Tell(new DriverInstanceActor.InitializeRequested("{}"));
+
+        AwaitCondition(
+            () => invoker.Calls.Any(c => c.Capability == DriverCapability.Initialize),
+            TimeSpan.FromSeconds(3));
+        invoker.Calls.ShouldContain(c =>
+            c.Capability == DriverCapability.Initialize && c.Host == driver.DriverInstanceId);
+    }
+
+    /// <summary>
+    ///     The open-breaker shape end to end at the actor: when the INVOKER itself throws on the
+    ///     Initialize route (exactly what an open circuit does — the call site is never reached),
+    ///     the actor treats it as an ordinary init failure — lands in Reconnecting, keeps the retry
+    ///     timer running, and recovers to Connected once the invoker lets an attempt through. This
+    ///     pins two things: the reconnect-RETRY path is wrapped too (not just the initial connect),
+    ///     and a breaker fast-fail can never wedge or crash the actor.
+    /// </summary>
+    [Fact]
+    public async Task An_invoker_fast_fail_on_init_keeps_the_actor_retrying_until_recovery()
+    {
+        var invoker = new FastFailingInitInvoker(failFirst: 2);
+        var driver = new HostResolvingWritableDriver();
+        var actor = Sys.ActorOf(DriverInstanceActor.Props(
+            driver, invoker: invoker, reconnectInterval: TimeSpan.FromMilliseconds(100)));
+
+        actor.Tell(new DriverInstanceActor.InitializeRequested("{}"));
+
+        // The first two init attempts die inside the invoker (open breaker); recovery is proven by a
+        // write succeeding — WriteAttribute fast-fails synchronously in Connecting/Reconnecting, so a
+        // Success reply is only possible from Connected.
+        DriverInstanceActor.WriteAttributeResult? reply = null;
+        for (var i = 0; i < 120 && reply is not { Success: true }; i++)
+        {
+            reply = await actor.Ask<DriverInstanceActor.WriteAttributeResult>(
+                new DriverInstanceActor.WriteAttribute("tag-1", 42), TimeSpan.FromSeconds(1));
+            if (reply is not { Success: true }) await Task.Delay(25);
+        }
+
+        reply.ShouldNotBeNull();
+        reply!.Success.ShouldBeTrue("the actor must recover to Connected once the breaker lets an attempt through");
+        invoker.InitializeAttempts.ShouldBeGreaterThanOrEqualTo(3,
+            "the reconnect-retry path must route through the invoker (initial attempt + at least the two fast-failed retries)");
+    }
+
+    /// <summary>Stands in for an OPEN circuit breaker on the Initialize route: the first
+    /// <paramref name="failFirst"/> Initialize calls throw WITHOUT invoking the call site (the
+    /// driver's connect never runs — exactly Polly's <c>BrokenCircuitException</c> shape), then the
+    /// breaker "closes" and calls pass through. All other capabilities always pass through.</summary>
+    private sealed class FastFailingInitInvoker : IDriverCapabilityInvoker
+    {
+        private readonly int _failFirst;
+        private int _initializeAttempts;
+
+        public FastFailingInitInvoker(int failFirst) => _failFirst = failFirst;
+
+        public int InitializeAttempts => Volatile.Read(ref _initializeAttempts);
+
+        public ValueTask<TResult> ExecuteAsync<TResult>(
+            DriverCapability capability, string hostName,
+            Func<CancellationToken, ValueTask<TResult>> callSite, CancellationToken cancellationToken)
+            => callSite(cancellationToken);
+
+        public ValueTask ExecuteAsync(
+            DriverCapability capability, string hostName,
+            Func<CancellationToken, ValueTask> callSite, CancellationToken cancellationToken)
+        {
+            if (capability == DriverCapability.Initialize
+                && Interlocked.Increment(ref _initializeAttempts) <= _failFirst)
+            {
+                throw new InvalidOperationException("circuit open (simulated fast-fail)");
+            }
+            return callSite(cancellationToken);
+        }
+
+        public ValueTask<TResult> ExecuteWriteAsync<TResult>(
+            string hostName, bool isIdempotent,
+            Func<CancellationToken, ValueTask<TResult>> callSite, CancellationToken cancellationToken)
+            => callSite(cancellationToken);
+    }
+
    /// <summary>Records every capability + resolved-host key routed through it, then delegates to the
    /// real call-site so the driver still runs. Stands in for <c>CapabilityInvoker</c> without pulling
    /// Polly / Core into the Runtime test.</summary>