feat(resilience): route IDriver.InitializeAsync through the capability invoker (R2-10 T10 closed)
The connect path was the one capability call the resilience pipeline never saw, so a driver that could not connect contributed zero breaker input and the /hosts resilience panel read 'All targets healthy' against a dead endpoint. InitializeAsync now routes through the invoker as DriverCapability.Initialize, keyed on the driver-instance id. - No pipeline retry: the actor's RetryConnect timer is the retry loop (parser forces retryCount overrides to 0, same funnel as write-shaped). - Breaker threshold 2, not the Read-sized 5: the breaker samples 30 s and the reconnect timer fires every 10 s, so at most three attempts fit one window — threshold 5 could never open. Pinned by its own test. - An open breaker fast-fails into the same InitializeFailed path as a real connect failure; the 15 s break window costs at most one extra tick. The best-effort teardown ShutdownAsync stays unwrapped. - Companion fix: DriverCapabilityInvokerFactory.Create now invalidates the instance's tracker snapshots with its pipelines — a respawn during an outage discarded the only breaker whose OnClosed could clear the OPEN flag, stranding a false 'Breaker OPEN' chip forever. Live gate PASSED on docker-dev (image symbol-verified per container): rtu-gate against dead 10.100.0.35:5021 opened its Initialize breaker on both centrals with the red 'BREAKER OPEN' chip on /hosts; a socat 5021→5020 forwarder recovered it (both breakers closed within 1 s, panel back to 'All targets healthy'); healthy MAIN-modbus recorded zero breaker events. Full suite 50/50 green. Evidence: docs/plans/2026-08-07-adminui-followups-and-live-gates.md §7.
This commit is contained in:
@@ -270,6 +270,40 @@ re-entered this path; the watchdog makes it automatic and unbounded, so a device
|
||||
have leaked ~360 transports at the 10 s retry interval. The shutdown is best-effort: a driver that throws
|
||||
on it still gets its initialise attempt.
|
||||
|
||||
### The connect path feeds the circuit breaker (R2-10 T10 closure)
|
||||
|
||||
`IDriver.InitializeAsync` is routed through the capability invoker as
|
||||
**`DriverCapability.Initialize`**, keyed on the driver-instance id (init spans the whole driver,
|
||||
like a bulk subscribe). Before this, the connect path was the ONE capability call the invoker never
|
||||
saw: a driver that could not connect contributed **zero** breaker input, so the AdminUI resilience
|
||||
panel read *"All targets healthy"* against a dead endpoint — live-proven during the R2-10 T10 gate,
|
||||
and the reason that gate was structurally unsatisfiable for a year of attempts. Things to know
|
||||
before touching it:
|
||||
|
||||
- **The Initialize policy never retries** — this actor's `RetryConnect` timer *is* the retry loop,
|
||||
and pipeline retries would stack inside each timer tick (and stretch one attempt past the timer
|
||||
interval so attempts overlap). The parser forces a `retryCount` override back to 0, same funnel
|
||||
as the write-shaped rule.
|
||||
- ⚠️ **The breaker threshold is 2, and raising it silently re-breaks the feature.** The breaker
|
||||
samples over 30 s and the reconnect timer fires every 10 s, so at most **three** init attempts fit
|
||||
one sampling window — a Read-sized threshold of 5 can literally never open on connect failures,
|
||||
which reads exactly like "the panel doesn't work" with nothing logged. Pinned by
|
||||
`Initialize_BreakerThreshold_IsReachableAtTheReconnectCadence` and, behaviourally, by
|
||||
`Initialize_BreakerOpens_AtTierDefaults_AndFastFailsTheNextAttempt`.
|
||||
- **An open breaker cannot wedge the reconnect loop.** A fast-failed attempt
|
||||
(`BrokenCircuitException`) lands in the same catch as a real connect failure →
|
||||
`InitializeFailed` → the timer keeps ticking. The 15 s break window is shorter than two 10 s
|
||||
ticks, so recovery costs at most one extra tick. The teardown `ShutdownAsync` stays deliberately
|
||||
UNWRAPPED — best-effort cleanup of an already-dead transport must not count toward the breaker.
|
||||
- **A respawn resets the instance's tracker snapshots** (`DriverResilienceStatusTracker.Invalidate`,
|
||||
called by `DriverCapabilityInvokerFactory.Create` alongside the pipeline invalidation). The
|
||||
pipelines carry the actual breaker state; once they are discarded, a retained "Breaker OPEN"
|
||||
snapshot could never self-correct — only the discarded pipeline's `OnClosed` stamps the close —
|
||||
so a deploy during an outage would otherwise strand a false OPEN chip forever. This was
|
||||
unreachable while breakers never opened; the Initialize wrap made it reachable.
|
||||
- `ConsecutiveFailures` stays 0 during a connect outage **by design** — the tracker counts retry
|
||||
attempts, and Initialize has none. The breaker-OPEN chip is the operator signal.
|
||||
|
||||
---
|
||||
|
||||
## IDriverConfigEditor — custom AdminUI config editor (plug-point)
|
||||
|
||||
Reference in New Issue
Block a user