feat(resilience): route IDriver.InitializeAsync through the capability invoker (R2-10 T10 closed)

The connect path was the one capability call the resilience pipeline never
saw, so a driver that could not connect contributed zero breaker input and
the /hosts resilience panel read 'All targets healthy' against a dead
endpoint. InitializeAsync now routes through the invoker as
DriverCapability.Initialize, keyed on the driver-instance id.

- No pipeline retry: the actor's RetryConnect timer is the retry loop
  (parser forces retryCount overrides to 0, same funnel as write-shaped).
- Breaker threshold 2, not the Read-sized 5: the breaker samples 30 s and
  the reconnect timer fires every 10 s, so at most three attempts fit one
  window — threshold 5 could never open. Pinned by its own test.
- An open breaker fast-fails into the same InitializeFailed path as a real
  connect failure; the 15 s break window costs at most one extra tick.
  The best-effort teardown ShutdownAsync stays unwrapped.
- Companion fix: DriverCapabilityInvokerFactory.Create now invalidates the
  instance's tracker snapshots with its pipelines — a respawn during an
  outage discarded the only breaker whose OnClosed could clear the OPEN
  flag, stranding a false 'Breaker OPEN' chip forever.

Live gate PASSED on docker-dev (image symbol-verified per container):
rtu-gate against dead 10.100.0.35:5021 opened its Initialize breaker on
both centrals with the red 'BREAKER OPEN' chip on /hosts; a socat
5021→5020 forwarder recovered it (both breakers closed within 1 s, panel
back to 'All targets healthy'); healthy MAIN-modbus recorded zero breaker
events. Full suite 50/50 green. Evidence:
docs/plans/2026-08-07-adminui-followups-and-live-gates.md §7.
This commit is contained in:
Joseph Doherty
2026-08-08 05:45:08 -04:00
parent c9f182b2d7
commit dac7e1885d
18 changed files with 445 additions and 8 deletions
+34
View File
@@ -270,6 +270,40 @@ re-entered this path; the watchdog makes it automatic and unbounded, so a device
have leaked ~360 transports at the 10 s retry interval. The shutdown is best-effort: a driver that throws
on it still gets its initialise attempt.
### The connect path feeds the circuit breaker (R2-10 T10 closure)
`IDriver.InitializeAsync` is routed through the capability invoker as
**`DriverCapability.Initialize`**, keyed on the driver-instance id (init spans the whole driver,
like a bulk subscribe). Before this, the connect path was the ONE capability call the invoker never
saw: a driver that could not connect contributed **zero** breaker input, so the AdminUI resilience
panel read *"All targets healthy"* against a dead endpoint — live-proven during the R2-10 T10 gate,
and the reason that gate was structurally unsatisfiable for a year of attempts. Things to know
before touching it:
- **The Initialize policy never retries** — this actor's `RetryConnect` timer *is* the retry loop,
and pipeline retries would stack inside each timer tick (and stretch one attempt past the timer
interval so attempts overlap). The parser forces a `retryCount` override back to 0, same funnel
as the write-shaped rule.
- ⚠️ **The breaker threshold is 2, and raising it silently re-breaks the feature.** The breaker
samples over 30 s and the reconnect timer fires every 10 s, so at most **three** init attempts fit
one sampling window — a Read-sized threshold of 5 can literally never open on connect failures,
which reads exactly like "the panel doesn't work" with nothing logged. Pinned by
`Initialize_BreakerThreshold_IsReachableAtTheReconnectCadence` and, behaviourally, by
`Initialize_BreakerOpens_AtTierDefaults_AndFastFailsTheNextAttempt`.
- **An open breaker cannot wedge the reconnect loop.** A fast-failed attempt
(`BrokenCircuitException`) lands in the same catch as a real connect failure →
`InitializeFailed` → the timer keeps ticking. The 15 s break window is shorter than two 10 s
ticks, so recovery costs at most one extra tick. The teardown `ShutdownAsync` stays deliberately
UNWRAPPED — best-effort cleanup of an already-dead transport must not count toward the breaker.
- **A respawn resets the instance's tracker snapshots** (`DriverResilienceStatusTracker.Invalidate`,
called by `DriverCapabilityInvokerFactory.Create` alongside the pipeline invalidation). The
pipelines carry the actual breaker state; once they are discarded, a retained "Breaker OPEN"
snapshot could never self-correct — only the discarded pipeline's `OnClosed` stamps the close —
so a deploy during an outage would otherwise strand a false OPEN chip forever. This was
unreachable while breakers never opened; the Initialize wrap made it reachable.
- `ConsecutiveFailures` stays 0 during a connect outage **by design** — the tracker counts retry
attempts, and Initialize has none. The breaker-OPEN chip is the operator signal.
---
## IDriverConfigEditor — custom AdminUI config editor (plug-point)