feat(resilience): route IDriver.InitializeAsync through the capability invoker (R2-10 T10 closed)

The connect path was the one capability call the resilience pipeline never
saw, so a driver that could not connect contributed zero breaker input and
the /hosts resilience panel read 'All targets healthy' against a dead
endpoint. InitializeAsync now routes through the invoker as
DriverCapability.Initialize, keyed on the driver-instance id.

- No pipeline retry: the actor's RetryConnect timer is the retry loop
  (parser forces retryCount overrides to 0, same funnel as write-shaped).
- Breaker threshold 2, not the Read-sized 5: the breaker samples 30 s and
  the reconnect timer fires every 10 s, so at most three attempts fit one
  window — threshold 5 could never open. Pinned by its own test.
- An open breaker fast-fails into the same InitializeFailed path as a real
  connect failure; the 15 s break window costs at most one extra tick.
  The best-effort teardown ShutdownAsync stays unwrapped.
- Companion fix: DriverCapabilityInvokerFactory.Create now invalidates the
  instance's tracker snapshots with its pipelines — a respawn during an
  outage discarded the only breaker whose OnClosed could clear the OPEN
  flag, stranding a false 'Breaker OPEN' chip forever.

Live gate PASSED on docker-dev (image symbol-verified per container):
rtu-gate against dead 10.100.0.35:5021 opened its Initialize breaker on
both centrals with the red 'BREAKER OPEN' chip on /hosts; a socat
5021→5020 forwarder recovered it (both breakers closed within 1 s, panel
back to 'All targets healthy'); healthy MAIN-modbus recorded zero breaker
events. Full suite 50/50 green. Evidence:
docs/plans/2026-08-07-adminui-followups-and-live-gates.md §7.
This commit is contained in:
Joseph Doherty
2026-08-08 05:45:08 -04:00
parent c9f182b2d7
commit dac7e1885d
18 changed files with 445 additions and 8 deletions
+34
View File
@@ -270,6 +270,40 @@ re-entered this path; the watchdog makes it automatic and unbounded, so a device
have leaked ~360 transports at the 10 s retry interval. The shutdown is best-effort: a driver that throws
on it still gets its initialise attempt.
### The connect path feeds the circuit breaker (R2-10 T10 closure)
`IDriver.InitializeAsync` is routed through the capability invoker as
**`DriverCapability.Initialize`**, keyed on the driver-instance id (init spans the whole driver,
like a bulk subscribe). Before this, the connect path was the ONE capability call the invoker never
saw: a driver that could not connect contributed **zero** breaker input, so the AdminUI resilience
panel read *"All targets healthy"* against a dead endpoint — live-proven during the R2-10 T10 gate,
and the reason that gate was structurally unsatisfiable for a year of attempts. Things to know
before touching it:
- **The Initialize policy never retries** — this actor's `RetryConnect` timer *is* the retry loop,
and pipeline retries would stack inside each timer tick (and stretch one attempt past the timer
interval so attempts overlap). The parser forces a `retryCount` override back to 0, same funnel
as the write-shaped rule.
- ⚠️ **The breaker threshold is 2, and raising it silently re-breaks the feature.** The breaker
samples over 30 s and the reconnect timer fires every 10 s, so at most **three** init attempts fit
one sampling window — a Read-sized threshold of 5 can literally never open on connect failures,
which reads exactly like "the panel doesn't work" with nothing logged. Pinned by
`Initialize_BreakerThreshold_IsReachableAtTheReconnectCadence` and, behaviourally, by
`Initialize_BreakerOpens_AtTierDefaults_AndFastFailsTheNextAttempt`.
- **An open breaker cannot wedge the reconnect loop.** A fast-failed attempt
(`BrokenCircuitException`) lands in the same catch as a real connect failure →
`InitializeFailed` → the timer keeps ticking. The 15 s break window is shorter than two 10 s
ticks, so recovery costs at most one extra tick. The teardown `ShutdownAsync` stays deliberately
UNWRAPPED — best-effort cleanup of an already-dead transport must not count toward the breaker.
- **A respawn resets the instance's tracker snapshots** (`DriverResilienceStatusTracker.Invalidate`,
called by `DriverCapabilityInvokerFactory.Create` alongside the pipeline invalidation). The
pipelines carry the actual breaker state; once they are discarded, a retained "Breaker OPEN"
snapshot could never self-correct — only the discarded pipeline's `OnClosed` stamps the close —
so a deploy during an outage would otherwise strand a false OPEN chip forever. This was
unreachable while breakers never opened; the Initialize wrap made it reachable.
- `ConsecutiveFailures` stays 0 during a connect outage **by design** — the tracker counts retry
attempts, and Initialize has none. The breaker-OPEN chip is the operator signal.
---
## IDriverConfigEditor — custom AdminUI config editor (plug-point)
@@ -215,3 +215,67 @@ original one-line body, so it mirrors `MainPressure` again rather than the `×6`
One anomaly recorded rather than explained: both centrals were **recreated** at 09:49:40/09:49:50
(new `Created`, `RestartCount=0`) by something this session did not issue. It postdates the
observations above and the old container's logs went with it, so it is noted, not attributed.
---
## 7. Addendum 2026-08-08 — R2-10 T10 ✅ PASSED after fixing the structural gap
§4 concluded the gate was not achievable as specified because the connect path never entered the
resilience pipeline. The user chose to fix that ("do option 1"), and the gate then passed against
the fixed build. Branch `feat/wrap-driver-init-in-resilience`.
### The fix
- **`DriverCapability.Initialize`** — `IDriver.InitializeAsync` routes through the capability
invoker in `DriverInstanceActor.InitializeAsync` (both the initial connect and the
`Reconnecting` retry path; the best-effort teardown `ShutdownAsync` stays deliberately
unwrapped). Keyed on the driver-instance id, like a bulk subscribe.
- **No pipeline retry for Initialize** — the actor's `RetryConnect` timer is the retry loop.
Enforced at the parser funnel (`retryCount` override forced to 0, same shape as the
write-shaped rule) and in every tier default.
- **Breaker threshold 2, not the Read-sized 5** — the breaker samples 30 s and the reconnect
timer fires every 10 s, so at most three init attempts fit one window; threshold 5 could never
open. Pinned by `Initialize_BreakerThreshold_IsReachableAtTheReconnectCadence` +
`Initialize_BreakerOpens_AtTierDefaults_AndFastFailsTheNextAttempt`.
- **Companion fix — respawn resets tracker snapshots.** `DriverCapabilityInvokerFactory.Create`
now calls `DriverResilienceStatusTracker.Invalidate(driverInstanceId)` alongside the pipeline
invalidation. Without it, a deploy during an outage discards the only breaker whose `OnClosed`
could clear the OPEN flag, stranding a false "Breaker OPEN" chip forever. Unreachable before
this change made breaker-open reachable; found by asking what clears the flag, not by a test.
- Wiring pinned by `Initialize_routes_through_the_capability_invoker` +
`An_invoker_fast_fail_on_init_keeps_the_actor_retrying_until_recovery` (falsification-controlled:
both go red with the actor wrap stashed; `Create_resets_the_instances_tracker_state_with_its_pipelines`
goes red with the factory call stashed).
### The gate run (docker-dev, image `df04c76ce06b`, symbol-verified on both centrals)
The staged failure was free: `rtu-gate` (`DRV-3616122eb53d`) was already authored against
`10.100.0.35:5021`, where nothing listens — connection refused on every init.
1. **Breaker OPEN, both replicas** — within ~20 s of boot, both centrals logged
`circuit-breaker OPENED: instance=DRV-3616122eb53d … capability=Initialize … Connection refused`,
re-opening each half-open probe cycle. Bonus coverage: `DRV-opcua1` (invalid config → init
throws) also tripped — a permanently-misconfigured driver was equally invisible before.
2. **Panel chip** — `/hosts` → rtu-gate → Controls showed the red **"BREAKER OPEN · OPENED 19S
AGO"** chip in the RESILIENCE section, row FAULTED with "Connection refused".
3. **Control** — healthy `MAIN-modbus` recorded zero breaker events throughout.
4. **Recovery clear** — a socat forwarder (`5021 → 5020` on the docker host) brought the endpoint
alive; both centrals logged `circuit-breaker closed (recovered)` + `reconnected` within 1 s of
each other, and the panel dropped to "All targets healthy."
5. **Teardown** — socat removed; 5021 dead again (as-found). rtu-gate returns to a truthfully-OPEN
breaker chip, which is the feature working.
### Two nuances against the original T10 wording
- **"Both-replica chips"** resolves as both nodes' *trackers* tripping (proven in both logs)
rendered through the single **active** AdminUI. Traefik's `/health/active` check deliberately
routes the UI to the active admin node only — the standby returns 503
(`Standby: active node for role 'admin' is …`), so there is never a second replica serving
chips. Discovered while trying to force the sticky-cookie LB onto central-1.
- **The staleness-dim leg was not exercised.** Both pair nodes publish under the same
`(instance, host)` key, so staleness requires *every* publisher for a key to stop — deliberate
node-kill churn this gate did not do. The dim is a pure render predicate (`IsStale` → opacity).
Also corrected while here: §4's "ConnectionBackoff throttles retries" explanation (the previously
recorded root cause) is confirmed wrong — the breaker config opens on 2 failures in 30 s; the
operative cause was always that no failure reached the pipeline at all.