feat(resilience): route IDriver.InitializeAsync through the capability invoker (R2-10 T10 closed)
The connect path was the one capability call the resilience pipeline never saw, so a driver that could not connect contributed zero breaker input and the /hosts resilience panel read 'All targets healthy' against a dead endpoint. InitializeAsync now routes through the invoker as DriverCapability.Initialize, keyed on the driver-instance id. - No pipeline retry: the actor's RetryConnect timer is the retry loop (parser forces retryCount overrides to 0, same funnel as write-shaped). - Breaker threshold 2, not the Read-sized 5: the breaker samples 30 s and the reconnect timer fires every 10 s, so at most three attempts fit one window — threshold 5 could never open. Pinned by its own test. - An open breaker fast-fails into the same InitializeFailed path as a real connect failure; the 15 s break window costs at most one extra tick. The best-effort teardown ShutdownAsync stays unwrapped. - Companion fix: DriverCapabilityInvokerFactory.Create now invalidates the instance's tracker snapshots with its pipelines — a respawn during an outage discarded the only breaker whose OnClosed could clear the OPEN flag, stranding a false 'Breaker OPEN' chip forever. Live gate PASSED on docker-dev (image symbol-verified per container): rtu-gate against dead 10.100.0.35:5021 opened its Initialize breaker on both centrals with the red 'BREAKER OPEN' chip on /hosts; a socat 5021→5020 forwarder recovered it (both breakers closed within 1 s, panel back to 'All targets healthy'); healthy MAIN-modbus recorded zero breaker events. Full suite 50/50 green. Evidence: docs/plans/2026-08-07-adminui-followups-and-live-gates.md §7.
This commit is contained in:
@@ -270,6 +270,40 @@ re-entered this path; the watchdog makes it automatic and unbounded, so a device
|
||||
have leaked ~360 transports at the 10 s retry interval. The shutdown is best-effort: a driver that throws
|
||||
on it still gets its initialise attempt.
|
||||
|
||||
### The connect path feeds the circuit breaker (R2-10 T10 closure)
|
||||
|
||||
`IDriver.InitializeAsync` is routed through the capability invoker as
|
||||
**`DriverCapability.Initialize`**, keyed on the driver-instance id (init spans the whole driver,
|
||||
like a bulk subscribe). Before this, the connect path was the ONE capability call the invoker never
|
||||
saw: a driver that could not connect contributed **zero** breaker input, so the AdminUI resilience
|
||||
panel read *"All targets healthy"* against a dead endpoint — live-proven during the R2-10 T10 gate,
|
||||
and the reason that gate was structurally unsatisfiable for a year of attempts. Things to know
|
||||
before touching it:
|
||||
|
||||
- **The Initialize policy never retries** — this actor's `RetryConnect` timer *is* the retry loop,
|
||||
and pipeline retries would stack inside each timer tick (and stretch one attempt past the timer
|
||||
interval so attempts overlap). The parser forces a `retryCount` override back to 0, same funnel
|
||||
as the write-shaped rule.
|
||||
- ⚠️ **The breaker threshold is 2, and raising it silently re-breaks the feature.** The breaker
|
||||
samples over 30 s and the reconnect timer fires every 10 s, so at most **three** init attempts fit
|
||||
one sampling window — a Read-sized threshold of 5 can literally never open on connect failures,
|
||||
which reads exactly like "the panel doesn't work" with nothing logged. Pinned by
|
||||
`Initialize_BreakerThreshold_IsReachableAtTheReconnectCadence` and, behaviourally, by
|
||||
`Initialize_BreakerOpens_AtTierDefaults_AndFastFailsTheNextAttempt`.
|
||||
- **An open breaker cannot wedge the reconnect loop.** A fast-failed attempt
|
||||
(`BrokenCircuitException`) lands in the same catch as a real connect failure →
|
||||
`InitializeFailed` → the timer keeps ticking. The 15 s break window is shorter than two 10 s
|
||||
ticks, so recovery costs at most one extra tick. The teardown `ShutdownAsync` stays deliberately
|
||||
UNWRAPPED — best-effort cleanup of an already-dead transport must not count toward the breaker.
|
||||
- **A respawn resets the instance's tracker snapshots** (`DriverResilienceStatusTracker.Invalidate`,
|
||||
called by `DriverCapabilityInvokerFactory.Create` alongside the pipeline invalidation). The
|
||||
pipelines carry the actual breaker state; once they are discarded, a retained "Breaker OPEN"
|
||||
snapshot could never self-correct — only the discarded pipeline's `OnClosed` stamps the close —
|
||||
so a deploy during an outage would otherwise strand a false OPEN chip forever. This was
|
||||
unreachable while breakers never opened; the Initialize wrap made it reachable.
|
||||
- `ConsecutiveFailures` stays 0 during a connect outage **by design** — the tracker counts retry
|
||||
attempts, and Initialize has none. The breaker-OPEN chip is the operator signal.
|
||||
|
||||
---
|
||||
|
||||
## IDriverConfigEditor — custom AdminUI config editor (plug-point)
|
||||
|
||||
@@ -215,3 +215,67 @@ original one-line body, so it mirrors `MainPressure` again rather than the `×6`
|
||||
One anomaly recorded rather than explained: both centrals were **recreated** at 09:49:40/09:49:50
|
||||
(new `Created`, `RestartCount=0`) by something this session did not issue. It postdates the
|
||||
observations above and the old container's logs went with it, so it is noted, not attributed.
|
||||
|
||||
---
|
||||
|
||||
## 7. Addendum 2026-08-08 — R2-10 T10 ✅ PASSED after fixing the structural gap
|
||||
|
||||
§4 concluded the gate was not achievable as specified because the connect path never entered the
|
||||
resilience pipeline. The user chose to fix that ("do option 1"), and the gate then passed against
|
||||
the fixed build. Branch `feat/wrap-driver-init-in-resilience`.
|
||||
|
||||
### The fix
|
||||
|
||||
- **`DriverCapability.Initialize`** — `IDriver.InitializeAsync` routes through the capability
|
||||
invoker in `DriverInstanceActor.InitializeAsync` (both the initial connect and the
|
||||
`Reconnecting` retry path; the best-effort teardown `ShutdownAsync` stays deliberately
|
||||
unwrapped). Keyed on the driver-instance id, like a bulk subscribe.
|
||||
- **No pipeline retry for Initialize** — the actor's `RetryConnect` timer is the retry loop.
|
||||
Enforced at the parser funnel (`retryCount` override forced to 0, same shape as the
|
||||
write-shaped rule) and in every tier default.
|
||||
- **Breaker threshold 2, not the Read-sized 5** — the breaker samples 30 s and the reconnect
|
||||
timer fires every 10 s, so at most three init attempts fit one window; threshold 5 could never
|
||||
open. Pinned by `Initialize_BreakerThreshold_IsReachableAtTheReconnectCadence` +
|
||||
`Initialize_BreakerOpens_AtTierDefaults_AndFastFailsTheNextAttempt`.
|
||||
- **Companion fix — respawn resets tracker snapshots.** `DriverCapabilityInvokerFactory.Create`
|
||||
now calls `DriverResilienceStatusTracker.Invalidate(driverInstanceId)` alongside the pipeline
|
||||
invalidation. Without it, a deploy during an outage discards the only breaker whose `OnClosed`
|
||||
could clear the OPEN flag, stranding a false "Breaker OPEN" chip forever. Unreachable before
|
||||
this change made breaker-open reachable; found by asking what clears the flag, not by a test.
|
||||
- Wiring pinned by `Initialize_routes_through_the_capability_invoker` +
|
||||
`An_invoker_fast_fail_on_init_keeps_the_actor_retrying_until_recovery` (falsification-controlled:
|
||||
both go red with the actor wrap stashed; `Create_resets_the_instances_tracker_state_with_its_pipelines`
|
||||
goes red with the factory call stashed).
|
||||
|
||||
### The gate run (docker-dev, image `df04c76ce06b`, symbol-verified on both centrals)
|
||||
|
||||
The staged failure was free: `rtu-gate` (`DRV-3616122eb53d`) was already authored against
|
||||
`10.100.0.35:5021`, where nothing listens — connection refused on every init.
|
||||
|
||||
1. **Breaker OPEN, both replicas** — within ~20 s of boot, both centrals logged
|
||||
`circuit-breaker OPENED: instance=DRV-3616122eb53d … capability=Initialize … Connection refused`,
|
||||
re-opening each half-open probe cycle. Bonus coverage: `DRV-opcua1` (invalid config → init
|
||||
throws) also tripped — a permanently-misconfigured driver was equally invisible before.
|
||||
2. **Panel chip** — `/hosts` → rtu-gate → Controls showed the red **"BREAKER OPEN · OPENED 19S
|
||||
AGO"** chip in the RESILIENCE section, row FAULTED with "Connection refused".
|
||||
3. **Control** — healthy `MAIN-modbus` recorded zero breaker events throughout.
|
||||
4. **Recovery clear** — a socat forwarder (`5021 → 5020` on the docker host) brought the endpoint
|
||||
alive; both centrals logged `circuit-breaker closed (recovered)` + `reconnected` within 1 s of
|
||||
each other, and the panel dropped to "All targets healthy."
|
||||
5. **Teardown** — socat removed; 5021 dead again (as-found). rtu-gate returns to a truthfully-OPEN
|
||||
breaker chip, which is the feature working.
|
||||
|
||||
### Two nuances against the original T10 wording
|
||||
|
||||
- **"Both-replica chips"** resolves as both nodes' *trackers* tripping (proven in both logs)
|
||||
rendered through the single **active** AdminUI. Traefik's `/health/active` check deliberately
|
||||
routes the UI to the active admin node only — the standby returns 503
|
||||
(`Standby: active node for role 'admin' is …`), so there is never a second replica serving
|
||||
chips. Discovered while trying to force the sticky-cookie LB onto central-1.
|
||||
- **The staleness-dim leg was not exercised.** Both pair nodes publish under the same
|
||||
`(instance, host)` key, so staleness requires *every* publisher for a key to stop — deliberate
|
||||
node-kill churn this gate did not do. The dim is a pure render predicate (`IsStale` → opacity).
|
||||
|
||||
Also corrected while here: §4's "ConnectionBackoff throttles retries" explanation (the previously
|
||||
recorded root cause) is confirmed wrong — the breaker config opens on 2 failures in 30 s; the
|
||||
operative cause was always that no failure reached the pipeline at all.
|
||||
|
||||
Reference in New Issue
Block a user