feat(resilience): route IDriver.InitializeAsync through the capability invoker (R2-10 T10 closed)
The connect path was the one capability call the resilience pipeline never saw, so a driver that could not connect contributed zero breaker input and the /hosts resilience panel read 'All targets healthy' against a dead endpoint. InitializeAsync now routes through the invoker as DriverCapability.Initialize, keyed on the driver-instance id. - No pipeline retry: the actor's RetryConnect timer is the retry loop (parser forces retryCount overrides to 0, same funnel as write-shaped). - Breaker threshold 2, not the Read-sized 5: the breaker samples 30 s and the reconnect timer fires every 10 s, so at most three attempts fit one window — threshold 5 could never open. Pinned by its own test. - An open breaker fast-fails into the same InitializeFailed path as a real connect failure; the 15 s break window costs at most one extra tick. The best-effort teardown ShutdownAsync stays unwrapped. - Companion fix: DriverCapabilityInvokerFactory.Create now invalidates the instance's tracker snapshots with its pipelines — a respawn during an outage discarded the only breaker whose OnClosed could clear the OPEN flag, stranding a false 'Breaker OPEN' chip forever. Live gate PASSED on docker-dev (image symbol-verified per container): rtu-gate against dead 10.100.0.35:5021 opened its Initialize breaker on both centrals with the red 'BREAKER OPEN' chip on /hosts; a socat 5021→5020 forwarder recovered it (both breakers closed within 1 s, panel back to 'All targets healthy'); healthy MAIN-modbus recorded zero breaker events. Full suite 50/50 green. Evidence: docs/plans/2026-08-07-adminui-followups-and-live-gates.md §7.
This commit is contained in:
@@ -215,3 +215,67 @@ original one-line body, so it mirrors `MainPressure` again rather than the `×6`
|
||||
One anomaly recorded rather than explained: both centrals were **recreated** at 09:49:40/09:49:50
|
||||
(new `Created`, `RestartCount=0`) by something this session did not issue. It postdates the
|
||||
observations above and the old container's logs went with it, so it is noted, not attributed.
|
||||
|
||||
---
|
||||
|
||||
## 7. Addendum 2026-08-08 — R2-10 T10 ✅ PASSED after fixing the structural gap
|
||||
|
||||
§4 concluded the gate was not achievable as specified because the connect path never entered the
|
||||
resilience pipeline. The user chose to fix that ("do option 1"), and the gate then passed against
|
||||
the fixed build. Branch `feat/wrap-driver-init-in-resilience`.
|
||||
|
||||
### The fix
|
||||
|
||||
- **`DriverCapability.Initialize`** — `IDriver.InitializeAsync` routes through the capability
|
||||
invoker in `DriverInstanceActor.InitializeAsync` (both the initial connect and the
|
||||
`Reconnecting` retry path; the best-effort teardown `ShutdownAsync` stays deliberately
|
||||
unwrapped). Keyed on the driver-instance id, like a bulk subscribe.
|
||||
- **No pipeline retry for Initialize** — the actor's `RetryConnect` timer is the retry loop.
|
||||
Enforced at the parser funnel (`retryCount` override forced to 0, same shape as the
|
||||
write-shaped rule) and in every tier default.
|
||||
- **Breaker threshold 2, not the Read-sized 5** — the breaker samples 30 s and the reconnect
|
||||
timer fires every 10 s, so at most three init attempts fit one window; threshold 5 could never
|
||||
open. Pinned by `Initialize_BreakerThreshold_IsReachableAtTheReconnectCadence` +
|
||||
`Initialize_BreakerOpens_AtTierDefaults_AndFastFailsTheNextAttempt`.
|
||||
- **Companion fix — respawn resets tracker snapshots.** `DriverCapabilityInvokerFactory.Create`
|
||||
now calls `DriverResilienceStatusTracker.Invalidate(driverInstanceId)` alongside the pipeline
|
||||
invalidation. Without it, a deploy during an outage discards the only breaker whose `OnClosed`
|
||||
could clear the OPEN flag, stranding a false "Breaker OPEN" chip forever. Unreachable before
|
||||
this change made breaker-open reachable; found by asking what clears the flag, not by a test.
|
||||
- Wiring pinned by `Initialize_routes_through_the_capability_invoker` +
|
||||
`An_invoker_fast_fail_on_init_keeps_the_actor_retrying_until_recovery` (falsification-controlled:
|
||||
both go red with the actor wrap stashed; `Create_resets_the_instances_tracker_state_with_its_pipelines`
|
||||
goes red with the factory call stashed).
|
||||
|
||||
### The gate run (docker-dev, image `df04c76ce06b`, symbol-verified on both centrals)
|
||||
|
||||
The staged failure was free: `rtu-gate` (`DRV-3616122eb53d`) was already authored against
|
||||
`10.100.0.35:5021`, where nothing listens — connection refused on every init.
|
||||
|
||||
1. **Breaker OPEN, both replicas** — within ~20 s of boot, both centrals logged
|
||||
`circuit-breaker OPENED: instance=DRV-3616122eb53d … capability=Initialize … Connection refused`,
|
||||
re-opening each half-open probe cycle. Bonus coverage: `DRV-opcua1` (invalid config → init
|
||||
throws) also tripped — a permanently-misconfigured driver was equally invisible before.
|
||||
2. **Panel chip** — `/hosts` → rtu-gate → Controls showed the red **"BREAKER OPEN · OPENED 19S
|
||||
AGO"** chip in the RESILIENCE section, row FAULTED with "Connection refused".
|
||||
3. **Control** — healthy `MAIN-modbus` recorded zero breaker events throughout.
|
||||
4. **Recovery clear** — a socat forwarder (`5021 → 5020` on the docker host) brought the endpoint
|
||||
alive; both centrals logged `circuit-breaker closed (recovered)` + `reconnected` within 1 s of
|
||||
each other, and the panel dropped to "All targets healthy."
|
||||
5. **Teardown** — socat removed; 5021 dead again (as-found). rtu-gate returns to a truthfully-OPEN
|
||||
breaker chip, which is the feature working.
|
||||
|
||||
### Two nuances against the original T10 wording
|
||||
|
||||
- **"Both-replica chips"** resolves as both nodes' *trackers* tripping (proven in both logs)
|
||||
rendered through the single **active** AdminUI. Traefik's `/health/active` check deliberately
|
||||
routes the UI to the active admin node only — the standby returns 503
|
||||
(`Standby: active node for role 'admin' is …`), so there is never a second replica serving
|
||||
chips. Discovered while trying to force the sticky-cookie LB onto central-1.
|
||||
- **The staleness-dim leg was not exercised.** Both pair nodes publish under the same
|
||||
`(instance, host)` key, so staleness requires *every* publisher for a key to stop — deliberate
|
||||
node-kill churn this gate did not do. The dim is a pure render predicate (`IsStale` → opacity).
|
||||
|
||||
Also corrected while here: §4's "ConnectionBackoff throttles retries" explanation (the previously
|
||||
recorded root cause) is confirmed wrong — the breaker config opens on 2 failures in 30 s; the
|
||||
operative cause was always that no failure reached the pipeline at all.
|
||||
|
||||
Reference in New Issue
Block a user