feat(resilience): route IDriver.InitializeAsync through the capability invoker (R2-10 T10 closed)

The connect path was the one capability call the resilience pipeline never
saw, so a driver that could not connect contributed zero breaker input and
the /hosts resilience panel read 'All targets healthy' against a dead
endpoint. InitializeAsync now routes through the invoker as
DriverCapability.Initialize, keyed on the driver-instance id.

- No pipeline retry: the actor's RetryConnect timer is the retry loop
  (parser forces retryCount overrides to 0, same funnel as write-shaped).
- Breaker threshold 2, not the Read-sized 5: the breaker samples 30 s and
  the reconnect timer fires every 10 s, so at most three attempts fit one
  window — threshold 5 could never open. Pinned by its own test.
- An open breaker fast-fails into the same InitializeFailed path as a real
  connect failure; the 15 s break window costs at most one extra tick.
  The best-effort teardown ShutdownAsync stays unwrapped.
- Companion fix: DriverCapabilityInvokerFactory.Create now invalidates the
  instance's tracker snapshots with its pipelines — a respawn during an
  outage discarded the only breaker whose OnClosed could clear the OPEN
  flag, stranding a false 'Breaker OPEN' chip forever.

Live gate PASSED on docker-dev (image symbol-verified per container):
rtu-gate against dead 10.100.0.35:5021 opened its Initialize breaker on
both centrals with the red 'BREAKER OPEN' chip on /hosts; a socat
5021→5020 forwarder recovered it (both breakers closed within 1 s, panel
back to 'All targets healthy'); healthy MAIN-modbus recorded zero breaker
events. Full suite 50/50 green. Evidence:
docs/plans/2026-08-07-adminui-followups-and-live-gates.md §7.
This commit is contained in:
Joseph Doherty
2026-08-08 05:45:08 -04:00
parent c9f182b2d7
commit dac7e1885d
18 changed files with 445 additions and 8 deletions
@@ -215,3 +215,67 @@ original one-line body, so it mirrors `MainPressure` again rather than the `×6`
One anomaly recorded rather than explained: both centrals were **recreated** at 09:49:40/09:49:50
(new `Created`, `RestartCount=0`) by something this session did not issue. It postdates the
observations above and the old container's logs went with it, so it is noted, not attributed.
---
## 7. Addendum 2026-08-08 — R2-10 T10 ✅ PASSED after fixing the structural gap
§4 concluded the gate was not achievable as specified because the connect path never entered the
resilience pipeline. The user chose to fix that ("do option 1"), and the gate then passed against
the fixed build. Branch `feat/wrap-driver-init-in-resilience`.
### The fix
- **`DriverCapability.Initialize`** — `IDriver.InitializeAsync` routes through the capability
invoker in `DriverInstanceActor.InitializeAsync` (both the initial connect and the
`Reconnecting` retry path; the best-effort teardown `ShutdownAsync` stays deliberately
unwrapped). Keyed on the driver-instance id, like a bulk subscribe.
- **No pipeline retry for Initialize** — the actor's `RetryConnect` timer is the retry loop.
Enforced at the parser funnel (`retryCount` override forced to 0, same shape as the
write-shaped rule) and in every tier default.
- **Breaker threshold 2, not the Read-sized 5** — the breaker samples 30 s and the reconnect
timer fires every 10 s, so at most three init attempts fit one window; threshold 5 could never
open. Pinned by `Initialize_BreakerThreshold_IsReachableAtTheReconnectCadence` +
`Initialize_BreakerOpens_AtTierDefaults_AndFastFailsTheNextAttempt`.
- **Companion fix — respawn resets tracker snapshots.** `DriverCapabilityInvokerFactory.Create`
now calls `DriverResilienceStatusTracker.Invalidate(driverInstanceId)` alongside the pipeline
invalidation. Without it, a deploy during an outage discards the only breaker whose `OnClosed`
could clear the OPEN flag, stranding a false "Breaker OPEN" chip forever. Unreachable before
this change made breaker-open reachable; found by asking what clears the flag, not by a test.
- Wiring pinned by `Initialize_routes_through_the_capability_invoker` +
`An_invoker_fast_fail_on_init_keeps_the_actor_retrying_until_recovery` (falsification-controlled:
both go red with the actor wrap stashed; `Create_resets_the_instances_tracker_state_with_its_pipelines`
goes red with the factory call stashed).
### The gate run (docker-dev, image `df04c76ce06b`, symbol-verified on both centrals)
The staged failure was free: `rtu-gate` (`DRV-3616122eb53d`) was already authored against
`10.100.0.35:5021`, where nothing listens — connection refused on every init.
1. **Breaker OPEN, both replicas** — within ~20 s of boot, both centrals logged
`circuit-breaker OPENED: instance=DRV-3616122eb53d … capability=Initialize … Connection refused`,
re-opening each half-open probe cycle. Bonus coverage: `DRV-opcua1` (invalid config → init
throws) also tripped — a permanently-misconfigured driver was equally invisible before.
2. **Panel chip** — `/hosts` → rtu-gate → Controls showed the red **"BREAKER OPEN · OPENED 19S
AGO"** chip in the RESILIENCE section, row FAULTED with "Connection refused".
3. **Control** — healthy `MAIN-modbus` recorded zero breaker events throughout.
4. **Recovery clear** — a socat forwarder (`5021 → 5020` on the docker host) brought the endpoint
alive; both centrals logged `circuit-breaker closed (recovered)` + `reconnected` within 1 s of
each other, and the panel dropped to "All targets healthy."
5. **Teardown** — socat removed; 5021 dead again (as-found). rtu-gate returns to a truthfully-OPEN
breaker chip, which is the feature working.
### Two nuances against the original T10 wording
- **"Both-replica chips"** resolves as both nodes' *trackers* tripping (proven in both logs)
rendered through the single **active** AdminUI. Traefik's `/health/active` check deliberately
routes the UI to the active admin node only — the standby returns 503
(`Standby: active node for role 'admin' is …`), so there is never a second replica serving
chips. Discovered while trying to force the sticky-cookie LB onto central-1.
- **The staleness-dim leg was not exercised.** Both pair nodes publish under the same
`(instance, host)` key, so staleness requires *every* publisher for a key to stop — deliberate
node-kill churn this gate did not do. The dim is a pure render predicate (`IsStale` → opacity).
Also corrected while here: §4's "ConnectionBackoff throttles retries" explanation (the previously
recorded root cause) is confirmed wrong — the breaker config opens on 2 failures in 30 s; the
operative cause was always that no failure reached the pipeline at all.