feat(resilience): route IDriver.InitializeAsync through the capability invoker (R2-10 T10 closed)

The connect path was the one capability call the resilience pipeline never
saw, so a driver that could not connect contributed zero breaker input and
the /hosts resilience panel read 'All targets healthy' against a dead
endpoint. InitializeAsync now routes through the invoker as
DriverCapability.Initialize, keyed on the driver-instance id.

- No pipeline retry: the actor's RetryConnect timer is the retry loop
  (parser forces retryCount overrides to 0, same funnel as write-shaped).
- Breaker threshold 2, not the Read-sized 5: the breaker samples 30 s and
  the reconnect timer fires every 10 s, so at most three attempts fit one
  window — threshold 5 could never open. Pinned by its own test.
- An open breaker fast-fails into the same InitializeFailed path as a real
  connect failure; the 15 s break window costs at most one extra tick.
  The best-effort teardown ShutdownAsync stays unwrapped.
- Companion fix: DriverCapabilityInvokerFactory.Create now invalidates the
  instance's tracker snapshots with its pipelines — a respawn during an
  outage discarded the only breaker whose OnClosed could clear the OPEN
  flag, stranding a false 'Breaker OPEN' chip forever.

Live gate PASSED on docker-dev (image symbol-verified per container):
rtu-gate against dead 10.100.0.35:5021 opened its Initialize breaker on
both centrals with the red 'BREAKER OPEN' chip on /hosts; a socat
5021→5020 forwarder recovered it (both breakers closed within 1 s, panel
back to 'All targets healthy'); healthy MAIN-modbus recorded zero breaker
events. Full suite 50/50 green. Evidence:
docs/plans/2026-08-07-adminui-followups-and-live-gates.md §7.
This commit is contained in:
Joseph Doherty
2026-08-08 05:45:08 -04:00
parent c9f182b2d7
commit dac7e1885d
18 changed files with 445 additions and 8 deletions
@@ -21,7 +21,7 @@ namespace ZB.MOM.WW.OtOpcUa.AdminUI.Components.Shared.Drivers;
public sealed class ResilienceFormModel
{
public static readonly string[] Capabilities =
["Read", "Write", "Discover", "Subscribe", "Probe", "AlarmSubscribe", "AlarmAcknowledge", "HistoryRead"];
["Read", "Write", "Discover", "Subscribe", "Probe", "AlarmSubscribe", "AlarmAcknowledge", "HistoryRead", "Initialize"];
// RecycleIntervalSeconds is GONE (Gitea #522) — the scheduled-recycle machinery it authored was
// deleted, so the field configured nothing. It is deliberately NOT explicitly stripped from stored
@@ -687,6 +687,7 @@ public sealed class DriverInstanceActor : ReceiveActor, IWithTimers
var log = _log;
var driverInstanceId = _driverInstanceId;
var driver = _driver;
var invoker = _invoker;
_ = Task.Run(async () =>
{
try
@@ -695,6 +696,8 @@ public sealed class DriverInstanceActor : ReceiveActor, IWithTimers
{
try
{
// Deliberately UNWRAPPED: teardown is best-effort cleanup of an already-dead
// transport — a shutdown failure must not count toward the Initialize breaker.
await driver.ShutdownAsync(CancellationToken.None);
}
catch (Exception ex)
@@ -704,7 +707,20 @@ public sealed class DriverInstanceActor : ReceiveActor, IWithTimers
"initialising anyway", driverInstanceId);
}
}
await driver.InitializeAsync(driverConfigJson, CancellationToken.None);
// R2-10 T10: the connect path was the ONE capability call the invoker never saw, so a
// driver that could not connect contributed zero breaker input and the resilience panel
// stayed quiet for a dead endpoint. Keyed on the driver-instance id (init spans the
// whole driver, like a bulk subscribe). The Initialize policy never retries — this
// actor's RetryConnect timer is the retry loop — and its pipeline timeout now bounds a
// hanging connect via the ct the call site must honour. An open breaker fast-fails the
// attempt with BrokenCircuitException, which lands in the catch below exactly like a
// real connect failure: InitializeFailed → keep retrying on the timer. The 15 s break
// window is shorter than two 10 s timer ticks, so recovery costs at most one extra tick.
await invoker.ExecuteAsync(
DriverCapability.Initialize,
driverInstanceId,
async ct => await driver.InitializeAsync(driverConfigJson, ct),
CancellationToken.None);
self.Tell(new InitializeSucceeded(generation));
}
catch (Exception ex)