lmxopcua/archreview/plans/artifacts/456-retry-breaker-live-finding-2026-07-15.md

#456 item 1 — live retry/breaker log-line capture: structural finding (2026-07-15)

Outcome: the log line was not captured, but the "grind" produced a definitive structural finding that supersedes the issue's acceptance framing: the trigger the issue proposes (an idempotent WriteAsync to a dead endpoint) is structurally incapable of emitting a retry/breaker line, and the line is reachable in production only for a session-based driver (OpcUaClient) faulting mid-Discover/Subscribe — a racy transient that cannot be deterministically forced on the dev rig. This is why three sessions with polling-driver fixtures never observed it.

What was actually driven live (the grind)

Full end-to-end wrapped-dispatch exercise on the docker-dev central cluster (central-1/central-2, MAIN):

1. Brought up the modbus sim on the shared host (10.100.0.35:5020, otopcua-pymodbus-standard).
2. Authored a complete Modbus equipment hierarchy via SQL (MAIN cluster): Equipment-kind Namespace ns-mb-eq → DriverInstance drv-mb (Modbus, {"Host":"10.100.0.35","Port":5020,"UnitId":1}) → Device → UnsArea/UnsLine → Equipment EQ-111111111111 → writable Tag reg1 ({"region":"HoldingRegisters","address":1,"dataType":"Int16","byteOrder":"BigEndian","writable":true}, AccessLevel=ReadWrite, WriteIdempotent=1).
3. Deployed headless (POST :9200/api/deployments). Runtime log: spawned Modbus driver drv-mb → DriverInstance drv-mb: connected → subscribed to 1 refs. Node materialized at ns=2;s=EQ-111111111111/reg1.
4. Baseline (sim up): live read Good (value 1); LDAP-authed (multi-role, WriteOperate role via the shared GLAuth data-plane GroupToRole) write = 123 → Write successful. The wrapped write path is fully live end-to-end (OPC UA write → NodeWriteRouter → DriverHostActor.RouteNodeWrite → DriverInstanceActor.WriteAsync invoker-wrapped site → ModbusDriver → sim).
5. Fault: docker pause otopcua-pymodbus-standard (frozen peer — driver stays Connected), then drove 4 LDAP-authed writes into the frozen peer.
6. Result: every write logged Operator write to EQ-111111111111/reg1 rejected: StatusCode=0x80020000 (BadInternalError), node reverted. grep "Driver resilience retry|circuit-breaker OPENED" on BOTH centrals = 0 matches despite 4 exercised, failing wrapped writes.

Why the write trigger is structurally impossible

• ModbusDriver.WriteAsync swallows every exception and returns a bad WriteResult status — it never throws (ModbusDriver.cs:964-967: catch (Exception) { results[i] = new WriteResult(StatusBadInternalError); } → 0x80020000, exactly the observed reject status).
• The CapabilityInvoker only feeds exceptions to the Polly pipeline (CapabilityInvoker.cs:84/112/147: await pipeline.ExecuteAsync(callSite, …) — the callSite is the driver's WriteAsync; a non-throwing return is a pipeline success). The invoker does not inspect the returned WriteResult status.
• The circuit breaker's ShouldHandle is Handle<Exception> (DriverResiliencePipelineBuilder.cs), and Write retry is pinned to 0 as an invariant (R2-02/S-8, DriverResilienceOptions.cs:64-66).
• ∴ a failing wrapped write produces no exception → no retry, no breaker count, no log line — by construction, for any driver that maps write failures to a status (all polling drivers do). The issue's hypothesis ("the only invoker-wrapped Modbus site that throws is an idempotent WriteAsync to a dead endpoint") is incorrect — even an idempotent write to a truly dead endpoint returns StatusBadInternalError rather than throwing.

Where the line IS reachable (and why it's elusive)

The retry/breaker pipeline fires only when a wrapped capability call throws. Of the 6 wrapped sites in DriverInstanceActor (Write, Acknowledge, Subscribe, Unsubscribe, SubscribeAlarms, Discover):

• Polling drivers (Modbus/S7/AbCip/AbLegacy/TwinCAT/Focas): WriteAsync/AcknowledgeAsync swallow → status; SubscribeAsync is a lazy poll-registration (no connect, no throw); DiscoverAsync is offline/config-driven (enumerates authored tags, no connect). None of their wrapped sites throw a transient exception through the seam.
• Session drivers (OpcUaClient): DiscoverAsync (remote browse) and SubscribeAsync (CreateSubscription/CreateMonitoredItems) require a live session and do throw on a session fault. The post-connect discovery loop (DriverInstanceActor.StartDiscovery, honouring RediscoverPolicy) runs DiscoverAsync under a 30 s Polly timeout on each Connected entry.

So the only way to emit the line is a session driver whose session faults while a wrapped Discover/Subscribe is in flight (e.g. docker pause of a live OPC UA endpoint mid-discovery → the 30 s timeout throws TimeoutRejectedException → Discover retry ×2 → Driver resilience retry). This is a genuine production event under real network faults, but it is a timing race — connect must succeed, then the fault must land during the wrapped call, before the session-keepalive drops the driver to Reconnecting (where the wrapped sites no longer fire). It cannot be deterministically forced on the rig with the available fixtures (opc-plc was down; no on-demand rediscovery trigger).

Conclusion / recommendation

• Item 2 (the real code gap — ResilienceConfig in the deploy artifact) is done + live-verified on both nodes (PR #461).
• Item 1 is deterministically proven by the DriverResiliencePipelineBuilder logging unit test (the OnRetry/OnOpened handlers emit the exact lines) plus CapabilityInvoker retry tests and the factory override test. The live capture is not deterministically achievable from the dev rig — and, more importantly, the grind proved the issue's proposed write-trigger is structurally impossible. The line is an OpcUaClient-only, transient-fault-mid-Discover/Subscribe event.
• Recommend closing item 1 on this finding: the observability surface is unit-proven and the wrapped dispatch path is now live-confirmed end-to-end (authored driver → connected → LDAP write → wrapped WriteAsync → faulted → surfaced). The remaining "eyeball the raw line" is an operator/staging observation during a real OpcUaClient session fault, not a code deliverable.