Ran the full item-1 grind (authored a live Modbus equipment tag, connected
to the sim, LDAP-authed writes, docker pause the peer). 4 failing wrapped
writes produced 0 retry/breaker lines on either central.
Root cause (corrects the issue's hypothesis): ModbusDriver.WriteAsync swallows
all exceptions and returns WriteResult(StatusBadInternalError) — never throws;
CapabilityInvoker feeds only exceptions to Polly; breaker ShouldHandle is
Handle<Exception>; Write retry pinned to 0 (R2-02/S-8). So a failing wrapped
write emits no line by construction, for any polling driver. The line is
reachable ONLY for a session driver (OpcUaClient) faulting mid-Discover/Subscribe
(30s Polly timeout throws) — a production timing race, not deterministically
forcible on the rig. Behaviour stays unit-proven by the pipeline-builder test.
- New: archreview/plans/artifacts/456-retry-breaker-live-finding-2026-07-15.md
- FOLLOWUP-10 updated with the structural finding + recommendation to close item 1.
2-container docker-dev rig: hard-killing the oldest node makes the younger
survivor down ITSELF (SBR DownReachable including myself -> CoordinatedShutdown)
-> no failover, total outage. Reproduced twice. Confirms #459 is REAL and
disproves Option A (keep keep-oldest). Baseline ServiceLevel 250(leader)/240(follower)
captured before the kill. Documents why (textbook 2-node keep-oldest limitation;
down-if-alone doesn't rescue a crashed oldest) and why the in-process
HardKillFailoverTests gave false confidence (Transport.Shutdown keeps the
ActorSystem alive, not a real crash). Fix needs Option B (witness+keep-majority)
or Option C (DB-lease arbiter).
Captures the docker-dev live verification for follow-up #10 acceptance item 2:
a non-default per-instance ResilienceConfig (Subscribe.retryCount:999) authored on
MAIN rode the deploy artifact into BOTH central-1/central-2 runtimes, which parsed
and clamped it — the authored value appears verbatim in the spawn-time diagnostic.
Confirms the artifact bytes carry ResilienceConfig and the runtime pipeline applies
authored (not tier-default) policy. Item 1 (raw retry/breaker-on-fault log line)
remains, with the connect-then-idempotent-write recipe documented.
Joseph Doherty