lmxopcua/archreview/plans/R2-09-driver-fleet-batch-plan.md

Design + Implementation Plan — R2-09 Driver Fleet Carried Batch

• Source report: archreview/05-protocol-drivers.md (2026-07-12 re-review)
• Overall action item: #9 in archreview/00-OVERALL.md — "Driver fleet carried batch: Modbus fatal-classification, AbCip/FOCAS locking, ResolveHost, onError, TwinCAT replay retry + Bad-quality"
• Plan verified against tree at: f6eaa267 (master, clean) — every anchor below re-opened and re-located in the working tree on 2026-07-12
• Round-1 plan built on: archreview/plans/05-protocol-drivers-plan.md Parts A (A3/A4/A5) and B (B1 PollGroupEngine v2, B3 ResolveHost-via-resolver, B4 ConnectionBackoff) — designs verified still valid and extended below
• Scope: src/Drivers/ fleet findings — 05/STAB-3, 05/STAB-4, 05/STAB-5, 05/STAB-8 (fleet part), 05/STAB-9 + CONV-1 + PERF-3 + STAB-6-S7 (PollGroupEngine v2 batch), CONV-4 (ResolveHost), 05/STAB-16, 05/STAB-17, 05/STAB-12 (TwinCAT-compounded part only)
• Scope boundary (do NOT duplicate):
  • Plan R2-01 owns the S7-specific OCE/fatal-classification fixes (STAB-14 ensure-wrapper OCE filter at S7Driver.cs:488/:1000, STAB-15 framing/cancel fatal-classification + probe mark-dead, and the S7 EnsureConnectedAsync attempt-throttle wiring). This plan hands R2-01 two seams: the ConnectionBackoff primitive (sub-batch B3/B4) and the engine-level OCE hardening that makes the S7 fork retirement safe (B3). Sequencing notes inline.
  • Plan R2-11 owns ReadEnum/TagConfig parse-strictness consolidation (CONV-2/UNDER-6). B5 adds exactly one key (unitId) to ModbusEquipmentTagParser because per-host breaker keying is impossible without it; all other parser-parity work stays with R2-11.

---

Verification summary

Every finding was re-opened at its cited file:line in the working tree at f6eaa267. All 10 findings CONFIRMED; 2 anchors drifted (details below); 0 stale.

Finding	Cited anchor	Verified at f6eaa267
STAB-3	ModbusTcpTransport.cs:155-165, 233-235, 257-261	EXACT — retry catch :155-165 (only IsSocketLevelFailure), per-op linked CTS CancelAfter :226-227, TxId-mismatch InvalidDataException :233-235, truncated-length :236-237, IsSocketLevelFailure :257-261, TearDownAsync :263-270
STAB-4	AbCipDriver.cs:544-575, 614-637, 708-726 (prompt's "~548-579 +4" was conservative)	EXACT to the report — ReadSingleAsync critical section :544-575, ReadGroupAsync :614-637, WriteAsync Encode→Write→GetStatus :708-726; zero runtime locking (only GetRmwLock at DeviceState:1243, bit-RMW only); alarm-projection read loop AbCipAlarmProjection.cs:230-232 hits the same handles; AbLegacy's guard confirmed at AbLegacyDriver.cs:227-244 (read) / :331-342 (write) / _runtimeLocks + GetRuntimeLock :773-787 / dispose :841-842
STAB-5	FocasDriver.cs:1184-1194, 752-839, 912-945	EXACT — plain Dictionary fields LastFixedSnapshots :1184, LastTimers :1190, LastServoLoads :1192, LastSpindleLoads :1194; loop writes :700, :752, :793, :836-839, :862-863, :874; reader-thread reads :878, :912, :920, :945; contrast _parsedAddressesByTagName already ConcurrentDictionary :40
STAB-9 (dead onError)	5 ctor sites	EXACT — ModbusDriver.cs:115, AbCipDriver.cs:133, AbLegacyDriver.cs:65, TwinCATDriver.cs:69, FocasDriver.cs:72 — all five pass reader:/onChange: only; PollGroupEngine.cs onError param :59, ReportError :164-169 dead fleet-wide
CONV-1/PERF-3/STAB-6-S7 (S7 fork)	S7Driver.cs:1373-1412, 1447, 1469, 1350-1354	EXACT — fork PollLoopAsync :1373-1412 (bare catch (OCE) { return; } :1383, :1397, :1404), HandlePollFailure :1421-1437, ComputeBackoffDelay :1447-1456, reference-equality diff :1469, UnsubscribeAsync cancel+dispose-without-drain :1350-1354. Engine already has everything the fork lacks: StopState drain PollGroupEngine.cs:99-112, structural array compare :203-209, ObjectDisposedException guard :136 — and lacks what the fork has: backoff + failure→health
STAB-8 (fleet)	AbCip :891-897, FOCAS :1089-1120, TwinCAT :706-721	CONFIRMED — EvictRuntime AbCipDriver.cs:891-897 + EnsureTagRuntimeAsync :852-882 recreate at full Forward-Open cost per call; FOCAS EnsureConnectedAsync :1089-1120 full two-socket reconnect per call, no throttle; TwinCAT EnsureConnectedAsync :706-721 / EnsureConnectedUnderGateAsync :731-775 connect per call, no throttle. Only Modbus transport (ConnectWithBackoffAsync :179-208) and the S7 poll loop have any backoff
CONV-4 (ResolveHost)	5 sites	EXACT — AbCipDriver.cs:483-488, AbLegacyDriver.cs:496-501, TwinCATDriver.cs:689-696 (falls back to UnresolvedHostSentinel "", :686), FocasDriver.cs:1082-1087, ModbusDriver.cs:125-131 (BuildSlaveHostName :134, ResolveUnitId :474). All consult _tagsByName only. All four multi-device TDefs carry DeviceHostAddress and all four equipment parsers parse deviceHostAddress (AbCip parser :39, AbLegacy :42, TwinCAT :30, FOCAS :32 — FOCAS defaults to "" when absent). ModbusEquipmentTagParser has no unitId key (grep-confirmed)
STAB-16	TwinCATDriver.cs:801-816	CONFIRMED, minor drift — the replay-failure catch is :801-807 (log-only; SwapHandle runs only in the success path :797), summary log :810-813. Failed intent keeps its dead handle, OnChange is Action<string, object?> (no quality channel, :575), replay runs only from EnsureConnectedUnderGateAsync :773 (i.e. only when a NEW client is installed — a healthy client never retries)
STAB-17	TwinCATDriver.cs:608-613, 786-803	EXACT — NativeRegistration.Dispose :607-613 is gate-free (registry TryRemove :610, handle exchange+dispose :611-612); replay snapshots intents :786 and SwapHandles :797 with no ownership re-check, so a dispose between snapshot and swap leaves the fresh handle ownerless and the ADS notification live
STAB-12 (compounded part)	TwinCATDriver.cs:835	DRIFTED −9 — the ConnectGate.WaitAsync(CancellationToken.None) is at TwinCATDriver.cs:826 (method RecycleClientAsync :824-840); callers are the probe loop :640 and :648, both holding a live ct

Drifted anchors (2): STAB-12's RecycleClientAsync gate-wait :835 → :826; STAB-16's cited span :801-816 is really catch :801-807 + summary log :810-813 (same method, cosmetic). Everything else verified exact.

Test infrastructure verified: PollGroupEngineTests.cs (Core.Abstractions.Tests), ModbusTcpReconnectTests.cs with in-proc FlakeyModbusServer (TcpListener-based — extendable for stall/wrong-TxId/truncation), AbLegacyRuntimeConcurrencyTests.cs (the template for B2), FakeAbCipTag.cs, FakeFocasClient.cs, TwinCATReconnectReplayTests.cs (fake ITwinCATClient), S7DriverReconnectTests.cs + S7DiscoveryAndSubscribeTests.cs (fake IS7PlcFactory), env-gated S7_1500ReconnectTests.cs.

Live gates: docker fixtures on 10.100.0.35 via lmxopcua-fix: modbus (:5020, lmxopcua-fix up modbus standard; the exception_injector container can force write/FC failures — see the write-outcome memory recipe), abcip (:44818, up abcip controllogix), s7 (:1102, up s7 s7_1500; bounce = down/up for reconnect recipes), opcuaclient (:50000). TwinCAT, FOCAS, and AbLegacy have NO fixture — the fake-client unit suites are authoritative there (accepted in the STAB-2 remediation and re-affirmed by UNDER-7).

---

Sub-batch B1 — Modbus transport fatal-classification (STAB-3, High)

Independently shippable. No dependencies.

STAB-3 — Modbus transport desynchronizes permanently after a response timeout or TxId mismatch

Restatement: a per-op timeout (OperationCanceledException from the linked CancelAfter CTS) or a framing violation (TxId mismatch / truncated header → InvalidDataException) is not classified socket-fatal, so the single-flight socket is kept with a half-read or still-in-flight response. The next transaction reads the stale bytes, throws on the TxId guard — also not socket-fatal — and the stream stays desynchronized forever (until a real socket error or the default-off idle-disconnect).

Verification: CONFIRMED at src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Modbus/ModbusTcpTransport.cs. SendAsync's reconnect-retry catch (:155) filters on IsSocketLevelFailure (:257-261 — only EndOfStreamException/IOException/SocketException/ObjectDisposedException). The per-op deadline is a linked CTS CancelAfter(_timeout) (:226-227) whose cancellation surfaces as OCE; the TxId guard throws InvalidDataException (:234-235); the length guard throws InvalidDataException (:237). None reach TearDownAsync (:263-270). Note the 05 report explicitly pairs this with S7's STAB-15 ("same defect class in two drivers") — the S7 side is R2-01's; only the Modbus transport is fixed here.

Root cause: the transport conflates "protocol-layer error, socket still coherent" (Modbus exception PDU — correctly kept) with "framing/timeout error, unknown bytes may be in flight — the socket is unusable". Timeout and framing violations require teardown regardless of exception type.

Proposed design (unchanged from round-1 plan A3, verified still correct): treat per-op timeout and any framing violation as connection-fatal — tear down before propagating, normalized as an IOException subclass so the existing retry classifier covers it with zero changes:

• New ModbusTransportDesyncException : IOException (carries the desync reason: Timeout, TxIdMismatch, TruncatedFrame).
• In SendOnceAsync (:210-250), wrap the write/read/parse body:
  • catch (OperationCanceledException) when (!ct.IsCancellationRequested) — the linked timeout fired, not the caller ⇒ await TearDownAsync() ⇒ throw new ModbusTransportDesyncException(...). Caller cancellation (ct.IsCancellationRequested) propagates as OCE with NO teardown — a legitimate shutdown must not look like a desync.
  • catch (InvalidDataException ide) ⇒ await TearDownAsync() ⇒ throw new ModbusTransportDesyncException(..., ide).
• Because ModbusTransportDesyncException is an IOException, SendAsync's existing single-retry (:155-165) now reconnects and resends once when _autoReconnect is on; when _autoReconnect is off the socket is still torn down (mandatory — a desynced socket must never be reused) and the exception propagates to the driver's status mapping (BadCommunicationError).

Alternative considered: add OperationCanceledException/InvalidDataException to IsSocketLevelFailure — rejected: the retry catch runs teardown only when auto-reconnect is on, and OCE must be split caller-vs-timeout at the throw site anyway. The teardown-at-throw-site design covers auto-reconnect-off and keeps IsSocketLevelFailure a pure type test.

Implementation steps:

1. src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Modbus/ModbusTransportDesyncException.cs — new file, sealed class ModbusTransportDesyncException : IOException.
2. src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Modbus/ModbusTcpTransport.cs — wrap SendOnceAsync body per design (two catch clauses + teardown). No change to SendAsync, IsSocketLevelFailure, or ConnectWithBackoffAsync.
3. XML-doc the timeout-vs-caller-cancel distinction on SendOnceAsync.

Tests (extend tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Modbus.Tests/ModbusTcpReconnectTests.cs — its FlakeyModbusServer (TcpListener, :22-35) gains scriptable behaviors):

• SendAsync_AfterResponseTimeout_TearsDownAndReconnects — server stalls past the transport timeout on request N, answers correctly on N+1; assert request N throws (Bad path), the socket was replaced (server sees a new connection), and N+1 succeeds with the CORRECT response (no stale-read).
• SendAsync_AfterTxIdMismatch_TearsDownSocket — server answers with a wrong TxId; assert ModbusTransportDesyncException (or successful single-retry when auto-reconnect on) and a fresh connection for the next call.
• SendAsync_AfterTruncatedHeader_TearsDownSocket — server sends a short/garbage header then closes write side.
• SendAsync_CallerCancellation_DoesNotTearDown — cancel the caller token mid-request; assert OCE propagates and the server does NOT see a reconnect on the next call.
• Live gate (optional, modbus fixture): the exception_injector rig scenario from the round-1 plan — force a non-responding unit, assert recovery on the next poll.

Effort: S. Risk: Low-Medium — the timeout-vs-caller-cancel filter is the one subtlety; an inverted filter turns every shutdown into a spurious reconnect (harmless but noisy) or every timeout into a permanent desync (the bug). The four unit tests pin both directions.

---

Sub-batch B2 — Silent-corruption locking: AbCip runtime lock + FOCAS concurrent caches (STAB-4 + STAB-5, High)

Independently shippable. No dependencies. Concurrency fixes — high-risk classification.

STAB-4 — AbCip runs concurrent operations on shared libplctag handles with no serialization

Restatement: ReadSingleAsync/ReadGroupAsync/WriteAsync execute Read→GetStatus→Decode / Encode→Write→GetStatus on cached IAbCipTagRuntime handles with zero locking, while the server read path, every poll-group loop, and the alarm-projection loop can hit the same handle concurrently. libplctag GetStatus returns the last operation's status → torn/cross-attributed values decoded with Good status. AbLegacy documents and guards this exact hazard; the fix never propagated.

Verification: CONFIRMED. Unlocked critical sections at src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.AbCip/AbCipDriver.cs:544-575 (ReadSingleAsync: EnsureTagRuntimeAsync→ReadAsync→GetStatus→DecodeValue/DecodeArray), :614-637 (ReadGroupAsync: same on the parent + DecodeValueAt per member), :708-726 (WriteAsync: EncodeValue→WriteAsync→GetStatus). Handles are cached in DeviceState.Runtimes (ConcurrentDictionary, :1226 — the map is safe, the handles are not). Concurrent callers confirmed: AbCipAlarmProjection.RunPollLoopAsync reads through _driver.ReadAsync (AbCipAlarmProjection.cs:230-232), plus PollGroupEngine loops (one per subscription) and server-path reads/writes. The AbLegacy template: _runtimeLocks ConcurrentDictionary<string, SemaphoreSlim> + GetRuntimeLock (AbLegacyDriver.cs:773-787), applied around the read sequence (:227-244) and the write sequence (:331-342), disposed at teardown (:841-842). The RMW path (WriteBitInDIntAsync :787-818) already serializes via GetRmwLock (DeviceState:1238-1244) but only RMW-vs-RMW, and it operates on a separate handle cache (ParentRuntimes :1235) — so the runtime lock and the RMW lock govern disjoint handle sets and do not need merging (the STAB-11 direct-parent-write-vs-RMW lost-update is a device-data race across two handles, explicitly out of this batch — carried under STAB-11).

Root cause: the AbLegacy fix never propagated to AbCip (report theme: fixes don't flow back to siblings). AbCip caches per-tag runtimes but never serializes operations on one.

Proposed design (round-1 plan A4, port AbLegacy's per-runtime operation lock — verbatim pattern):

• DeviceState gains private readonly ConcurrentDictionary<string, SemaphoreSlim> _runtimeLocks + public SemaphoreSlim GetRuntimeLock(string tagName) (mirror AbLegacyDriver.cs:786-787), disposed in DisposeHandles() (mirror :841-842).
• ReadSingleAsync: acquire device.GetRuntimeLock(def.Name) around the ReadAsync→GetStatus→Decode* sequence (:545-573); status-mapping/health/result-stamping can stay outside the lock (they touch no handle) but keeping the AbLegacy shape (compute status/value inside, act outside) is the cleanest port.
• ReadGroupAsync: acquire device.GetRuntimeLock(parent.Name) around ReadAsync→GetStatus→per-member DecodeValueAt (:615-636) — the members decode from the parent handle's buffer, so they belong inside the same critical section.
• WriteAsync: acquire device.GetRuntimeLock(def.Name) around EncodeValue→WriteAsync→GetStatus (:709-712).
• Eviction interplay: EvictRuntime (:891-897) removes+disposes the handle while the evicting caller still holds the lock; the lock entry is deliberately NOT removed — a fresh runtime under the same name reuses it (AbLegacy precedent, avoids a remove/recreate race on the lock map).
• Locks keyed per tag name ⇒ cross-tag parallelism preserved. Alternative (per-device lock): rejected — serializes independent tags, and AbLegacy's field-proven granularity is per-runtime.

Implementation steps:

1. AbCipDriver.cs DeviceState (:1187-1281): add _runtimeLocks, GetRuntimeLock, dispose in DisposeHandles.
2. Wrap the three op bodies (ReadSingleAsync, ReadGroupAsync, WriteAsync) in WaitAsync(ct)/finally Release.
3. XML-doc on GetRuntimeLock quoting the AbLegacy rationale ("a libplctag Tag handle is not safe for concurrent Read/GetStatus/DecodeValue").

Tests:

• New tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.AbCip.Tests/AbCipRuntimeConcurrencyTests.cs mirroring AbLegacyRuntimeConcurrencyTests: extend/derive from FakeAbCipTag with a reentrancy guard (an Interlocked in-op counter that records a violation when >1 thread is inside Read/GetStatus/Decode/Encode/Write on the same instance); hammer concurrent ReadAsync + WriteAsync + a group read against one tag; assert zero overlap violations and coherent statuses.
• Regression: full existing AbCip suite (subscription/read/write/RMW tests) must stay green — the lock must not deadlock the RMW path (RMW uses ParentRuntimes handles, never Runtimes, so no nesting occurs; assert with the existing AbCipBoolInDIntRmwTests).
• Live gate (optional): abcip fixture — concurrent subscribe + write on overlapping tags, values stay coherent.

Effort: S. Risk: Medium (concurrency change on the hot data path, but strictly additive locking with a field-proven sibling template; deadlock risk nil — single lock level, no nesting).

STAB-5 — FOCAS fixed-tree caches are plain Dictionarys mutated across threads

Restatement: four per-device caches are written by the background fixed-tree loop while ReadAsync callers read them concurrently — undefined behaviour on Dictionary (corruption or throw on resize).

Verification: CONFIRMED at src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.FOCAS/FocasDriver.cs: LastFixedSnapshots (Dictionary<string,double>, :1184), LastTimers (Dictionary<FocasTimerKind,FocasTimer> = [], :1190), LastServoLoads (Dictionary<string,double>, :1192), LastSpindleLoads (Dictionary<int,int> = [], :1194). Writes from FixedTreeLoopAsync and its helpers (:700, :752, :793, :836-839, :862-863, :874); reads from TryReadFixedTree on ReadAsync callers' threads (:878, :912, :920, :945). The neighbouring _parsedAddressesByTagName already got the ConcurrentDictionary treatment (:40). All call sites use indexer-set and TryGetValue only — both supported by ConcurrentDictionary unchanged. (LastProgramInfo/LastProgramAxisRef are single reference-typed properties — atomic writes, not part of this finding.)

Root cause: partial application of the concurrent-collection fix.

Proposed design (round-1 plan A5): convert the four fields to ConcurrentDictionary<,>; []/new(...) initializers become new()/new(StringComparer.OrdinalIgnoreCase). Alternative (immutable snapshot swap per loop pass): cleaner isolation, more churn — rejected for the minimal type-only change matching the in-tree precedent at :40.

Implementation steps: change the four field types on FocasDriver.DeviceState (:1184-1194) + initializers. Zero call-site changes.

Tests: new FocasFixedTreeConcurrencyTests.cs in tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.FOCAS.Tests/: a stress test spinning writer iterations (mimic PublishAxisSnapshot-style keyed writes forcing dictionary growth/resize) against concurrent TryGetValue readers on the same DeviceState; N-thousand iterations, assert no exception. (On the plain Dictionary this reliably throws "operations that change non-concurrent collections must have exclusive access" / corrupts under resize.) No FOCAS fixture exists — this unit test is the authoritative gate.

Effort: S. Risk: Very low — type-only change, FOCAS-local.

---

Sub-batch B3 — PollGroupEngine v2: backoff + OCE hardening + onError fleet wiring + S7 fork retirement (STAB-9, STAB-8-poll-part, CONV-1, PERF-3, STAB-6-S7 — High/Medium)

The highest-leverage structural item. Contains concurrency-adjacent loop changes — high-risk classification for the S7 migration. Introduces the shared ConnectionBackoff primitive that B4 and plan R2-01 consume.

Verification of the pieces

• Engine (src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/PollGroupEngine.cs) already has: the onError sink (:59, :164-169), StopState drain-before-CTS-dispose (:99-112), ObjectDisposedException-on-Delay guard (:136), structural array compare (:203-209), reader-contract validation (:178-181). It lacks: failure backoff (fixed Task.Delay(state.Interval) :131) and — newly relevant — its PollLoopAsync treats any OCE as teardown (catch (OperationCanceledException) { return; } at :122, :132, :139), the exact shape that turned S7's connect-timeout OCE into permanent subscription death (STAB-14). If S7 migrates onto the engine without this filter, STAB-14 reproduces inside the shared engine.
• STAB-9 CONFIRMED: all five consumers construct without onError (ModbusDriver.cs:115, AbCipDriver.cs:133, AbLegacyDriver.cs:65, TwinCATDriver.cs:69, FocasDriver.cs:72) — a reader-contract violation or reader exception is silently swallowed fleet-wide.
• CONV-1/PERF-3/STAB-6-S7 CONFIRMED: S7's fork (S7Driver.cs:1373-1412) has capped exponential backoff (ComputeBackoffDelay :1447-1456, cap PollBackoffCap :1363 = 30 s) and failure→health (HandlePollFailure :1421-1437, threshold :1371, never-downgrade-Faulted nuance :1432-1435) that the engine lacks; and lacks the structural array diff (Equals(lastSeen?.Value, current.Value) :1469 — every subscribed array tag fires every tick) and the drain-before-dispose (UnsubscribeAsync :1350-1354 cancels + disposes the CTS without awaiting PollTask; the loop's Task.Delay has no ODE guard) that the engine has. SubscribeAsync :1328-1345, SubscriptionState :1477-1492, teardown drains at :269-270/:1605-1606.
• STAB-8 poll-part CONFIRMED: the engine polls at fixed cadence regardless of failures — a dead device pays full-cost reads every tick on Modbus/AbCip/AbLegacy/TwinCAT-fallback/FOCAS.

Root cause (CONV-1): template divergence — each side owns fixes the other needs. Single home = extend the engine with the fork's two genuine improvements, then delete the fork.

Proposed design

1. ConnectionBackoff primitive — new src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/ConnectionBackoff.cs:

• public static TimeSpan ComputeDelay(TimeSpan baseInterval, int consecutiveFailures, TimeSpan cap) — the S7 math ported verbatim (S7Driver.cs:1447-1456: shift-capped-at-30, tick overflow guard, saturate at cap; failures <= 0 ⇒ baseInterval).
• public sealed class ConnectionBackoff — the per-device attempt-throttle B4/R2-01 consume: ctor (TimeSpan baseDelay, TimeSpan maxDelay); bool ShouldAttempt(DateTime nowUtc) (true when outside the current backoff window), void RecordFailure(DateTime nowUtc), void RecordSuccess() (reset to zero — recovery must never be delayed). Not internally synchronized — documented contract: callers invoke it under their existing per-device gate (ConnectGate/_gate/probe lock), which every intended call site already holds. Alternative (lock-free Interlocked state): rejected as needless complexity while every consumer already serializes.
• This is the seam handoff to R2-01: S7's EnsureConnectedAsync attempt-throttle (STAB-8's "slightly worse on S7" leg) is wired by R2-01 using this type; B3 lands the primitive, B4 wires TwinCAT/FOCAS/AbCip.

2. Engine v2 (PollGroupEngine) — three additive changes, all preserving current consumer behavior by default:

• Ctor gains TimeSpan? backoffCap = null. PollLoopAsync tracks consecutiveFailures (reset on a successful PollOnceAsync, incremented in the catch), and the delay becomes backoffCap is null ? state.Interval : ConnectionBackoff.ComputeDelay(state.Interval, consecutiveFailures, backoffCap.Value). Null cap ⇒ byte-identical current cadence.
• OCE hardening: the three catch (OperationCanceledException) { return; } sites (:122, :132, :139) gain when (ct.IsCancellationRequested); a non-caller OCE thrown by the reader (a driver-internal timeout CTS — the STAB-14/STAB-15 class) falls through to the generic when (!IsFatal(ex)) handler ⇒ ReportError + backoff + continue. The Task.Delay catch at :132 keeps its bare form (Delay's OCE can only be the loop token) — only the two PollOnceAsync catches change. This is the guard that makes the S7 migration safe regardless of R2-01's landing order.
• Keep onError optional in the signature (fleet consumers all pass it after this batch; making it required is a breaking nicety deferred — noted, not done, to keep the change additive).

3. onError wiring — all five consumers. Each construction gains onError: ex => { /* degrade health, log */ } implementing the S7 nuance: log warning; set Degraded preserving LastSuccessfulRead; never downgrade Faulted (_health.State != DriverState.Faulted guard — S7Driver.cs:1432-1435). Per-driver health idiom follows each driver's existing convention (Modbus WriteHealth Volatile helper; AbCip/TwinCAT plain _health field; AbLegacy volatile; FOCAS Volatile) — CONV-5 idiom unification is explicitly NOT this batch.

• Also pass backoffCap: TimeSpan.FromSeconds(30) in all five (adopting S7's proven cap fleet-wide) — this closes the STAB-8 subscriber-side leg for the whole fleet.

4. S7 fork retirement. Replace _subscriptions/SubscribeAsync/UnsubscribeAsync/PollLoopAsync/PollOnceAsync/HandlePollFailure/SubscriptionState/S7SubscriptionHandle/PollBackoffCap/PollFailureHealthThreshold (S7Driver.cs:1328-1499 + field :59 + teardown drains :269-270, :1605-1606) with a PollGroupEngine instance constructed like the other five (reader ReadAsync — method-group reference, same OTOPCUA0001-clean shape the other five use; onChange forwards OnDataChange; onError = the extracted failure handler; minInterval default 100 ms — matches the fork's floor :1335-1337; backoffCap 30 s). ComputeBackoffDelay moves to ConnectionBackoff.ComputeDelay (S7's copy deleted; its unit asserts move to ConnectionBackoffTests). Teardown paths swap the manual drain for _poll.DisposeAsync(). This deletes PERF-3 (structural compare inherited), STAB-6-S7 (StopState drain inherited), and the fork's bare-OCE poll-death (superseded by the hardened engine catches).

Sequencing vs R2-01: independent in either order. If R2-01 lands first, its poll-loop OCE-filter edits (:1383/:1397/:1404) are simply deleted by this migration (its ensure-wrapper fixes at :488/:1000 are on the read/write path and survive untouched). If B3 lands first, the engine's hardened OCE catch already prevents timeout-OCE from killing the loop, and R2-01's remaining scope is the wrapper rethrow filter + fatal classification + probe mark-dead. Cross-reference this in both PRs.

Implementation steps

1. New Core.Abstractions/ConnectionBackoff.cs (static ComputeDelay + instance throttle) + tests/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions.Tests/ConnectionBackoffTests.cs (schedule parity with the S7 math: 0 failures ⇒ base, doubling, cap saturation, overflow guard; ShouldAttempt window semantics; RecordSuccess resets immediately).
2. PollGroupEngine.cs: backoffCap param + consecutiveFailures + ConnectionBackoff.ComputeDelay; OCE when (ct.IsCancellationRequested) filters on the two PollOnceAsync catches. Extend tests/Core/.../PollGroupEngineTests.cs.
3. Wire onError (+ 30 s backoffCap) into ModbusDriver.cs:115, AbCipDriver.cs:133, AbLegacyDriver.cs:65, TwinCATDriver.cs:69, FocasDriver.cs:72 — each with a small private HandlePollError(Exception) following the driver's health idiom.
4. S7: construct the engine in the ctor; delete the fork block (:1328-1499 minus what moves); route SubscribeAsync/UnsubscribeAsync to _poll; swap teardown drains for _poll.DisposeAsync(); delete ComputeBackoffDelay (now in ConnectionBackoff).

Tests

• ConnectionBackoffTests: as step 1 (port the existing S7 backoff asserts — grep ComputeBackoffDelay under tests/Drivers/.../S7.Tests).
• PollGroupEngineTests additions: Backoff_SustainedReaderFailures_StretchesCadenceToCap + Backoff_SuccessResetsToInterval (virtual-time via short intervals + tick counting); ReaderOperationCanceled_WithoutCallerCancellation_TreatedAsFailureNotTeardown (reader throws new OperationCanceledException() with the caller token un-cancelled; assert the loop survives, onError fired, next tick polls again) — this is the STAB-14-class regression guard at the engine level; CallerCancellation_StillExitsPromptly.
• Per-driver onError tests (one per driver, in each driver's .Tests): reader forced to throw (e.g. contract-violation via a fake returning a short snapshot list, or a throwing fake client) ⇒ assert health transitions to Degraded with LastSuccessfulRead preserved, and a pre-existing Faulted is not downgraded.
• S7 migration guards (in S7.Tests): ArrayTag_UnchangedBetweenPolls_DoesNotRefire (PERF-3 regression — fresh array instances, identical contents, assert one initial push only); RapidSubscribeUnsubscribe_NoObjectDisposedException (STAB-6 regression); SustainedPollFailure_DegradesHealth_WithBackoff (parity with the deleted HandlePollFailure); full existing S7DiscoveryAndSubscribeTests + S7DriverReconnectTests stay green.
• Live gate: s7 fixture (lmxopcua-fix up s7 s7_1500, :1102) — subscribe, bounce the container, assert values resume (the existing env-gated S7_1500ReconnectTests recipe now also exercises the engine-based loop).

Effort: M (engine S; five wirings S; S7 migration M). Risk: Medium-High — the S7 migration touches its subscription surface and alters array publish cadence (intended, PERF-3); the engine OCE filter changes loop-exit semantics (guarded by the new engine tests). Blast radius otherwise bounded: null backoffCap consumers (none remain in-tree after step 3) and existing engine consumers keep byte-identical behavior until they opt in.

---

Sub-batch B4 — Connect-attempt throttles for the lazy-reconnect drivers (STAB-8 fleet part, Medium)

Depends on B3 step 1 (ConnectionBackoff). Independently shippable after that. S7 wiring explicitly excluded — R2-01 seam.

STAB-8 — unthrottled reconnect attempts against a dead device

Restatement: a dead device pays full reconnect cost on every data call / poll tick: TwinCAT a connect attempt per call, FOCAS a full two-socket reconnect per tick of each of three loops, AbCip a full Tag create + Forward Open per tag per tick via evict-recreate. (S7's new per-call reconnect is the same defect — R2-01's wiring, same primitive.)

Verification: CONFIRMED — TwinCAT EnsureConnectedAsync/EnsureConnectedUnderGateAsync (TwinCATDriver.cs:706-775) attempts a connect on every call while down; FOCAS EnsureConnectedAsync (FocasDriver.cs:1089-1120) unconditionally builds + connects a fresh SynchronizedFocasClient; AbCip EnsureTagRuntimeAsync (AbCipDriver.cs:852-882) re-creates + re-initializes (Forward Open) a runtime per call after eviction (EvictRuntime :891-897). B3 closes the subscriber-side cost (poll backoff stretches tick cadence); this closes the connect-side cost for server-initiated reads/writes and multi-loop drivers.

Proposed design: per-device ConnectionBackoff gate on the connect/create path, with the probe loop as the recovery agent:

• Data-path callers consult backoff.ShouldAttempt(now) before an actual connect/create attempt; inside the window they fail fast with the same exception type the failed connect would have thrown (drivers already map it to BadCommunicationError snapshots / write results) — no wire traffic.
• The probe loop bypasses the throttle (its own interval is already a bounded cadence) and drives RecordFailure/RecordSuccess. Success on any path (probe or a data call that squeaked through at a window boundary) resets the backoff instantly — recovery is never delayed beyond one probe interval, and typically not at all (first post-recovery data call inside an expired window succeeds and resets).
• Defaults: base 1 s, max 30 s (matches Modbus transport + S7 poll conventions). Not config-exposed in this pass (no options-surface churn; note for CONV-8's future options Validate() work).

Per driver:

• TwinCAT: DeviceState gains ConnectionBackoff Backoff; EnsureConnectedUnderGateAsync (under ConnectGate — the documented synchronization home) checks ShouldAttempt before building a client when a bypass flag is not set, records failure in its existing connect catch (:759-767) and success after device.Client = client (:768); EnsureConnectedAsync gains a bypassBackoff parameter the probe loop (:631) sets true.
• FOCAS: same shape on FocasDriver.DeviceState + EnsureConnectedAsync (:1089); the probe loop's EnsureConnectedAsync call passes bypass; the fixed-tree and recycle loops go through the throttled path (they're the per-tick hammer the finding cites).
• AbCip: DeviceState gains ConnectionBackoff Backoff; EnsureTagRuntimeAsync checks ShouldAttempt only on the create path (cache hits unaffected, :855); RecordFailure in the init catch (:867-875 — only for transport-shaped failures: keep it simple and record on any InitializeAsync throw), RecordSuccess after a successful TryAdd; the probe loop bypasses (its own runtime create) and records success/failure. Eviction (EvictRuntime) does not touch the backoff — the next create attempt decides.

Alternative — throttle inside PollGroupEngine: rejected; B3's poll backoff already stretches subscription cadence, and the connect cost also burns on server-initiated reads which never pass through the engine.

Implementation steps:

1. (ConnectionBackoff instance API landed in B3 task 1.)
2. TwinCAT: field + EnsureConnectedUnderGateAsync/EnsureConnectedAsync bypass param + probe wiring.
3. FOCAS: field + EnsureConnectedAsync gate + probe bypass.
4. AbCip: field + EnsureTagRuntimeAsync create-gate + probe wiring.
5. Cross-reference comment in S7Driver.EnsureConnectedAsync pointing R2-01 at the primitive (comment only — no S7 behavior change here).

Tests (all fake-client — TwinCAT/FOCAS have no fixture; AbCip's is optional):

• Per driver: DeadDevice_ConnectAttemptsFollowBackoffSchedule — failing fake connect; N rapid data calls; assert the fake's connect/create invocation count matches the schedule (1 attempt + fail-fasts inside the window), not N.
• Per driver: ProbeBypassesBackoff_AndSuccessResets — device recovers; assert the next probe tick connects despite an open window and the following data call succeeds immediately (no residual delay).
• AbCip: CacheHit_NeverConsultsBackoff (healthy runtime reads unaffected).

Effort: M. Risk: Low-Medium — the failure mode to guard is delayed recovery; the probe-bypass + instant-reset design plus the reset tests pin it. Fail-fast inside the window changes dead-device error latency (faster Bad, less wire noise) — intended.

---

Sub-batch B5 — ResolveHost via the resolver: per-host breaker isolation (CONV-4, Medium)

Independently shippable. No dependencies.

CONV-4 — ResolveHost mis-keys equipment-tag references in every multi-device driver

Restatement: every ResolveHost consults only _tagsByName (authored names); an equipment-tag reference (raw TagConfig JSON) never matches and falls back to the first device's host — so per-host Polly bulkhead/circuit-breaker keys are wrong for the primary authoring model, and a broken device B trips device A's breaker. Now production-live via the CapabilityInvoker dispatch wiring (#10).

Verification: CONFIRMED at all five sites — AbCipDriver.cs:483-488, AbLegacyDriver.cs:496-501, TwinCATDriver.cs:689-696 (fallback UnresolvedHostSentinel = ""), FocasDriver.cs:1082-1087, ModbusDriver.cs:125-131 (per-unit host BuildSlaveHostName :134; ResolveUnitId :474 = per-tag override or driver default). All four multi-device TDef records carry DeviceHostAddress, and all four equipment parsers populate it from the deviceHostAddress key (verified: AbCip :39/:56, AbLegacy :42/:51, TwinCAT :30/:38, FOCAS :32/:34 — FOCAS coalesces a missing key to ""). ModbusEquipmentTagParser parses no unitId (grep-confirmed), so Modbus equipment tags currently cannot key per-unit at all. EquipmentTagRefResolver (Core.Abstractions/EquipmentTagRefResolver.cs) already caches parses (positive and negative) — routing ResolveHost through it adds no per-call parse cost after first use.

Root cause: ResolveHost predates the equipment-tag path and never learned about _parseRef.

Proposed design (round-1 plan B3, confirmed): route through the resolver, derive the host from the parsed def:

• AbCip / AbLegacy / TwinCAT / FOCAS: ResolveHost(ref) becomes _resolver.TryResolve(ref, out var def) && !string.IsNullOrEmpty(def.DeviceHostAddress) ? def.DeviceHostAddress : <current fallback> (first-device / UnresolvedHostSentinel / DriverInstanceId — each driver keeps its exact current fallback; the empty-string guard covers FOCAS's ?? "" default so an address-less equipment tag falls back rather than keying an "" host that TwinCAT alone treats as a documented sentinel).
• Modbus: ResolveHost becomes _resolver.TryResolve(ref, out var tag) ? BuildSlaveHostName(ResolveUnitId(tag)) : HostName; and ModbusEquipmentTagParser gains an optional unitId (byte, absent ⇒ null ⇒ driver-level _options.UnitId via ResolveUnitId) so multi-unit equipment tags key their own breaker. Scope note: unitId only — the parser's other gaps (writable, deadband, address-grammar…) are UNDER-6 and the strictness tiers are R2-11's.
• Optional resolver sugar (ResolveHost(ref, Func<TDef,string>) overload on EquipmentTagRefResolver) — skipped: five one-line bodies don't justify a new generic seam; revisit if a sixth pattern appears.

Implementation steps:

1. Rewrite the four multi-device ResolveHost bodies (files/lines above).
2. src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Modbus.Contracts/ModbusEquipmentTagParser.cs: parse optional unitId into ModbusTagDefinition.UnitId; rewrite ModbusDriver.ResolveHost.
3. XML-doc each ResolveHost: the key is consumed by per-host resilience (bulkhead/breaker) keying — equipment tags MUST resolve their authored device.

Tests:

• Per multi-device driver (in each .Tests project, pattern: two configured devices A+B, an equipment-tag ref whose TagConfig JSON names device B): ResolveHost_EquipmentTagRef_ReturnsItsOwnDeviceHost — asserts B (today: A). Plus ResolveHost_UnknownRef_KeepsCurrentFallback (regression: fallback string unchanged per driver).
• FOCAS: equipment ref with no deviceHostAddress falls back (not "").
• Modbus: EquipmentTag_WithUnitId_ResolvesPerUnitHostName (host:port/unitN) + parser round-trip test in the existing ModbusEquipmentTagParser suite; absent unitId keeps driver-level behavior.

Effort: S-M. Risk: Low — corrects a mis-key; the visible change is that per-host breaker keys become per-device for equipment tags (strictly the intended semantics). Verify no dashboard/consumer grouped on the old first-device key (the /hosts status table keys on HostConnectivityStatus.HostName, which already reports per-device — this fix aligns the two).

---

Sub-batch B6 — TwinCAT replay hardening (STAB-16 Medium, STAB-17 Low, STAB-12-compound Medium)

Independently shippable. No dependencies. Concurrency fixes — high-risk classification. No TC3 fixture exists — the fake-ITwinCATClient suite (TwinCATReconnectReplayTests) is authoritative.

STAB-16 — replay failure is silent to subscribers and never retried while the client stays up

Restatement: ReplayNativeRegistrationsAsync logs and skips a failed intent (deliberately — one bad symbol must not abort recovery of the rest), but the failed registration keeps its dead handle, nothing can push a Bad snapshot (the OnChange shape has no quality channel), and nothing retries until the next client swap. A transient AddNotificationAsync failure (ADS notification-quota blip, symbol mid-redeploy) kills that tag's push feed with only a warning log.

Verification: CONFIRMED — catch at TwinCATDriver.cs:801-807 (log-only; SwapHandle success-path-only :797); OnChange is Action<string, object?> building Good-only snapshots in the SubscribeAsync closure (:477-479); replay is invoked solely from EnsureConnectedUnderGateAsync :773 (new-client installs only); the probe loop's success path (:631-651) never re-examines registrations.

Root cause: the replay commit treated per-intent failure as purely a logging concern; the registration's lifecycle has no "dead, needs re-registration" state.

Proposed design:

• Dead-handle state: NativeRegistration gains public bool HasLiveHandle => Volatile.Read(ref _handle) is not null; and public ITwinCATNotificationHandle? MarkHandleDead() => Interlocked.Exchange(ref _handle, null);. On replay failure: reg.MarkHandleDead() + dispose the returned (already-dead) handle best-effort.
• Bad-quality surfacing: NativeRegistration gains a second stored callback Action OnUnavailable (built in SubscribeAsync beside OnChange, closing over the same (handle, reference)): () => OnDataChange?.Invoke(this, new DataChangeEventArgs(handle, reference, new DataValueSnapshot(null, TwinCATStatusMapper.BadCommunicationError, null, DateTime.UtcNow))). The replay-failure catch invokes it (defensively wrapped) and degrades _health to Degraded preserving LastSuccessfulRead. Subscribers see the tag go Bad instead of frozen-Good-stale.
• Probe-tick retry: new RetryDeadRegistrationsAsync(DeviceState device, CancellationToken ct): under ConnectGate, if device.Client is { IsConnected: true } client, for each reg in device.NativeRegistrations.Values with !reg.HasLiveHandle, attempt AddNotificationAsync → SwapHandle → ownership re-check (shared with STAB-17 below); on success, count + log (recovery of the value itself arrives via ADS's initial-notification push on register — same mechanism initial subscribe relies on). Called from ProbeLoopAsync after a successful probe (success == true, after :640's block). Failures just stay dead for the next tick — bounded retry cadence = probe interval, no extra timers.
• Alternative — retry inside a dedicated timer: rejected; the probe loop is the existing per-device liveness cadence and already holds the right gate discipline.

STAB-17 — unsubscribe racing a replay leaks a live ADS notification

Restatement: NativeRegistration.Dispose runs gate-free (registry remove + handle dispose). A concurrent replay has already snapshotted the intents (:786) and may complete AddNotificationAsync after the dispose; SwapHandle installs the fresh handle into a registration nobody owns — the ADS notification stays live on the new client for its lifetime.

Verification: CONFIRMED (:607-613 vs :786-798). Narrow window, bounded blast radius (one orphan per race), but a genuine native-resource leak on the PLC side.

Proposed design (the report's recommended shape): post-swap ownership re-check — in ReplayNativeRegistrationsAsync (and the new RetryDeadRegistrationsAsync), after reg.SwapHandle(newHandle): if (!device.NativeRegistrations.ContainsKey(reg.Id)) { var orphan = reg.MarkHandleDead(); try { orphan?.Dispose(); } catch { } } — the registration was unsubscribed mid-replay; retract the fresh notification immediately. Alternative — take ConnectGate in Dispose: rejected: Dispose is called from UnsubscribeAsync (server-facing, should stay non-blocking) and would wait behind a wedged connect — exactly the STAB-12 hazard class this same sub-batch is shrinking.

STAB-12 (compounded part) — RecycleClientAsync waits the gate uncancellably

Restatement: the probe loop's RecycleClientAsync waits ConnectGate with CancellationToken.None, so a connect wedged under the gate (TwinCAT's blocking SDK Connect, AdsTwinCATClient.cs:73-80 — the un-fixed root of STAB-12, carried) now also wedges the probe loop beyond cancellation.

Verification: CONFIRMED at TwinCATDriver.cs:826 (drifted from cited :835). Both call sites (:640, :648) are in ProbeLoopAsync and hold ct.

Proposed design: RecycleClientAsync(DeviceState device, CancellationToken ct) — pass the probe token into ConnectGate.WaitAsync(ct). Both call sites pass the loop ct; the :648 site sits inside a catch block, so wrap its await in try { … } catch (OperationCanceledException) { break-equivalent } (return from the loop body iteration) so a shutdown mid-recycle exits the probe task cleanly instead of faulting it. The root blocking-Connect fix (AdsTwinCATClient) stays carried under STAB-12 — out of this batch.

Implementation steps

1. TwinCATDriver.cs NativeRegistration (:558-614): add OnUnavailable ctor param + property, HasLiveHandle, MarkHandleDead.
2. SubscribeAsync (:474-479): build + pass the OnUnavailable closure.
3. ReplayNativeRegistrationsAsync (:784-814): failure catch → MarkHandleDead + OnUnavailable + _health = Degraded(...); success path → post-swap ownership re-check (STAB-17); keep the per-intent isolation and tally log.
4. New RetryDeadRegistrationsAsync; call from ProbeLoopAsync after a successful probe.
5. RecycleClientAsync signature + both call sites + the in-catch OCE wrap.

Tests (extend tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.TwinCAT.Tests/TwinCATReconnectReplayTests.cs — fake client authoritative)

• ReplayFailure_EmitsBadQuality_AndDegradesHealth — fake client fails AddNotificationAsync for one symbol during replay; assert an OnDataChange with BadCommunicationError fired for that reference, health Degraded (LastSuccessfulRead preserved), sibling intents replayed.
• ReplayFailure_RetriedOnNextSuccessfulProbeTick — after the failed replay, drive a probe tick with the fake now accepting registrations; assert AddNotificationAsync re-invoked with the identical intent, a subsequent fake push reaches OnDataChange with Good, and no duplicate registration exists.
• UnsubscribeDuringReplay_DisposesFreshHandle_NoOrphan — fake client blocks AddNotificationAsync on a test-controlled gate; start replay, UnsubscribeAsync mid-flight, release the gate; assert the freshly-created handle was disposed and the fake's live-notification count is zero (fake tracks adds/disposes).
• RecycleClient_HonorsCancellation — wedge the gate (hold ConnectGate from the test), cancel the probe token; assert RecycleClientAsync throws/returns promptly and the probe task completes (no uncancellable hang; bound with WaitAsync(seconds) so a regression fails rather than hangs the suite).
• Regression: the existing 4 replay tests stay green.

Effort: M. Risk: Medium — concurrency around ConnectGate/Interlocked handle state; mitigated by the fake-client suite and by keeping every new mutation inside the existing gate discipline (retry under gate; ownership re-check is lock-free Interlocked + registry read, matching the existing swap idiom).

---

Task breakdown

Rules: tasks are bite-sized (≤ ~5 min of implementation each — TDD steps split test-writing from implementation); sub-batch boundaries are PR boundaries (one branch + one merge per sub-batch: fix/r2-09-b1-modbus-desync, fix/r2-09-b2-abcip-focas-locks, fix/r2-09-b3-pollengine-v2, fix/r2-09-b4-connect-backoff, fix/r2-09-b5-resolvehost, fix/r2-09-b6-twincat-replay). B1/B2/B5/B6 are mutually independent and parallelizable; B4 depends on B3.1-3.2 only; within a sub-batch, tasks are sequential unless noted. Parallel implementers on this shared tree must worktree-isolate (house rule).

B1 — Modbus fatal-classification (sub-batch boundary — PR 1)

Task B1.1 — Failing transport-desync tests

Classification: test-first (low-risk) Estimated implement time: 5 min Parallelizable with: B2., B3., B5., B6. Files: tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Modbus.Tests/ModbusTcpReconnectTests.cs

1. Extend FlakeyModbusServer with scriptable per-request behaviors: stall (no reply), wrong-TxId reply, truncated header.
2. Add the four tests from the B1 design (…ResponseTimeout_TearsDownAndReconnects, …TxIdMismatch…, …TruncatedHeader…, …CallerCancellation_DoesNotTearDown).
3. dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Modbus.Tests --filter "FullyQualifiedName~ModbusTcpReconnectTests" → FAIL (first three; caller-cancel may pass — keep as regression pin).
4. Commit: test(modbus): failing repro for STAB-3 timeout/framing stream desync.

Task B1.2 — Teardown-before-propagate in SendOnceAsync

Classification: transport fault-path (medium-risk) Estimated implement time: 5 min Parallelizable with: — Files: src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Modbus/ModbusTransportDesyncException.cs (new), src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Modbus/ModbusTcpTransport.cs

1. Add ModbusTransportDesyncException : IOException.
2. Wrap SendOnceAsync body: OCE when (!ct.IsCancellationRequested) + InvalidDataException ⇒ await TearDownAsync() ⇒ throw desync exception.
3. Same filter as B1.1 step 3 → PASS (all four).
4. Commit: fix(modbus): classify per-op timeout + framing violations connection-fatal (05/STAB-3).

Task B1.3 — Full Modbus suite + PR

Classification: verification Estimated implement time: 3 min Parallelizable with: — Files: —

1. dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Modbus.Tests and dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Modbus.Addressing.Tests → green.
2. Optional live gate: modbus fixture + exception_injector non-responding-unit scenario.
3. Merge PR 1.

B2 — AbCip lock + FOCAS caches (sub-batch boundary — PR 2)

Task B2.1 — Failing AbCip concurrency test

Classification: concurrency (high-risk) Estimated implement time: 5 min Parallelizable with: B1., B2.4, B3., B5., B6. Files: tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.AbCip.Tests/AbCipRuntimeConcurrencyTests.cs (new), tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.AbCip.Tests/FakeAbCipTag.cs

1. Add a reentrancy-violation counter to FakeAbCipTag (Interlocked in-op depth; record violation when >1).
2. Write the hammer test (concurrent Read + Write + group-read on one tag; assert zero violations) — mirror AbLegacyRuntimeConcurrencyTests.
3. dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.AbCip.Tests --filter "FullyQualifiedName~AbCipRuntimeConcurrencyTests" → FAIL.
4. Commit: test(abcip): failing repro for STAB-4 unserialized shared-handle ops.

Task B2.2 — GetRuntimeLock on AbCip DeviceState

Classification: concurrency (high-risk) Estimated implement time: 3 min Parallelizable with: — Files: src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.AbCip/AbCipDriver.cs (DeviceState :1187-1281)

1. Port _runtimeLocks + GetRuntimeLock + dispose-in-DisposeHandles from AbLegacyDriver.cs:773-787, 841-842, with the hazard doc-comment.
2. Commit: feat(abcip): per-runtime operation lock on DeviceState (AbLegacy pattern).

Task B2.3 — Wrap the three AbCip op bodies

Classification: concurrency (high-risk) Estimated implement time: 5 min Parallelizable with: — Files: src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.AbCip/AbCipDriver.cs (:544-575, :614-637, :708-726)

1. WaitAsync(ct)/finally Release around ReadSingle / ReadGroup / Write critical sections.
2. B2.1's filter → PASS; then full AbCip suite (dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.AbCip.Tests) → green (RMW suite proves no nesting/deadlock).
3. Commit: fix(abcip): serialize Read/GetStatus/Decode + Encode/Write/GetStatus per runtime (05/STAB-4).

Task B2.4 — Failing FOCAS cache-concurrency test

Classification: concurrency (high-risk) Estimated implement time: 5 min Parallelizable with: B2.1-B2.3 (different driver) Files: tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.FOCAS.Tests/FocasFixedTreeConcurrencyTests.cs (new)

1. Stress test: writer thread grows/updates the four DeviceState caches (force resizes) vs concurrent TryGetValue readers; assert no exception over N iterations.
2. dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.FOCAS.Tests --filter "FullyQualifiedName~FocasFixedTreeConcurrencyTests" → FAIL (flaky-fail is expected on UB — loop iterations high enough to be deterministic in practice).
3. Commit: test(focas): failing repro for STAB-5 cross-thread Dictionary mutation.

Task B2.5 — Convert the four FOCAS caches

Classification: concurrency (high-risk, type-only) Estimated implement time: 3 min Parallelizable with: — Files: src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.FOCAS/FocasDriver.cs (:1184-1194)

1. Four fields → ConcurrentDictionary<,>; initializers adjusted. No call-site edits.
2. B2.4's filter → PASS; full FOCAS suite green.
3. Commit: fix(focas): ConcurrentDictionary for fixed-tree caches (05/STAB-5). Merge PR 2.

B3 — PollGroupEngine v2 + S7 fork retirement (sub-batch boundary — PR 3)

Task B3.1 — ConnectionBackoff primitive + tests

Classification: shared primitive (medium-risk) Estimated implement time: 5 min Parallelizable with: B1., B2., B5., B6. Files: src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/ConnectionBackoff.cs (new), tests/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions.Tests/ConnectionBackoffTests.cs (new)

1. Write ConnectionBackoffTests first (schedule parity with S7Driver.ComputeBackoffDelay — port the S7 asserts; ShouldAttempt/RecordFailure/RecordSuccess window + instant-reset semantics). dotnet test tests/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions.Tests --filter "FullyQualifiedName~ConnectionBackoffTests" → FAIL (doesn't compile → add type) → implement → PASS.
2. Commit: feat(core): ConnectionBackoff primitive — shared capped-exponential backoff + attempt throttle (05/STAB-8; seam for R2-01 S7 wiring).

Task B3.2 — Engine backoff + OCE hardening (tests first)

Classification: concurrency (high-risk — shared loop semantics) Estimated implement time: 5 min (tests) + 5 min (impl) Parallelizable with: — (after B3.1) Files: tests/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions.Tests/PollGroupEngineTests.cs, src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/PollGroupEngine.cs

1. Add Backoff_SustainedReaderFailures_StretchesCadenceToCap, Backoff_SuccessResetsToInterval, ReaderOperationCanceled_WithoutCallerCancellation_TreatedAsFailureNotTeardown, CallerCancellation_StillExitsPromptly. Filter FullyQualifiedName~PollGroupEngineTests → FAIL.
2. Implement: backoffCap ctor param, consecutiveFailures, ConnectionBackoff.ComputeDelay, when (ct.IsCancellationRequested) on the two PollOnceAsync OCE catches (:122, :139). → PASS (plus all existing engine tests).
3. Commit: feat(core): PollGroupEngine v2 — failure backoff + caller-token-filtered OCE exit (CONV-1/STAB-8; STAB-14-class engine guard).

Task B3.3 — onError + backoffCap wiring: Modbus, AbCip, AbLegacy

Classification: driver wiring (medium-risk) Estimated implement time: 5 min per driver (tests-first each) Parallelizable with: B3.4 Files: src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Modbus/ModbusDriver.cs:115, src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.AbCip/AbCipDriver.cs:133, src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.AbLegacy/AbLegacyDriver.cs:65 + one test per driver's .Tests project

1. Per driver: failing test PollReaderFailure_DegradesHealth_PreservesLastSuccessfulRead_NeverDowngradesFaulted (e.g. --filter "FullyQualifiedName~PollErrorHealthTests" per project) → FAIL.
2. Add HandlePollError(Exception) (driver's own health idiom) + pass onError: and backoffCap: TimeSpan.FromSeconds(30). → PASS.
3. Commit per driver or one commit: fix(drivers): wire PollGroupEngine onError→health + 30s poll backoff (05/STAB-9) [modbus,abcip,ablegacy].

Task B3.4 — onError + backoffCap wiring: TwinCAT, FOCAS

Classification: driver wiring (medium-risk) Estimated implement time: 5 min per driver Parallelizable with: B3.3 Files: src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.TwinCAT/TwinCATDriver.cs:69, src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.FOCAS/FocasDriver.cs:72 + tests

1. Same TDD shape as B3.3.
2. Commit: fix(drivers): wire PollGroupEngine onError→health + 30s poll backoff (05/STAB-9) [twincat,focas].

Task B3.5 — S7 migration guard tests (failing)

Classification: concurrency (high-risk) Estimated implement time: 5 min Parallelizable with: — Files: tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.S7.Tests/S7PollEngineMigrationTests.cs (new)

1. Write ArrayTag_UnchangedBetweenPolls_DoesNotRefire (PERF-3), RapidSubscribeUnsubscribe_NoObjectDisposedException (STAB-6-S7), SustainedPollFailure_DegradesHealth_WithBackoff against the fake IS7PlcFactory seam.
2. dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.S7.Tests --filter "FullyQualifiedName~S7PollEngineMigrationTests" → FAIL (array test fails on the fork's Equals diff; unsub test fails/flakes on the disposed CTS).
3. Commit: test(s7): failing guards for the poll-fork retirement (PERF-3, STAB-6).

Task B3.6 — Retire the S7 fork onto the engine

Classification: concurrency (high-risk — subscription surface rewrite) Estimated implement time: 3 × 5 min (ctor+subscribe/unsubscribe swap; fork deletion + teardown swap; ComputeBackoffDelay relocation) Parallelizable with: — Files: src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.S7/S7Driver.cs (:59, :269-270, :1328-1499, :1605-1606), S7 backoff test relocation into ConnectionBackoffTests

1. Construct _poll = new PollGroupEngine(reader: ReadAsync, onChange: …, minInterval: 100ms, onError: HandlePollError, backoffCap: 30s); route SubscribeAsync/UnsubscribeAsync through it; delete PollLoopAsync/PollOnceAsync/HandlePollFailure/SubscriptionState/S7SubscriptionHandle/_subscriptions/PollBackoffCap/PollFailureHealthThreshold/ComputeBackoffDelay; swap both teardown drains for _poll.DisposeAsync().
2. B3.5's filter → PASS; full S7 suite (dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.S7.Tests) → green, incl. S7DriverReconnectTests + S7DiscoveryAndSubscribeTests.
3. Commit: refactor(s7): retire the bespoke poll fork onto PollGroupEngine v2 (CONV-1; fixes PERF-3 + STAB-6-S7; deletes the fork's bare-OCE poll-death — cross-ref plan R2-01).

Task B3.7 — Cross-solution verification + PR

Classification: verification Estimated implement time: 5 min Parallelizable with: —

1. dotnet test tests/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions.Tests + all five driver unit suites → green.
2. Live gate (recommended for the S7 leg): s7 fixture bounce recipe (lmxopcua-fix down s7 && lmxopcua-fix up s7 s7_1500) with the env-gated S7_1500ReconnectTests.
3. Merge PR 3. Note in the PR body: R2-01 sequencing paragraph from the B3 design.

B4 — Connect-attempt throttles (sub-batch boundary — PR 4; blocked by B3.1-B3.2 only)

Task B4.1 — TwinCAT connect throttle (tests first)

Classification: concurrency (high-risk — connect-path gating) Estimated implement time: 5 min (tests) + 5 min (impl) Parallelizable with: B4.2, B4.3 Files: src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.TwinCAT/TwinCATDriver.cs (DeviceState, EnsureConnectedAsync :706, EnsureConnectedUnderGateAsync :731, probe :631), tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.TwinCAT.Tests/TwinCATConnectBackoffTests.cs (new)

1. Failing tests: DeadDevice_ConnectAttemptsFollowBackoffSchedule, ProbeBypassesBackoff_AndSuccessResets (filter FullyQualifiedName~TwinCATConnectBackoffTests) → FAIL → implement per B4 design → PASS.
2. Commit: fix(twincat): per-device connect-attempt backoff, probe-driven recovery (05/STAB-8).

Task B4.2 — FOCAS connect throttle (tests first)

Classification: concurrency (high-risk) Estimated implement time: 5 min + 5 min Parallelizable with: B4.1, B4.3 Files: src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.FOCAS/FocasDriver.cs (DeviceState, EnsureConnectedAsync :1089, probe loop), tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.FOCAS.Tests/FocasConnectBackoffTests.cs (new)

1. Same TDD shape (fake IFocasClientFactory counting connects). FAIL → implement → PASS.
2. Commit: fix(focas): per-device connect-attempt backoff across the three loops (05/STAB-8).

Task B4.3 — AbCip runtime-create throttle (tests first)

Classification: concurrency (high-risk) Estimated implement time: 5 min + 5 min Parallelizable with: B4.1, B4.2 Files: src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.AbCip/AbCipDriver.cs (DeviceState, EnsureTagRuntimeAsync :852, probe loop), tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.AbCip.Tests/AbCipConnectBackoffTests.cs (new)

1. Tests: schedule on the create path, CacheHit_NeverConsultsBackoff, probe bypass + reset. FAIL → implement → PASS.
2. Commit: fix(abcip): throttle evict-recreate Forward-Open attempts per device (05/STAB-8).

Task B4.4 — S7 seam note + PR

Classification: docs/verification Estimated implement time: 3 min Files: src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.S7/S7Driver.cs (comment on EnsureConnectedAsync)

1. Comment pointing R2-01 at ConnectionBackoff for the S7 attempt-throttle. Full three driver suites green. Merge PR 4.

B5 — ResolveHost via resolver (sub-batch boundary — PR 5)

Task B5.1 — Failing ResolveHost tests (4 multi-device drivers)

Classification: correctness (medium-risk) Estimated implement time: 2 × 5 min Parallelizable with: B1., B2., B3., B6. Files: one test file (or additions) per .Tests project: AbCip, AbLegacy, TwinCAT, FOCAS

1. Per driver: ResolveHost_EquipmentTagRef_ReturnsItsOwnDeviceHost (two devices, equipment ref naming device B → expect B) + ResolveHost_UnknownRef_KeepsCurrentFallback; FOCAS adds the empty-deviceHostAddress fallback case. Suggested shared name: --filter "FullyQualifiedName~ResolveHostTests" per project → FAIL (4×).
2. Commit: test(drivers): failing repro for CONV-4 equipment-tag ResolveHost mis-keying.

Task B5.2 — Rewrite the four ResolveHost bodies

Classification: correctness (medium-risk) Estimated implement time: 5 min Parallelizable with: B5.3 Files: AbCipDriver.cs:483-488, AbLegacyDriver.cs:496-501, TwinCATDriver.cs:689-696, FocasDriver.cs:1082-1087

1. Route through _resolver.TryResolve + empty-host guard; keep per-driver fallbacks. B5.1 filters → PASS.
2. Commit: fix(drivers): ResolveHost resolves equipment-tag refs via EquipmentTagRefResolver — per-host breaker isolation (CONV-4).

Task B5.3 — Modbus unitId + ResolveHost

Classification: correctness (medium-risk) Estimated implement time: 5 min (test) + 5 min (impl) Parallelizable with: B5.2 Files: src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Modbus.Contracts/ModbusEquipmentTagParser.cs, src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Modbus/ModbusDriver.cs:125-131, Modbus parser + driver tests

1. Failing tests: parser unitId round-trip; EquipmentTag_WithUnitId_ResolvesPerUnitHostName; absent-unitId regression. → FAIL → implement → PASS. (Scope: unitId ONLY — R2-11 owns parser strictness/parity.)
2. Commit: fix(modbus): equipment tags carry unitId; ResolveHost keys per-unit via the resolver (CONV-4).

Task B5.4 — Suites + PR

Classification: verification Estimated implement time: 3 min

1. All five driver unit suites green. Merge PR 5.

B6 — TwinCAT replay hardening (sub-batch boundary — PR 6)

Task B6.1 — Failing test: replay failure surfaces Bad + degrades health

Classification: concurrency (high-risk) Estimated implement time: 5 min Parallelizable with: B1., B2., B3., B5. Files: tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.TwinCAT.Tests/TwinCATReconnectReplayTests.cs

1. ReplayFailure_EmitsBadQuality_AndDegradesHealth per B6 design. dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.TwinCAT.Tests --filter "FullyQualifiedName~TwinCATReconnectReplayTests" → FAIL.
2. Commit: test(twincat): failing repro for STAB-16 silent replay failure.

Task B6.2 — Dead-handle state + OnUnavailable + replay-catch surfacing

Classification: concurrency (high-risk) Estimated implement time: 2 × 5 min Parallelizable with: — Files: src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.TwinCAT/TwinCATDriver.cs (NativeRegistration :558-614, SubscribeAsync :474-479, ReplayNativeRegistrationsAsync :784-814)

1. HasLiveHandle/MarkHandleDead/OnUnavailable + closure in SubscribeAsync + failure-catch: mark dead, emit Bad, degrade health.
2. B6.1's filter → PASS. Commit: fix(twincat): replay failure marks the registration dead, publishes Bad quality, degrades health (05/STAB-16).

Task B6.3 — Probe-tick retry of dead registrations (tests first)

Classification: concurrency (high-risk) Estimated implement time: 5 min + 5 min Parallelizable with: — Files: same driver file (ProbeLoopAsync :622-656, new RetryDeadRegistrationsAsync), same test file

1. Failing ReplayFailure_RetriedOnNextSuccessfulProbeTick → implement RetryDeadRegistrationsAsync (under ConnectGate, re-register !HasLiveHandle intents, tally log) + probe-success call site → PASS.
2. Commit: fix(twincat): retry failed replay intents on the next successful probe tick (05/STAB-16).

Task B6.4 — STAB-17 ownership re-check (tests first)

Classification: concurrency (high-risk) Estimated implement time: 5 min + 5 min Parallelizable with: — Files: same driver file (ReplayNativeRegistrationsAsync success path + RetryDeadRegistrationsAsync), same test file (+ fake-client live-notification tracking)

1. Failing UnsubscribeDuringReplay_DisposesFreshHandle_NoOrphan (gate-blocked fake AddNotificationAsync) → implement the post-SwapHandle NativeRegistrations.ContainsKey re-check + orphan dispose in both replay paths → PASS.
2. Commit: fix(twincat): retract the fresh ADS notification when unsubscribe raced the replay (05/STAB-17).

Task B6.5 — Cancellable RecycleClientAsync (tests first)

Classification: fault-path (medium-risk) Estimated implement time: 5 min Parallelizable with: — Files: same driver file (RecycleClientAsync :824-840, probe call sites :640, :648), same test file

1. Failing RecycleClient_HonorsCancellation (held gate + cancelled token; bounded wait so regression fails, not hangs) → thread ct through ConnectGate.WaitAsync + OCE-wrap the in-catch call site → PASS.
2. Commit: fix(twincat): probe recycle honors cancellation — no uncancellable gate wait (05/STAB-12 compounded part).

Task B6.6 — Full TwinCAT suite + PR

Classification: verification Estimated implement time: 3 min

1. dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.TwinCAT.Tests → green (incl. the pre-existing 4 replay tests). No TC3 fixture — this suite is the authoritative gate (UNDER-7 noted).
2. Merge PR 6.

---

Sequencing & effort roll-up

PR	Sub-batch	Findings	Effort	Risk	Depends on	Live gate
1	B1 Modbus fatal-classification	STAB-3	S	Low-Med	—	modbus fixture (+exception_injector), optional
2	B2 AbCip lock + FOCAS caches	STAB-4, STAB-5	S-M	Med (concurrency)	—	abcip fixture optional; FOCAS unit-authoritative
3	B3 PollGroupEngine v2 + S7 fork retirement (+ConnectionBackoff)	STAB-9, CONV-1, PERF-3, STAB-6-S7, STAB-8-poll	M	Med-High	— (coordinate with R2-01)	s7 fixture bounce, recommended
4	B4 connect-attempt throttles (TwinCAT/FOCAS/AbCip)	STAB-8-fleet	M	Low-Med	B3.1-B3.2	none needed (fake-client)
5	B5 ResolveHost via resolver (+Modbus unitId)	CONV-4	S-M	Low	—	none needed
6	B6 TwinCAT replay hardening	STAB-16, STAB-17, STAB-12-part	M	Med (concurrency)	—	none exists — unit-authoritative

Overall effort: ≈ 5.5–7 dev-days across 6 independently shippable PRs (B1/B2/B5/B6 parallelizable from day one; B4 unblocks after B3's first two tasks).

Explicit hand-offs: ConnectionBackoff → R2-01 (S7 EnsureConnectedAsync throttle + the STAB-14/15 wrapper/classification fixes; B3.6's fork deletion removes the poll-loop half of STAB-14's blast surface in whichever order the two plans land). Parser strictness / ReadEnum / remaining equipment-parser parity → R2-11. Carried and untouched here: STAB-10, STAB-11, full STAB-12 (FOCAS close-PDU + TwinCAT blocking Connect), CONV-3 (ILoggerFactory — worth an early ride-along in any of these PRs per the report, but owned elsewhere), PERF-1/2, CONV-5 idioms.