lmxopcua/archreview/plans/FOLLOWUP-10-resilience-dispatch-gap.md

Follow-up #10 — Wire CapabilityInvoker into the production dispatch layer (RESILIENCE-DISPATCH-GAP)

Status: IMPLEMENTED (code-complete + unit/analyzer/context-verified) · live behavioral gate PENDING · Branch: fix/archreview-crit10-wire-capability-invoker @ 62556c24 (off fix/archreview-c1-wire-analyzer, NOT master — the analyzer is the standing guard + the pragmas to remove live there) · Surfaced by: guard 07/C-1 · Task: #10 · Severity: High · Risk: Medium (behavior-changing hot path). See STATUS.md.

OUTCOME (2026-07-08) — implemented as a Core.Abstractions SEAM, not a direct thread

The filed plan below was INFEASIBLE as written. It assumed DriverInstanceActor could reference CapabilityInvoker directly. It cannot: Runtime is deliberately Polly-free — it references Core.Abstractions, NOT Core (where CapabilityInvoker + Polly live), the exact boundary IDriverFactory documents ("so the Runtime project doesn't pull in Core which would drag in Polly + driver hosting"). Threading the concrete invoker in would reverse that architectural decision.

Fix shipped = a seam mirroring IDriverFactory:

• Core.Abstractions: IDriverCapabilityInvoker (+ NullDriverCapabilityInvoker pass-through), IDriverCapabilityInvokerFactory (+ null factory).
• Core: CapabilityInvoker implements the interface; new DriverCapabilityInvokerFactory builds a per-instance invoker over the singleton pipeline builder + status tracker + DriverFactoryRegistry.GetTier.
• Runtime: DriverInstanceActor takes an IDriverCapabilityInvoker (default = pass-through) and wraps its 6 dispatch sites; DriverHostActor.SpawnChild injects the real invoker built by the factory (resolved from DI in the Runtime SCE like IDriverFactory).
• Host: DriverFactoryBootstrap registers the tracker + builder + concrete factory.
• Analyzer: IDriverCapabilityInvoker.ExecuteAsync/ExecuteWriteAsync added as wrapper homes (interface-typed invoker calls must suppress OTOPCUA0001 too). All 6 RESILIENCE-DISPATCH-GAP pragmas removed.

Two triage corrections vs. the filed table below:

1. Only 6 sites are real (all in DriverInstanceActor). The 7th — GenericDriverNodeManager — is test-only scaffolding (zero production references; the Server has its own address-space path). Its call is NOT wired; the pragma is re-annotated accordingly, not left as a "tracked gap".
2. Tier-DEFAULT policy only. DriverInstanceSpec (the deploy artifact) does not carry the per-instance ResilienceConfig JSON, so overrides aren't applied yet. Sub-gap → new follow-up: plumb ResilienceConfig through the composer/artifact so per-instance overrides reach the invoker.

Observability finding + fix. The resilience-status tracker has no reader (Admin /hosts, Phase 6.1 Stream E.2/E.3, was never built) and the pipeline builder logged nothing — so the pipeline ran invisibly. Added retry / breaker-open / breaker-close structured logging to DriverResiliencePipelineBuilder (optional ILogger, wired in the Host DI). This is the operator-facing observability surface AND the prerequisite for any live-verify.

Verification done (deterministic + local):

• Negative control: unwrapping any site fails the Runtime build with OTOPCUA0001.
• Recording-invoker Runtime test: write routes via ExecuteWriteAsync with the IPerCallHostResolver host; subscribe via ExecuteAsync — proves runtime routing + capability + host key.
• Real-CapabilityInvoker actor-context test: a genuinely-yielding discovery driven through the real invoker publishes DiscoveredNodesReady — proving the invoker's internal ConfigureAwait(false) does NOT leak to the actor's await (the one path pass-through can't cover; needed a test-only Core ref in Runtime.Tests).
• Analyzer test: the interface is a valid wrapper home. Pipeline-builder test: retry events are logged.
• Full solution builds clean (0 errors); Runtime.Tests 358, Core.Tests 238, Analyzers 32 green; pass-through default keeps existing dispatch tests byte-for-byte unchanged.

RESIDUAL — live behavioral gate (pending, recipe below). The two real risks (routing + actor-context) are now covered deterministically. What's unrun is an end-to-end rig check. Recipe:

1. docker compose -f docker-dev/docker-compose.yml build central-1 central-2 (builds the Host FROM SOURCE → picks up this branch; note: the container restore hits the private Gitea feed — may need feed creds/nuget.config in the build context).
2. Deploy a Modbus driver (AdminUI :9200 or POST /api/deployments) pointing at the modbus sim (10.100.0.35:5020).
3. Force an exception on an invoker-wrapped call: bounce the sim (ssh dohertj2@10.100.0.35 docker restart <sim>) while the actor is (re)subscribing/discovering — reads are subscription-based (NOT invoker-wrapped) and device rejects return bad-status (no exception), so only connection-level exceptions on subscribe/discover/write trip retry/breaker.
4. Observe the Driver resilience retry / circuit-breaker OPENED log lines in the central-1 container logs, and recovery (circuit-breaker closed). Also confirms the Host DI resolves + injects the factory end-to-end.

---

(Original filed analysis — retained for history; superseded by OUTCOME above)

The finding

Wiring the custom UnwrappedCapabilityCallAnalyzer (OTOPCUA0001) tree-wide surfaced that the entire Phase 6.1 CapabilityInvoker resilience pipeline is constructed only in tests — it is instantiated nowhere in production. The production dispatch layer calls driver-capability methods (IReadable/IWritable/ISubscribable/ITagDiscovery/IAlarmSource) directly, bypassing the retry / circuit-breaker / bulkhead / tracker-telemetry pipeline entirely.

CapabilityInvoker's own XML doc falsely claims: "The server's dispatch layer routes every capability call … through this invoker." It does not. This is the exact "built-but-never-wired" class the whole arch review targets — the analyzer did its job on the first wiring.

Evidence

• grep -rn "new CapabilityInvoker" → only tests/Core/… (7 test files). Zero production constructions.
• DriverInstanceActor holds a raw IDriver _driver and calls it directly at 6 sites.
• GenericDriverNodeManager.BuildAddressSpaceAsync calls discovery.DiscoverAsync directly (1 site).

The 7 deferred sites (all marked #pragma warning disable OTOPCUA0001 // RESILIENCE-DISPATCH-GAP)

Grep the marker to find them: grep -rn "RESILIENCE-DISPATCH-GAP" src.

File	~line	Call
src/Server/…/Runtime/Drivers/DriverInstanceActor.cs	575	writable.WriteAsync
src/Server/…/Runtime/Drivers/DriverInstanceActor.cs	623	src.AcknowledgeAsync
src/Server/…/Runtime/Drivers/DriverInstanceActor.cs	660	subscribable.SubscribeAsync
src/Server/…/Runtime/Drivers/DriverInstanceActor.cs	685	subscribable.UnsubscribeAsync
src/Server/…/Runtime/Drivers/DriverInstanceActor.cs	762	src.SubscribeAlarmsAsync
src/Server/…/Runtime/Drivers/DriverInstanceActor.cs	821	discovery.DiscoverAsync
src/Core/…/Core/OpcUa/GenericDriverNodeManager.cs	71	discovery.DiscoverAsync

These are distinct from the 3 driver-internal self-call sites (AbCipAlarmProjection ×2, S7Driver ×1) which are #pragma'd as intentional — a driver must not re-wrap its own internal poll/ack calls; the invoker wraps the driver from the dispatch layer OUTWARD. Do NOT touch those.

Why it wasn't fixed inside 07/C-1

07/C-1's scope was "wire the analyzer + triage." Actually routing all 7 sites through CapabilityInvoker is a substantial, behavior-changing hot-path remediation (activates retry/breaker/bulkhead on every live driver read/write/subscribe/discover), which must be its own verified change — exactly the discipline the review espouses. So the sites were suppressed as a tracked gap (not "intentional"), keeping the tree green and the analyzer live everywhere else, and this follow-up was filed.

Proposed remediation

1. Construct a per-DriverInstance CapabilityInvoker and thread it into DriverInstanceActor (via its Props/ctor) and GenericDriverNodeManager. It needs:
   • the process-singleton DriverResiliencePipelineBuilder,
   • a Func<DriverResilienceOptions> options accessor (source the per-instance resilience options — confirm where they live / how Admin edits them),
   • the driverType string (already on the actor),
   • the DriverResilienceStatusTracker (so Admin /hosts shows in-flight/bulkhead depth),
   • the per-call host resolver (IPerCallHostResolver) for multi-host drivers.
2. Route each of the 7 sites through ExecuteAsync / ExecuteWriteAsync (writes/acks use the write variant). Remove the RESILIENCE-DISPATCH-GAP pragma at each site as it is wired.
3. Fix CapabilityInvoker's XML doc once the claim becomes true.

Verification bar (unit-green ≠ wired)

• Unit: an actor test asserting the capability call goes through a fake/recording invoker (not the raw driver). The analyzer itself becomes the standing guard — once the pragmas are gone, any regression that re-introduces a raw call fails the build.
• Live (decisive): on the docker-dev / driver rig, prove the pipeline is actually engaged — e.g. use the modbus exception_injector (see the write-outcome memory) or a flaky-read fixture to observe a retry / breaker-open transition and the /hosts in-flight counter moving. A behavior change on the live data path must not ship on unit tests alone.

Decision needed

• Confirm this is worth doing now (it changes runtime latency/behavior on every driver call) vs. deferring.
• If deferring, keep the RESILIENCE-DISPATCH-GAP pragmas + this file as the standing record.
• Own branch off master (like the Criticals).