# Follow-up #10 — Wire CapabilityInvoker into the production dispatch layer (RESILIENCE-DISPATCH-GAP)

> **Status:** IMPLEMENTED (code-complete + unit/analyzer/context-verified) · live behavioral gate PENDING ·
> **Branch:** `fix/archreview-crit10-wire-capability-invoker` @ `62556c24` (off `fix/archreview-c1-wire-analyzer`,
> NOT master — the analyzer is the standing guard + the pragmas to remove live there) · **Surfaced by:** guard
> 07/C-1 · **Task:** #10 · **Severity:** High · **Risk:** Medium (behavior-changing hot path). See [`STATUS.md`](STATUS.md).

## OUTCOME (2026-07-08) — implemented as a Core.Abstractions SEAM, not a direct thread

**The filed plan below was INFEASIBLE as written.** It assumed `DriverInstanceActor` could reference
`CapabilityInvoker` directly. It cannot: **Runtime is deliberately Polly-free** — it references
`Core.Abstractions`, NOT `Core` (where `CapabilityInvoker` + Polly live), the exact boundary `IDriverFactory`
documents ("so the Runtime project doesn't pull in Core which would drag in Polly + driver hosting"). Threading
the concrete invoker in would reverse that architectural decision.

**Fix shipped = a seam mirroring `IDriverFactory`:**
- `Core.Abstractions`: `IDriverCapabilityInvoker` (+ `NullDriverCapabilityInvoker` pass-through),
  `IDriverCapabilityInvokerFactory` (+ null factory).
- `Core`: `CapabilityInvoker` implements the interface; new `DriverCapabilityInvokerFactory` builds a per-instance
  invoker over the singleton pipeline builder + status tracker + `DriverFactoryRegistry.GetTier`.
- `Runtime`: `DriverInstanceActor` takes an `IDriverCapabilityInvoker` (default = pass-through) and wraps its
  **6** dispatch sites; `DriverHostActor.SpawnChild` injects the real invoker built by the factory (resolved from
  DI in the Runtime SCE like `IDriverFactory`).
- `Host`: `DriverFactoryBootstrap` registers the tracker + builder + concrete factory.
- **Analyzer**: `IDriverCapabilityInvoker.ExecuteAsync/ExecuteWriteAsync` added as wrapper homes (interface-typed
  invoker calls must suppress OTOPCUA0001 too). All 6 `RESILIENCE-DISPATCH-GAP` pragmas removed.

**Two triage corrections vs. the filed table below:**
1. Only **6** sites are real (all in `DriverInstanceActor`). The 7th — `GenericDriverNodeManager` — is **test-only
   scaffolding** (zero production references; the Server has its own address-space path). Its call is NOT wired;
   the pragma is re-annotated accordingly, not left as a "tracked gap".
2. ~~**Tier-DEFAULT policy only.** `DriverInstanceSpec` (the deploy artifact) does not carry the per-instance
   `ResilienceConfig` JSON, so overrides aren't applied yet.~~ **✅ RESOLVED by task #13**
   (`fix/archreview-crit13-resilience-config-artifact` `75403caa`): the artifact already emitted the column (the
   composer serializes the whole `DriverInstance` entity; AdminUI already authored+persisted it) — only the read
   path dropped it. `DriverInstanceSpec` now carries `ResilienceConfig`, `Create` takes + layers it (invalidating
   the instance's cached pipelines so a respawn takes effect), and `DriverSpawnPlanner` respawns on a
   ResilienceConfig change. Per-instance overrides now reach the invoker. See
   [`FOLLOWUP-13`](FOLLOWUP-13-resilience-config-artifact.md).

**Observability finding + fix.** The resilience-status tracker has **no reader** (Admin `/hosts`, Phase 6.1 Stream
E.2/E.3, was never built) and the pipeline builder logged nothing — so the pipeline ran **invisibly**. Added
retry / breaker-open / breaker-close **structured logging** to `DriverResiliencePipelineBuilder` (optional ILogger,
wired in the Host DI). This is the operator-facing observability surface AND the prerequisite for any live-verify.

**Verification done (deterministic + local):**
- Negative control: unwrapping any site fails the Runtime build with OTOPCUA0001.
- Recording-invoker Runtime test: write routes via `ExecuteWriteAsync` with the `IPerCallHostResolver` host;
  subscribe via `ExecuteAsync` — proves runtime routing + capability + host key.
- **Real-`CapabilityInvoker` actor-context test**: a genuinely-yielding discovery driven through the real invoker
  publishes `DiscoveredNodesReady` — proving the invoker's internal `ConfigureAwait(false)` does NOT leak to the
  actor's `await` (the one path pass-through can't cover; needed a test-only Core ref in Runtime.Tests).
- Analyzer test: the interface is a valid wrapper home. Pipeline-builder test: retry events are logged.
- Full solution builds clean (0 errors); Runtime.Tests 358, Core.Tests 238, Analyzers 32 green; pass-through
  default keeps existing dispatch tests byte-for-byte unchanged.

**RESIDUAL — live behavioral gate (pending, recipe below).** The two real risks (routing + actor-context) are now
covered deterministically. What's unrun is an end-to-end rig check. Recipe:
1. `docker compose -f docker-dev/docker-compose.yml build central-1 central-2` (builds the Host FROM SOURCE →
   picks up this branch; note: the container restore hits the private Gitea feed — may need feed creds/nuget.config
   in the build context).
2. Deploy a Modbus driver (AdminUI `:9200` or `POST /api/deployments`) pointing at the modbus sim
   (`10.100.0.35:5020`).
3. Force an **exception** on an invoker-wrapped call: bounce the sim (`ssh dohertj2@10.100.0.35 docker restart <sim>`)
   while the actor is (re)subscribing/discovering — reads are subscription-based (NOT invoker-wrapped) and device
   rejects return bad-status (no exception), so only connection-level exceptions on subscribe/discover/write trip
   retry/breaker.
4. Observe the `Driver resilience retry` / `circuit-breaker OPENED` log lines in the central-1 container logs, and
   recovery (`circuit-breaker closed`). Also confirms the Host DI resolves + injects the factory end-to-end.

---

## (Original filed analysis — retained for history; superseded by OUTCOME above)

## The finding

Wiring the custom `UnwrappedCapabilityCallAnalyzer` (OTOPCUA0001) tree-wide surfaced that the entire
**Phase 6.1 `CapabilityInvoker` resilience pipeline is constructed only in tests** — it is instantiated
nowhere in production. The production dispatch layer calls driver-capability methods
(`IReadable`/`IWritable`/`ISubscribable`/`ITagDiscovery`/`IAlarmSource`) **directly**, bypassing the
retry / circuit-breaker / bulkhead / tracker-telemetry pipeline entirely.

`CapabilityInvoker`'s own XML doc falsely claims: *"The server's dispatch layer routes every capability
call … through this invoker."* It does not. This is the exact "built-but-never-wired" class the whole
arch review targets — the analyzer did its job on the first wiring.

### Evidence
- `grep -rn "new CapabilityInvoker"` → only `tests/Core/…` (7 test files). Zero production constructions.
- `DriverInstanceActor` holds a raw `IDriver _driver` and calls it directly at 6 sites.
- `GenericDriverNodeManager.BuildAddressSpaceAsync` calls `discovery.DiscoverAsync` directly (1 site).

### The 7 deferred sites (all marked `#pragma warning disable OTOPCUA0001 // RESILIENCE-DISPATCH-GAP`)
Grep the marker to find them: `grep -rn "RESILIENCE-DISPATCH-GAP" src`.

| File | ~line | Call |
|---|---|---|
| `src/Server/…/Runtime/Drivers/DriverInstanceActor.cs` | 575 | `writable.WriteAsync` |
| `src/Server/…/Runtime/Drivers/DriverInstanceActor.cs` | 623 | `src.AcknowledgeAsync` |
| `src/Server/…/Runtime/Drivers/DriverInstanceActor.cs` | 660 | `subscribable.SubscribeAsync` |
| `src/Server/…/Runtime/Drivers/DriverInstanceActor.cs` | 685 | `subscribable.UnsubscribeAsync` |
| `src/Server/…/Runtime/Drivers/DriverInstanceActor.cs` | 762 | `src.SubscribeAlarmsAsync` |
| `src/Server/…/Runtime/Drivers/DriverInstanceActor.cs` | 821 | `discovery.DiscoverAsync` |
| `src/Core/…/Core/OpcUa/GenericDriverNodeManager.cs` | 71 | `discovery.DiscoverAsync` |

> These are distinct from the 3 **driver-internal self-call** sites (AbCipAlarmProjection ×2, S7Driver ×1)
> which are `#pragma`'d as *intentional* — a driver must not re-wrap its own internal poll/ack calls; the
> invoker wraps the driver from the dispatch layer OUTWARD. Do NOT touch those.

## Why it wasn't fixed inside 07/C-1

07/C-1's scope was "wire the analyzer + triage." Actually routing all 7 sites through `CapabilityInvoker`
is a substantial, behavior-changing hot-path remediation (activates retry/breaker/bulkhead on every live
driver read/write/subscribe/discover), which must be its own verified change — exactly the discipline the
review espouses. So the sites were suppressed as a *tracked* gap (not "intentional"), keeping the tree
green and the analyzer live everywhere else, and this follow-up was filed.

## Proposed remediation

1. **Construct a per-`DriverInstance` `CapabilityInvoker`** and thread it into `DriverInstanceActor` (via
   its `Props`/ctor) and `GenericDriverNodeManager`. It needs:
   - the process-singleton `DriverResiliencePipelineBuilder`,
   - a `Func<DriverResilienceOptions>` options accessor (source the per-instance resilience options —
     confirm where they live / how Admin edits them),
   - the `driverType` string (already on the actor),
   - the `DriverResilienceStatusTracker` (so Admin `/hosts` shows in-flight/bulkhead depth),
   - the per-call host resolver (`IPerCallHostResolver`) for multi-host drivers.
2. **Route each of the 7 sites** through `ExecuteAsync` / `ExecuteWriteAsync` (writes/acks use the write
   variant). Remove the `RESILIENCE-DISPATCH-GAP` pragma at each site as it is wired.
3. **Fix `CapabilityInvoker`'s XML doc** once the claim becomes true.

## Verification bar (unit-green ≠ wired)

- **Unit:** an actor test asserting the capability call goes through a fake/recording invoker (not the raw
  driver). The analyzer itself becomes the standing guard — once the pragmas are gone, any regression that
  re-introduces a raw call fails the build.
- **Live (decisive):** on the docker-dev / driver rig, prove the pipeline is actually engaged — e.g. use
  the modbus `exception_injector` (see the write-outcome memory) or a flaky-read fixture to observe a retry
  / breaker-open transition and the `/hosts` in-flight counter moving. A behavior change on the live data
  path must not ship on unit tests alone.

## Decision needed

- Confirm this is worth doing now (it changes runtime latency/behavior on every driver call) vs. deferring.
- If deferring, keep the `RESILIENCE-DISPATCH-GAP` pragmas + this file as the standing record.
- Own branch off `master` (like the Criticals).