Joseph Doherty
051a59845d
- New FOLLOWUP-13: the DriverInstance.ResilienceConfig column was authored/persisted and already in the artifact; only the driver-node read path dropped it. Read-path plumbing + invalidate-on-change + respawn-on-change; deterministic verification. - FOLLOWUP-10 sub-gap (a) 'tier-defaults only' → RESOLVED by #13 (sub-gap (b) tracker reader still open). - STATUS.md: task #13 → completed; crit13 branch topology row; Completed-table row; findings #10 bullet updated; FOLLOWUP-13 linked.
146 lines
10 KiB
Markdown
146 lines
10 KiB
Markdown
# Follow-up #10 — Wire CapabilityInvoker into the production dispatch layer (RESILIENCE-DISPATCH-GAP)
|
||
|
||
> **Status:** IMPLEMENTED (code-complete + unit/analyzer/context-verified) · live behavioral gate PENDING ·
|
||
> **Branch:** `fix/archreview-crit10-wire-capability-invoker` @ `62556c24` (off `fix/archreview-c1-wire-analyzer`,
|
||
> NOT master — the analyzer is the standing guard + the pragmas to remove live there) · **Surfaced by:** guard
|
||
> 07/C-1 · **Task:** #10 · **Severity:** High · **Risk:** Medium (behavior-changing hot path). See [`STATUS.md`](STATUS.md).
|
||
|
||
## OUTCOME (2026-07-08) — implemented as a Core.Abstractions SEAM, not a direct thread
|
||
|
||
**The filed plan below was INFEASIBLE as written.** It assumed `DriverInstanceActor` could reference
|
||
`CapabilityInvoker` directly. It cannot: **Runtime is deliberately Polly-free** — it references
|
||
`Core.Abstractions`, NOT `Core` (where `CapabilityInvoker` + Polly live), the exact boundary `IDriverFactory`
|
||
documents ("so the Runtime project doesn't pull in Core which would drag in Polly + driver hosting"). Threading
|
||
the concrete invoker in would reverse that architectural decision.
|
||
|
||
**Fix shipped = a seam mirroring `IDriverFactory`:**
|
||
- `Core.Abstractions`: `IDriverCapabilityInvoker` (+ `NullDriverCapabilityInvoker` pass-through),
|
||
`IDriverCapabilityInvokerFactory` (+ null factory).
|
||
- `Core`: `CapabilityInvoker` implements the interface; new `DriverCapabilityInvokerFactory` builds a per-instance
|
||
invoker over the singleton pipeline builder + status tracker + `DriverFactoryRegistry.GetTier`.
|
||
- `Runtime`: `DriverInstanceActor` takes an `IDriverCapabilityInvoker` (default = pass-through) and wraps its
|
||
**6** dispatch sites; `DriverHostActor.SpawnChild` injects the real invoker built by the factory (resolved from
|
||
DI in the Runtime SCE like `IDriverFactory`).
|
||
- `Host`: `DriverFactoryBootstrap` registers the tracker + builder + concrete factory.
|
||
- **Analyzer**: `IDriverCapabilityInvoker.ExecuteAsync/ExecuteWriteAsync` added as wrapper homes (interface-typed
|
||
invoker calls must suppress OTOPCUA0001 too). All 6 `RESILIENCE-DISPATCH-GAP` pragmas removed.
|
||
|
||
**Two triage corrections vs. the filed table below:**
|
||
1. Only **6** sites are real (all in `DriverInstanceActor`). The 7th — `GenericDriverNodeManager` — is **test-only
|
||
scaffolding** (zero production references; the Server has its own address-space path). Its call is NOT wired;
|
||
the pragma is re-annotated accordingly, not left as a "tracked gap".
|
||
2. ~~**Tier-DEFAULT policy only.** `DriverInstanceSpec` (the deploy artifact) does not carry the per-instance
|
||
`ResilienceConfig` JSON, so overrides aren't applied yet.~~ **✅ RESOLVED by task #13**
|
||
(`fix/archreview-crit13-resilience-config-artifact` `75403caa`): the artifact already emitted the column (the
|
||
composer serializes the whole `DriverInstance` entity; AdminUI already authored+persisted it) — only the read
|
||
path dropped it. `DriverInstanceSpec` now carries `ResilienceConfig`, `Create` takes + layers it (invalidating
|
||
the instance's cached pipelines so a respawn takes effect), and `DriverSpawnPlanner` respawns on a
|
||
ResilienceConfig change. Per-instance overrides now reach the invoker. See
|
||
[`FOLLOWUP-13`](FOLLOWUP-13-resilience-config-artifact.md).
|
||
|
||
**Observability finding + fix.** The resilience-status tracker has **no reader** (Admin `/hosts`, Phase 6.1 Stream
|
||
E.2/E.3, was never built) and the pipeline builder logged nothing — so the pipeline ran **invisibly**. Added
|
||
retry / breaker-open / breaker-close **structured logging** to `DriverResiliencePipelineBuilder` (optional ILogger,
|
||
wired in the Host DI). This is the operator-facing observability surface AND the prerequisite for any live-verify.
|
||
|
||
**Verification done (deterministic + local):**
|
||
- Negative control: unwrapping any site fails the Runtime build with OTOPCUA0001.
|
||
- Recording-invoker Runtime test: write routes via `ExecuteWriteAsync` with the `IPerCallHostResolver` host;
|
||
subscribe via `ExecuteAsync` — proves runtime routing + capability + host key.
|
||
- **Real-`CapabilityInvoker` actor-context test**: a genuinely-yielding discovery driven through the real invoker
|
||
publishes `DiscoveredNodesReady` — proving the invoker's internal `ConfigureAwait(false)` does NOT leak to the
|
||
actor's `await` (the one path pass-through can't cover; needed a test-only Core ref in Runtime.Tests).
|
||
- Analyzer test: the interface is a valid wrapper home. Pipeline-builder test: retry events are logged.
|
||
- Full solution builds clean (0 errors); Runtime.Tests 358, Core.Tests 238, Analyzers 32 green; pass-through
|
||
default keeps existing dispatch tests byte-for-byte unchanged.
|
||
|
||
**RESIDUAL — live behavioral gate (pending, recipe below).** The two real risks (routing + actor-context) are now
|
||
covered deterministically. What's unrun is an end-to-end rig check. Recipe:
|
||
1. `docker compose -f docker-dev/docker-compose.yml build central-1 central-2` (builds the Host FROM SOURCE →
|
||
picks up this branch; note: the container restore hits the private Gitea feed — may need feed creds/nuget.config
|
||
in the build context).
|
||
2. Deploy a Modbus driver (AdminUI `:9200` or `POST /api/deployments`) pointing at the modbus sim
|
||
(`10.100.0.35:5020`).
|
||
3. Force an **exception** on an invoker-wrapped call: bounce the sim (`ssh dohertj2@10.100.0.35 docker restart <sim>`)
|
||
while the actor is (re)subscribing/discovering — reads are subscription-based (NOT invoker-wrapped) and device
|
||
rejects return bad-status (no exception), so only connection-level exceptions on subscribe/discover/write trip
|
||
retry/breaker.
|
||
4. Observe the `Driver resilience retry` / `circuit-breaker OPENED` log lines in the central-1 container logs, and
|
||
recovery (`circuit-breaker closed`). Also confirms the Host DI resolves + injects the factory end-to-end.
|
||
|
||
---
|
||
|
||
## (Original filed analysis — retained for history; superseded by OUTCOME above)
|
||
|
||
## The finding
|
||
|
||
Wiring the custom `UnwrappedCapabilityCallAnalyzer` (OTOPCUA0001) tree-wide surfaced that the entire
|
||
**Phase 6.1 `CapabilityInvoker` resilience pipeline is constructed only in tests** — it is instantiated
|
||
nowhere in production. The production dispatch layer calls driver-capability methods
|
||
(`IReadable`/`IWritable`/`ISubscribable`/`ITagDiscovery`/`IAlarmSource`) **directly**, bypassing the
|
||
retry / circuit-breaker / bulkhead / tracker-telemetry pipeline entirely.
|
||
|
||
`CapabilityInvoker`'s own XML doc falsely claims: *"The server's dispatch layer routes every capability
|
||
call … through this invoker."* It does not. This is the exact "built-but-never-wired" class the whole
|
||
arch review targets — the analyzer did its job on the first wiring.
|
||
|
||
### Evidence
|
||
- `grep -rn "new CapabilityInvoker"` → only `tests/Core/…` (7 test files). Zero production constructions.
|
||
- `DriverInstanceActor` holds a raw `IDriver _driver` and calls it directly at 6 sites.
|
||
- `GenericDriverNodeManager.BuildAddressSpaceAsync` calls `discovery.DiscoverAsync` directly (1 site).
|
||
|
||
### The 7 deferred sites (all marked `#pragma warning disable OTOPCUA0001 // RESILIENCE-DISPATCH-GAP`)
|
||
Grep the marker to find them: `grep -rn "RESILIENCE-DISPATCH-GAP" src`.
|
||
|
||
| File | ~line | Call |
|
||
|---|---|---|
|
||
| `src/Server/…/Runtime/Drivers/DriverInstanceActor.cs` | 575 | `writable.WriteAsync` |
|
||
| `src/Server/…/Runtime/Drivers/DriverInstanceActor.cs` | 623 | `src.AcknowledgeAsync` |
|
||
| `src/Server/…/Runtime/Drivers/DriverInstanceActor.cs` | 660 | `subscribable.SubscribeAsync` |
|
||
| `src/Server/…/Runtime/Drivers/DriverInstanceActor.cs` | 685 | `subscribable.UnsubscribeAsync` |
|
||
| `src/Server/…/Runtime/Drivers/DriverInstanceActor.cs` | 762 | `src.SubscribeAlarmsAsync` |
|
||
| `src/Server/…/Runtime/Drivers/DriverInstanceActor.cs` | 821 | `discovery.DiscoverAsync` |
|
||
| `src/Core/…/Core/OpcUa/GenericDriverNodeManager.cs` | 71 | `discovery.DiscoverAsync` |
|
||
|
||
> These are distinct from the 3 **driver-internal self-call** sites (AbCipAlarmProjection ×2, S7Driver ×1)
|
||
> which are `#pragma`'d as *intentional* — a driver must not re-wrap its own internal poll/ack calls; the
|
||
> invoker wraps the driver from the dispatch layer OUTWARD. Do NOT touch those.
|
||
|
||
## Why it wasn't fixed inside 07/C-1
|
||
|
||
07/C-1's scope was "wire the analyzer + triage." Actually routing all 7 sites through `CapabilityInvoker`
|
||
is a substantial, behavior-changing hot-path remediation (activates retry/breaker/bulkhead on every live
|
||
driver read/write/subscribe/discover), which must be its own verified change — exactly the discipline the
|
||
review espouses. So the sites were suppressed as a *tracked* gap (not "intentional"), keeping the tree
|
||
green and the analyzer live everywhere else, and this follow-up was filed.
|
||
|
||
## Proposed remediation
|
||
|
||
1. **Construct a per-`DriverInstance` `CapabilityInvoker`** and thread it into `DriverInstanceActor` (via
|
||
its `Props`/ctor) and `GenericDriverNodeManager`. It needs:
|
||
- the process-singleton `DriverResiliencePipelineBuilder`,
|
||
- a `Func<DriverResilienceOptions>` options accessor (source the per-instance resilience options —
|
||
confirm where they live / how Admin edits them),
|
||
- the `driverType` string (already on the actor),
|
||
- the `DriverResilienceStatusTracker` (so Admin `/hosts` shows in-flight/bulkhead depth),
|
||
- the per-call host resolver (`IPerCallHostResolver`) for multi-host drivers.
|
||
2. **Route each of the 7 sites** through `ExecuteAsync` / `ExecuteWriteAsync` (writes/acks use the write
|
||
variant). Remove the `RESILIENCE-DISPATCH-GAP` pragma at each site as it is wired.
|
||
3. **Fix `CapabilityInvoker`'s XML doc** once the claim becomes true.
|
||
|
||
## Verification bar (unit-green ≠ wired)
|
||
|
||
- **Unit:** an actor test asserting the capability call goes through a fake/recording invoker (not the raw
|
||
driver). The analyzer itself becomes the standing guard — once the pragmas are gone, any regression that
|
||
re-introduces a raw call fails the build.
|
||
- **Live (decisive):** on the docker-dev / driver rig, prove the pipeline is actually engaged — e.g. use
|
||
the modbus `exception_injector` (see the write-outcome memory) or a flaky-read fixture to observe a retry
|
||
/ breaker-open transition and the `/hosts` in-flight counter moving. A behavior change on the live data
|
||
path must not ship on unit tests alone.
|
||
|
||
## Decision needed
|
||
|
||
- Confirm this is worth doing now (it changes runtime latency/behavior on every driver call) vs. deferring.
|
||
- If deferring, keep the `RESILIENCE-DISPATCH-GAP` pragmas + this file as the standing record.
|
||
- Own branch off `master` (like the Criticals).
|