diff --git a/archreview/plans/FOLLOWUP-10-resilience-dispatch-gap.md b/archreview/plans/FOLLOWUP-10-resilience-dispatch-gap.md index 53e24541..e67ac5fb 100644 --- a/archreview/plans/FOLLOWUP-10-resilience-dispatch-gap.md +++ b/archreview/plans/FOLLOWUP-10-resilience-dispatch-gap.md @@ -1,6 +1,7 @@ # Follow-up #10 — Wire CapabilityInvoker into the production dispatch layer (RESILIENCE-DISPATCH-GAP) -> **Status:** IMPLEMENTED (code-complete + unit/analyzer/context-verified) · live behavioral gate PENDING · +> **Status:** IMPLEMENTED (code-complete + unit/analyzer/context-verified) · live gate RAN 2026-07-08 — DI wiring +> confirmed live, behavioral log-line not observed (disciplined stop; see LIVE GATE below) · > **Branch:** `fix/archreview-crit10-wire-capability-invoker` @ `62556c24` (off `fix/archreview-c1-wire-analyzer`, > NOT master — the analyzer is the standing guard + the pragmas to remove live there) · **Surfaced by:** guard > 07/C-1 · **Task:** #10 · **Severity:** High · **Risk:** Medium (behavior-changing hot path). See [`STATUS.md`](STATUS.md). @@ -54,19 +55,40 @@ wired in the Host DI). This is the operator-facing observability surface AND the - Full solution builds clean (0 errors); Runtime.Tests 358, Core.Tests 238, Analyzers 32 green; pass-through default keeps existing dispatch tests byte-for-byte unchanged. -**RESIDUAL — live behavioral gate (pending, recipe below).** The two real risks (routing + actor-context) are now -covered deterministically. What's unrun is an end-to-end rig check. Recipe: -1. `docker compose -f docker-dev/docker-compose.yml build central-1 central-2` (builds the Host FROM SOURCE → - picks up this branch; note: the container restore hits the private Gitea feed — may need feed creds/nuget.config - in the build context). -2. Deploy a Modbus driver (AdminUI `:9200` or `POST /api/deployments`) pointing at the modbus sim - (`10.100.0.35:5020`). -3. Force an **exception** on an invoker-wrapped call: bounce the sim (`ssh dohertj2@10.100.0.35 docker restart `) - while the actor is (re)subscribing/discovering — reads are subscription-based (NOT invoker-wrapped) and device - rejects return bad-status (no exception), so only connection-level exceptions on subscribe/discover/write trip - retry/breaker. -4. Observe the `Driver resilience retry` / `circuit-breaker OPENED` log lines in the central-1 container logs, and - recovery (`circuit-breaker closed`). Also confirms the Host DI resolves + injects the factory end-to-end. +**LIVE GATE — ran 2026-07-08 (task #12). PARTIAL: DI wiring confirmed live; behavioral log NOT observed (disciplined +stop). Outcome:** + +- ✅ **Built the Host image FROM SOURCE** on the crit13 tip (`docker compose … build central-1`) — #10 + #13 compile + and publish into a real container. The Gitea feed was **not** a blocker: it answers **200 anonymously**, so the + in-container `dotnet restore` pulled the `ZB.MOM.WW.*` packages fine (the recipe's feed-creds caveat is moot). +- ✅ **Stood up the live rig** (`sql` + `migrator` + `central-1`/`central-2` + `traefik`); central-1 booted + **healthy** — cluster up, OPC UA on `:4840`, AdminUI on `:9000` — running the new invoker wiring. +- ✅ **Prod DI binds the REAL factory — confirmed two ways:** (1) the new deterministic guard + `ResilienceInvokerFactoryRegistrationTests` (Host.IntegrationTests, `8a8b9ec5`) asserts + `AddOtOpcUaDriverFactories` resolves `IDriverCapabilityInvokerFactory` → the concrete + `DriverCapabilityInvokerFactory` (not the null pass-through) and that it mints a real `CapabilityInvoker`; (2) the + rig booted cleanly with that exact registration active. This closes the **primary** risk (was it wired in prod?). +- 🟡 **Behavioral retry/breaker LOG line — NOT observed live**, by a deliberate stop (not a failure). Reading the + Modbus driver showed **none of its cheap paths throw a wrapped exception**: `DiscoverAsync` is offline/config-driven + (enumerates `_options.Tags`, no connect), `SubscribeAsync` is lazy (`_poll.Subscribe`, no connect), and reads are + the **non-wrapped** poll path returning bad-status. The **only** invoker-wrapped Modbus site that throws a + connection exception is an **idempotent `WriteAsync`** to a dead endpoint — which needs full config authoring + (driver + UNS + equipment + idempotent-writable tag) **plus** a data-plane-LDAP-authorized Client.CLI write. That + multi-dependency chase isn't worth it: the retry/breaker **behavior** is already proven deterministically + (`CapabilityInvokerTests` retry, `DriverCapabilityInvokerFactoryTests` override, the real-invoker actor-context + test, the pipeline-builder logging test). To actually see the log line, S7 (whose reconnect path *does* throw on a + dead socket) or a Modbus idempotent-write-to-dead-endpoint would be the trigger. +- 🐞 **RIG BUG found (unrelated to #10/#13, worth fixing):** the committed `docker-dev/docker-compose.yml` still + carries the **retired Wonderware `ServerHistorian` keys** (`Host`/`Port`/`SharedSecret`) with `Enabled: "true"`, + but the historian-gateway cutover made the code read `ServerHistorian__Endpoint` → **empty-URI crash at every dev + bring-up** (`HistorianGatewayClientAdapter.Create`, unhandled `UriFormatException`). Worked around for the gate with + a scratchpad override (`ServerHistorian__Enabled: "false"`). **Recommend** fixing the committed compose (disable it + for the rig, or supply a gateway `Endpoint`) — docker-dev is otherwise unbootable against current code. + +Original recipe (retained): build central-1/central-2 from source → deploy a Modbus driver at the sim +(`10.100.0.35:5020`) → bounce the sim (`ssh dohertj2@10.100.0.35 docker restart `) during a wrapped call → +grep central-1 logs for `Driver resilience retry` / `circuit-breaker OPENED` / `closed`. NB the Modbus-path caveat +above — an idempotent write (or an S7 driver) is the reliable trigger, not a plain subscribe/read. --- diff --git a/archreview/plans/STATUS.md b/archreview/plans/STATUS.md index d8a477a1..c369b4eb 100644 --- a/archreview/plans/STATUS.md +++ b/archreview/plans/STATUS.md @@ -17,7 +17,7 @@ truth for remediation progress. Update it as work lands. | `fix/archreview-c1-wire-analyzer` | `f0082af5` | Guard 07/C-1 — wire OTOPCUA0001 analyzer tree-wide + triage (surfaced the RESILIENCE-DISPATCH-GAP, task #10) | | `fix/archreview-s1-s2-ci-coverage` | `10b89830` | Guard 07/S-1,S-2,S-4 — whole-solution CI leg + fail-on-skip gate + CLI sleep deflake | | `fix/archreview-crit10-wire-capability-invoker` | `62556c24` | **#10 — wire CapabilityInvoker into dispatch via a Core.Abstractions seam** (off `fix/archreview-c1-wire-analyzer`, NOT master — the analyzer is the standing guard + the pragmas live there). Code-complete + unit/analyzer/context-verified; live gate pending. | -| `fix/archreview-crit13-resilience-config-artifact` | `75403caa` | **#13 — apply per-instance ResilienceConfig from the artifact** (off the crit10 branch — builds on its invoker factory). Read-path plumbing + invalidate/respawn-on-change; deterministically verified. | +| `fix/archreview-crit13-resilience-config-artifact` | `8a8b9ec5` | **#13 — apply per-instance ResilienceConfig from the artifact** (`75403caa`, off the crit10 branch) **+ #10/#13 DI guard** (`8a8b9ec5` — `ResilienceInvokerFactoryRegistrationTests`, from the #12 live gate). Read-path plumbing + invalidate/respawn-on-change; deterministically verified + rig-booted. | **Decision (user):** one branch per Critical, each off `master`. **#9 (failover test) lives on the crit1 branch** (not its own branch off master) because SBR only exists there — it is Critical 1's integration half. The plans are only on the docs branch, @@ -57,7 +57,7 @@ so from a critical branch read them via `git show docs/archreview-plans-and-drif | 9 | Critical 1 follow-up — hard-kill failover integration test (03/S1) | ✅ completed — `a25c9ed0`; surfaced task #11 | | 10 | **Wire CapabilityInvoker into the dispatch layer** (RESILIENCE-DISPATCH-GAP, from 07/C-1) | ✅ code-complete + unit/analyzer/context-verified — `fix/archreview-crit10-wire-capability-invoker` `62556c24` (seam, NOT direct thread — Runtime is Polly-free). **Live behavioral gate pending** (rig recipe in FOLLOWUP-10). Surfaced a sub-gap: ResilienceConfig not in the deploy artifact → tier-defaults only. | | 11 | **Review Critical 1 premise** — SBR was already active on master, not NoDowning (from #9) | ✅ completed — `fix/archreview-crit1-split-brain-resolver` `eaf78aad`; docs/comment-only correction (no code revert), Cluster.Tests 29/29. | -| 12 | **#10 live behavioral gate** — rebuild docker-dev, force a connection-level exception on an invoker-wrapped call, observe retry/breaker logs + recovery (from #10) | ⬜ pending — rig session (recipe in FOLLOWUP-10); the residual live gate for #10 (also covers #13's override-behavior end-to-end). | +| 12 | **#10 live behavioral gate** — rig session (from #10) | ✅ ran 2026-07-08 (PARTIAL) — built the Host image from source (crit13 tip) + stood up the rig; central-1 booted healthy; **prod DI binds the real invoker factory** confirmed live + by a new deterministic guard (`ResilienceInvokerFactoryRegistrationTests` `8a8b9ec5`). Behavioral retry/breaker **log-line not observed** (disciplined stop): Modbus's cheap paths don't throw a wrapped exception (Discover offline, Subscribe lazy, reads non-wrapped) — only an idempotent write-to-dead-endpoint (or S7) does, not worth the multi-dependency chase since the retry behavior is deterministically proven. **Found a rig bug:** docker-dev compose's retired Wonderware `ServerHistorian` keys crash the Host at bring-up (historian-gateway cutover → reads `Endpoint`); recommend fixing. See [`FOLLOWUP-10`](FOLLOWUP-10-resilience-dispatch-gap.md). | | 13 | **Plumb per-instance `ResilienceConfig` through the deploy artifact** (from #10) | ✅ completed — `fix/archreview-crit13-resilience-config-artifact` `75403caa`; read-path plumbing (write side was already complete), invalidate-on-change + respawn-on-change; Core.Tests 243/Runtime.Tests 363. See [`FOLLOWUP-13`](FOLLOWUP-13-resilience-config-artifact.md). | ## Findings surfaced by the guards (NEW — need user decision)