dohertj2/lmxopcua
1
0
Fork 0
Code Issues 1 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

docs(archreview #12): record live-gate outcome (DI wiring confirmed live; behavioral log a disciplined stop)

Browse Source 
Ran the #10 live gate: built the Host image from source (crit13 tip; Gitea feed answers
200 anonymously so restore worked), stood up the docker-dev rig, central-1 booted healthy
on the new invoker wiring. Prod DI binds the REAL DriverCapabilityInvokerFactory —
confirmed live + by a new deterministic guard (ResilienceInvokerFactoryRegistrationTests,
crit13 8a8b9ec5). The behavioral retry/breaker log-line was NOT observed: reading the
Modbus driver showed its cheap paths don't throw a wrapped exception (Discover offline,
Subscribe lazy, reads non-wrapped) — only an idempotent write-to-dead-endpoint or S7 does,
not worth the multi-dependency chase since the retry behavior is deterministically proven.

Also found a rig bug: docker-dev compose's retired Wonderware ServerHistorian keys
(Host/Port/SharedSecret, Enabled=true) crash the Host at bring-up (historian-gateway cutover
now reads Endpoint -> empty UriFormatException). Worked around with an override; recommend
fixing the committed compose. FOLLOWUP-10 + STATUS updated; crit13 topology head -> 8a8b9ec5.
This commit is contained in:
Joseph Doherty
2026-07-08 23:25:31 -04:00
parent 051a59845d
commit 347446d627
2 changed files with 38 additions and 16 deletions
Download Patch File Download Diff File Expand all files Collapse all files
archreview/plans/FOLLOWUP-10-resilience-dispatch-gap.md
+36 -14
View File
@@ -1,6 +1,7 @@
# Follow-up #10 — Wire CapabilityInvoker into the production dispatch layer (RESILIENCE-DISPATCH-GAP)
> **Status:** IMPLEMENTED (code-complete + unit/analyzer/context-verified) · live behavioral gate PENDING ·
> **Status:** IMPLEMENTED (code-complete + unit/analyzer/context-verified) · live gate RAN 2026-07-08 — DI wiring
> confirmed live, behavioral log-line not observed (disciplined stop; see LIVE GATE below) ·
> **Branch:** `fix/archreview-crit10-wire-capability-invoker` @ `62556c24` (off `fix/archreview-c1-wire-analyzer`,
> NOT master — the analyzer is the standing guard + the pragmas to remove live there) · **Surfaced by:** guard
> 07/C-1 · **Task:** #10 · **Severity:** High · **Risk:** Medium (behavior-changing hot path). See [`STATUS.md`](STATUS.md).
@@ -54,19 +55,40 @@ wired in the Host DI). This is the operator-facing observability surface AND the
- Full solution builds clean (0 errors); Runtime.Tests 358, Core.Tests 238, Analyzers 32 green; pass-through
default keeps existing dispatch tests byte-for-byte unchanged.
**RESIDUAL — live behavioral gate (pending, recipe below).** The two real risks (routing + actor-context) are now
covered deterministically. What's unrun is an end-to-end rig check. Recipe:
1. `docker compose -f docker-dev/docker-compose.yml build central-1 central-2` (builds the Host FROM SOURCE →
picks up this branch; note: the container restore hits the private Gitea feed — may need feed creds/nuget.config
in the build context).
2. Deploy a Modbus driver (AdminUI `:9200` or `POST /api/deployments`) pointing at the modbus sim
(`10.100.0.35:5020`).
3. Force an **exception** on an invoker-wrapped call: bounce the sim (`ssh dohertj2@10.100.0.35 docker restart <sim>`)
while the actor is (re)subscribing/discovering — reads are subscription-based (NOT invoker-wrapped) and device
rejects return bad-status (no exception), so only connection-level exceptions on subscribe/discover/write trip
retry/breaker.
4. Observe the `Driver resilience retry` / `circuit-breaker OPENED` log lines in the central-1 container logs, and
recovery (`circuit-breaker closed`). Also confirms the Host DI resolves + injects the factory end-to-end.
**LIVE GATE — ran 2026-07-08 (task #12). PARTIAL: DI wiring confirmed live; behavioral log NOT observed (disciplined
stop). Outcome:**
- ✅ **Built the Host image FROM SOURCE** on the crit13 tip (`docker compose … build central-1`) — #10 + #13 compile
and publish into a real container. The Gitea feed was **not** a blocker: it answers **200 anonymously**, so the
in-container `dotnet restore` pulled the `ZB.MOM.WW.*` packages fine (the recipe's feed-creds caveat is moot).
- ✅ **Stood up the live rig** (`sql` + `migrator` + `central-1`/`central-2` + `traefik`); central-1 booted
**healthy** — cluster up, OPC UA on `:4840`, AdminUI on `:9000` — running the new invoker wiring.
- ✅ **Prod DI binds the REAL factory — confirmed two ways:** (1) the new deterministic guard
`ResilienceInvokerFactoryRegistrationTests` (Host.IntegrationTests, `8a8b9ec5`) asserts
`AddOtOpcUaDriverFactories` resolves `IDriverCapabilityInvokerFactory` → the concrete
`DriverCapabilityInvokerFactory` (not the null pass-through) and that it mints a real `CapabilityInvoker`; (2) the
rig booted cleanly with that exact registration active. This closes the **primary** risk (was it wired in prod?).
- 🟡 **Behavioral retry/breaker LOG line — NOT observed live**, by a deliberate stop (not a failure). Reading the
Modbus driver showed **none of its cheap paths throw a wrapped exception**: `DiscoverAsync` is offline/config-driven
(enumerates `_options.Tags`, no connect), `SubscribeAsync` is lazy (`_poll.Subscribe`, no connect), and reads are
the **non-wrapped** poll path returning bad-status. The **only** invoker-wrapped Modbus site that throws a
connection exception is an **idempotent `WriteAsync`** to a dead endpoint — which needs full config authoring
(driver + UNS + equipment + idempotent-writable tag) **plus** a data-plane-LDAP-authorized Client.CLI write. That
multi-dependency chase isn't worth it: the retry/breaker **behavior** is already proven deterministically
(`CapabilityInvokerTests` retry, `DriverCapabilityInvokerFactoryTests` override, the real-invoker actor-context
test, the pipeline-builder logging test). To actually see the log line, S7 (whose reconnect path *does* throw on a
dead socket) or a Modbus idempotent-write-to-dead-endpoint would be the trigger.
- 🐞 **RIG BUG found (unrelated to #10/#13, worth fixing):** the committed `docker-dev/docker-compose.yml` still
carries the **retired Wonderware `ServerHistorian` keys** (`Host`/`Port`/`SharedSecret`) with `Enabled: "true"`,
but the historian-gateway cutover made the code read `ServerHistorian__Endpoint` → **empty-URI crash at every dev
bring-up** (`HistorianGatewayClientAdapter.Create`, unhandled `UriFormatException`). Worked around for the gate with
a scratchpad override (`ServerHistorian__Enabled: "false"`). **Recommend** fixing the committed compose (disable it
for the rig, or supply a gateway `Endpoint`) — docker-dev is otherwise unbootable against current code.
Original recipe (retained): build central-1/central-2 from source → deploy a Modbus driver at the sim
(`10.100.0.35:5020`) → bounce the sim (`ssh dohertj2@10.100.0.35 docker restart <sim>`) during a wrapped call →
grep central-1 logs for `Driver resilience retry` / `circuit-breaker OPENED` / `closed`. NB the Modbus-path caveat
above — an idempotent write (or an S7 driver) is the reliable trigger, not a plain subscribe/read.
---
archreview/plans/STATUS.md
+2 -2
View File
@@ -17,7 +17,7 @@ truth for remediation progress. Update it as work lands.
| `fix/archreview-c1-wire-analyzer` | `f0082af5` | Guard 07/C-1 — wire OTOPCUA0001 analyzer tree-wide + triage (surfaced the RESILIENCE-DISPATCH-GAP, task #10) |
| `fix/archreview-s1-s2-ci-coverage` | `10b89830` | Guard 07/S-1,S-2,S-4 — whole-solution CI leg + fail-on-skip gate + CLI sleep deflake |
| `fix/archreview-crit10-wire-capability-invoker` | `62556c24` | **#10 — wire CapabilityInvoker into dispatch via a Core.Abstractions seam** (off `fix/archreview-c1-wire-analyzer`, NOT master — the analyzer is the standing guard + the pragmas live there). Code-complete + unit/analyzer/context-verified; live gate pending. |
| `fix/archreview-crit13-resilience-config-artifact` | `75403caa` | **#13 — apply per-instance ResilienceConfig from the artifact** (off the crit10 branch — builds on its invoker factory). Read-path plumbing + invalidate/respawn-on-change; deterministically verified. |
| `fix/archreview-crit13-resilience-config-artifact` | `8a8b9ec5` | **#13 — apply per-instance ResilienceConfig from the artifact** (`75403caa`, off the crit10 branch) **+ #10/#13 DI guard** (`8a8b9ec5` — `ResilienceInvokerFactoryRegistrationTests`, from the #12 live gate). Read-path plumbing + invalidate/respawn-on-change; deterministically verified + rig-booted. |
**Decision (user):** one branch per Critical, each off `master`. **#9 (failover test) lives on the crit1 branch**
(not its own branch off master) because SBR only exists there — it is Critical 1's integration half. The plans are only on the docs branch,
@@ -57,7 +57,7 @@ so from a critical branch read them via `git show docs/archreview-plans-and-drif
| 9 | Critical 1 follow-up — hard-kill failover integration test (03/S1) | ✅ completed — `a25c9ed0`; surfaced task #11 |
| 10 | **Wire CapabilityInvoker into the dispatch layer** (RESILIENCE-DISPATCH-GAP, from 07/C-1) | ✅ code-complete + unit/analyzer/context-verified — `fix/archreview-crit10-wire-capability-invoker` `62556c24` (seam, NOT direct thread — Runtime is Polly-free). **Live behavioral gate pending** (rig recipe in FOLLOWUP-10). Surfaced a sub-gap: ResilienceConfig not in the deploy artifact → tier-defaults only. |
| 11 | **Review Critical 1 premise** — SBR was already active on master, not NoDowning (from #9) | ✅ completed — `fix/archreview-crit1-split-brain-resolver` `eaf78aad`; docs/comment-only correction (no code revert), Cluster.Tests 29/29. |
| 12 | **#10 live behavioral gate** — rebuild docker-dev, force a connection-level exception on an invoker-wrapped call, observe retry/breaker logs + recovery (from #10) | ⬜ pending — rig session (recipe in FOLLOWUP-10); the residual live gate for #10 (also covers #13's override-behavior end-to-end). |
| 12 | **#10 live behavioral gate** — rig session (from #10) | ✅ ran 2026-07-08 (PARTIAL) — built the Host image from source (crit13 tip) + stood up the rig; central-1 booted healthy; **prod DI binds the real invoker factory** confirmed live + by a new deterministic guard (`ResilienceInvokerFactoryRegistrationTests` `8a8b9ec5`). Behavioral retry/breaker **log-line not observed** (disciplined stop): Modbus's cheap paths don't throw a wrapped exception (Discover offline, Subscribe lazy, reads non-wrapped) — only an idempotent write-to-dead-endpoint (or S7) does, not worth the multi-dependency chase since the retry behavior is deterministically proven. **Found a rig bug:** docker-dev compose's retired Wonderware `ServerHistorian` keys crash the Host at bring-up (historian-gateway cutover → reads `Endpoint`); recommend fixing. See [`FOLLOWUP-10`](FOLLOWUP-10-resilience-dispatch-gap.md). |
| 13 | **Plumb per-instance `ResilienceConfig` through the deploy artifact** (from #10) | ✅ completed — `fix/archreview-crit13-resilience-config-artifact` `75403caa`; read-path plumbing (write side was already complete), invalidate-on-change + respawn-on-change; Core.Tests 243/Runtime.Tests 363. See [`FOLLOWUP-13`](FOLLOWUP-13-resilience-config-artifact.md). |
## Findings surfaced by the guards (NEW — need user decision)