dohertj2/lmxopcua
1
0
Fork 0
Code Issues 1 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

[archreview #10] Observe resilience retry/breaker firing in live dispatch (+ ResilienceConfig not in deploy artifact) #456

New Issue
Closed
opened 2026-07-15 08:01:42 -04:00 by dohertj2 · 2 comments
dohertj2 commented 2026-07-15 08:01:42 -04:00
Owner
Copy Link
Copy Source

Status: non-blocking follow-up. The dispatch wiring itself is merged (62556c24, seam IDriverCapabilityInvoker/IDriverCapabilityInvokerFactory in Core.Abstractions; DriverInstanceActor wraps its 6 dispatch sites) and DI-wiring was live-confirmed on docker-dev (2026-07-08). Two things remain:

  1. Behavioral log-line never observed. No one has watched a retry/breaker log line actually fire during a live driver fault in dispatch — a "disciplined stop" was taken. Retry/breaker logging was added to the pipeline builder as the observability surface (the tracker has no reader — Streams E.2/E.3 never shipped), so it currently runs invisibly.
  2. Sub-gap — ResilienceConfig is not in the deploy artifact, so the pipeline runs tier-defaults only. Operator-authored per-driver ResilienceConfig is authored in AdminUI but never reaches the runtime pipeline. Until it does, a behavioral live gate only proves plumbing, not operator policy.

Source of record: archreview/plans/FOLLOWUP-10-resilience-dispatch-gap.md (rig recipe there), archreview/plans/STATUS.md (#10 row).

Acceptance:

  • Drive a driver fault on docker-dev and capture the retry/breaker log line from the pipeline builder.
  • Thread ResilienceConfig into the deploy artifact so the runtime pipeline uses authored policy (not just tier-defaults), then re-verify the behavioral gate with a non-default policy.
**Status:** non-blocking follow-up. The dispatch wiring itself is **merged** (`62556c24`, seam `IDriverCapabilityInvoker`/`IDriverCapabilityInvokerFactory` in `Core.Abstractions`; `DriverInstanceActor` wraps its 6 dispatch sites) and DI-wiring was live-confirmed on docker-dev (2026-07-08). Two things remain: 1. **Behavioral log-line never observed.** No one has watched a retry/breaker log line actually fire during a live driver fault in dispatch — a "disciplined stop" was taken. Retry/breaker **logging** was added to the pipeline builder as the observability surface (the tracker has no reader — Streams E.2/E.3 never shipped), so it currently runs invisibly. 2. **Sub-gap — `ResilienceConfig` is not in the deploy artifact**, so the pipeline runs **tier-defaults only**. Operator-authored per-driver `ResilienceConfig` is authored in AdminUI but never reaches the runtime pipeline. Until it does, a behavioral live gate only proves plumbing, not operator policy. **Source of record:** `archreview/plans/FOLLOWUP-10-resilience-dispatch-gap.md` (rig recipe there), `archreview/plans/STATUS.md` (#10 row). **Acceptance:** - [ ] Drive a driver fault on docker-dev and capture the retry/breaker log line from the pipeline builder. - [ ] Thread `ResilienceConfig` into the deploy artifact so the runtime pipeline uses authored policy (not just tier-defaults), then re-verify the behavioral gate with a non-default policy.
dohertj2 commented 2026-07-15 08:50:25 -04:00
Author
Owner
Copy Link
Copy Source

Update 2026-07-15 — sub-gap CLOSED (item 2 done, live-verified on both nodes); narrowed to item 1. master b364a499 (PR #461).

Finding: the artifact-threading was already wired; the gap was coverage + live re-verify

The issue body carried the pre-#13 banner. In current master the per-instance ResilienceConfig already threads end-to-end: ConfigComposer.SnapshotAndFlattenAsync serialises the whole DriverInstance entity → DeploymentArtifact.ParseDriverInstances recovers it onto DriverInstanceSpec.ResilienceConfig (:173) → DriverHostActor.SpawnChild (:1727) passes it to DriverCapabilityInvokerFactory.Create, which parses + layers it onto the driver's Polly pipeline; DriverSpawnPlan respawns on change. The only untested leg was the real composer→artifact serialization (the existing DeploymentArtifactTests hand-build the JSON).

✅ Item 2 — DONE (threading + non-default-policy re-verify)

  1. New regression test — ConfigComposerTests.ResilienceConfig_survives_ConfigComposer_to_ParseDriverInstances_round_trip: seeds a DriverInstance WITH a non-default override + one WITHOUT, runs the real SnapshotAndFlattenAsync, asserts ParseDriverInstances recovers the override byte-for-byte (guards against a future projection / [JsonIgnore] silently reverting authored policy to tier defaults). ControlPlane.Tests ConfigComposerTests 6/6 green.
  2. LIVE re-verify with a non-default policy (docker-dev, both nodes). Seeded a Subscribe.retryCount:999 override, deployed headless. Proof (archreview/plans/artifacts/456-resilience-config-live-proof-2026-07-15.md):
    • Persisted Deployment.ArtifactBlob (4055 B) contains ResilienceConfig + 999.
    • Runtime, central-1 AND central-2: [WRN] Driver resilience config for instance=drv-res type=OpcUaClient: Subscribe.retryCount 999 exceeds the cap (100); clamped. → [INF] spawned OpcUaClient driver drv-res.
    • The authored value 999 appears verbatim in the runtime log. If the artifact didn't carry ResilienceConfig, spec.ResilienceConfig would be null, the parser would take pure tier-defaults, and no clamp diagnostic would fire. So the diagnostic firing = the authored policy rode DB → composer → artifact → runtime pipeline, on both nodes.

⏳ Item 1 — remaining (raw retry/breaker log line on an actual fault)

Not captured. It needs a connect-then-fault-mid-flight trigger: the invoker-wrapped capability calls (Subscribe/Discover/Write) only fire after a driver reaches Connected (DriverInstanceActor.ResubscribeDesired runs on the Become(Connected) transition), so a driver at a dead endpoint stays in Reconnecting and never exercises the wrapped path — the same wall the 2026-07-08 run hit. Reliable trigger: an idempotent Write to a Connected-then-killed driver (LDAP-gated Client.CLI write) or an S7 dead-socket reconnect. The retry/breaker behaviour is already proven deterministically (CapabilityInvokerTests retry, the pipeline-builder logging test, DriverCapabilityInvokerFactoryTests override-applies-to-behaviour), so this remaining line is operator-confidence only.

Acceptance:

  • Drive a driver fault on docker-dev and capture the retry/breaker log line (connect-then-idempotent-write trigger).
  • Thread ResilienceConfig into the deploy artifact + re-verify with a non-default policy — done + live-verified on both nodes.
**Update 2026-07-15 — sub-gap CLOSED (item 2 done, live-verified on both nodes); narrowed to item 1.** master `b364a499` (PR #461). ## Finding: the artifact-threading was already wired; the gap was coverage + live re-verify The issue body carried the pre-#13 banner. In current master the per-instance `ResilienceConfig` **already threads end-to-end**: `ConfigComposer.SnapshotAndFlattenAsync` serialises the whole `DriverInstance` entity → `DeploymentArtifact.ParseDriverInstances` recovers it onto `DriverInstanceSpec.ResilienceConfig` (`:173`) → `DriverHostActor.SpawnChild` (`:1727`) passes it to `DriverCapabilityInvokerFactory.Create`, which parses + layers it onto the driver's Polly pipeline; `DriverSpawnPlan` respawns on change. The only untested leg was the **real composer→artifact** serialization (the existing `DeploymentArtifactTests` hand-build the JSON). ## ✅ Item 2 — DONE (threading + non-default-policy re-verify) 1. **New regression test** — `ConfigComposerTests.ResilienceConfig_survives_ConfigComposer_to_ParseDriverInstances_round_trip`: seeds a `DriverInstance` WITH a non-default override + one WITHOUT, runs the real `SnapshotAndFlattenAsync`, asserts `ParseDriverInstances` recovers the override byte-for-byte (guards against a future projection / `[JsonIgnore]` silently reverting authored policy to tier defaults). ControlPlane.Tests `ConfigComposerTests` 6/6 green. 2. **LIVE re-verify with a non-default policy (docker-dev, both nodes).** Seeded a `Subscribe.retryCount:999` override, deployed headless. Proof (`archreview/plans/artifacts/456-resilience-config-live-proof-2026-07-15.md`): - Persisted `Deployment.ArtifactBlob` (4055 B) contains `ResilienceConfig` + `999`. - Runtime, **central-1 AND central-2**: `[WRN] Driver resilience config for instance=drv-res type=OpcUaClient: Subscribe.retryCount 999 exceeds the cap (100); clamped.` → `[INF] spawned OpcUaClient driver drv-res`. - The authored value `999` appears verbatim in the runtime log. If the artifact didn't carry `ResilienceConfig`, `spec.ResilienceConfig` would be null, the parser would take pure tier-defaults, and **no clamp diagnostic would fire**. So the diagnostic firing = the authored policy rode DB → composer → artifact → runtime pipeline, on both nodes. ## ⏳ Item 1 — remaining (raw retry/breaker log line on an actual fault) Not captured. It needs a **connect-then-fault-mid-flight** trigger: the invoker-wrapped capability calls (`Subscribe`/`Discover`/`Write`) only fire after a driver reaches **Connected** (`DriverInstanceActor.ResubscribeDesired` runs on the `Become(Connected)` transition), so a driver at a dead endpoint stays in Reconnecting and never exercises the wrapped path — the same wall the 2026-07-08 run hit. Reliable trigger: an **idempotent Write to a Connected-then-killed driver** (LDAP-gated Client.CLI write) or an S7 dead-socket reconnect. The retry/breaker **behaviour** is already proven deterministically (`CapabilityInvokerTests` retry, the pipeline-builder logging test, `DriverCapabilityInvokerFactoryTests` override-applies-to-behaviour), so this remaining line is operator-confidence only. **Acceptance:** - [ ] Drive a driver fault on docker-dev and capture the retry/breaker log line (connect-then-idempotent-write trigger). - [x] Thread `ResilienceConfig` into the deploy artifact + re-verify with a non-default policy — **done + live-verified on both nodes.**
dohertj2 commented 2026-07-15 14:49:05 -04:00
Author
Owner
Copy Link
Copy Source

Item 1 — live grind done; write-trigger proven structurally impossible. Closing. master 46bc458f (PR #466).

The grind (full end-to-end, docker-dev)

Authored a live Modbus equipment hierarchy via SQL (Equipment-kind ns → drv-mb Modbus @ 10.100.0.35:5020 → Device → Area/Line → EQ-111111111111 → writable reg1). Deployed headless → spawned Modbus driver drv-mb → connected → subscribed. Baseline: live read Good, LDAP-authed (multi-role/WriteOperate) write =123 succeeded — the wrapped write path is live end-to-end (OPC UA write → NodeWriteRouter → DriverHostActor.RouteNodeWrite → DriverInstanceActor.WriteAsync invoker-wrapped → ModbusDriver → sim). Then docker paused the peer and drove 4 writes.

Result: 4 failing wrapped writes → 0 retry/breaker lines on either central

Each write logged Operator write … rejected: StatusCode=0x80020000 (node reverted), but grep 'Driver resilience retry|circuit-breaker OPENED' = 0.

Why — the issue's write-trigger is structurally impossible

  • ModbusDriver.WriteAsync catches every exception → returns WriteResult(StatusBadInternalError) = 0x80020000; it never throws (ModbusDriver.cs:964-967).
  • CapabilityInvoker feeds only exceptions to the Polly pipeline (CapabilityInvoker.cs:84/112/147) — it does not inspect the returned status. Breaker ShouldHandle is Handle<Exception>; Write retry is pinned to 0 (R2-02/S-8).
  • ∴ a failing wrapped write emits no line by construction, for any polling driver (all map write faults to a status). Even an idempotent write to a dead endpoint returns a bad status, not a throw — so the issue-body hypothesis ("idempotent WriteAsync to a dead endpoint throws") is incorrect.

Where the line actually lives

Only a session driver (OpcUaClient) can emit it — its DiscoverAsync (remote browse) / SubscribeAsync throw on a session fault, and the post-connect discovery loop runs DiscoverAsync under a 30s Polly timeout. The line requires a session fault landing mid-Discover/Subscribe before keepalive drops the driver to Reconnecting — a genuine production timing race, not deterministically forcible on the rig (polling drivers' Subscribe/Discover are lazy/offline and never throw). This is exactly why three sessions with polling-driver fixtures never observed it.

Finding artifact · FOLLOWUP-10 updated.

Disposition

  • Item 2 (ResilienceConfig in the deploy artifact) — done + live-verified on both nodes (PR #461).
  • Item 1 — the retry/breaker behaviour is deterministically proven by the DriverResiliencePipelineBuilder logging unit test + CapabilityInvoker retry tests + the factory override test; the wrapped dispatch path is now live-confirmed end-to-end; and the write-based live trigger is structurally impossible. The only real live trigger is an OpcUaClient session fault mid-Discover/Subscribe — an operator/staging observation during a real network blip, not a code deliverable.

Closing as resolved.

**Item 1 — live grind done; write-trigger proven structurally impossible. Closing.** master `46bc458f` (PR #466). ## The grind (full end-to-end, docker-dev) Authored a live Modbus equipment hierarchy via SQL (Equipment-kind ns → `drv-mb` Modbus @ `10.100.0.35:5020` → Device → Area/Line → `EQ-111111111111` → writable `reg1`). Deployed headless → `spawned Modbus driver drv-mb` → `connected` → `subscribed`. Baseline: live read `Good`, LDAP-authed (`multi-role`/`WriteOperate`) write `=123` succeeded — **the wrapped write path is live end-to-end** (OPC UA write → `NodeWriteRouter` → `DriverHostActor.RouteNodeWrite` → `DriverInstanceActor.WriteAsync` invoker-wrapped → `ModbusDriver` → sim). Then `docker pause`d the peer and drove 4 writes. ## Result: 4 failing wrapped writes → **0** retry/breaker lines on either central Each write logged `Operator write … rejected: StatusCode=0x80020000` (node reverted), but `grep 'Driver resilience retry|circuit-breaker OPENED'` = 0. ## Why — the issue's write-trigger is structurally impossible - `ModbusDriver.WriteAsync` **catches every exception → returns `WriteResult(StatusBadInternalError)` = 0x80020000; it never throws** (`ModbusDriver.cs:964-967`). - `CapabilityInvoker` feeds **only exceptions** to the Polly pipeline (`CapabilityInvoker.cs:84/112/147`) — it does not inspect the returned status. Breaker `ShouldHandle` is `Handle<Exception>`; `Write` retry is pinned to 0 (R2-02/S-8). - ∴ a failing wrapped **write** emits **no line by construction**, for **any** polling driver (all map write faults to a status). Even an idempotent write to a dead endpoint returns a bad status, not a throw — so the issue-body hypothesis ("idempotent WriteAsync to a dead endpoint throws") is **incorrect**. ## Where the line actually lives Only a **session driver (OpcUaClient)** can emit it — its `DiscoverAsync` (remote browse) / `SubscribeAsync` throw on a session fault, and the post-connect discovery loop runs `DiscoverAsync` under a 30s Polly timeout. The line requires a session fault landing *mid-`Discover`/`Subscribe`* before keepalive drops the driver to `Reconnecting` — a genuine production **timing race**, not deterministically forcible on the rig (polling drivers' `Subscribe`/`Discover` are lazy/offline and never throw). This is exactly why three sessions with polling-driver fixtures never observed it. [Finding artifact](https://gitea.dohertylan.com/dohertj2/lmxopcua/src/branch/master/archreview/plans/artifacts/456-retry-breaker-live-finding-2026-07-15.md) · FOLLOWUP-10 updated. ## Disposition - **Item 2** (ResilienceConfig in the deploy artifact) — done + live-verified on both nodes (PR #461). - **Item 1** — the retry/breaker **behaviour** is deterministically proven by the `DriverResiliencePipelineBuilder` logging unit test + `CapabilityInvoker` retry tests + the factory override test; the wrapped dispatch path is now **live-confirmed end-to-end**; and the write-based live trigger is **structurally impossible**. The only real live trigger is an OpcUaClient session fault mid-Discover/Subscribe — an operator/staging observation during a real network blip, not a code deliverable. Closing as resolved.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
auto-managed
queue management
 cross-driver
queue management
 driver/abcip
queue management
 driver/ablegacy
queue management
 driver/focas
queue management
 driver/opcuaclient
queue management
 driver/s7
queue management
 driver/twincat
queue management
 phase/1
queue management
 phase/2
queue management
 phase/3
queue management
 phase/4
queue management
 phase/5
queue management
 phase/6
queue management
 queue/blocked
queue management
 queue/done
queue management
 queue/failed
queue management
 queue/in-progress
queue management
 queue/queued
queue management
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
dohertj2 (dohertj2)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: dohertj2/lmxopcua#456
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?