feat(resilience): route IDriver.InitializeAsync through the capability invoker (R2-10 T10 closed)

The connect path was the one capability call the resilience pipeline never
saw, so a driver that could not connect contributed zero breaker input and
the /hosts resilience panel read 'All targets healthy' against a dead
endpoint. InitializeAsync now routes through the invoker as
DriverCapability.Initialize, keyed on the driver-instance id.

- No pipeline retry: the actor's RetryConnect timer is the retry loop
  (parser forces retryCount overrides to 0, same funnel as write-shaped).
- Breaker threshold 2, not the Read-sized 5: the breaker samples 30 s and
  the reconnect timer fires every 10 s, so at most three attempts fit one
  window — threshold 5 could never open. Pinned by its own test.
- An open breaker fast-fails into the same InitializeFailed path as a real
  connect failure; the 15 s break window costs at most one extra tick.
  The best-effort teardown ShutdownAsync stays unwrapped.
- Companion fix: DriverCapabilityInvokerFactory.Create now invalidates the
  instance's tracker snapshots with its pipelines — a respawn during an
  outage discarded the only breaker whose OnClosed could clear the OPEN
  flag, stranding a false 'Breaker OPEN' chip forever.

Live gate PASSED on docker-dev (image symbol-verified per container):
rtu-gate against dead 10.100.0.35:5021 opened its Initialize breaker on
both centrals with the red 'BREAKER OPEN' chip on /hosts; a socat
5021→5020 forwarder recovered it (both breakers closed within 1 s, panel
back to 'All targets healthy'); healthy MAIN-modbus recorded zero breaker
events. Full suite 50/50 green. Evidence:
docs/plans/2026-08-07-adminui-followups-and-live-gates.md §7.
This commit is contained in:
Joseph Doherty
2026-08-08 05:45:08 -04:00
parent c9f182b2d7
commit dac7e1885d
18 changed files with 445 additions and 8 deletions
@@ -74,13 +74,14 @@
{
"id": "T10",
"subject": "Live-/run gate on docker-dev: rebuild BOTH centrals, S7 dead-endpoint breaker-open, both-replica chips, recovery clear, staleness dim",
"status": "blocked-structural",
"status": "completed",
"blockedBy": [
"T6",
"T8",
"T9"
],
"openNote": "PARTIAL \u2014 the resilience PIPELINE is live-verified (Polly CircuitBreaker -> Retry -> Timeout -> CapabilityInvoker in central-1 stack traces), but breaker-OPEN state was never forced: R2-09's ConnectionBackoff throttles retries so failures don't accumulate fast enough to open the Polly breaker. The reliable trigger is a rapid idempotent write to a dead endpoint. Confirmed still open 2026-07-27 (deferment.md \u00a78.5).",
"passNote": "PASSED 2026-08-08 after the structural gap was FIXED (feat/wrap-driver-init-in-resilience): IDriver.InitializeAsync now routes through the invoker as DriverCapability.Initialize (no retry \u2014 the actor's RetryConnect timer is the retry loop; tier-default breaker threshold 2, because at the 10 s reconnect cadence at most three attempts fit the 30 s sampling window and a threshold of 5 could never open). Live on docker-dev (image df04c76ce06b, symbol-verified on both centrals): rtu-gate (DRV-3616122eb53d, endpoint 10.100.0.35:5021 with no listener) opened its Initialize breaker on BOTH centrals ('circuit-breaker OPENED \u2026 capability=Initialize \u2026 Connection refused'), the /hosts RESILIENCE panel showed the red 'BREAKER OPEN \u00b7 OPENED 19S AGO' chip, and after a socat 5021\u21925020 forwarder brought the endpoint alive both breakers closed within 1 s of each other ('circuit-breaker closed (recovered)' + 'reconnected') and the panel cleared to 'All targets healthy.' Control: healthy MAIN-modbus recorded zero breaker events. Two nuances: (1) 'both-replica chips' resolves as both nodes' TRACKERS tripping (proven in both logs) rendered through the single ACTIVE AdminUI \u2014 Traefik's /health/active check deliberately routes the UI to the active admin node only (standby returns 503 'Standby: active node for role admin is \u2026'), so there is never a second replica serving chips; (2) the staleness-dim leg was NOT exercised \u2014 both pair nodes publish under the same key, so staleness needs every publisher for a key to stop, which requires deliberate node-kill churn this gate did not do. Companion fix shipped in the same change: DriverCapabilityInvokerFactory.Create now Invalidates the instance's TRACKER snapshots alongside its pipelines, because a respawn during an outage discarded the only breaker whose OnClosed could clear the OPEN flag \u2014 a deploy mid-outage would have stranded a false OPEN chip forever (unreachable before this change made breaker-open reachable). Evidence: docs/plans/2026-08-07-adminui-followups-and-live-gates.md \u00a77.",
"closureNote": "ATTEMPTED 2026-08-07, NOT ACHIEVABLE AS SPECIFIED \u2014 and the reason is structural, not environmental. The breaker only sees failures that propagate out of a CapabilityInvoker-wrapped call, and the dead-endpoint path produces none: (1) IDriver.InitializeAsync is NOT wrapped (DriverInstanceActor.cs:707 calls it directly; the invoker wraps Read/Write/Subscribe/Discover/AlarmAcknowledge only), so a driver that cannot connect contributes zero breaker input \u2014 refuting this task's own premise; (2) ModbusDriver swallows per-tag failures in BOTH directions \u2014 ReadAsync -> Degraded health (:365-375,:838-855), WriteAsync -> WriteResult status \u2014 so even the fallback trigger this plan suggests (rapid idempotent write-to-dead-endpoint) cannot work. Live-proven: MAIN-modbus pointed at a TCP blackhole went DEGRADED with every read timing out (0x80050000) while the tracker held ConsecutiveFailures=0 and the panel showed \"All targets healthy.\" The breaker is configured to open on 2 failures in a 30 s window at ratio 1.0, so the previously-recorded ConnectionBackoff-throttling explanation is not the operative one. VERIFIED LIVE regardless: panel renders on /hosts (rehomed by #546; this plan's \"8 driver pages\" placement is stale), RESILIENCE block present, and the publisher emits real tracker state every 5 s. To close: wrap the connect path, stop drivers swallowing capability failures, or use a driver whose wrapped call fails AFTER a successful init. Evidence: docs/plans/2026-08-07-adminui-followups-and-live-gates.md \u00a74."
},
{
+15
View File
@@ -272,6 +272,21 @@ line + resilience chip were not triggered.)*
> blackhole showed `ConsecutiveFailures = 0` and a panel reading *"All targets healthy."*
> Full evidence: [`docs/plans/2026-08-07-adminui-followups-and-live-gates.md`](../../docs/plans/2026-08-07-adminui-followups-and-live-gates.md).
> **Update 2026-08-08 — R2-10 T10 ✅ PASSED, by fixing the structural gap the entry above diagnosed.**
> `IDriver.InitializeAsync` now routes through the invoker as **`DriverCapability.Initialize`**
> (branch `feat/wrap-driver-init-in-resilience`): no pipeline retry (the actor's reconnect timer is
> the retry loop, parser-enforced), breaker threshold **2** (at the 10 s reconnect cadence at most
> three attempts fit the 30 s sampling window — a Read-sized 5 could never open; pinned by its own
> test), and `DriverCapabilityInvokerFactory.Create` now invalidates the instance's **tracker**
> snapshots with its pipelines (else a deploy mid-outage strands a false OPEN chip forever).
> Live on docker-dev: `rtu-gate` vs. dead `:5021` → breaker OPEN on **both** centrals + the red
> "BREAKER OPEN" chip on `/hosts`; socat `5021→5020` up → both breakers closed within 1 s and the
> panel cleared to "All targets healthy." (healthy `MAIN-modbus` control: zero breaker events).
> This is also the **first live sighting of the `circuit-breaker OPENED` log line**, closing
> FOLLOWUP-10's behavioral gate. Two narrowings vs. this plan's T10 wording: the chips render on the
> single **active** AdminUI (Traefik `/health/active` 503s the standby by design — "both replicas"
> means both trackers), and the staleness-dim leg was not exercised. Evidence: the same doc, §7.
### Residual follow-ups — tracked in Gitea (2026-07-15)
The remaining non-blocking items are filed as issues (none gate the remediation, which is complete):
+31 -5
View File
@@ -53,7 +53,7 @@ So everything below is **untracked work**. Nothing in it is a data-loss or outag
| What | Nature | Where |
|---|---|---|
| **4 live gates blocked on hardware/licence** — archreview R2-03 (VT Good→Bad on a data-fed rig), R2-10 (force breaker-OPEN), Galaxy `ScanStateProbeParityTests` (licence-gated), v2 GA exit criteria (~~FOCAS live-CNC smoke #54~~ *closed — live-validated on a real 31i-B 2026-06-25, see `docs/plans/2026-06-25-focas-pdu-v3-30i-b-support.md`*, OPC UA CTT pass, Ignition 8.3 cutover) | Cannot be closed from this box. Not decisions — inputs. | §5 |
| **4 live gates blocked on hardware/licence** — ~~archreview R2-03 (VT Good→Bad on a data-fed rig)~~ ✅ *PASSED 2026-08-07*, ~~R2-10 (force breaker-OPEN)~~ ✅ *PASSED 2026-08-08 by fixing the structural gap (`DriverCapability.Initialize`)*, Galaxy `ScanStateProbeParityTests` (licence-gated), v2 GA exit criteria (~~FOCAS live-CNC smoke #54~~ *closed — live-validated on a real 31i-B 2026-06-25, see `docs/plans/2026-06-25-focas-pdu-v3-30i-b-support.md`*, OPC UA CTT pass, Ignition 8.3 cutover) | Cannot be closed from this box. Not decisions — inputs. *(The two archreview gates turned out to be closable from this box after all — §5.)* | §5 |
| ~~**1 deferred live-verify task** — `2026-06-14-driver-reconfigure-while-faulted` Task 3~~ ✅ **PASSED 2026-08-07.** A faulted driver ("initialize failed: Connection refused") adopted a corrected config and connected while faulted, with the node's `startedAt` byte-identical before and after and `RestartCount=0` — no restart. | §5 |
| **G-7 residual** — Sql / MTConnect / Calculation / OpcUaClient have no Contracts-side `Inspect` | **The deploy gate is weaker than the authoring gate.** `TagConfigValidator` covers 11 of 12, so config arriving by **CSV import or API** — the paths that skip the UI — meets only the weaker gate. | §1.2 |
| ~~**3 carried-open archreview findings** — the failure-visibility trio~~ **Update 2026-08-07 (truth sweep): mostly CLOSED.** 01/S-1 **FIXED** — `AddressSpaceApplier.cs:95-97` tallies swallowed removal-pass failures into `AddressSpaceApplyOutcome.FailedNodes` (comment cites 01/S-1). 03/S4 **FIXED** — `PrimaryGatePolicy.cs` default-denies on unknown role when a driver peer exists (doc-comment "Fixes archreview 03/S4"), live-proven R2-04 T15. 06/S-1 **PARTIAL** — fail-closed advise path + `galaxy.writes.unconfirmed` meter shipped in `GatewayGalaxyDataWriter.cs`; residual = gateway-side WriteComplete correlation (cross-repo, mxaccessgw). | Residual is only the 06/S-1 cross-repo half. | §6.3 |
@@ -87,7 +87,7 @@ themes into tracked decisions. What remains:
| **G-7** `EquipmentTagConfigInspector` coverage | ◐ **Narrowed + guarded** (2026-07-28): MQTT wired (it shipped an `Inspect` nobody called), and a reflection guard now fails the build if a reachable driver ships one and is not mapped. **Residual:** Sql / MTConnect / Calculation / OpcUaClient have no Contracts-side `Inspect` at all, so the deploy gate stays blind for them while the AdminUI validates them — see §1.2. | §1.2 |
| ~~**13 `EquipmentTagsDarkBatch4` skipped tests**~~ | ✅ **Done** — 6 revived onto the raw/UNS shape, 3 folded into a widened live parity corpus, 4 deleted as retired. The reason constant is gone. **Skips now: `Runtime.Tests` 13 → 3, `OpcUaServer.Tests` 4 → 1** — all remaining are deliberate `EquipmentDeviceBindingRetired` tombstones. | §9 (2026-07-28, skips) |
| 17 other tracker issues | Driver features + deferred work, unchanged by this remediation. | §2 |
| Open live gates | ~~#491 continuous historization, #468 browser tree~~ ✅ **both PASSED since** (#468 `c2c3c522`, #491 `e77ea4aa` + the #537/#538 gate `4bdce910`); archreview R2-03 / R2-10 remain. *(The un-gated drift detector's gate closed by deletion — see #523.)* | §5 |
| Open live gates | ~~#491 continuous historization, #468 browser tree~~ ✅ **both PASSED since** (#468 `c2c3c522`, #491 `e77ea4aa` + the #537/#538 gate `4bdce910`); ~~archreview R2-03 / R2-10~~ ✅ **both PASSED** (R2-03 2026-08-07; R2-10 2026-08-08, closed by fixing the structural gap). *(The un-gated drift detector's gate closed by deletion — see #523.)* | §5 |
✅ **The tracker is now clean of stale-open issues** (2026-07-30). **#516**, **#518**, **#507** and
**#523** were closed, each with a resolution comment recording what was actually done — which differs
@@ -411,7 +411,7 @@ mere env-var presence, so it cannot pass vacuously.
| **Continuous-historization value capture** (#491) | ✅ **PASSED** `e77ea4aa` (2026-08-04) — and it found a **production-fatal defect the code-complete reading missed**: `ConfigureAwait(false)` in the Akka handler destroyed `Self`, the actor restarted, and the restart discarded the (empty-seeded) ref set. One value captured, then permanent silence. Two follow-ups filed and closed the same day: **#537** (capture had no Primary gate — both pair nodes wrote duplicates) and **#538** (a poison tag pinned the FIFO head, re-writing every healthy tag forever). | `docs/plans/2026-08-01-continuous-historization-live-gate.md`; `ad744734`, `e77ea4aa`, `e7fbe977`, `4bdce910` |
| **Wave-0 universal discovery browser** full tree render (#468) | ✅ **PASSED** `c2c3c522` (2026-08-01) — unblocked by **building** the missing sim capability (an `ab_server` patch) rather than waiting for hardware. The gate then found a decoder defect **350 unit tests agreed with**. | tracking doc §Wave 0 |
| **archreview R2-03** (T11 + T12) — VT Good→Bad on a data-fed rig | ✅ **PASSED 2026-08-07.** The rig is no longer data-less — the Modbus fixture is reachable from inside it — so the VT was staged Good and tripped **without a deployment** (a register was flipped on the fixture, so the VT child survived and could hold a last value): `42 → 0x80020000 → 42`, plus the P7 identical-redeploy spot-check (zero disturbance). ⚠️ **Found:** the Bad publish's "carrying the last-known value" is **unobservable by any client** — subscription *and* raw Read both return NULL. The actor is correct (new regression test pins it); the docs overstate what a client sees. | `docs/plans/2026-08-07-adminui-followups-and-live-gates.md` §2–3 |
| **archreview R2-10** — force breaker-OPEN live | ❌ **STILL OPEN — but the recorded cause was wrong, and the real one is structural.** Not ConnectionBackoff throttling (the breaker opens on 2 failures / 30 s): the dead-endpoint failure mode **never enters the resilience pipeline**. `IDriver.InitializeAsync` is not invoker-wrapped (`DriverInstanceActor.cs:707`), and `ModbusDriver` swallows per-tag failures in **both** directions, so even the plan's own fallback trigger (write-to-dead-endpoint) cannot work. Live-proven: a driver timing out every read against a blackhole reported `ConsecutiveFailures = 0` and *"All targets healthy."* — **the panel is permanently quiet for the failure operators most care about.** | `docs/plans/2026-08-07-adminui-followups-and-live-gates.md` §4 |
| **archreview R2-10** — force breaker-OPEN live | ✅ **PASSED 2026-08-08 — by FIXING the structural gap** (`feat/wrap-driver-init-in-resilience`), the day after the 2026-08-07 run re-diagnosed it. `IDriver.InitializeAsync` now routes through the invoker as **`DriverCapability.Initialize`** (no retry — the actor's timer is the retry loop; **breaker threshold 2**, because at the 10 s reconnect cadence at most three attempts fit the 30 s sampling window and a Read-sized 5 could literally never open). Live: `rtu-gate` against dead `10.100.0.35:5021` opened its breaker on **both** centrals, the `/hosts` panel showed the red **"BREAKER OPEN"** chip, and a socat 5021→5020 forwarder recovered it — both breakers closed within 1 s, panel back to *"All targets healthy."* Control: healthy `MAIN-modbus`, zero breaker events. Companion fix: a respawn now `Invalidate`s the instance's **tracker** snapshots with its pipelines, or a deploy mid-outage would strand a false OPEN chip forever. Caveats recorded: the UI is served by the **active admin node only** (Traefik `/health/active` 503s the standby — "both-replica chips" means both *trackers*, one UI), and the staleness-dim leg was not exercised. | `docs/plans/2026-08-07-adminui-followups-and-live-gates.md` §7 |
| **Driver-reconfigure-while-faulted** Task 3 | ✅ **PASSED 2026-08-07** (the block indeed no longer held — docker-dev runs `DisableLogin`). A driver faulted on a bad device config adopted a corrected config **while faulted** and connected, with the node's `startedAt` byte-identical before/after and `RestartCount=0` — no restart, which is the whole point (pre-fix it retried the stale config forever). | `docs/plans/2026-08-07-adminui-followups-and-live-gates.md` §5 |
| Galaxy `ScanStateProbeParityTests` | **OPEN** — licence-gated (one `$WinPlatform` on this box) | `docs/v2/Galaxy.ParityRig.md:180` |
| **v2 GA exit criteria** — FOCAS live-CNC wire smoke (#54), OPC UA CTT/compliance pass, Ignition 8.3 redundancy cutover | **OPEN** — manual/hardware/external-tool | `docs/v2/v2-release-readiness.md:105-120` |
@@ -421,8 +421,9 @@ mere env-var presence, so it cannot pass vacuously.
**Note on the 14 archreview `deferred-live` tasks across 8 plans:** `archreview/plans/STATUS.md:242-305`
records a 2026-07-13 live pass closing several (R2-02, R2-07, R2-11, R2-04, R2-05-partial) and
2026-07-15 closures for R2-01/R2-08/R2-06 — **but the `.tasks.json` files were never updated.** Only
R2-03 and R2-10 have no closure record anywhere.
2026-07-15 closures for R2-01/R2-08/R2-06 — **but the `.tasks.json` files were never updated.** ~~Only
R2-03 and R2-10 have no closure record anywhere.~~ *(Both now closed with full records: R2-03
2026-08-07, R2-10 2026-08-08 — their `.tasks.json` files carry the closure notes.)*
---
@@ -1279,6 +1280,31 @@ drivers), 3 carried-open archreview findings led by 01/S-1 (`AddressSpaceApplier
---
### 2026-08-08 — R2-10 closed by fixing the structural gap (`DriverCapability.Initialize`)
The 2026-08-07 entry below left R2-10 T10 "still open — structural." The user chose the fix over the
other three options (gate through a different driver / drivers reporting out-of-band / retire the gate),
and the gate then **PASSED** against the fixed build. Branch `feat/wrap-driver-init-in-resilience`;
evidence in [`docs/plans/2026-08-07-adminui-followups-and-live-gates.md`](docs/plans/2026-08-07-adminui-followups-and-live-gates.md) §7.
- **`IDriver.InitializeAsync` now routes through the invoker** as `DriverCapability.Initialize` — the
one capability call the pipeline never saw, which is why a dead endpoint produced a quiet panel. No
pipeline retry (the actor's `RetryConnect` timer is the retry loop; parser-enforced), and **breaker
threshold 2** — at the 10 s reconnect cadence at most three attempts fit the 30 s sampling window,
so the Read-sized 5 could literally never open. That threshold arithmetic is the load-bearing detail
and is pinned by its own test.
- **Live:** `rtu-gate` against dead `:5021` → breaker OPEN on both centrals + the red panel chip;
socat forwarder up → both breakers closed within 1 s, panel cleared. Healthy `MAIN-modbus`: zero
breaker events (control). Bonus: a config-invalid OpcUaClient driver also now trips — equally
invisible before.
- **Companion fix found by asking "what clears the flag":** a respawn (`Create`) now `Invalidate`s the
instance's tracker snapshots along with its pipelines — a deploy mid-outage discards the only
breaker whose `OnClosed` could clear the OPEN flag, so the old behaviour would have stranded a false
OPEN chip forever. Unreachable until this change made breaker-open reachable at all.
- **Two honest narrowings of the original T10 wording:** "both-replica chips" = both *trackers* (one
active AdminUI — Traefik's `/health/active` deliberately 503s the standby, discovered fighting the
sticky cookie); the staleness-dim leg was not exercised (needs every publisher for a key to stop).
### 2026-08-07 — adminui-followups closed by audit; three of four runnable gates PASSED
Took the two items §0.1 named as the remaining untracked engineering. Full evidence:
+34
View File
@@ -270,6 +270,40 @@ re-entered this path; the watchdog makes it automatic and unbounded, so a device
have leaked ~360 transports at the 10 s retry interval. The shutdown is best-effort: a driver that throws
on it still gets its initialise attempt.
### The connect path feeds the circuit breaker (R2-10 T10 closure)
`IDriver.InitializeAsync` is routed through the capability invoker as
**`DriverCapability.Initialize`**, keyed on the driver-instance id (init spans the whole driver,
like a bulk subscribe). Before this, the connect path was the ONE capability call the invoker never
saw: a driver that could not connect contributed **zero** breaker input, so the AdminUI resilience
panel read *"All targets healthy"* against a dead endpoint — live-proven during the R2-10 T10 gate,
and the reason that gate was structurally unsatisfiable for a year of attempts. Things to know
before touching it:
- **The Initialize policy never retries** — this actor's `RetryConnect` timer *is* the retry loop,
and pipeline retries would stack inside each timer tick (and stretch one attempt past the timer
interval so attempts overlap). The parser forces a `retryCount` override back to 0, same funnel
as the write-shaped rule.
- ⚠️ **The breaker threshold is 2, and raising it silently re-breaks the feature.** The breaker
samples over 30 s and the reconnect timer fires every 10 s, so at most **three** init attempts fit
one sampling window — a Read-sized threshold of 5 can literally never open on connect failures,
which reads exactly like "the panel doesn't work" with nothing logged. Pinned by
`Initialize_BreakerThreshold_IsReachableAtTheReconnectCadence` and, behaviourally, by
`Initialize_BreakerOpens_AtTierDefaults_AndFastFailsTheNextAttempt`.
- **An open breaker cannot wedge the reconnect loop.** A fast-failed attempt
(`BrokenCircuitException`) lands in the same catch as a real connect failure →
`InitializeFailed` → the timer keeps ticking. The 15 s break window is shorter than two 10 s
ticks, so recovery costs at most one extra tick. The teardown `ShutdownAsync` stays deliberately
UNWRAPPED — best-effort cleanup of an already-dead transport must not count toward the breaker.
- **A respawn resets the instance's tracker snapshots** (`DriverResilienceStatusTracker.Invalidate`,
called by `DriverCapabilityInvokerFactory.Create` alongside the pipeline invalidation). The
pipelines carry the actual breaker state; once they are discarded, a retained "Breaker OPEN"
snapshot could never self-correct — only the discarded pipeline's `OnClosed` stamps the close —
so a deploy during an outage would otherwise strand a false OPEN chip forever. This was
unreachable while breakers never opened; the Initialize wrap made it reachable.
- `ConsecutiveFailures` stays 0 during a connect outage **by design** — the tracker counts retry
attempts, and Initialize has none. The breaker-OPEN chip is the operator signal.
---
## IDriverConfigEditor — custom AdminUI config editor (plug-point)
@@ -215,3 +215,67 @@ original one-line body, so it mirrors `MainPressure` again rather than the `×6`
One anomaly recorded rather than explained: both centrals were **recreated** at 09:49:40/09:49:50
(new `Created`, `RestartCount=0`) by something this session did not issue. It postdates the
observations above and the old container's logs went with it, so it is noted, not attributed.
---
## 7. Addendum 2026-08-08 — R2-10 T10 ✅ PASSED after fixing the structural gap
§4 concluded the gate was not achievable as specified because the connect path never entered the
resilience pipeline. The user chose to fix that ("do option 1"), and the gate then passed against
the fixed build. Branch `feat/wrap-driver-init-in-resilience`.
### The fix
- **`DriverCapability.Initialize`** — `IDriver.InitializeAsync` routes through the capability
invoker in `DriverInstanceActor.InitializeAsync` (both the initial connect and the
`Reconnecting` retry path; the best-effort teardown `ShutdownAsync` stays deliberately
unwrapped). Keyed on the driver-instance id, like a bulk subscribe.
- **No pipeline retry for Initialize** — the actor's `RetryConnect` timer is the retry loop.
Enforced at the parser funnel (`retryCount` override forced to 0, same shape as the
write-shaped rule) and in every tier default.
- **Breaker threshold 2, not the Read-sized 5** — the breaker samples 30 s and the reconnect
timer fires every 10 s, so at most three init attempts fit one window; threshold 5 could never
open. Pinned by `Initialize_BreakerThreshold_IsReachableAtTheReconnectCadence` +
`Initialize_BreakerOpens_AtTierDefaults_AndFastFailsTheNextAttempt`.
- **Companion fix — respawn resets tracker snapshots.** `DriverCapabilityInvokerFactory.Create`
now calls `DriverResilienceStatusTracker.Invalidate(driverInstanceId)` alongside the pipeline
invalidation. Without it, a deploy during an outage discards the only breaker whose `OnClosed`
could clear the OPEN flag, stranding a false "Breaker OPEN" chip forever. Unreachable before
this change made breaker-open reachable; found by asking what clears the flag, not by a test.
- Wiring pinned by `Initialize_routes_through_the_capability_invoker` +
`An_invoker_fast_fail_on_init_keeps_the_actor_retrying_until_recovery` (falsification-controlled:
both go red with the actor wrap stashed; `Create_resets_the_instances_tracker_state_with_its_pipelines`
goes red with the factory call stashed).
### The gate run (docker-dev, image `df04c76ce06b`, symbol-verified on both centrals)
The staged failure was free: `rtu-gate` (`DRV-3616122eb53d`) was already authored against
`10.100.0.35:5021`, where nothing listens — connection refused on every init.
1. **Breaker OPEN, both replicas** — within ~20 s of boot, both centrals logged
`circuit-breaker OPENED: instance=DRV-3616122eb53d … capability=Initialize … Connection refused`,
re-opening each half-open probe cycle. Bonus coverage: `DRV-opcua1` (invalid config → init
throws) also tripped — a permanently-misconfigured driver was equally invisible before.
2. **Panel chip** — `/hosts` → rtu-gate → Controls showed the red **"BREAKER OPEN · OPENED 19S
AGO"** chip in the RESILIENCE section, row FAULTED with "Connection refused".
3. **Control** — healthy `MAIN-modbus` recorded zero breaker events throughout.
4. **Recovery clear** — a socat forwarder (`5021 → 5020` on the docker host) brought the endpoint
alive; both centrals logged `circuit-breaker closed (recovered)` + `reconnected` within 1 s of
each other, and the panel dropped to "All targets healthy."
5. **Teardown** — socat removed; 5021 dead again (as-found). rtu-gate returns to a truthfully-OPEN
breaker chip, which is the feature working.
### Two nuances against the original T10 wording
- **"Both-replica chips"** resolves as both nodes' *trackers* tripping (proven in both logs)
rendered through the single **active** AdminUI. Traefik's `/health/active` check deliberately
routes the UI to the active admin node only — the standby returns 503
(`Standby: active node for role 'admin' is …`), so there is never a second replica serving
chips. Discovered while trying to force the sticky-cookie LB onto central-1.
- **The staleness-dim leg was not exercised.** Both pair nodes publish under the same
`(instance, host)` key, so staleness requires *every* publisher for a key to stop — deliberate
node-kill churn this gate did not do. The dim is a pure render predicate (`IsStale` → opacity).
Also corrected while here: §4's "ConnectionBackoff throttles retries" explanation (the previously
recorded root cause) is confirmed wrong — the breaker config opens on 2 failures in 30 s; the
operative cause was always that no failure reached the pipeline at all.
@@ -39,4 +39,15 @@ public enum DriverCapability
/// <summary><see cref="IHistoryProvider"/> reads (Raw/Processed/AtTime/Events). Retries by default.</summary>
HistoryRead,
/// <summary>
/// <c>IDriver.InitializeAsync</c> — connection establishment (R2-10 T10 closure: without this,
/// a driver that cannot connect contributes zero breaker input and the resilience panel stays
/// quiet for the failure operators most care about). Does NOT retry — the
/// <c>DriverInstanceActor</c> reconnect timer is the retry loop, and pipeline-level retries
/// would stack inside each timer tick. The tier-default breaker threshold is deliberately low
/// (2): at the 10 s reconnect cadence at most three attempts fit the breaker's 30 s sampling
/// window, so a Read-sized threshold of 5 could never open on connect failures.
/// </summary>
Initialize,
}
@@ -77,6 +77,10 @@ public sealed class DriverCapabilityInvokerFactory : IDriverCapabilityInvokerFac
// Cleanup (no longer the correctness mechanism): drop this instance's cached pipelines across all
// generations so lingering old-generation entries don't accumulate. No-op on first spawn.
_builder.Invalidate(driverInstanceId);
// The tracker's snapshots describe the breakers inside the pipelines just discarded, so they go
// too — a retained "Breaker OPEN" from a pre-respawn outage could never self-correct (only the
// discarded pipeline's OnClosed stamps the close) and would show a false OPEN chip forever.
_statusTracker.Invalidate(driverInstanceId);
return new CapabilityInvoker(
_builder,
@@ -82,6 +82,11 @@ public sealed record DriverResilienceOptions
[DriverCapability.AlarmSubscribe] = new(TimeoutSeconds: 5, RetryCount: 3, BreakerFailureThreshold: 5),
[DriverCapability.AlarmAcknowledge] = new(TimeoutSeconds: 5, RetryCount: 0, BreakerFailureThreshold: 5),
[DriverCapability.HistoryRead] = new(TimeoutSeconds: 30, RetryCount: 2, BreakerFailureThreshold: 5),
// Initialize: no retry (the DriverInstanceActor reconnect timer is the retry loop) and a
// threshold of 2, NOT the Read-sized 5 — the breaker samples over 30 s and the reconnect
// timer fires every 10 s, so at most three init attempts fit one window; a threshold of 5
// could never open on connect failures (R2-10 T10).
[DriverCapability.Initialize] = new(TimeoutSeconds: 5, RetryCount: 0, BreakerFailureThreshold: 2),
},
DriverTier.B => new Dictionary<DriverCapability, CapabilityPolicy>
{
@@ -93,6 +98,8 @@ public sealed record DriverResilienceOptions
[DriverCapability.AlarmSubscribe] = new(TimeoutSeconds: 8, RetryCount: 3, BreakerFailureThreshold: 5),
[DriverCapability.AlarmAcknowledge] = new(TimeoutSeconds: 8, RetryCount: 0, BreakerFailureThreshold: 5),
[DriverCapability.HistoryRead] = new(TimeoutSeconds: 60, RetryCount: 2, BreakerFailureThreshold: 5),
// See the Tier A Initialize note: threshold 2 is cadence-reachability, not a tuning choice.
[DriverCapability.Initialize] = new(TimeoutSeconds: 8, RetryCount: 0, BreakerFailureThreshold: 2),
},
DriverTier.C => new Dictionary<DriverCapability, CapabilityPolicy>
{
@@ -104,6 +111,8 @@ public sealed record DriverResilienceOptions
[DriverCapability.AlarmSubscribe] = new(TimeoutSeconds: 15, RetryCount: 1, BreakerFailureThreshold: 0),
[DriverCapability.AlarmAcknowledge] = new(TimeoutSeconds: 15, RetryCount: 0, BreakerFailureThreshold: 0),
[DriverCapability.HistoryRead] = new(TimeoutSeconds: 120, RetryCount: 1, BreakerFailureThreshold: 0),
// Tier C stays breaker-less like every other Tier C capability (process-level supervision).
[DriverCapability.Initialize] = new(TimeoutSeconds: 15, RetryCount: 0, BreakerFailureThreshold: 0),
},
_ => throw new ArgumentOutOfRangeException(nameof(tier), tier, $"No default policy table defined for tier {tier}."),
};
@@ -158,6 +158,17 @@ public static class DriverResilienceOptionsParser
retry = 0;
}
// Initialize's retry loop is the DriverInstanceActor reconnect timer, not the pipeline —
// pipeline retries would stack inside each timer tick (and, worse, stretch a single init
// attempt past the timer interval so attempts overlap). Same funnel-enforced invariant
// shape as the write-shaped rule above, different rationale.
if (capability == DriverCapability.Initialize && retry > 0)
{
AppendDiagnostic(ref diagnostic,
$"{capability} never retries (the reconnect timer is the retry loop); retryCount override ignored.");
retry = 0;
}
var breaker = merged.BreakerFailureThreshold;
if (breaker < ResiliencePolicyRanges.MinBreakerFailureThreshold)
{
@@ -83,6 +83,27 @@ public sealed class DriverResilienceStatusTracker
(_, existing) => existing with { LastBreakerClosedUtc = utcNow, LastSampledUtc = utcNow });
}
/// <summary>
/// Drop every snapshot for one driver instance. Called by
/// <c>DriverCapabilityInvokerFactory.Create</c> alongside
/// <c>DriverResiliencePipelineBuilder.Invalidate</c>: the pipelines carry the actual breaker
/// state, so once they are discarded on a respawn, a retained snapshot describes a breaker
/// that no longer exists — and a stale "Breaker OPEN" could never self-correct, because only
/// the discarded pipeline's <c>OnClosed</c> hook stamps the close.
/// </summary>
/// <param name="driverInstanceId">The driver instance whose snapshots should be dropped.</param>
/// <returns>The number of snapshots removed.</returns>
public int Invalidate(string driverInstanceId)
{
var removed = 0;
foreach (var key in _status.Keys)
{
if (key.DriverInstanceId == driverInstanceId && _status.TryRemove(key, out _))
removed++;
}
return removed;
}
/// <summary>Record a process recycle event (Tier C only).</summary>
/// <param name="driverInstanceId">The driver instance identifier.</param>
/// <param name="hostName">The host name.</param>
@@ -21,7 +21,7 @@ namespace ZB.MOM.WW.OtOpcUa.AdminUI.Components.Shared.Drivers;
public sealed class ResilienceFormModel
{
public static readonly string[] Capabilities =
["Read", "Write", "Discover", "Subscribe", "Probe", "AlarmSubscribe", "AlarmAcknowledge", "HistoryRead"];
["Read", "Write", "Discover", "Subscribe", "Probe", "AlarmSubscribe", "AlarmAcknowledge", "HistoryRead", "Initialize"];
// RecycleIntervalSeconds is GONE (Gitea #522) — the scheduled-recycle machinery it authored was
// deleted, so the field configured nothing. It is deliberately NOT explicitly stripped from stored
@@ -687,6 +687,7 @@ public sealed class DriverInstanceActor : ReceiveActor, IWithTimers
var log = _log;
var driverInstanceId = _driverInstanceId;
var driver = _driver;
var invoker = _invoker;
_ = Task.Run(async () =>
{
try
@@ -695,6 +696,8 @@ public sealed class DriverInstanceActor : ReceiveActor, IWithTimers
{
try
{
// Deliberately UNWRAPPED: teardown is best-effort cleanup of an already-dead
// transport — a shutdown failure must not count toward the Initialize breaker.
await driver.ShutdownAsync(CancellationToken.None);
}
catch (Exception ex)
@@ -704,7 +707,20 @@ public sealed class DriverInstanceActor : ReceiveActor, IWithTimers
"initialising anyway", driverInstanceId);
}
}
await driver.InitializeAsync(driverConfigJson, CancellationToken.None);
// R2-10 T10: the connect path was the ONE capability call the invoker never saw, so a
// driver that could not connect contributed zero breaker input and the resilience panel
// stayed quiet for a dead endpoint. Keyed on the driver-instance id (init spans the
// whole driver, like a bulk subscribe). The Initialize policy never retries — this
// actor's RetryConnect timer is the retry loop — and its pipeline timeout now bounds a
// hanging connect via the ct the call site must honour. An open breaker fast-fails the
// attempt with BrokenCircuitException, which lands in the catch below exactly like a
// real connect failure: InitializeFailed → keep retrying on the timer. The 15 s break
// window is shorter than two 10 s timer ticks, so recovery costs at most one extra tick.
await invoker.ExecuteAsync(
DriverCapability.Initialize,
driverInstanceId,
async ct => await driver.InitializeAsync(driverConfigJson, ct),
CancellationToken.None);
self.Tell(new InitializeSucceeded(generation));
}
catch (Exception ex)
@@ -22,6 +22,29 @@ public sealed class DriverCapabilityInvokerFactoryTests
ILogger? logger = null) =>
new(builder, new DriverResilienceStatusTracker(), _ => tier, logger);
/// <summary>
/// A respawn (every <c>Create</c>) resets the instance's tracker state alongside its pipeline
/// cache. The pipelines carry the actual breaker state and are discarded by the same call, so a
/// snapshot recorded against them — a "Breaker OPEN" from a pre-deploy outage — refers to a
/// breaker that no longer exists; keeping it would show a false OPEN chip forever, because only
/// the discarded pipeline's <c>OnClosed</c> could have cleared it.
/// </summary>
[Fact]
public void Create_resets_the_instances_tracker_state_with_its_pipelines()
{
var tracker = new DriverResilienceStatusTracker();
var factory = new DriverCapabilityInvokerFactory(
new DriverResiliencePipelineBuilder(), tracker, _ => DriverTier.A);
tracker.RecordBreakerOpen("d1", "d1", DateTime.UtcNow);
tracker.RecordBreakerOpen("d2", "d2", DateTime.UtcNow);
factory.Create("d1", "Modbus");
tracker.TryGet("d1", "d1").ShouldBeNull(
"the respawn discarded the pipeline whose breaker this snapshot described");
tracker.TryGet("d2", "d2")!.IsBreakerOpen.ShouldBeTrue("other instances must be untouched");
}
/// <summary>
/// End-to-end proof the per-instance override REACHES execution: a ResilienceConfig that sets
/// Read's retryCount to 0 must suppress the tier-A Read retry — a failing Read is attempted once,
@@ -331,6 +331,19 @@ public sealed class DriverResilienceOptionsParserTests
diag.ShouldContain("AlarmAcknowledge never retries");
}
/// <summary>An Initialize retryCount override is forced to 0 with a diagnostic — the
/// DriverInstanceActor reconnect timer is the retry loop, so pipeline retries would stack.</summary>
[Fact]
public void InitializeRetryOverride_IsForcedToZero_WithDiagnostic()
{
var options = DriverResilienceOptionsParser.ParseOrDefaults(
DriverTier.A, "{\"capabilityPolicies\":{\"Initialize\":{\"retryCount\":3}}}", out var diag);
options.Resolve(DriverCapability.Initialize).RetryCount.ShouldBe(0);
diag.ShouldNotBeNull();
diag.ShouldContain("Initialize never retries");
}
/// <summary>A Read (idempotent) retryCount override is still honored — the invariant is write-shaped only.</summary>
[Fact]
public void ReadRetryOverride_StillHonored()
@@ -78,6 +78,39 @@ public sealed class DriverResilienceOptionsTests
defaults[DriverCapability.AlarmAcknowledge].RetryCount.ShouldBe(0);
}
/// <summary>
/// Initialize never retries by default — the <c>DriverInstanceActor</c> reconnect timer IS the
/// retry loop; pipeline retries would stack inside each timer tick.
/// </summary>
/// <param name="tier">The driver tier to test.</param>
[Theory]
[InlineData(DriverTier.A)]
[InlineData(DriverTier.B)]
[InlineData(DriverTier.C)]
public void Initialize_NeverRetries_ByDefault(DriverTier tier)
{
var defaults = DriverResilienceOptions.GetTierDefaults(tier);
defaults[DriverCapability.Initialize].RetryCount.ShouldBe(0);
}
/// <summary>
/// The Initialize breaker threshold must be reachable at the reconnect cadence (R2-10 T10):
/// the breaker samples over 30 s and the reconnect timer fires every 10 s, so at most THREE
/// init attempts fit one sampling window. A threshold above 3 can never open on connect
/// failures — the exact silent-inertness this capability exists to fix — so the default is
/// pinned at 2 (Polly's minimum enabled throughput). Tier C stays breaker-less, consistent
/// with every other Tier C capability (the supervisor's process-level breaker owns it).
/// </summary>
[Theory]
[InlineData(DriverTier.A, 2)]
[InlineData(DriverTier.B, 2)]
[InlineData(DriverTier.C, 0)]
public void Initialize_BreakerThreshold_IsReachableAtTheReconnectCadence(DriverTier tier, int expectedThreshold)
{
var defaults = DriverResilienceOptions.GetTierDefaults(tier);
defaults[DriverCapability.Initialize].BreakerFailureThreshold.ShouldBe(expectedThreshold);
}
/// <summary>Verifies that idempotent capabilities retry by default.</summary>
/// <param name="tier">The driver tier to test.</param>
/// <param name="capability">The driver capability to test.</param>
@@ -315,6 +315,47 @@ public sealed class DriverResiliencePipelineBuilderTests
snap.IsBreakerOpen.ShouldBeFalse("the breaker must read closed after a successful recovery probe");
}
/// <summary>
/// R2-10 T10 closure: with the TIER DEFAULTS (no override), consecutive Initialize failures
/// open the breaker — and a subsequent attempt fast-fails WITHOUT reaching the call site.
/// The load-bearing half is the threshold: at the 10 s reconnect cadence at most three init
/// attempts fit the 30 s sampling window, so this test drives exactly the tier-default
/// threshold of failures and demands an open breaker — it goes red if anyone raises the
/// Initialize threshold back into never-opens territory (5), the silent inertness the
/// live gate proved.
/// </summary>
[Fact]
public async Task Initialize_BreakerOpens_AtTierDefaults_AndFastFailsTheNextAttempt()
{
var tracker = new DriverResilienceStatusTracker();
var builder = new DriverResiliencePipelineBuilder(statusTracker: tracker);
var pipeline = builder.GetOrCreate("drv-init", "drv-init", DriverCapability.Initialize, TierAOptions);
var threshold = TierAOptions.Resolve(DriverCapability.Initialize).BreakerFailureThreshold;
for (var i = 0; i < threshold; i++)
{
await Should.ThrowAsync<InvalidOperationException>(async () =>
await pipeline.ExecuteAsync(async _ =>
{
await Task.Yield();
throw new InvalidOperationException("connect refused");
}));
}
tracker.TryGet("drv-init", "drv-init")!.IsBreakerOpen.ShouldBeTrue(
"the tier-default threshold of consecutive connect failures must open the Initialize breaker");
// The open breaker fast-fails the next attempt without invoking the connect at all.
var reached = false;
await Should.ThrowAsync<BrokenCircuitException>(async () =>
await pipeline.ExecuteAsync(_ =>
{
reached = true;
return ValueTask.CompletedTask;
}));
reached.ShouldBeFalse("an open breaker must not let the connect attempt through");
}
/// <summary>Verifies that the tracker isolates counters per host.</summary>
[Fact]
public async Task Tracker_IsolatesCounters_PerHost()
@@ -90,6 +90,27 @@ public sealed class DriverResilienceStatusTrackerTests
tracker.TryGet("drv", "host")!.IsBreakerOpen.ShouldBeTrue();
}
/// <summary>
/// Invalidate drops every entry for one instance and leaves other instances alone. Without
/// this, a driver-child respawn (deploy) while a breaker is OPEN strands a permanently-stale
/// "Breaker OPEN" snapshot: the respawn discards the pipelines whose <c>OnClosed</c> is the
/// only thing that stamps the close, so nothing can ever clear the flag again.
/// </summary>
[Fact]
public void Invalidate_DropsTheInstance_LeavesOthers()
{
var tracker = new DriverResilienceStatusTracker();
tracker.RecordBreakerOpen("drv", "host-a", Now);
tracker.RecordBreakerOpen("drv", "host-b", Now);
tracker.RecordBreakerOpen("other", "host-a", Now);
tracker.Invalidate("drv");
tracker.TryGet("drv", "host-a").ShouldBeNull();
tracker.TryGet("drv", "host-b").ShouldBeNull();
tracker.TryGet("other", "host-a")!.IsBreakerOpen.ShouldBeTrue();
}
/// <summary>Verifies that RecordRecycle populates the LastRecycleUtc timestamp.</summary>
[Fact]
public void RecordRecycle_Populates_LastRecycleUtc()
@@ -80,6 +80,100 @@ public sealed class DriverInstanceActorResilienceWiringTests : RuntimeActorTestB
invoker.WriteCalls.ShouldContain(c => c.Host == "plc-7" && c.IsIdempotent == false);
}
/// <summary>
/// R2-10 T10 closure: <c>IDriver.InitializeAsync</c> routes through the invoker as
/// <see cref="DriverCapability.Initialize"/> keyed on the driver-instance id (init spans the
/// whole driver, like a bulk subscribe). Before this wiring the connect path was the ONE
/// capability call the invoker never saw — a driver that could not connect contributed zero
/// breaker input, so the resilience panel stayed quiet for a dead endpoint.
/// </summary>
[Fact]
public void Initialize_routes_through_the_capability_invoker()
{
var invoker = new RecordingInvoker();
var driver = new HostResolvingWritableDriver();
var actor = Sys.ActorOf(DriverInstanceActor.Props(driver, invoker: invoker));
actor.Tell(new DriverInstanceActor.InitializeRequested("{}"));
AwaitCondition(
() => invoker.Calls.Any(c => c.Capability == DriverCapability.Initialize),
TimeSpan.FromSeconds(3));
invoker.Calls.ShouldContain(c =>
c.Capability == DriverCapability.Initialize && c.Host == driver.DriverInstanceId);
}
/// <summary>
/// The open-breaker shape end to end at the actor: when the INVOKER itself throws on the
/// Initialize route (exactly what an open circuit does — the call site is never reached),
/// the actor treats it as an ordinary init failure — lands in Reconnecting, keeps the retry
/// timer running, and recovers to Connected once the invoker lets an attempt through. This
/// pins two things: the reconnect-RETRY path is wrapped too (not just the initial connect),
/// and a breaker fast-fail can never wedge or crash the actor.
/// </summary>
[Fact]
public async Task An_invoker_fast_fail_on_init_keeps_the_actor_retrying_until_recovery()
{
var invoker = new FastFailingInitInvoker(failFirst: 2);
var driver = new HostResolvingWritableDriver();
var actor = Sys.ActorOf(DriverInstanceActor.Props(
driver, invoker: invoker, reconnectInterval: TimeSpan.FromMilliseconds(100)));
actor.Tell(new DriverInstanceActor.InitializeRequested("{}"));
// The first two init attempts die inside the invoker (open breaker); recovery is proven by a
// write succeeding — WriteAttribute fast-fails synchronously in Connecting/Reconnecting, so a
// Success reply is only possible from Connected.
DriverInstanceActor.WriteAttributeResult? reply = null;
for (var i = 0; i < 120 && reply is not { Success: true }; i++)
{
reply = await actor.Ask<DriverInstanceActor.WriteAttributeResult>(
new DriverInstanceActor.WriteAttribute("tag-1", 42), TimeSpan.FromSeconds(1));
if (reply is not { Success: true }) await Task.Delay(25);
}
reply.ShouldNotBeNull();
reply!.Success.ShouldBeTrue("the actor must recover to Connected once the breaker lets an attempt through");
invoker.InitializeAttempts.ShouldBeGreaterThanOrEqualTo(3,
"the reconnect-retry path must route through the invoker (initial attempt + at least the two fast-failed retries)");
}
/// <summary>Stands in for an OPEN circuit breaker on the Initialize route: the first
/// <paramref name="failFirst"/> Initialize calls throw WITHOUT invoking the call site (the
/// driver's connect never runs — exactly Polly's <c>BrokenCircuitException</c> shape), then the
/// breaker "closes" and calls pass through. All other capabilities always pass through.</summary>
private sealed class FastFailingInitInvoker : IDriverCapabilityInvoker
{
private readonly int _failFirst;
private int _initializeAttempts;
public FastFailingInitInvoker(int failFirst) => _failFirst = failFirst;
public int InitializeAttempts => Volatile.Read(ref _initializeAttempts);
public ValueTask<TResult> ExecuteAsync<TResult>(
DriverCapability capability, string hostName,
Func<CancellationToken, ValueTask<TResult>> callSite, CancellationToken cancellationToken)
=> callSite(cancellationToken);
public ValueTask ExecuteAsync(
DriverCapability capability, string hostName,
Func<CancellationToken, ValueTask> callSite, CancellationToken cancellationToken)
{
if (capability == DriverCapability.Initialize
&& Interlocked.Increment(ref _initializeAttempts) <= _failFirst)
{
throw new InvalidOperationException("circuit open (simulated fast-fail)");
}
return callSite(cancellationToken);
}
public ValueTask<TResult> ExecuteWriteAsync<TResult>(
string hostName, bool isIdempotent,
Func<CancellationToken, ValueTask<TResult>> callSite, CancellationToken cancellationToken)
=> callSite(cancellationToken);
}
/// <summary>Records every capability + resolved-host key routed through it, then delegates to the
/// real call-site so the driver still runs. Stands in for <c>CapabilityInvoker</c> without pulling
/// Polly / Core into the Runtime test.</summary>