lmxopcua/archreview/plans/R2-10-resilience-observability-plan.md

# Design + Implementation Plan — R2-10 Resilience Observability: an operator-facing reader for `DriverResilienceStatusTracker`

> **Status (2026-07-12):** PLANNED — not started.

- **Source reports:** `archreview/03-server-runtime.md` (U10) · `archreview/04-adminui.md` (U-5) · `archreview/00-OVERALL.md` prioritized action list **item #10** ("Resilience observability: a `/hosts` reader for `DriverResilienceStatusTracker` (Stream E.2/E.3)" — 03 U10 + 04 U-5, Medium, "operators currently blind").
- **History:** `archreview/plans/FOLLOWUP-10-resilience-dispatch-gap.md` — #10 (`bacea1a4`) wired `CapabilityInvoker` into all 6 production dispatch sites; #13 (`75403caa`) made AdminUI-authored per-instance `ResilienceConfig` overrides live. The Phase 6.1 "Stream E.2" (snapshot-persisting HostedService) and "E.3" (SignalR push + Blazor `/hosts` panel) readers were never built; the interim mitigation was retry/breaker structured logging in `DriverResiliencePipelineBuilder` (added with #10).
- **Plan verified against tree at:** `f6eaa267` (master, clean) — every anchor below was opened and re-read at this commit.
- **Scope:** `Core/Resilience` (tracker + pipeline builder + invoker — additive only), `Commons/Messages/Drivers` (one new message), `Host/Drivers` (one new publisher hosted service + DI), `AdminUI/Hubs` (one new store + one new bridge actor), `AdminUI/Components/Shared/Drivers/DriverStatusPanel.razor` (one new section).

## Verification summary

Every claim in U10/U-5 was re-verified at `f6eaa267`. **All confirmed — none stale.**

- **The tracker has no reader.** `grep -rn "DriverResilienceStatusTracker" src --include="*.cs"` matches only `Core/Resilience/{DriverResilienceStatusTracker, DriverResiliencePipelineBuilder, CapabilityInvoker, DriverCapabilityInvokerFactory}.cs` and `Host/Drivers/DriverFactoryBootstrap.cs` (the DI registration, `:57`). Zero AdminUI/Runtime references. `grep -rn Resilience src/Server/ZB.MOM.WW.OtOpcUa.AdminUI` matches only the authoring surface (`DriverResilienceSection.razor`, `ResilienceFormModel.cs`, the 8 driver pages). The only production consumers of tracker *state* are its own writers.
- **What the tracker holds** (`DriverResilienceStatusTracker.cs`): a `ConcurrentDictionary<(DriverInstanceId, HostName), ResilienceStatusSnapshot>` — `ConsecutiveFailures`, `LastBreakerOpenUtc`, `LastRecycleUtc`, footprint bytes, `CurrentInFlight`, `LastSampledUtc`. Thread-safe (immutable record values, `AddOrUpdate`); `Snapshot()` returns a copy safe under concurrent writes — ideal for a periodic reader. **Note the key: the tracker aggregates per `(instance, host)`, NOT per capability** — pipelines are per `(instance, host, capability)` (`DriverResiliencePipelineBuilder.PipelineKey`) but all telemetry callbacks fold into the `(instance, host)` counter. The read model below therefore keys per **instance × host** (mirroring the tracker) rather than instance × capability; adding capability granularity would mean re-keying the tracker and every invoker call site — rejected as not the simplest thing (capability is already visible in the #10 log lines when drill-down is needed).
- **Feeding is real and live:** `CapabilityInvoker` calls `RecordCallStart/Complete` around every wrapped dispatch (`CapabilityInvoker.cs:70/80/94/104…`); `DriverResiliencePipelineBuilder.Build` wires `OnRetry → RecordFailure` (`:131`), `OnOpened → RecordBreakerOpen` (`:155`), `OnClosed → RecordSuccess` (`:165`). Registered singleton + threaded into the builder and invoker factory in `DriverFactoryBootstrap.AddOtOpcUaDriverFactories` (`:57-71`).
- **Two truthfulness gaps found in the tracker itself** (must fix for a display to be honest):
  1. **No breaker-close marker.** `RecordBreakerOpen` stamps `LastBreakerOpenUtc` but nothing ever records "closed" — `OnClosed` only calls `RecordSuccess` (resets `ConsecutiveFailures`). "Is the breaker open *now*?" is underivable from the snapshot.
  2. **`RecordSuccess` has exactly one caller — the breaker's `OnClosed`.** A successful capability call never resets `ConsecutiveFailures`, so after a transient blip with 2 retries the counter reads 2 *forever* (until a full breaker open/close cycle). The tracker's own xmldoc ("Reset … on a successful pipeline execution") shows the intended-but-unwired call from the invoker's success path — another edge of the house "built-but-never-wired" pattern (00-OVERALL theme #1).
- **Topology determination (where the tracker lives vs where AdminUI renders):** `Host/Program.cs` gates by role: the `hasDriver` block (`:104-262`) calls `AddOtOpcUaDriverFactories()` (`:195`) — **the tracker singleton exists only in driver-role processes**. The `hasAdmin` block mounts the AdminUI (`:278-293`) and spawns the DPS→SignalR bridge actors via `WithOtOpcUaSignalRBridges()` inside the Akka configurator (`:269-273`). In the split-role MAIN cluster (admin-a/admin-b vs driver nodes) tracker and AdminUI are **different processes** — state must travel over the cluster. Even on the fused docker-dev rig (central-1/central-2 both `admin,driver`), `:9200` **round-robins** the two containers while a given `DriverHostActor` child runs on one of them — so a direct in-process tracker read from the Blazor circuit shows the wrong (often empty) node's state. **A cluster transport is mandatory in every deployed topology.**
- **The precedent to mirror (found, verified end-to-end):** driver-health already makes exactly this trip. `DriverInstanceActor:994` → `AkkaDriverHealthPublisher.Publish` → DPS topic `driver-health` (`DriverHealthChanged` record in `Commons/Messages/Drivers`, `TopicName` const on the contract) → per-admin-node `DriverStatusSignalRBridge` (spawned by `WithOtOpcUaSignalRBridges`, `HubServiceCollectionExtensions.cs:69-72`) → `IDriverStatusSnapshotStore` singleton upsert + `SnapshotChanged` event → `DriverStatusPanel.razor` reads the store **in-process** (`:190-198`), explicitly *not* via a self-targeted SignalR `HubConnection` (the house ban — a server-side circuit behind Traefik cannot dial its own public URL; documented at `IDriverStatusSnapshotStore.cs:28-36` and `DriverStatusPanel.razor:182-189`). This plan mirrors that flow one-for-one.
- **UI placement facts:** `DriverStatusPanel` is embedded on all 8 driver pages (`Components/Pages/Clusters/Drivers/*DriverPage.razor`) — the **same pages** where `DriverResilienceSection` authors the `ResilienceConfig` overrides that #13 made live. `/hosts` (`Hosts.razor`) shows cluster members + a cluster-scoped driver-health table built by `HostsDriverView.Build(DriverStore.GetAll(), …)` (`:291`).
- **Test landscape:** tracker/builder/invoker suites exist at `tests/Core/ZB.MOM.WW.OtOpcUa.Core.Tests/Resilience/` (incl. `DriverResilienceStatusTrackerTests.cs`, `DriverResiliencePipelineBuilderTests.cs`, `InFlightCounterTests.cs`); the DI wiring-guard template is `tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/ResilienceInvokerFactoryRegistrationTests.cs` (pure-DI, no Docker fixture); the store-test template is `tests/Server/ZB.MOM.WW.OtOpcUa.AdminUI.Tests/DriverStatusSnapshotStoreTests.cs`. **AdminUI has no bUnit** — razor rendering changes are verified only by the live-`/run` pass (docker-dev `:9200` round-robins central-1/central-2; rebuild BOTH).

---

## Restatement

`DriverResilienceStatusTracker` is registered, fed by every wrapped capability call, retry, and breaker transition — and consumed by nothing. Since #10/#13, retry/breaker/bulkhead behavior is production-live **and operator-authorable** (per-instance `ResilienceConfig` on all 8 driver pages), yet the only way to see a breaker reject calls (`BrokenCircuitException`) or a retry storm is to grep server logs. Operators author live resilience config completely blind. Ship the smallest reader that gives them sight of breaker-open and retry storms, on the page where they author the config.

## Verification

Confirmed at `f6eaa267` — see Verification summary. Anchors: `src/Core/ZB.MOM.WW.OtOpcUa.Core/Resilience/DriverResilienceStatusTracker.cs` (no non-writer reference anywhere), `DriverResiliencePipelineBuilder.cs:124-170` (the interim log-only surface), `src/Server/ZB.MOM.WW.OtOpcUa.Host/Drivers/DriverFactoryBootstrap.cs:57` (singleton registration, driver nodes only).

## Root cause

Phase 6.1 shipped Stream E.1 (the tracker + telemetry hooks) and deferred E.2 (snapshot persistence) / E.3 (push + panel) to "a follow-up visual-review PR" (tracker xmldoc, `:12-16`) that never happened. #10 wired the *writer* side into production and added log lines as an interim reader; the review (03/U10, 04/U-5) now tracks the missing operator surface. Two smaller instances of the same pattern hide inside the tracker itself (no breaker-close marker; `RecordSuccess` unwired from the invoker success path).

## Proposed design

Mirror the driver-health flow end-to-end. Four pieces, smallest-correct at each hop:

**(a) Read model** — `ResilienceStatusSnapshot` gains `LastBreakerClosedUtc` + a derived `IsBreakerOpen` (`LastBreakerOpenUtc` is set and later than any close). One new wire message in Commons:

```csharp
// Commons/Messages/Drivers/DriverResilienceStatusChanged.cs
public sealed record DriverResilienceStatusChanged(
    string DriverInstanceId,
    string HostName,
    bool BreakerOpen,
    int ConsecutiveFailures,
    int CurrentInFlight,
    DateTime? LastBreakerOpenUtc,
    DateTime LastSampledUtc,
    DateTime PublishedUtc)
{
    public const string TopicName = "driver-resilience-status";
}
```

Keyed per **(instance, host)** — matches the tracker key (see Verification summary for why not per-capability). No `ClusterId`: `DriverInstanceId` is globally unique and the panel filters on it alone (same as `DriverStatusPanel` today). Tracker truthfulness fixes ride along: `RecordBreakerClose` (called from the builder's `OnClosed` beside the existing `RecordSuccess`) and `RecordSuccess` on the invoker's success paths so `ConsecutiveFailures` is a true consecutive counter (nonzero only during an active storm).

*Known display caveat (documented, accepted):* Polly half-opens the breaker after `BreakDuration` without firing `OnClosed`; `OnClosed` fires only when a probe call succeeds. An idle driver therefore shows "breaker OPEN" until the next call — truthful enough (the next call is at best a probe) and self-corrects on traffic.

**(b) Transport (driver node → admin surface)** — a periodic full-snapshot publisher on driver nodes: `DriverResilienceStatusPublisherService : BackgroundService` in `Host/Drivers/`, ctor `(DriverResilienceStatusTracker, Func<ActorSystem>, ILogger<…>, TimeSpan? interval = null /* default 5 s */)`. Every tick: `tracker.Snapshot()` → map each triple to a `DriverResilienceStatusChanged` (`PublishedUtc = now`) → `DistributedPubSub.Get(system).Mediator.Tell(new Publish(TopicName, msg))` per entry. Loop body in try/catch (a transient DPS/actor-system hiccup must never kill the service); skip empty snapshots. Publishing the **full set every tick** (not deltas) deliberately self-heals late-subscribing admin nodes — the known redundancy-state "late-subscriber" gotcha — and gives staleness detection for free. Volume is negligible (entries = active instances × hosts, typically < a few dozen, every 5 s). Registered in `DriverFactoryBootstrap.AddOtOpcUaDriverFactories` next to the tracker it reads (driver nodes only, exactly where the tracker exists), with a `TryAddSingleton<Func<ActorSystem>>` for self-containedness (idempotent with `Program.cs:260`).

Why `Host`, not `Runtime`: Runtime is deliberately **Polly-free** — it references `Core.Abstractions`, not `Core` where the tracker lives (the exact boundary FOLLOWUP-10 discovered the hard way). Host already references Core, Commons, and Akka (`OtOpcUaServerHostedService` resolves the DPS mediator via the same lazy `Func<ActorSystem>` pattern). No new seam needed.

**Rejected transport alternatives:**
- *Direct in-process tracker read from the AdminUI/Blazor circuit* — wrong in every deployed topology (split-role MAIN cluster: different process; fused docker-dev: `:9200` round-robin lands circuits on a node that isn't running the driver instance). Rejected.
- *Persist snapshots to a `DriverInstanceResilienceStatus` table* (the original Stream E.2 shape, still named in the tracker xmldoc) — needs a migration, a poller, and row cleanup; the house idiom has since evolved to DPS → in-memory store, and 04/U-5's own recommendation names that idiom. Rejected; fix the stale xmldoc instead.
- *Event-driven publish from inside the tracker* (hook every `RecordX`) — would thread an Akka publisher seam into Polly-free/Akka-free Core, and is chattier under a retry storm exactly when the cluster is stressed. Periodic snapshot is simpler and bounded. Rejected.
- *Extend `DriverHealthChanged`* — mutates an existing wire contract and conflates two feeds with different cadences (event-driven vs periodic). Separate topic = zero blast radius. Rejected.
- *Pull via `AdminOperationsActor` ask* — request/response through the admin-pinned singleton plus panel-side polling; more moving parts than the established push idiom, and the singleton is one node while the store idiom feeds every admin replica (which the round-robin requires). Rejected.

**(c) Admin surface** — mirror the driver-status pair:
- `IDriverResilienceStatusStore` + `InMemoryDriverResilienceStatusStore` in `AdminUI/Hubs/`: keyed `(DriverInstanceId, HostName)`, `Upsert`, `GetForInstance(driverInstanceId)` (per-host list), `GetAll()`, `SnapshotChanged` event (capture-then-invoke, same as `InMemoryDriverStatusSnapshotStore`). Registered in `HubServiceCollectionExtensions.AddOtOpcUaDriverStatusServices`.
- `DriverResilienceStatusBridge` (`ReceiveActor`, `AdminUI/Hubs/`): `PreStart` subscribes to the DPS topic; on message, `store.Upsert(msg)`. Spawned per admin node in `WithOtOpcUaSignalRBridges` beside the other four bridges. **No new SignalR hub** — the only consumer is the server-side panel, which reads the store in-process per the house ban on self-HubConnections; `DriverStatusHub` exists for browser-JS clients and resilience has none. Add a hub later only if an external consumer appears (rejected for v1 as speculative surface).

**(d) UI placement** — extend **`DriverStatusPanel.razor`** (recommended; embedded on all 8 driver pages — the exact pages where `DriverResilienceSection` authors the config, closing the authoring feedback loop). New "Resilience" block under the existing status chips: one row per host with
- `Breaker OPEN` chip (`chip-bad`) with "opened Xs ago" when `BreakerOpen`, else nothing (healthy is quiet);
- `N consecutive failures` chip (`chip-warn`) when `ConsecutiveFailures > 0` (the retry-storm signal);
- `in-flight N` (`chip-idle`) when `CurrentInFlight > 0`;
- **staleness:** row dims + shows a `stale` chip when `PublishedUtc` is older than 15 s (3× publish interval) — covers driver-node death, where entries freeze; reuses the panel's existing 5 s re-render timer;
- muted "No resilience activity recorded yet" when the store has no entries for this instance (the tracker only populates once wrapped capability calls run).

Subscription lifecycle copies the existing pattern exactly: inject the store, subscribe `SnapshotChanged` in `OnInitializedAsync`, filter by `DriverInstanceId`, marshal via `InvokeAsync(StateHasChanged)`, unsubscribe first in `DisposeAsync`.

*Rejected UI alternatives:* a `/hosts` column (fleet-wide join of a second store into `HostsDriverView`; more code, and `/hosts` rows are cluster-scoped health — the operator authoring overrides is on the driver page; fine follow-on once the store exists) · a dedicated `/resilience` page (new nav surface for a v1 that needs two chips). The store makes both trivially addable later.

## Implementation steps (exact paths)

1. **Core read-model truthfulness** — `src/Core/ZB.MOM.WW.OtOpcUa.Core/Resilience/DriverResilienceStatusTracker.cs`: add `LastBreakerClosedUtc` to `ResilienceStatusSnapshot`, derived `IsBreakerOpen`, and `RecordBreakerClose(driverInstanceId, hostName, utcNow)`. Update the stale class xmldoc (`:5-16` still promises the never-built snapshot-persisting HostedService/table).
2. **Builder close-hook** — `src/Core/ZB.MOM.WW.OtOpcUa.Core/Resilience/DriverResiliencePipelineBuilder.cs` (`OnClosed`, `:161-170`): add `tracker?.RecordBreakerClose(...)` beside the existing `RecordSuccess`. Refresh the ctor `logger` xmldoc (`:38-42`, "until the Admin /hosts reader ships" becomes stale once this lands).
3. **Invoker success-reset** — `src/Core/ZB.MOM.WW.OtOpcUa.Core/Resilience/CapabilityInvoker.cs`: on each `Execute*` success path (after `pipeline.ExecuteAsync` returns, before the `finally`), call `_statusTracker?.RecordSuccess(_driverInstanceId, hostName, utcNow)`. (Clock: reuse the invoker's existing time source if present, else `DateTime.UtcNow` matching current tracker call sites.)
4. **Wire message** — new `src/Core/ZB.MOM.WW.OtOpcUa.Commons/Messages/Drivers/DriverResilienceStatusChanged.cs` (record above). Plain Commons record = same DPS serialization path `DriverHealthChanged` already rides.
5. **Publisher** — new `src/Server/ZB.MOM.WW.OtOpcUa.Host/Drivers/DriverResilienceStatusPublisherService.cs` with `public static IReadOnlyList<DriverResilienceStatusChanged> BuildMessages(DriverResilienceStatusTracker tracker, DateTime publishedUtc)` (pure, unit-testable) + the `BackgroundService` timer/publish/try-catch shell.
6. **Publisher DI** — `src/Server/ZB.MOM.WW.OtOpcUa.Host/Drivers/DriverFactoryBootstrap.cs` (`AddOtOpcUaDriverFactories`): `services.TryAddSingleton<Func<ActorSystem>>(sp => () => sp.GetRequiredService<ActorSystem>());` + `services.AddHostedService<DriverResilienceStatusPublisherService>();` with a comment naming this the tracker's production reader.
7. **Admin store** — new `src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/Hubs/IDriverResilienceStatusStore.cs` + `InMemoryDriverResilienceStatusStore.cs`; register in `HubServiceCollectionExtensions.AddOtOpcUaDriverStatusServices`.
8. **Bridge actor** — new `src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/Hubs/DriverResilienceStatusBridge.cs`; spawn + `ActorRegistry` key in `HubServiceCollectionExtensions.WithOtOpcUaSignalRBridges` (name const `driver-resilience-status-bridge`).
9. **Panel** — `src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/Components/Shared/Drivers/DriverStatusPanel.razor`: inject the store, resilience section per (d), subscribe/unsubscribe per the existing pattern.
10. **Docs/ledger** — update `archreview/03-server-runtime.md` (U10), `archreview/04-adminui.md` (U-5), `archreview/plans/STATUS.md`, and FOLLOWUP-10's "no reader" notes once merged; `docs/` has no resilience guide to touch (the tracker/builder xmldocs updated in steps 1-2 are the canonical doc surface).

## Tests

- **Unit — read model (Core.Tests, existing suites):**
  - `DriverResilienceStatusTrackerTests`: `RecordBreakerClose` stamps the close time; `IsBreakerOpen` is false initially, true after open, false after later close, true again after re-open.
  - `DriverResiliencePipelineBuilderTests`: driving a pipeline to breaker-open then recovery records open **and close** on the tracker (extend the existing breaker test).
  - `CapabilityInvokerTests` (or `InFlightCounterTests` home): a successful `ExecuteAsync` resets `ConsecutiveFailures` previously raised by a failure.
- **Unit — publisher mapping (Host.IntegrationTests, pure-DI/no fixture):** `BuildMessages` maps a fed tracker to one message per (instance, host) with correct `BreakerOpen`/counter/`PublishedUtc`; empty tracker → empty list.
- **Wiring guard — the tracker now HAS a production reader** (the built-but-never-wired class of bug, mirroring `ResilienceInvokerFactoryRegistrationTests`): assert `AddOtOpcUaDriverFactories()` registers an `IHostedService` descriptor whose implementation type is `DriverResilienceStatusPublisherService` — a regression that unregisters the reader fails this test, not production.
- **Unit — store (AdminUI.Tests, mirroring `DriverStatusSnapshotStoreTests`):** upsert keyed by (instance, host) with last-write-wins; `GetForInstance` returns all hosts for one instance only; `SnapshotChanged` raised per upsert.
- **Bridge actor:** logic is one line (subscribe → upsert), same as the untested sibling bridges; covered by the live gate.
- **Razor (no bUnit — live-`/run` is the verification):** docker-dev rig, **rebuild BOTH central-1 and central-2** (`:9200` round-robins them):
  1. `docker compose build central-1 central-2 && docker compose up -d` in `docker-dev/`.
  2. On `http://localhost:9200`, open (or author) an **S7 driver** and point its endpoint at a dead port (e.g. `10.100.0.35:9999`) — per FOLLOWUP-10, S7's connect path throws on a dead socket, unlike Modbus's cheap paths; alternatively `lmxopcua-fix down s7` a live fixture. In `DriverResilienceSection`, author overrides `retryCount: 2`, `breakerFailureThreshold: 2` (small numbers so the breaker opens fast). Deploy.
  3. On the driver page, watch the panel: consecutive-failures chip climbs, then `Breaker OPEN` appears; correlate with `docker logs` central grep `circuit-breaker OPENED` (the #10 log line — first-ever live observation of it, closing FOLLOWUP-10's deferred behavioral gate as a bonus).
  4. Refresh until the circuit lands on the **other** central (round-robin) — chips must show there too (proves the per-node DPS bridge/store, not a lucky co-located read).
  5. Restore the endpoint (`lmxopcua-fix up s7 s7_1500`); on the next successful call the breaker closes → chips clear.
  6. Stop the driver node (fused rig: stop central-2 while circuit is on central-1): rows dim to `stale` within ~15 s.

## Effort + Risk

**Effort: Medium-small** — ~10 bite-sized tasks; one focused day including the live gate. **Risk: Low.** Everything is additive: three small Core additions (new snapshot field, one hook call, one success-reset call — the only behavior deltas, all confined to telemetry counters), a new message type, a new hosted service, a new store + bridge, and one razor section. No dispatch-path, artifact, or config-shape changes. Blast radius of a publisher bug = missing/wrong chips, never driver behavior. The main live-verify hazard is the known one: razor binding errors that unit tests can't see — hence the mandatory live-`/run` pass on both centrals.

---

## Task breakdown

> Branch: `fix/archreview-r2-10-resilience-observability` off `master`. TDD throughout: write the failing test, see FAIL, implement, see PASS. Commit per task (or per labeled group).

### T1 — Tracker: breaker-close marker + `IsBreakerOpen`
**Classification:** Core read model (unit-testable) · **Estimated implement time:** 5 min · **Parallelizable with:** T4, T7 · **Files:** `src/Core/ZB.MOM.WW.OtOpcUa.Core/Resilience/DriverResilienceStatusTracker.cs`, `tests/Core/ZB.MOM.WW.OtOpcUa.Core.Tests/Resilience/DriverResilienceStatusTrackerTests.cs`
1. Add tests: `RecordBreakerClose_StampsCloseTime`, `IsBreakerOpen_TracksOpenCloseReopen` → `dotnet test tests/Core/ZB.MOM.WW.OtOpcUa.Core.Tests --filter "FullyQualifiedName~DriverResilienceStatusTrackerTests"` → **FAIL** (no such members).
2. Add `LastBreakerClosedUtc` init-prop + computed `IsBreakerOpen` to `ResilienceStatusSnapshot`; add `RecordBreakerClose`. Fix the stale class xmldoc (`:5-16`).
3. Re-run filter → **PASS**. Commit: `feat(resilience): breaker-close marker + IsBreakerOpen on ResilienceStatusSnapshot (R2-10)`.

### T2 — Builder: record breaker-close in `OnClosed`
**Classification:** Core wiring (unit-testable) · **Estimated implement time:** 5 min · **Parallelizable with:** — (needs T1) · **blockedBy:** T1 · **Files:** `src/Core/ZB.MOM.WW.OtOpcUa.Core/Resilience/DriverResiliencePipelineBuilder.cs`, `tests/Core/ZB.MOM.WW.OtOpcUa.Core.Tests/Resilience/DriverResiliencePipelineBuilderTests.cs`
1. Extend the existing breaker-telemetry test: after recovery, tracker snapshot has `IsBreakerOpen == false` with a close stamp → `dotnet test tests/Core/ZB.MOM.WW.OtOpcUa.Core.Tests --filter "FullyQualifiedName~DriverResiliencePipelineBuilderTests"` → **FAIL**.
2. Add `tracker?.RecordBreakerClose(...)` in `OnClosed` (`:161-170`); refresh the ctor logger xmldoc (`:38-42`).
3. → **PASS**. Commit: `feat(resilience): OnClosed records breaker-close on the status tracker (R2-10)`.

### T3 — Invoker: reset consecutive failures on success
**Classification:** Core wiring (unit-testable) · **Estimated implement time:** 5 min · **Parallelizable with:** T2 (after T1) · **blockedBy:** T1 · **Files:** `src/Core/ZB.MOM.WW.OtOpcUa.Core/Resilience/CapabilityInvoker.cs`, `tests/Core/ZB.MOM.WW.OtOpcUa.Core.Tests/Resilience/CapabilityInvokerTests.cs`
1. Test: `ExecuteAsync_OnSuccess_ResetsConsecutiveFailures` (record a failure, run a succeeding call, assert 0) → `dotnet test tests/Core/ZB.MOM.WW.OtOpcUa.Core.Tests --filter "FullyQualifiedName~CapabilityInvokerTests"` → **FAIL**.
2. Call `_statusTracker?.RecordSuccess(...)` on each `Execute*` success path.
3. → **PASS**. Commit: `fix(resilience): successful capability calls reset ConsecutiveFailures (R2-10)`.

### T4 — Commons: `DriverResilienceStatusChanged` wire message
**Classification:** Contract (compile-verified; consumed by T5/T8 tests) · **Estimated implement time:** 3 min · **Parallelizable with:** T1, T7 · **Files:** `src/Core/ZB.MOM.WW.OtOpcUa.Commons/Messages/Drivers/DriverResilienceStatusChanged.cs`
1. Add the record + `TopicName = "driver-resilience-status"` const, xmldoc naming publisher + bridge (mirror `DriverHealthChanged`'s single-source-of-truth note). Commit: `feat(resilience): DriverResilienceStatusChanged DPS contract (R2-10)`.

### T5 — Host: publisher service (`BuildMessages` + timer shell)
**Classification:** Transport (mapping unit-testable; shell live-verified) · **Estimated implement time:** 5 min · **Parallelizable with:** T7 · **blockedBy:** T1, T4 · **Files:** `src/Server/ZB.MOM.WW.OtOpcUa.Host/Drivers/DriverResilienceStatusPublisherService.cs`, `tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/DriverResilienceStatusPublisherServiceTests.cs`
1. Tests (pure, no fixture): `BuildMessages_MapsTrackerSnapshot` (fed tracker → one msg per (instance,host), correct BreakerOpen/counters/PublishedUtc), `BuildMessages_EmptyTracker_ReturnsEmpty` → `dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests --filter "FullyQualifiedName~DriverResilienceStatusPublisherServiceTests"` → **FAIL**.
2. Implement `public static BuildMessages(tracker, publishedUtc)` + `BackgroundService` (`PeriodicTimer` 5 s default via ctor param, per-tick try/catch, skip-empty, DPS `Publish` per message via `Func<ActorSystem>`).
3. → **PASS**. Commit: `feat(resilience): periodic DPS publisher for tracker snapshots (R2-10)`.

### T6 — Host: DI registration + reader wiring guard
**Classification:** Wiring + guard (the built-but-never-wired class) · **Estimated implement time:** 5 min · **blockedBy:** T5 · **Files:** `src/Server/ZB.MOM.WW.OtOpcUa.Host/Drivers/DriverFactoryBootstrap.cs`, `tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/ResilienceStatusReaderRegistrationTests.cs`
1. Guard test (mirror `ResilienceInvokerFactoryRegistrationTests`): `AddOtOpcUaDriverFactories` yields an `IHostedService` descriptor with implementation type `DriverResilienceStatusPublisherService` — **the assertion that the tracker now has a production reader** → `dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests --filter "FullyQualifiedName~ResilienceStatusReaderRegistrationTests"` → **FAIL**.
2. Register `TryAddSingleton<Func<ActorSystem>>` + `AddHostedService<DriverResilienceStatusPublisherService>()` in `AddOtOpcUaDriverFactories`, with a comment naming it the tracker's reader.
3. → **PASS**. Commit: `feat(resilience): register the tracker's production reader on driver nodes + wiring guard (R2-10)`.

### T7 — AdminUI: resilience status store
**Classification:** Admin read model (unit-testable) · **Estimated implement time:** 5 min · **Parallelizable with:** T1-T6 (needs only T4) · **blockedBy:** T4 · **Files:** `src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/Hubs/IDriverResilienceStatusStore.cs`, `.../InMemoryDriverResilienceStatusStore.cs`, `.../HubServiceCollectionExtensions.cs`, `tests/Server/ZB.MOM.WW.OtOpcUa.AdminUI.Tests/DriverResilienceStatusStoreTests.cs`
1. Tests (mirror `DriverStatusSnapshotStoreTests`): upsert last-write-wins per (instance,host); `GetForInstance` filters; `SnapshotChanged` fires per upsert → `dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.AdminUI.Tests --filter "FullyQualifiedName~DriverResilienceStatusStoreTests"` → **FAIL**.
2. Implement + register in `AddOtOpcUaDriverStatusServices`.
3. → **PASS**. Commit: `feat(adminui): in-process resilience status store (R2-10)`.

### T8 — AdminUI: DPS bridge actor + spawn
**Classification:** Admin wiring (one-line logic; live-verified) · **Estimated implement time:** 5 min · **blockedBy:** T7 · **Files:** `src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/Hubs/DriverResilienceStatusBridge.cs`, `.../HubServiceCollectionExtensions.cs`
1. Implement `ReceiveActor` (PreStart DPS `Subscribe(DriverResilienceStatusChanged.TopicName)`, `Receive<DriverResilienceStatusChanged>` → `store.Upsert`, swallow `SubscribeAck`); spawn in `WithOtOpcUaSignalRBridges` + `DriverResilienceStatusBridgeKey`. Build-verify (`dotnet build` AdminUI project); behavior covered by the live gate. Commit: `feat(adminui): driver-resilience-status DPS bridge (R2-10)`.

### T9 — AdminUI: `DriverStatusPanel` resilience section
**Classification:** Razor (NO bUnit — live-`/run` is the only verification for this task) · **Estimated implement time:** 5 min (code) + live pass in T10 · **blockedBy:** T7 · **Files:** `src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/Components/Shared/Drivers/DriverStatusPanel.razor`
1. Inject `IDriverResilienceStatusStore`; add the per-host chip rows (Breaker OPEN / consecutive failures / in-flight / stale-after-15 s / "No resilience activity recorded yet"); subscribe + filter by `DriverInstanceId`, `InvokeAsync(StateHasChanged)`, unsubscribe in `DisposeAsync` — copy the existing `SnapshotChanged` pattern in the same file verbatim. Remember the string-component-param-needs-`@` gotcha. Build-verify. Commit: `feat(adminui): breaker/retry chips on the driver status panel (R2-10)`.

### T10 — Live-`/run` gate (decisive for T8/T9)
**Classification:** Live verification (mandatory — razor + DPS delivery are invisible to unit tests) · **Estimated implement time:** 30-45 min · **blockedBy:** T6, T8, T9
1. `docker-dev`: rebuild **BOTH** central-1 + central-2 from source, `up -d`.
2. Author/point an S7 driver at a dead endpoint (or `lmxopcua-fix down s7`); set overrides `retryCount: 2`, `breakerFailureThreshold: 2`; deploy.
3. Observe: failures chip climbs → `Breaker OPEN` chip; grep central logs for `circuit-breaker OPENED` to correlate (also the first live observation of the #10 log line — note it against FOLLOWUP-10's deferred behavioral gate).
4. Refresh onto the other central (round-robin) — chips present there too.
5. Restore endpoint → breaker closes → chips clear. Stop one central → rows go `stale` in ~15 s.
6. Commit any live-found fixes; record the evidence in the PR/plan status line.

### T11 — Docs + review ledger
**Classification:** Docs · **Estimated implement time:** 5 min · **blockedBy:** T10 · **Files:** `archreview/03-server-runtime.md`, `archreview/04-adminui.md`, `archreview/plans/STATUS.md`, `archreview/plans/FOLLOWUP-10-resilience-dispatch-gap.md`
1. Mark U10/U-5 remediated with the branch/commit; update STATUS.md; annotate FOLLOWUP-10 that the reader shipped and (if observed in T10) the live behavioral log-line gate closed. Commit: `docs(archreview): R2-10 resilience observability shipped`.

**Full-suite gate before merge:** `dotnet test ZB.MOM.WW.OtOpcUa.slnx` green (integration suites need their fixtures per CLAUDE.md; the new tests themselves are fixture-free).