lmxopcua/archreview/plans/R2-10-resilience-observability-plan.md

Design + Implementation Plan — R2-10 Resilience Observability: an operator-facing reader for DriverResilienceStatusTracker

Status (2026-07-12): PLANNED — not started.

• Source reports: archreview/03-server-runtime.md (U10) · archreview/04-adminui.md (U-5) · archreview/00-OVERALL.md prioritized action list item #10 ("Resilience observability: a /hosts reader for DriverResilienceStatusTracker (Stream E.2/E.3)" — 03 U10 + 04 U-5, Medium, "operators currently blind").
• History: archreview/plans/FOLLOWUP-10-resilience-dispatch-gap.md — #10 (bacea1a4) wired CapabilityInvoker into all 6 production dispatch sites; #13 (75403caa) made AdminUI-authored per-instance ResilienceConfig overrides live. The Phase 6.1 "Stream E.2" (snapshot-persisting HostedService) and "E.3" (SignalR push + Blazor /hosts panel) readers were never built; the interim mitigation was retry/breaker structured logging in DriverResiliencePipelineBuilder (added with #10).
• Plan verified against tree at: f6eaa267 (master, clean) — every anchor below was opened and re-read at this commit.
• Scope: Core/Resilience (tracker + pipeline builder + invoker — additive only), Commons/Messages/Drivers (one new message), Host/Drivers (one new publisher hosted service + DI), AdminUI/Hubs (one new store + one new bridge actor), AdminUI/Components/Shared/Drivers/DriverStatusPanel.razor (one new section).

Verification summary

Every claim in U10/U-5 was re-verified at f6eaa267. All confirmed — none stale.

• The tracker has no reader. grep -rn "DriverResilienceStatusTracker" src --include="*.cs" matches only Core/Resilience/{DriverResilienceStatusTracker, DriverResiliencePipelineBuilder, CapabilityInvoker, DriverCapabilityInvokerFactory}.cs and Host/Drivers/DriverFactoryBootstrap.cs (the DI registration, :57). Zero AdminUI/Runtime references. grep -rn Resilience src/Server/ZB.MOM.WW.OtOpcUa.AdminUI matches only the authoring surface (DriverResilienceSection.razor, ResilienceFormModel.cs, the 8 driver pages). The only production consumers of tracker state are its own writers.
• What the tracker holds (DriverResilienceStatusTracker.cs): a ConcurrentDictionary<(DriverInstanceId, HostName), ResilienceStatusSnapshot> — ConsecutiveFailures, LastBreakerOpenUtc, LastRecycleUtc, footprint bytes, CurrentInFlight, LastSampledUtc. Thread-safe (immutable record values, AddOrUpdate); Snapshot() returns a copy safe under concurrent writes — ideal for a periodic reader. Note the key: the tracker aggregates per (instance, host), NOT per capability — pipelines are per (instance, host, capability) (DriverResiliencePipelineBuilder.PipelineKey) but all telemetry callbacks fold into the (instance, host) counter. The read model below therefore keys per instance × host (mirroring the tracker) rather than instance × capability; adding capability granularity would mean re-keying the tracker and every invoker call site — rejected as not the simplest thing (capability is already visible in the #10 log lines when drill-down is needed).
• Feeding is real and live: CapabilityInvoker calls RecordCallStart/Complete around every wrapped dispatch (CapabilityInvoker.cs:70/80/94/104…); DriverResiliencePipelineBuilder.Build wires OnRetry → RecordFailure (:131), OnOpened → RecordBreakerOpen (:155), OnClosed → RecordSuccess (:165). Registered singleton + threaded into the builder and invoker factory in DriverFactoryBootstrap.AddOtOpcUaDriverFactories (:57-71).
• Two truthfulness gaps found in the tracker itself (must fix for a display to be honest):
  1. No breaker-close marker. RecordBreakerOpen stamps LastBreakerOpenUtc but nothing ever records "closed" — OnClosed only calls RecordSuccess (resets ConsecutiveFailures). "Is the breaker open now?" is underivable from the snapshot.
  2. RecordSuccess has exactly one caller — the breaker's OnClosed. A successful capability call never resets ConsecutiveFailures, so after a transient blip with 2 retries the counter reads 2 forever (until a full breaker open/close cycle). The tracker's own xmldoc ("Reset … on a successful pipeline execution") shows the intended-but-unwired call from the invoker's success path — another edge of the house "built-but-never-wired" pattern (00-OVERALL theme #1).
• Topology determination (where the tracker lives vs where AdminUI renders): Host/Program.cs gates by role: the hasDriver block (:104-262) calls AddOtOpcUaDriverFactories() (:195) — the tracker singleton exists only in driver-role processes. The hasAdmin block mounts the AdminUI (:278-293) and spawns the DPS→SignalR bridge actors via WithOtOpcUaSignalRBridges() inside the Akka configurator (:269-273). In the split-role MAIN cluster (admin-a/admin-b vs driver nodes) tracker and AdminUI are different processes — state must travel over the cluster. Even on the fused docker-dev rig (central-1/central-2 both admin,driver), :9200 round-robins the two containers while a given DriverHostActor child runs on one of them — so a direct in-process tracker read from the Blazor circuit shows the wrong (often empty) node's state. A cluster transport is mandatory in every deployed topology.
• The precedent to mirror (found, verified end-to-end): driver-health already makes exactly this trip. DriverInstanceActor:994 → AkkaDriverHealthPublisher.Publish → DPS topic driver-health (DriverHealthChanged record in Commons/Messages/Drivers, TopicName const on the contract) → per-admin-node DriverStatusSignalRBridge (spawned by WithOtOpcUaSignalRBridges, HubServiceCollectionExtensions.cs:69-72) → IDriverStatusSnapshotStore singleton upsert + SnapshotChanged event → DriverStatusPanel.razor reads the store in-process (:190-198), explicitly not via a self-targeted SignalR HubConnection (the house ban — a server-side circuit behind Traefik cannot dial its own public URL; documented at IDriverStatusSnapshotStore.cs:28-36 and DriverStatusPanel.razor:182-189). This plan mirrors that flow one-for-one.
• UI placement facts: DriverStatusPanel is embedded on all 8 driver pages (Components/Pages/Clusters/Drivers/*DriverPage.razor) — the same pages where DriverResilienceSection authors the ResilienceConfig overrides that #13 made live. /hosts (Hosts.razor) shows cluster members + a cluster-scoped driver-health table built by HostsDriverView.Build(DriverStore.GetAll(), …) (:291).
• Test landscape: tracker/builder/invoker suites exist at tests/Core/ZB.MOM.WW.OtOpcUa.Core.Tests/Resilience/ (incl. DriverResilienceStatusTrackerTests.cs, DriverResiliencePipelineBuilderTests.cs, InFlightCounterTests.cs); the DI wiring-guard template is tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/ResilienceInvokerFactoryRegistrationTests.cs (pure-DI, no Docker fixture); the store-test template is tests/Server/ZB.MOM.WW.OtOpcUa.AdminUI.Tests/DriverStatusSnapshotStoreTests.cs. AdminUI has no bUnit — razor rendering changes are verified only by the live-/run pass (docker-dev :9200 round-robins central-1/central-2; rebuild BOTH).

---

Restatement

DriverResilienceStatusTracker is registered, fed by every wrapped capability call, retry, and breaker transition — and consumed by nothing. Since #10/#13, retry/breaker/bulkhead behavior is production-live and operator-authorable (per-instance ResilienceConfig on all 8 driver pages), yet the only way to see a breaker reject calls (BrokenCircuitException) or a retry storm is to grep server logs. Operators author live resilience config completely blind. Ship the smallest reader that gives them sight of breaker-open and retry storms, on the page where they author the config.

Verification

Confirmed at f6eaa267 — see Verification summary. Anchors: src/Core/ZB.MOM.WW.OtOpcUa.Core/Resilience/DriverResilienceStatusTracker.cs (no non-writer reference anywhere), DriverResiliencePipelineBuilder.cs:124-170 (the interim log-only surface), src/Server/ZB.MOM.WW.OtOpcUa.Host/Drivers/DriverFactoryBootstrap.cs:57 (singleton registration, driver nodes only).

Root cause

Phase 6.1 shipped Stream E.1 (the tracker + telemetry hooks) and deferred E.2 (snapshot persistence) / E.3 (push + panel) to "a follow-up visual-review PR" (tracker xmldoc, :12-16) that never happened. #10 wired the writer side into production and added log lines as an interim reader; the review (03/U10, 04/U-5) now tracks the missing operator surface. Two smaller instances of the same pattern hide inside the tracker itself (no breaker-close marker; RecordSuccess unwired from the invoker success path).

Proposed design

Mirror the driver-health flow end-to-end. Four pieces, smallest-correct at each hop:

(a) Read model — ResilienceStatusSnapshot gains LastBreakerClosedUtc + a derived IsBreakerOpen (LastBreakerOpenUtc is set and later than any close). One new wire message in Commons:

// Commons/Messages/Drivers/DriverResilienceStatusChanged.cs
public sealed record DriverResilienceStatusChanged(
    string DriverInstanceId,
    string HostName,
    bool BreakerOpen,
    int ConsecutiveFailures,
    int CurrentInFlight,
    DateTime? LastBreakerOpenUtc,
    DateTime LastSampledUtc,
    DateTime PublishedUtc)
{
    public const string TopicName = "driver-resilience-status";
}

Keyed per (instance, host) — matches the tracker key (see Verification summary for why not per-capability). No ClusterId: DriverInstanceId is globally unique and the panel filters on it alone (same as DriverStatusPanel today). Tracker truthfulness fixes ride along: RecordBreakerClose (called from the builder's OnClosed beside the existing RecordSuccess) and RecordSuccess on the invoker's success paths so ConsecutiveFailures is a true consecutive counter (nonzero only during an active storm).

Known display caveat (documented, accepted): Polly half-opens the breaker after BreakDuration without firing OnClosed; OnClosed fires only when a probe call succeeds. An idle driver therefore shows "breaker OPEN" until the next call — truthful enough (the next call is at best a probe) and self-corrects on traffic.

(b) Transport (driver node → admin surface) — a periodic full-snapshot publisher on driver nodes: DriverResilienceStatusPublisherService : BackgroundService in Host/Drivers/, ctor (DriverResilienceStatusTracker, Func<ActorSystem>, ILogger<…>, TimeSpan? interval = null /* default 5 s */). Every tick: tracker.Snapshot() → map each triple to a DriverResilienceStatusChanged (PublishedUtc = now) → DistributedPubSub.Get(system).Mediator.Tell(new Publish(TopicName, msg)) per entry. Loop body in try/catch (a transient DPS/actor-system hiccup must never kill the service); skip empty snapshots. Publishing the full set every tick (not deltas) deliberately self-heals late-subscribing admin nodes — the known redundancy-state "late-subscriber" gotcha — and gives staleness detection for free. Volume is negligible (entries = active instances × hosts, typically < a few dozen, every 5 s). Registered in DriverFactoryBootstrap.AddOtOpcUaDriverFactories next to the tracker it reads (driver nodes only, exactly where the tracker exists), with a TryAddSingleton<Func<ActorSystem>> for self-containedness (idempotent with Program.cs:260).

Why Host, not Runtime: Runtime is deliberately Polly-free — it references Core.Abstractions, not Core where the tracker lives (the exact boundary FOLLOWUP-10 discovered the hard way). Host already references Core, Commons, and Akka (OtOpcUaServerHostedService resolves the DPS mediator via the same lazy Func<ActorSystem> pattern). No new seam needed.

Rejected transport alternatives:

• Direct in-process tracker read from the AdminUI/Blazor circuit — wrong in every deployed topology (split-role MAIN cluster: different process; fused docker-dev: :9200 round-robin lands circuits on a node that isn't running the driver instance). Rejected.
• Persist snapshots to a DriverInstanceResilienceStatus table (the original Stream E.2 shape, still named in the tracker xmldoc) — needs a migration, a poller, and row cleanup; the house idiom has since evolved to DPS → in-memory store, and 04/U-5's own recommendation names that idiom. Rejected; fix the stale xmldoc instead.
• Event-driven publish from inside the tracker (hook every RecordX) — would thread an Akka publisher seam into Polly-free/Akka-free Core, and is chattier under a retry storm exactly when the cluster is stressed. Periodic snapshot is simpler and bounded. Rejected.
• Extend DriverHealthChanged — mutates an existing wire contract and conflates two feeds with different cadences (event-driven vs periodic). Separate topic = zero blast radius. Rejected.
• Pull via AdminOperationsActor ask — request/response through the admin-pinned singleton plus panel-side polling; more moving parts than the established push idiom, and the singleton is one node while the store idiom feeds every admin replica (which the round-robin requires). Rejected.

(c) Admin surface — mirror the driver-status pair:

• IDriverResilienceStatusStore + InMemoryDriverResilienceStatusStore in AdminUI/Hubs/: keyed (DriverInstanceId, HostName), Upsert, GetForInstance(driverInstanceId) (per-host list), GetAll(), SnapshotChanged event (capture-then-invoke, same as InMemoryDriverStatusSnapshotStore). Registered in HubServiceCollectionExtensions.AddOtOpcUaDriverStatusServices.
• DriverResilienceStatusBridge (ReceiveActor, AdminUI/Hubs/): PreStart subscribes to the DPS topic; on message, store.Upsert(msg). Spawned per admin node in WithOtOpcUaSignalRBridges beside the other four bridges. No new SignalR hub — the only consumer is the server-side panel, which reads the store in-process per the house ban on self-HubConnections; DriverStatusHub exists for browser-JS clients and resilience has none. Add a hub later only if an external consumer appears (rejected for v1 as speculative surface).

(d) UI placement — extend DriverStatusPanel.razor (recommended; embedded on all 8 driver pages — the exact pages where DriverResilienceSection authors the config, closing the authoring feedback loop). New "Resilience" block under the existing status chips: one row per host with

• Breaker OPEN chip (chip-bad) with "opened Xs ago" when BreakerOpen, else nothing (healthy is quiet);
• N consecutive failures chip (chip-warn) when ConsecutiveFailures > 0 (the retry-storm signal);
• in-flight N (chip-idle) when CurrentInFlight > 0;
• staleness: row dims + shows a stale chip when PublishedUtc is older than 15 s (3× publish interval) — covers driver-node death, where entries freeze; reuses the panel's existing 5 s re-render timer;
• muted "No resilience activity recorded yet" when the store has no entries for this instance (the tracker only populates once wrapped capability calls run).

Subscription lifecycle copies the existing pattern exactly: inject the store, subscribe SnapshotChanged in OnInitializedAsync, filter by DriverInstanceId, marshal via InvokeAsync(StateHasChanged), unsubscribe first in DisposeAsync.

Rejected UI alternatives: a /hosts column (fleet-wide join of a second store into HostsDriverView; more code, and /hosts rows are cluster-scoped health — the operator authoring overrides is on the driver page; fine follow-on once the store exists) · a dedicated /resilience page (new nav surface for a v1 that needs two chips). The store makes both trivially addable later.

Implementation steps (exact paths)

1. Core read-model truthfulness — src/Core/ZB.MOM.WW.OtOpcUa.Core/Resilience/DriverResilienceStatusTracker.cs: add LastBreakerClosedUtc to ResilienceStatusSnapshot, derived IsBreakerOpen, and RecordBreakerClose(driverInstanceId, hostName, utcNow). Update the stale class xmldoc (:5-16 still promises the never-built snapshot-persisting HostedService/table).
2. Builder close-hook — src/Core/ZB.MOM.WW.OtOpcUa.Core/Resilience/DriverResiliencePipelineBuilder.cs (OnClosed, :161-170): add tracker?.RecordBreakerClose(...) beside the existing RecordSuccess. Refresh the ctor logger xmldoc (:38-42, "until the Admin /hosts reader ships" becomes stale once this lands).
3. Invoker success-reset — src/Core/ZB.MOM.WW.OtOpcUa.Core/Resilience/CapabilityInvoker.cs: on each Execute* success path (after pipeline.ExecuteAsync returns, before the finally), call _statusTracker?.RecordSuccess(_driverInstanceId, hostName, utcNow). (Clock: reuse the invoker's existing time source if present, else DateTime.UtcNow matching current tracker call sites.)
4. Wire message — new src/Core/ZB.MOM.WW.OtOpcUa.Commons/Messages/Drivers/DriverResilienceStatusChanged.cs (record above). Plain Commons record = same DPS serialization path DriverHealthChanged already rides.
5. Publisher — new src/Server/ZB.MOM.WW.OtOpcUa.Host/Drivers/DriverResilienceStatusPublisherService.cs with public static IReadOnlyList<DriverResilienceStatusChanged> BuildMessages(DriverResilienceStatusTracker tracker, DateTime publishedUtc) (pure, unit-testable) + the BackgroundService timer/publish/try-catch shell.
6. Publisher DI — src/Server/ZB.MOM.WW.OtOpcUa.Host/Drivers/DriverFactoryBootstrap.cs (AddOtOpcUaDriverFactories): services.TryAddSingleton<Func<ActorSystem>>(sp => () => sp.GetRequiredService<ActorSystem>()); + services.AddHostedService<DriverResilienceStatusPublisherService>(); with a comment naming this the tracker's production reader.
7. Admin store — new src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/Hubs/IDriverResilienceStatusStore.cs + InMemoryDriverResilienceStatusStore.cs; register in HubServiceCollectionExtensions.AddOtOpcUaDriverStatusServices.
8. Bridge actor — new src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/Hubs/DriverResilienceStatusBridge.cs; spawn + ActorRegistry key in HubServiceCollectionExtensions.WithOtOpcUaSignalRBridges (name const driver-resilience-status-bridge).
9. Panel — src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/Components/Shared/Drivers/DriverStatusPanel.razor: inject the store, resilience section per (d), subscribe/unsubscribe per the existing pattern.
10. Docs/ledger — update archreview/03-server-runtime.md (U10), archreview/04-adminui.md (U-5), archreview/plans/STATUS.md, and FOLLOWUP-10's "no reader" notes once merged; docs/ has no resilience guide to touch (the tracker/builder xmldocs updated in steps 1-2 are the canonical doc surface).

Tests

• Unit — read model (Core.Tests, existing suites):
  • DriverResilienceStatusTrackerTests: RecordBreakerClose stamps the close time; IsBreakerOpen is false initially, true after open, false after later close, true again after re-open.
  • DriverResiliencePipelineBuilderTests: driving a pipeline to breaker-open then recovery records open and close on the tracker (extend the existing breaker test).
  • CapabilityInvokerTests (or InFlightCounterTests home): a successful ExecuteAsync resets ConsecutiveFailures previously raised by a failure.
• Unit — publisher mapping (Host.IntegrationTests, pure-DI/no fixture): BuildMessages maps a fed tracker to one message per (instance, host) with correct BreakerOpen/counter/PublishedUtc; empty tracker → empty list.
• Wiring guard — the tracker now HAS a production reader (the built-but-never-wired class of bug, mirroring ResilienceInvokerFactoryRegistrationTests): assert AddOtOpcUaDriverFactories() registers an IHostedService descriptor whose implementation type is DriverResilienceStatusPublisherService — a regression that unregisters the reader fails this test, not production.
• Unit — store (AdminUI.Tests, mirroring DriverStatusSnapshotStoreTests): upsert keyed by (instance, host) with last-write-wins; GetForInstance returns all hosts for one instance only; SnapshotChanged raised per upsert.
• Bridge actor: logic is one line (subscribe → upsert), same as the untested sibling bridges; covered by the live gate.
• Razor (no bUnit — live-/run is the verification): docker-dev rig, rebuild BOTH central-1 and central-2 (:9200 round-robins them):
  1. docker compose build central-1 central-2 && docker compose up -d in docker-dev/.
  2. On http://localhost:9200, open (or author) an S7 driver and point its endpoint at a dead port (e.g. 10.100.0.35:9999) — per FOLLOWUP-10, S7's connect path throws on a dead socket, unlike Modbus's cheap paths; alternatively lmxopcua-fix down s7 a live fixture. In DriverResilienceSection, author overrides retryCount: 2, breakerFailureThreshold: 2 (small numbers so the breaker opens fast). Deploy.
  3. On the driver page, watch the panel: consecutive-failures chip climbs, then Breaker OPEN appears; correlate with docker logs central grep circuit-breaker OPENED (the #10 log line — first-ever live observation of it, closing FOLLOWUP-10's deferred behavioral gate as a bonus).
  4. Refresh until the circuit lands on the other central (round-robin) — chips must show there too (proves the per-node DPS bridge/store, not a lucky co-located read).
  5. Restore the endpoint (lmxopcua-fix up s7 s7_1500); on the next successful call the breaker closes → chips clear.
  6. Stop the driver node (fused rig: stop central-2 while circuit is on central-1): rows dim to stale within ~15 s.

Effort + Risk

Effort: Medium-small — ~10 bite-sized tasks; one focused day including the live gate. Risk: Low. Everything is additive: three small Core additions (new snapshot field, one hook call, one success-reset call — the only behavior deltas, all confined to telemetry counters), a new message type, a new hosted service, a new store + bridge, and one razor section. No dispatch-path, artifact, or config-shape changes. Blast radius of a publisher bug = missing/wrong chips, never driver behavior. The main live-verify hazard is the known one: razor binding errors that unit tests can't see — hence the mandatory live-/run pass on both centrals.

---

Task breakdown

Branch: fix/archreview-r2-10-resilience-observability off master. TDD throughout: write the failing test, see FAIL, implement, see PASS. Commit per task (or per labeled group).

T1 — Tracker: breaker-close marker + IsBreakerOpen

Classification: Core read model (unit-testable) · Estimated implement time: 5 min · Parallelizable with: T4, T7 · Files: src/Core/ZB.MOM.WW.OtOpcUa.Core/Resilience/DriverResilienceStatusTracker.cs, tests/Core/ZB.MOM.WW.OtOpcUa.Core.Tests/Resilience/DriverResilienceStatusTrackerTests.cs

1. Add tests: RecordBreakerClose_StampsCloseTime, IsBreakerOpen_TracksOpenCloseReopen → dotnet test tests/Core/ZB.MOM.WW.OtOpcUa.Core.Tests --filter "FullyQualifiedName~DriverResilienceStatusTrackerTests" → FAIL (no such members).
2. Add LastBreakerClosedUtc init-prop + computed IsBreakerOpen to ResilienceStatusSnapshot; add RecordBreakerClose. Fix the stale class xmldoc (:5-16).
3. Re-run filter → PASS. Commit: feat(resilience): breaker-close marker + IsBreakerOpen on ResilienceStatusSnapshot (R2-10).

T2 — Builder: record breaker-close in OnClosed

Classification: Core wiring (unit-testable) · Estimated implement time: 5 min · Parallelizable with: — (needs T1) · blockedBy: T1 · Files: src/Core/ZB.MOM.WW.OtOpcUa.Core/Resilience/DriverResiliencePipelineBuilder.cs, tests/Core/ZB.MOM.WW.OtOpcUa.Core.Tests/Resilience/DriverResiliencePipelineBuilderTests.cs

1. Extend the existing breaker-telemetry test: after recovery, tracker snapshot has IsBreakerOpen == false with a close stamp → dotnet test tests/Core/ZB.MOM.WW.OtOpcUa.Core.Tests --filter "FullyQualifiedName~DriverResiliencePipelineBuilderTests" → FAIL.
2. Add tracker?.RecordBreakerClose(...) in OnClosed (:161-170); refresh the ctor logger xmldoc (:38-42).
3. → PASS. Commit: feat(resilience): OnClosed records breaker-close on the status tracker (R2-10).

T3 — Invoker: reset consecutive failures on success

Classification: Core wiring (unit-testable) · Estimated implement time: 5 min · Parallelizable with: T2 (after T1) · blockedBy: T1 · Files: src/Core/ZB.MOM.WW.OtOpcUa.Core/Resilience/CapabilityInvoker.cs, tests/Core/ZB.MOM.WW.OtOpcUa.Core.Tests/Resilience/CapabilityInvokerTests.cs

1. Test: ExecuteAsync_OnSuccess_ResetsConsecutiveFailures (record a failure, run a succeeding call, assert 0) → dotnet test tests/Core/ZB.MOM.WW.OtOpcUa.Core.Tests --filter "FullyQualifiedName~CapabilityInvokerTests" → FAIL.
2. Call _statusTracker?.RecordSuccess(...) on each Execute* success path.
3. → PASS. Commit: fix(resilience): successful capability calls reset ConsecutiveFailures (R2-10).

T4 — Commons: DriverResilienceStatusChanged wire message

Classification: Contract (compile-verified; consumed by T5/T8 tests) · Estimated implement time: 3 min · Parallelizable with: T1, T7 · Files: src/Core/ZB.MOM.WW.OtOpcUa.Commons/Messages/Drivers/DriverResilienceStatusChanged.cs

1. Add the record + TopicName = "driver-resilience-status" const, xmldoc naming publisher + bridge (mirror DriverHealthChanged's single-source-of-truth note). Commit: feat(resilience): DriverResilienceStatusChanged DPS contract (R2-10).

T5 — Host: publisher service (BuildMessages + timer shell)

Classification: Transport (mapping unit-testable; shell live-verified) · Estimated implement time: 5 min · Parallelizable with: T7 · blockedBy: T1, T4 · Files: src/Server/ZB.MOM.WW.OtOpcUa.Host/Drivers/DriverResilienceStatusPublisherService.cs, tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/DriverResilienceStatusPublisherServiceTests.cs

1. Tests (pure, no fixture): BuildMessages_MapsTrackerSnapshot (fed tracker → one msg per (instance,host), correct BreakerOpen/counters/PublishedUtc), BuildMessages_EmptyTracker_ReturnsEmpty → dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests --filter "FullyQualifiedName~DriverResilienceStatusPublisherServiceTests" → FAIL.
2. Implement public static BuildMessages(tracker, publishedUtc) + BackgroundService (PeriodicTimer 5 s default via ctor param, per-tick try/catch, skip-empty, DPS Publish per message via Func<ActorSystem>).
3. → PASS. Commit: feat(resilience): periodic DPS publisher for tracker snapshots (R2-10).

T6 — Host: DI registration + reader wiring guard

Classification: Wiring + guard (the built-but-never-wired class) · Estimated implement time: 5 min · blockedBy: T5 · Files: src/Server/ZB.MOM.WW.OtOpcUa.Host/Drivers/DriverFactoryBootstrap.cs, tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/ResilienceStatusReaderRegistrationTests.cs

1. Guard test (mirror ResilienceInvokerFactoryRegistrationTests): AddOtOpcUaDriverFactories yields an IHostedService descriptor with implementation type DriverResilienceStatusPublisherService — the assertion that the tracker now has a production reader → dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests --filter "FullyQualifiedName~ResilienceStatusReaderRegistrationTests" → FAIL.
2. Register TryAddSingleton<Func<ActorSystem>> + AddHostedService<DriverResilienceStatusPublisherService>() in AddOtOpcUaDriverFactories, with a comment naming it the tracker's reader.
3. → PASS. Commit: feat(resilience): register the tracker's production reader on driver nodes + wiring guard (R2-10).

T7 — AdminUI: resilience status store

Classification: Admin read model (unit-testable) · Estimated implement time: 5 min · Parallelizable with: T1-T6 (needs only T4) · blockedBy: T4 · Files: src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/Hubs/IDriverResilienceStatusStore.cs, .../InMemoryDriverResilienceStatusStore.cs, .../HubServiceCollectionExtensions.cs, tests/Server/ZB.MOM.WW.OtOpcUa.AdminUI.Tests/DriverResilienceStatusStoreTests.cs

1. Tests (mirror DriverStatusSnapshotStoreTests): upsert last-write-wins per (instance,host); GetForInstance filters; SnapshotChanged fires per upsert → dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.AdminUI.Tests --filter "FullyQualifiedName~DriverResilienceStatusStoreTests" → FAIL.
2. Implement + register in AddOtOpcUaDriverStatusServices.
3. → PASS. Commit: feat(adminui): in-process resilience status store (R2-10).

T8 — AdminUI: DPS bridge actor + spawn

Classification: Admin wiring (one-line logic; live-verified) · Estimated implement time: 5 min · blockedBy: T7 · Files: src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/Hubs/DriverResilienceStatusBridge.cs, .../HubServiceCollectionExtensions.cs

1. Implement ReceiveActor (PreStart DPS Subscribe(DriverResilienceStatusChanged.TopicName), Receive<DriverResilienceStatusChanged> → store.Upsert, swallow SubscribeAck); spawn in WithOtOpcUaSignalRBridges + DriverResilienceStatusBridgeKey. Build-verify (dotnet build AdminUI project); behavior covered by the live gate. Commit: feat(adminui): driver-resilience-status DPS bridge (R2-10).

T9 — AdminUI: DriverStatusPanel resilience section

Classification: Razor (NO bUnit — live-/run is the only verification for this task) · Estimated implement time: 5 min (code) + live pass in T10 · blockedBy: T7 · Files: src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/Components/Shared/Drivers/DriverStatusPanel.razor

1. Inject IDriverResilienceStatusStore; add the per-host chip rows (Breaker OPEN / consecutive failures / in-flight / stale-after-15 s / "No resilience activity recorded yet"); subscribe + filter by DriverInstanceId, InvokeAsync(StateHasChanged), unsubscribe in DisposeAsync — copy the existing SnapshotChanged pattern in the same file verbatim. Remember the string-component-param-needs-@ gotcha. Build-verify. Commit: feat(adminui): breaker/retry chips on the driver status panel (R2-10).

T10 — Live-/run gate (decisive for T8/T9)

Classification: Live verification (mandatory — razor + DPS delivery are invisible to unit tests) · Estimated implement time: 30-45 min · blockedBy: T6, T8, T9

1. docker-dev: rebuild BOTH central-1 + central-2 from source, up -d.
2. Author/point an S7 driver at a dead endpoint (or lmxopcua-fix down s7); set overrides retryCount: 2, breakerFailureThreshold: 2; deploy.
3. Observe: failures chip climbs → Breaker OPEN chip; grep central logs for circuit-breaker OPENED to correlate (also the first live observation of the #10 log line — note it against FOLLOWUP-10's deferred behavioral gate).
4. Refresh onto the other central (round-robin) — chips present there too.
5. Restore endpoint → breaker closes → chips clear. Stop one central → rows go stale in ~15 s.
6. Commit any live-found fixes; record the evidence in the PR/plan status line.

T11 — Docs + review ledger

Classification: Docs · Estimated implement time: 5 min · blockedBy: T10 · Files: archreview/03-server-runtime.md, archreview/04-adminui.md, archreview/plans/STATUS.md, archreview/plans/FOLLOWUP-10-resilience-dispatch-gap.md

1. Mark U10/U-5 remediated with the branch/commit; update STATUS.md; annotate FOLLOWUP-10 that the reader shipped and (if observed in T10) the live behavioral log-line gate closed. Commit: docs(archreview): R2-10 resilience observability shipped.

Full-suite gate before merge: dotnet test ZB.MOM.WW.OtOpcUa.slnx green (integration suites need their fixtures per CLAUDE.md; the new tests themselves are fixture-free).

---

Execution deviations (R2-10)

Executed on branch r2/10-resilience-observability (off master 46fedda3). T1–T9 + T11 completed; T10 deferred-live.

• T2 breaker-close test uses a controllable TimeProvider (ManualTimeProvider), not wall-clock waits. Driving the real Polly breaker to OnClosed requires elapsing the 15 s BreakDuration then a successful half-open probe. Rather than a 15 s Task.Delay, the test injects a TimeProvider that advances both GetUtcNow and GetTimestamp off one tick counter (Advance(16 s)), making the close deterministic and instant. Same self-contained-fake style the repo already uses (TriePermissionEvaluatorTests); no new package. This is the one deviation from the plan's "extend the existing breaker test" wording — the existing test only reaches breaker-open, and open→close needs time control.
• T9 panel renders healthy hosts quietly rather than one always-visible row per host. Per the plan's "healthy is quiet" intent, the resilience section shows: No resilience activity recorded yet. when the store has no entries for the instance; All targets healthy. when entries exist but none has an active signal; and a chip row only for hosts with breaker-OPEN / consecutive-failures>0 / in-flight>0 / stale. Keeps the panel silent under normal operation (the tracker populates an entry for every host that has ever had a successful call, so an always-on per-host row would be noisy).
• T10 (live-/run gate) deferred-live. Code-complete + offline-verified (unit + full-solution build). The docker-dev breaker-open observation on both centrals + recovery-clear + staleness-dim is a heavy serial pass left for the consolidated live session (also the first live sighting of the #10 circuit-breaker OPENED log line, closing FOLLOWUP-10's behavioral gate). Marked deferred-live in .tasks.json.
• T6/T8 registration: the publisher's Func<ActorSystem> is registered via TryAddSingleton (idempotent with the one Program.cs registers for the alarm-command router, per plan). The resilience bridge intentionally has no SignalR hub (the panel reads the store in-process; resilience has no browser-JS consumer) — a deliberate simplification the plan calls for.