R2-10 Resilience observability (arch-review round 2) #432

2026-07-13T10:51:45-04:00

dohertj2 commented

2026-07-13 10:51:45 -04:00

Round-2 arch-review plan R2-10 (finding 03/U10 = 04/U-5 — the resilience-status tracker had no reader; operators were blind to retry/breaker state).

Mirrors the driver-health flow end-to-end so retry/breaker/staleness is visible on AdminUI /hosts:

Core truthfulness fixes: RecordBreakerClose + derived IsBreakerOpen; OnClosed records the close; RecordSuccess wired onto every CapabilityInvoker success path (ConsecutiveFailures is now a true consecutive counter, not stuck-forever after a blip).
Contract: DriverResilienceStatusChanged (topic driver-resilience-status).
Transport: DriverResilienceStatusPublisherService (5s full-snapshot DPS publisher) + wiring guard (built-but-never-wired guard).
Admin: IDriverResilienceStatusStore + in-memory impl + DriverResilienceStatusBridge (in-process DPS subscriber, no self-dialed hub) + resilience chip section on DriverStatusPanel.razor.

10/11 tasks; T10 (docker-dev live /run breaker-open/recovery + FOLLOWUP-10 log-line gate) deferred to serial heavy pass. Build clean; Core.Tests resilience 128/128, Host.IntegrationTests publisher/guards 6/6, AdminUI.Tests store 8/8. Plan: archreview/plans/R2-10-resilience-observability-plan.md.

Round-2 arch-review plan **R2-10** (finding 03/U10 = 04/U-5 — the resilience-status tracker had no reader; operators were blind to retry/breaker state). Mirrors the driver-health flow end-to-end so retry/breaker/staleness is visible on AdminUI `/hosts`: - **Core truthfulness fixes:** `RecordBreakerClose` + derived `IsBreakerOpen`; `OnClosed` records the close; `RecordSuccess` wired onto every CapabilityInvoker success path (ConsecutiveFailures is now a true consecutive counter, not stuck-forever after a blip). - **Contract:** `DriverResilienceStatusChanged` (topic `driver-resilience-status`). - **Transport:** `DriverResilienceStatusPublisherService` (5s full-snapshot DPS publisher) + wiring guard (built-but-never-wired guard). - **Admin:** `IDriverResilienceStatusStore` + in-memory impl + `DriverResilienceStatusBridge` (in-process DPS subscriber, no self-dialed hub) + resilience chip section on `DriverStatusPanel.razor`. 10/11 tasks; **T10 (docker-dev live /run breaker-open/recovery + FOLLOWUP-10 log-line gate) deferred** to serial heavy pass. Build clean; Core.Tests resilience 128/128, Host.IntegrationTests publisher/guards 6/6, AdminUI.Tests store 8/8. Plan: `archreview/plans/R2-10-resilience-observability-plan.md`.

dohertj2 added 10 commits 2026-07-13 10:51:47 -04:00

fix(r2-10): breaker-close marker + IsBreakerOpen on ResilienceStatusSnapshot 205d398055

fix(r2-10): OnClosed records breaker-close on the status tracker 2cce61fc04

fix(r2-10): successful capability calls reset ConsecutiveFailures 0ba60db1e4

fix(r2-10): DriverResilienceStatusChanged DPS contract 2cb3f962b9

fix(r2-10): periodic DPS publisher for tracker snapshots 37f180c5a3

fix(r2-10): register the tracker's production reader on driver nodes + wiring guard 12b2ce078b

fix(r2-10): in-process resilience status store 99ce5b401e

fix(r2-10): driver-resilience-status DPS bridge 38c059dae7

fix(r2-10): breaker/retry chips on the driver status panel cc5ecb5868

docs(archreview): R2-10 resilience observability shipped

v2-ci / build (pull_request) Successful in 3m47s