{
  "planPath": "archreview/plans/R2-10-resilience-observability-plan.md",
  "lastUpdated": "2026-08-07",
  "tasks": [
    {
      "id": "T1",
      "subject": "Tracker: LastBreakerClosedUtc + IsBreakerOpen + RecordBreakerClose (Core, TDD in DriverResilienceStatusTrackerTests)",
      "status": "completed",
      "blockedBy": []
    },
    {
      "id": "T2",
      "subject": "Builder: OnClosed records breaker-close on the tracker (TDD in DriverResiliencePipelineBuilderTests)",
      "status": "completed",
      "blockedBy": [
        "T1"
      ]
    },
    {
      "id": "T3",
      "subject": "Invoker: successful Execute* resets ConsecutiveFailures (TDD in CapabilityInvokerTests)",
      "status": "completed",
      "blockedBy": [
        "T1"
      ]
    },
    {
      "id": "T4",
      "subject": "Commons: DriverResilienceStatusChanged record + driver-resilience-status TopicName",
      "status": "completed",
      "blockedBy": []
    },
    {
      "id": "T5",
      "subject": "Host: DriverResilienceStatusPublisherService (BuildMessages mapping + 5s DPS timer shell; TDD mapping in Host.IntegrationTests)",
      "status": "completed",
      "blockedBy": [
        "T1",
        "T4"
      ]
    },
    {
      "id": "T6",
      "subject": "Host DI: register the publisher in AddOtOpcUaDriverFactories + ResilienceStatusReaderRegistrationTests wiring guard (tracker HAS a production reader)",
      "status": "completed",
      "blockedBy": [
        "T5"
      ]
    },
    {
      "id": "T7",
      "subject": "AdminUI: IDriverResilienceStatusStore + InMemory impl + AddOtOpcUaDriverStatusServices registration (TDD mirroring DriverStatusSnapshotStoreTests)",
      "status": "completed",
      "blockedBy": [
        "T4"
      ]
    },
    {
      "id": "T8",
      "subject": "AdminUI: DriverResilienceStatusBridge DPS actor + spawn in WithOtOpcUaSignalRBridges",
      "status": "completed",
      "blockedBy": [
        "T7"
      ]
    },
    {
      "id": "T9",
      "subject": "AdminUI: resilience chip section in DriverStatusPanel.razor (in-process store read; no bUnit \u2014 verified by T10)",
      "status": "completed",
      "blockedBy": [
        "T7"
      ]
    },
    {
      "id": "T10",
      "subject": "Live-/run gate on docker-dev: rebuild BOTH centrals, S7 dead-endpoint breaker-open, both-replica chips, recovery clear, staleness dim",
      "status": "completed",
      "blockedBy": [
        "T6",
        "T8",
        "T9"
      ],
      "openNote": "PARTIAL \u2014 the resilience PIPELINE is live-verified (Polly CircuitBreaker -> Retry -> Timeout -> CapabilityInvoker in central-1 stack traces), but breaker-OPEN state was never forced: R2-09's ConnectionBackoff throttles retries so failures don't accumulate fast enough to open the Polly breaker. The reliable trigger is a rapid idempotent write to a dead endpoint. Confirmed still open 2026-07-27 (deferment.md \u00a78.5).",
      "passNote": "PASSED 2026-08-08 after the structural gap was FIXED (feat/wrap-driver-init-in-resilience): IDriver.InitializeAsync now routes through the invoker as DriverCapability.Initialize (no retry \u2014 the actor's RetryConnect timer is the retry loop; tier-default breaker threshold 2, because at the 10 s reconnect cadence at most three attempts fit the 30 s sampling window and a threshold of 5 could never open). Live on docker-dev (image df04c76ce06b, symbol-verified on both centrals): rtu-gate (DRV-3616122eb53d, endpoint 10.100.0.35:5021 with no listener) opened its Initialize breaker on BOTH centrals ('circuit-breaker OPENED \u2026 capability=Initialize \u2026 Connection refused'), the /hosts RESILIENCE panel showed the red 'BREAKER OPEN \u00b7 OPENED 19S AGO' chip, and after a socat 5021\u21925020 forwarder brought the endpoint alive both breakers closed within 1 s of each other ('circuit-breaker closed (recovered)' + 'reconnected') and the panel cleared to 'All targets healthy.' Control: healthy MAIN-modbus recorded zero breaker events. Two nuances: (1) 'both-replica chips' resolves as both nodes' TRACKERS tripping (proven in both logs) rendered through the single ACTIVE AdminUI \u2014 Traefik's /health/active check deliberately routes the UI to the active admin node only (standby returns 503 'Standby: active node for role admin is \u2026'), so there is never a second replica serving chips; (2) the staleness-dim leg was NOT exercised \u2014 both pair nodes publish under the same key, so staleness needs every publisher for a key to stop, which requires deliberate node-kill churn this gate did not do. Companion fix shipped in the same change: DriverCapabilityInvokerFactory.Create now Invalidates the instance's TRACKER snapshots alongside its pipelines, because a respawn during an outage discarded the only breaker whose OnClosed could clear the OPEN flag \u2014 a deploy mid-outage would have stranded a false OPEN chip forever (unreachable before this change made breaker-open reachable). Evidence: docs/plans/2026-08-07-adminui-followups-and-live-gates.md \u00a77.",
      "closureNote": "ATTEMPTED 2026-08-07, NOT ACHIEVABLE AS SPECIFIED \u2014 and the reason is structural, not environmental. The breaker only sees failures that propagate out of a CapabilityInvoker-wrapped call, and the dead-endpoint path produces none: (1) IDriver.InitializeAsync is NOT wrapped (DriverInstanceActor.cs:707 calls it directly; the invoker wraps Read/Write/Subscribe/Discover/AlarmAcknowledge only), so a driver that cannot connect contributes zero breaker input \u2014 refuting this task's own premise; (2) ModbusDriver swallows per-tag failures in BOTH directions \u2014 ReadAsync -> Degraded health (:365-375,:838-855), WriteAsync -> WriteResult status \u2014 so even the fallback trigger this plan suggests (rapid idempotent write-to-dead-endpoint) cannot work. Live-proven: MAIN-modbus pointed at a TCP blackhole went DEGRADED with every read timing out (0x80050000) while the tracker held ConsecutiveFailures=0 and the panel showed \"All targets healthy.\" The breaker is configured to open on 2 failures in a 30 s window at ratio 1.0, so the previously-recorded ConnectionBackoff-throttling explanation is not the operative one. VERIFIED LIVE regardless: panel renders on /hosts (rehomed by #546; this plan's \"8 driver pages\" placement is stale), RESILIENCE block present, and the publisher emits real tracker state every 5 s. To close: wrap the connect path, stop drivers swallowing capability failures, or use a driver whose wrapped call fails AFTER a successful init. Evidence: docs/plans/2026-08-07-adminui-followups-and-live-gates.md \u00a74."
    },
    {
      "id": "T11",
      "subject": "Docs + review ledger: mark 03/U10 + 04/U-5 remediated; update STATUS.md + FOLLOWUP-10",
      "status": "completed",
      "blockedBy": [
        "T10"
      ]
    }
  ]
}