Joseph Doherty
dac7e1885d
The connect path was the one capability call the resilience pipeline never saw, so a driver that could not connect contributed zero breaker input and the /hosts resilience panel read 'All targets healthy' against a dead endpoint. InitializeAsync now routes through the invoker as DriverCapability.Initialize, keyed on the driver-instance id. - No pipeline retry: the actor's RetryConnect timer is the retry loop (parser forces retryCount overrides to 0, same funnel as write-shaped). - Breaker threshold 2, not the Read-sized 5: the breaker samples 30 s and the reconnect timer fires every 10 s, so at most three attempts fit one window — threshold 5 could never open. Pinned by its own test. - An open breaker fast-fails into the same InitializeFailed path as a real connect failure; the 15 s break window costs at most one extra tick. The best-effort teardown ShutdownAsync stays unwrapped. - Companion fix: DriverCapabilityInvokerFactory.Create now invalidates the instance's tracker snapshots with its pipelines — a respawn during an outage discarded the only breaker whose OnClosed could clear the OPEN flag, stranding a false 'Breaker OPEN' chip forever. Live gate PASSED on docker-dev (image symbol-verified per container): rtu-gate against dead 10.100.0.35:5021 opened its Initialize breaker on both centrals with the red 'BREAKER OPEN' chip on /hosts; a socat 5021→5020 forwarder recovered it (both breakers closed within 1 s, panel back to 'All targets healthy'); healthy MAIN-modbus recorded zero breaker events. Full suite 50/50 green. Evidence: docs/plans/2026-08-07-adminui-followups-and-live-gates.md §7.
96 lines
6.7 KiB
JSON
96 lines
6.7 KiB
JSON
{
|
|
"planPath": "archreview/plans/R2-10-resilience-observability-plan.md",
|
|
"lastUpdated": "2026-08-07",
|
|
"tasks": [
|
|
{
|
|
"id": "T1",
|
|
"subject": "Tracker: LastBreakerClosedUtc + IsBreakerOpen + RecordBreakerClose (Core, TDD in DriverResilienceStatusTrackerTests)",
|
|
"status": "completed",
|
|
"blockedBy": []
|
|
},
|
|
{
|
|
"id": "T2",
|
|
"subject": "Builder: OnClosed records breaker-close on the tracker (TDD in DriverResiliencePipelineBuilderTests)",
|
|
"status": "completed",
|
|
"blockedBy": [
|
|
"T1"
|
|
]
|
|
},
|
|
{
|
|
"id": "T3",
|
|
"subject": "Invoker: successful Execute* resets ConsecutiveFailures (TDD in CapabilityInvokerTests)",
|
|
"status": "completed",
|
|
"blockedBy": [
|
|
"T1"
|
|
]
|
|
},
|
|
{
|
|
"id": "T4",
|
|
"subject": "Commons: DriverResilienceStatusChanged record + driver-resilience-status TopicName",
|
|
"status": "completed",
|
|
"blockedBy": []
|
|
},
|
|
{
|
|
"id": "T5",
|
|
"subject": "Host: DriverResilienceStatusPublisherService (BuildMessages mapping + 5s DPS timer shell; TDD mapping in Host.IntegrationTests)",
|
|
"status": "completed",
|
|
"blockedBy": [
|
|
"T1",
|
|
"T4"
|
|
]
|
|
},
|
|
{
|
|
"id": "T6",
|
|
"subject": "Host DI: register the publisher in AddOtOpcUaDriverFactories + ResilienceStatusReaderRegistrationTests wiring guard (tracker HAS a production reader)",
|
|
"status": "completed",
|
|
"blockedBy": [
|
|
"T5"
|
|
]
|
|
},
|
|
{
|
|
"id": "T7",
|
|
"subject": "AdminUI: IDriverResilienceStatusStore + InMemory impl + AddOtOpcUaDriverStatusServices registration (TDD mirroring DriverStatusSnapshotStoreTests)",
|
|
"status": "completed",
|
|
"blockedBy": [
|
|
"T4"
|
|
]
|
|
},
|
|
{
|
|
"id": "T8",
|
|
"subject": "AdminUI: DriverResilienceStatusBridge DPS actor + spawn in WithOtOpcUaSignalRBridges",
|
|
"status": "completed",
|
|
"blockedBy": [
|
|
"T7"
|
|
]
|
|
},
|
|
{
|
|
"id": "T9",
|
|
"subject": "AdminUI: resilience chip section in DriverStatusPanel.razor (in-process store read; no bUnit \u2014 verified by T10)",
|
|
"status": "completed",
|
|
"blockedBy": [
|
|
"T7"
|
|
]
|
|
},
|
|
{
|
|
"id": "T10",
|
|
"subject": "Live-/run gate on docker-dev: rebuild BOTH centrals, S7 dead-endpoint breaker-open, both-replica chips, recovery clear, staleness dim",
|
|
"status": "completed",
|
|
"blockedBy": [
|
|
"T6",
|
|
"T8",
|
|
"T9"
|
|
],
|
|
"openNote": "PARTIAL \u2014 the resilience PIPELINE is live-verified (Polly CircuitBreaker -> Retry -> Timeout -> CapabilityInvoker in central-1 stack traces), but breaker-OPEN state was never forced: R2-09's ConnectionBackoff throttles retries so failures don't accumulate fast enough to open the Polly breaker. The reliable trigger is a rapid idempotent write to a dead endpoint. Confirmed still open 2026-07-27 (deferment.md \u00a78.5).",
|
|
"passNote": "PASSED 2026-08-08 after the structural gap was FIXED (feat/wrap-driver-init-in-resilience): IDriver.InitializeAsync now routes through the invoker as DriverCapability.Initialize (no retry \u2014 the actor's RetryConnect timer is the retry loop; tier-default breaker threshold 2, because at the 10 s reconnect cadence at most three attempts fit the 30 s sampling window and a threshold of 5 could never open). Live on docker-dev (image df04c76ce06b, symbol-verified on both centrals): rtu-gate (DRV-3616122eb53d, endpoint 10.100.0.35:5021 with no listener) opened its Initialize breaker on BOTH centrals ('circuit-breaker OPENED \u2026 capability=Initialize \u2026 Connection refused'), the /hosts RESILIENCE panel showed the red 'BREAKER OPEN \u00b7 OPENED 19S AGO' chip, and after a socat 5021\u21925020 forwarder brought the endpoint alive both breakers closed within 1 s of each other ('circuit-breaker closed (recovered)' + 'reconnected') and the panel cleared to 'All targets healthy.' Control: healthy MAIN-modbus recorded zero breaker events. Two nuances: (1) 'both-replica chips' resolves as both nodes' TRACKERS tripping (proven in both logs) rendered through the single ACTIVE AdminUI \u2014 Traefik's /health/active check deliberately routes the UI to the active admin node only (standby returns 503 'Standby: active node for role admin is \u2026'), so there is never a second replica serving chips; (2) the staleness-dim leg was NOT exercised \u2014 both pair nodes publish under the same key, so staleness needs every publisher for a key to stop, which requires deliberate node-kill churn this gate did not do. Companion fix shipped in the same change: DriverCapabilityInvokerFactory.Create now Invalidates the instance's TRACKER snapshots alongside its pipelines, because a respawn during an outage discarded the only breaker whose OnClosed could clear the OPEN flag \u2014 a deploy mid-outage would have stranded a false OPEN chip forever (unreachable before this change made breaker-open reachable). Evidence: docs/plans/2026-08-07-adminui-followups-and-live-gates.md \u00a77.",
|
|
"closureNote": "ATTEMPTED 2026-08-07, NOT ACHIEVABLE AS SPECIFIED \u2014 and the reason is structural, not environmental. The breaker only sees failures that propagate out of a CapabilityInvoker-wrapped call, and the dead-endpoint path produces none: (1) IDriver.InitializeAsync is NOT wrapped (DriverInstanceActor.cs:707 calls it directly; the invoker wraps Read/Write/Subscribe/Discover/AlarmAcknowledge only), so a driver that cannot connect contributes zero breaker input \u2014 refuting this task's own premise; (2) ModbusDriver swallows per-tag failures in BOTH directions \u2014 ReadAsync -> Degraded health (:365-375,:838-855), WriteAsync -> WriteResult status \u2014 so even the fallback trigger this plan suggests (rapid idempotent write-to-dead-endpoint) cannot work. Live-proven: MAIN-modbus pointed at a TCP blackhole went DEGRADED with every read timing out (0x80050000) while the tracker held ConsecutiveFailures=0 and the panel showed \"All targets healthy.\" The breaker is configured to open on 2 failures in a 30 s window at ratio 1.0, so the previously-recorded ConnectionBackoff-throttling explanation is not the operative one. VERIFIED LIVE regardless: panel renders on /hosts (rehomed by #546; this plan's \"8 driver pages\" placement is stale), RESILIENCE block present, and the publisher emits real tracker state every 5 s. To close: wrap the connect path, stop drivers swallowing capability failures, or use a driver whose wrapped call fails AFTER a successful init. Evidence: docs/plans/2026-08-07-adminui-followups-and-live-gates.md \u00a74."
|
|
},
|
|
{
|
|
"id": "T11",
|
|
"subject": "Docs + review ledger: mark 03/U10 + 04/U-5 remediated; update STATUS.md + FOLLOWUP-10",
|
|
"status": "completed",
|
|
"blockedBy": [
|
|
"T10"
|
|
]
|
|
}
|
|
]
|
|
} |