Commit Graph

14 Commits

Author SHA1 Message Date
Joseph Doherty dac7e1885d feat(resilience): route IDriver.InitializeAsync through the capability invoker (R2-10 T10 closed)
The connect path was the one capability call the resilience pipeline never
saw, so a driver that could not connect contributed zero breaker input and
the /hosts resilience panel read 'All targets healthy' against a dead
endpoint. InitializeAsync now routes through the invoker as
DriverCapability.Initialize, keyed on the driver-instance id.

- No pipeline retry: the actor's RetryConnect timer is the retry loop
  (parser forces retryCount overrides to 0, same funnel as write-shaped).
- Breaker threshold 2, not the Read-sized 5: the breaker samples 30 s and
  the reconnect timer fires every 10 s, so at most three attempts fit one
  window — threshold 5 could never open. Pinned by its own test.
- An open breaker fast-fails into the same InitializeFailed path as a real
  connect failure; the 15 s break window costs at most one extra tick.
  The best-effort teardown ShutdownAsync stays unwrapped.
- Companion fix: DriverCapabilityInvokerFactory.Create now invalidates the
  instance's tracker snapshots with its pipelines — a respawn during an
  outage discarded the only breaker whose OnClosed could clear the OPEN
  flag, stranding a false 'Breaker OPEN' chip forever.

Live gate PASSED on docker-dev (image symbol-verified per container):
rtu-gate against dead 10.100.0.35:5021 opened its Initialize breaker on
both centrals with the red 'BREAKER OPEN' chip on /hosts; a socat
5021→5020 forwarder recovered it (both breakers closed within 1 s, panel
back to 'All targets healthy'); healthy MAIN-modbus recorded zero breaker
events. Full suite 50/50 green. Evidence:
docs/plans/2026-08-07-adminui-followups-and-live-gates.md §7.
2026-08-08 05:45:08 -04:00
Joseph Doherty 1e02661598 chore(gates): close adminui-followups by audit; R2-03 T11/T12 + reconfigure Task 3 PASSED, R2-10 re-diagnosed
The two items deferment.md §0.1 named as the remaining untracked engineering.

adminui-followups (19 pending / 0 completed) — CLOSED by the overlap audit the plan
itself asks for first, with no code written. All 19 tasks are shipped, superseded by
v3, or moot. WS1's premise had also expired: it assumed driver-embedded tag lists in
DriverConfig are the runtime source of truth, which v3 ended (the artifact populates
RawTags). Its capability shipped in the v3 shape — TagConfigEditorMap + DeviceFormMap,
both verified reachable from RawTagModal/DeviceModal rather than merely present.

Live gates on docker-dev (image rebuilt: the rig was running a build 3 h older than the
last code commit; parity asserted per container plus a positive check that the symbols
under test are in the shipped DLLs):

- R2-03 T11 PASSED — build 0 errors, Runtime.Tests 555/555, RoslynVirtualTagEvaluator 20/20.
- R2-03 T12 PASSED — VT 42 -> 0x80020000 -> 42, staged WITHOUT a deployment (a Modbus
  register flipped on the fixture) so the VT child survived; a deploy-driven trip creates
  a fresh child with no last value, which is correct-by-design and masked the result twice.
  P7 confirmed: two byte-identical redeploys, zero disturbance.
- driver-reconfigure Task 3 PASSED — a faulted driver adopted a corrected config and
  connected with the node's startedAt unchanged and RestartCount=0.
- R2-10 T10 still OPEN, but the recorded cause was wrong. Not ConnectionBackoff
  throttling (the breaker opens on 2 failures / 30 s): the dead-endpoint failure mode
  never enters the resilience pipeline. IDriver.InitializeAsync is not invoker-wrapped
  (DriverInstanceActor.cs:707) and ModbusDriver swallows per-tag failures in BOTH
  directions, so even the plan's fallback trigger cannot work. Live-proven: a driver
  timing out every read against a blackhole reported ConsecutiveFailures=0 while the
  panel read "All targets healthy."

Finding from T12: the Bad publish's documented "carrying the last-known value" is
unobservable by any client — subscription and a raw Read service call both return NULL.
The actor is correct; every existing failure assertion checked StatusCode alone, so the
value half was asserted nowhere. Adds
VirtualTagActorTests.The_Bad_result_carries_the_last_known_value, which proves the actor
emits it. The documentation is what overstates.
2026-08-07 06:28:22 -04:00
Joseph Doherty 88ce8df099 docs: close the plan-bookkeeping drift and state the tier truth (§8.5, §8.6, #522)
Bookkeeping: 13 .tasks.json files closed — the 8 the audit named, plus
mesh-phase4 (Task 9 landed 3a590a0c), the 4th stillpending phase file, and
historian-tcp-transport closed as OBSOLETE (it targets the retired Wonderware
sidecar; there is no Wonderware backend in the tree). Each carries a
closureNote naming the evidence rather than just a status flip.

11 archreview live gates written back from STATUS.md, which had recorded them
passed since 2026-07-13/15 without ever updating the task files: R2-01 #11,
R2-02 #15/#18, R2-05 T15, R2-06 T12, R2-07 T5/T6/T12/T14, R2-11 T22/T24.

R2-03 and R2-10 were deliberately NOT closed. STATUS.md's headline says "every
Round-2 live gate is now GREEN", but its own per-item detail says R2-03 is
"live-blocked on this data-less rig" and R2-10 verified the pipeline with
"breaker-OPEN state not forced". Both now carry an openNote recording the
actual blocker.

The point of the sweep: ~140 phantom pending tasks are gone, so the genuinely
unexecuted 19-task AdminUI follow-ups plan is now the largest open item in
docs/plans/ instead of being buried under work that shipped months ago.

Tier truth: docs/v2/driver-stability.md now states that its tier table is
aspirational — every driver runs Tier A because no factory passes a tier, so
MemoryRecycle and ScheduledRecycleScheduler have never engaged, and the config
parser validates a RecycleIntervalSeconds that does nothing. It also corrects a
staleness the audit missed: the separate-Windows-service hosting it describes
for Tier C no longer exists (Galaxy moved to the mxaccessgw sidecar in PR 7.2,
FOCAS went in-process with its managed wire client). docs/drivers/README.md
already said Tier A for both, so the two no longer contradict.

No runtime change — enabling recycle on two live drivers wants its own live
gate, not a docs-pass side effect. Keep-or-delete tracked as Gitea #522.
2026-07-27 19:54:41 -04:00
Joseph Doherty 1e3552dbd5 docs(archreview): R2-10 resilience observability shipped
v2-ci / build (pull_request) Successful in 3m47s
v2-ci / unit-tests (pull_request) Failing after 8m12s
2026-07-13 10:50:02 -04:00
Joseph Doherty cc5ecb5868 fix(r2-10): breaker/retry chips on the driver status panel 2026-07-13 10:46:47 -04:00
Joseph Doherty 38c059dae7 fix(r2-10): driver-resilience-status DPS bridge 2026-07-13 10:44:44 -04:00
Joseph Doherty 99ce5b401e fix(r2-10): in-process resilience status store 2026-07-13 10:43:47 -04:00
Joseph Doherty 12b2ce078b fix(r2-10): register the tracker's production reader on driver nodes + wiring guard 2026-07-13 10:42:18 -04:00
Joseph Doherty 37f180c5a3 fix(r2-10): periodic DPS publisher for tracker snapshots 2026-07-13 10:40:54 -04:00
Joseph Doherty 2cb3f962b9 fix(r2-10): DriverResilienceStatusChanged DPS contract 2026-07-13 10:39:18 -04:00
Joseph Doherty 0ba60db1e4 fix(r2-10): successful capability calls reset ConsecutiveFailures 2026-07-13 10:38:51 -04:00
Joseph Doherty 2cce61fc04 fix(r2-10): OnClosed records breaker-close on the status tracker 2026-07-13 10:37:12 -04:00
Joseph Doherty 205d398055 fix(r2-10): breaker-close marker + IsBreakerOpen on ResilienceStatusSnapshot 2026-07-13 10:35:41 -04:00
Joseph Doherty 1891f5d6a7 docs(archreview): 2026-07-12 re-review at f6eaa267 + 12 round-2 remediation plans
Re-ran all seven domain reviews at master f6eaa267 (reports rewritten in
place, each with a prior-finding status table): all 4 round-1 Criticals
verified closed; new top findings are the S7 connect-timeout OCE regression
(05/STAB-14), the ResilienceConfig operator-authorable brick (01/S-6), and
a batch of resilience-seam Mediums. 00-OVERALL.md carries the updated
maturity matrix + 12-item action list.

Adds R2-01..R2-12 design/implementation plans (one per action item, house
format + bite-sized TDD task breakdowns + co-located .tasks.json; 193 tasks,
~18-24 dev-days). STATUS.md updated: round-1 topology marked historical
(all merged+pushed), re-review findings table + plan pointers added.
2026-07-12 23:52:23 -04:00