lmxopcua/archreview/plans/STATUS.md

# Arch-Review Remediation — Status & Handoff

**Snapshot:** 2026-07-08 · **Review baseline:** master `9cad9ed0` · this file is the single source of
truth for remediation progress. Update it as work lands.

## Branch topology (all LOCAL — nothing pushed, nothing on master)

| Branch | Head | Contains |
|---|---|---|
| `master` | `9cad9ed0` | untouched baseline |
| `docs/archreview-plans-and-drift-fixes` | `9fadead6` | the 7 domain plans + `00-INDEX.md` + this `STATUS.md` + the two CLAUDE.md doc-drift fixes. **(the plans live ONLY here)** |
| `fix/archreview-crit1-split-brain-resolver` | `a25c9ed0` | Critical 1 (03/S1) typed SBR option + guard test + Redundancy.md; **+ #9 hard-kill failover test** (`a25c9ed0`) |
| `fix/archreview-crit2-vt-timeout` | `7fd44f0f` | Critical 2 (02/U2 + 02/U3) + tests |
| `fix/archreview-crit3-s7-reconnect` | `25c0c6f6` | Critical 3 (05/STAB-1) IS7Plc seam + lazy reconnect + unit guards + live bounce test |
| `fix/archreview-crit4-twincat-replay` | `af318fb4` | Critical 4 (05/STAB-2) native-notification replay + probe-recycle + unit guards |
| `fix/archreview-u2-deferred-sink-forwarding-guard` | `a65c2ced` | Guard 03/U2 — reflection-exhaustive deferred-sink forwarding test (+ DeferredServiceLevelPublisher) |
| `fix/archreview-c1-wire-analyzer` | `f0082af5` | Guard 07/C-1 — wire OTOPCUA0001 analyzer tree-wide + triage (surfaced the RESILIENCE-DISPATCH-GAP, task #10) |
| `fix/archreview-s1-s2-ci-coverage` | `10b89830` | Guard 07/S-1,S-2,S-4 — whole-solution CI leg + fail-on-skip gate + CLI sleep deflake |

**Decision (user):** one branch per Critical, each off `master`. **#9 (failover test) lives on the crit1 branch**
(not its own branch off master) because SBR only exists there — it is Critical 1's integration half. The plans are only on the docs branch,
so from a critical branch read them via `git show docs/archreview-plans-and-drift-fixes:archreview/plans/0X-<name>.md`.

## Completed

| Item | Finding | Branch @ commit | Verification |
|---|---|---|---|
| **Critical 1 — split-brain resolver activated** | 03/S1 | `fix/archreview-crit1-split-brain-resolver` `a81dea10` | `BuildClusterOptions` sets `KeepOldestOption{DownIfAlone=true}`; 3 unit guards (Cluster.Tests 29/29); **e2e-verified** the resolved ActorSystem config carries `SplitBrainResolverProvider` + `stable-after=15s` survives the HOCON/Hosting merge (throwaway spike, removed). HOCON cross-ref comment + Redundancy.md corrected. |
| **Critical 2a — real VT script timeout** | 02/U2 | `fix/archreview-crit2-vt-timeout` `7fd44f0f` | Routes through `TimedScriptEvaluator`; infinite-loop test (bounded by WaitAsync) proves no hang. Host.Integration VT 18/18. |
| **Critical 2b — ALC-safe compile cache** | 02/U3 | same `7fd44f0f` | Swapped raw dict → `CompiledScriptCache`; new `IScriptCacheOwner` seam; apply-boundary clear in `VirtualTagHostActor`; wiring guard test. Runtime VT 12/12. |
| **Doc drift — KNOWN LIMITATION 2 stale** | 03/U1 = 06/U-1 | `docs/…` `9fadead6` (CLAUDE.md) | historized-ref feed IS wired; rewrote to "value-capture code-complete, live-verify pending". |
| **Doc drift — ScriptAnalysis policy** | theme #5 | same `9fadead6` (CLAUDE.md) | corrected `FleetAdmin` → `Administrator,Designer` roles. |
| **Critical 3 — S7 lazy reconnect** | 05/STAB-1 | `fix/archreview-crit3-s7-reconnect` `25c0c6f6` | New `IS7Plc`/`IS7PlcFactory` seam + `EnsureConnectedAsync` (dispose-dead-handle → reopen on next data call); connection-fatal faults (socket/`ErrorCode.ConnectionError`) mark the handle dead, data-address errors do NOT; probe loop routes through it as a backstop. 4 unit guards + full S7 suite 234/234; **LIVE-verified** against real python-snap7 — bounced the container (`docker restart`), driver observed the outage and resumed Good reads with no redeploy (`S7_1500ReconnectTests`, double-gated on sim + `S7_RECONNECT_BOUNCE_CMD`). Also closes the S7 TEST-1 gap. |
| **Critical 4 — TwinCAT native-notification replay** | 05/STAB-2 | `fix/archreview-crit4-twincat-replay` `af318fb4` | Native ADS notifications were opaque handles (no replayable intent) → silently orphaned on a client swap. Now `NativeRegistration` (symbol/type/bit/interval/onChange + swappable handle) hangs off `DeviceState`; `EnsureConnectedAsync` split into a gate wrapper + under-gate core that **replays every intent onto a freshly-built client** (register+replay both under `ConnectGate`); probe-failure `RecycleClientAsync` disposes+nulls the wire-dead-but-port-open client so the next tick rebuilds+replays. 4 unit guards + full TwinCAT suite 174/174. **Unit-only — no TwinCAT docker fixture exists** (integration needs a real TC3 XAR), so the fake-client tests are authoritative (no live gate like S7 had). |
| **Guard — reflection-exhaustive deferred-sink forwarding** | 03/U2 | `fix/archreview-u2-deferred-sink-forwarding-guard` `a65c2ced` | New `DeferredSinkForwardingReflectionTests` (Commons.Tests) reflects over **every** method of every forwarding interface `DeferredAddressSpaceSink` implements (`IOpcUaAddressSpaceSink` + `ISurgicalAddressSpaceSink`) — plus the sibling `DeferredServiceLevelPublisher` — invokes each on the wrapper over a `DispatchProxy` recording inner, and asserts the call **reached** the inner. A `DispatchProxy` recorder auto-implements future members (zero maintenance); a guard-of-the-guard asserts the wrapper implements exactly the known forwarding-interface set, so a brand-new capability *interface* trips too. **This closes the F10b/PR#423 trap class and is the prerequisite gate for 03/P1 remove methods + 02/P `WriteValues`.** 3 new tests green (Commons.Tests 57/57); **negative control verified** — stubbing `UpdateFolderDisplayName` to a non-forwarding no-op fails the guard with the exact member-named diagnostic. NO production change. |
| **Guard — wire OTOPCUA0001 analyzer + triage** | 07/C-1 | `fix/archreview-c1-wire-analyzer` `f0082af5` | Injected the analyzer tree-wide via `Directory.Build.props` (`OutputItemType="Analyzer"`), then triaged ~280 surfaced hits: **(1) RESILIENCE-DISPATCH-GAP** — 7 production sites (`DriverInstanceActor` ×6 + `GenericDriverNodeManager`) scoped-`#pragma`'d as a **real tracked gap** (CapabilityInvoker built but never wired into dispatch; **new task #10**); **(2) driver-internal self-calls** — 3 sites (AbCip/S7 own poll/ack) `#pragma`'d as intentional; **(3) 12 wire-level test/CLI projects** → commented `NoWarn`. Full build green, 0 hits; analyzer 31/31; **negative control** — dropping one dispatch-gap pragma re-fires OTOPCUA0001 and fails the build, proving it's live tree-wide. |
| **Guard — CI to all suites + fail-on-skip + deflake** | 07/S-1,S-2,S-4 | `fix/archreview-s1-s2-ci-coverage` `10b89830` | S-1: replaced the 5-project `unit-tests` matrix with ONE whole-solution leg (self-maintaining; now covers Client's 388 + Analyzers' 31 + all drivers). S-2: `scripts/ci/assert-not-all-skipped.sh` fail-on-skip gate (turns "green==all-skipped" red) — strict on the unit leg, report-only on the fixtureless integration leg. S-4: readiness-signal (`SubscribeInvoked`/`SubscribeAlarmsInvoked`) replaces 11 fixed-sleep startup races in the newly-CI'd Client.CLI.Tests. Skip-gate proven locally (real trx OK / all-skip trx exit 1 / report-only / edge cases); Client.CLI.Tests 104/104. **CI job execution itself is verifiable only on push (nothing pushed).** |
| **#9 — hard-kill failover integration test** | 03/S1 | `fix/archreview-crit1-split-brain-resolver` `a25c9ed0` | Critical 1's load-bearing integration half. `TwoNodeClusterHarness.HardKillNodeAAsync` (transport-shutdown crash sim → node B sees A Unreachable, no graceful Leave) + `HardKillFailoverTests` asserting B downs the alone oldest node and takes over as `driver` role-leader. **LIVE-verified in-process** (~35s); **negative control** (force `downing-provider-class=""`) makes it correctly time out. ⚠️ The negative control also surfaced a **Crit-1 premise finding (task #11): SBR was already active on master** (Akka.Cluster.Hosting applies `SplitBrainResolverOption.Default` when the typed option is null → reads the pre-existing akka.conf keep-oldest block), so the cluster was NOT NoDowning before Crit-1 — the typed option is reinforcing, not the activator. |

## Task list (TaskCreate IDs)

| # | Task | Status |
|---|---|---|
| 1 | Critical 1 — split-brain resolver | ✅ completed |
| 2 | Critical 2a — VT timeout | ✅ completed |
| 3 | Critical 2b — CompiledScriptCache | ✅ completed |
| 4 | Critical 3 — S7 reconnect (05/STAB-1) | ✅ completed |
| 5 | Critical 4 — TwinCAT ADS re-registration (05/STAB-2) | ✅ completed |
| 6 | Guard — DeferredAddressSpaceSink reflection forwarding test (03/U2) | ✅ completed — `a65c2ced`; gate for 03/P1 + 02/P now in place |
| 7 | Guard — wire OTOPCUA0001 analyzer + triage (07/C-1) | ✅ completed — `f0082af5`; surfaced task #10 |
| 8 | Guard — CI to all unit suites + fail-on-skip (07/S-1,S-2, pair S-4) | ✅ completed — `10b89830` |
| 9 | Critical 1 follow-up — hard-kill failover integration test (03/S1) | ✅ completed — `a25c9ed0`; surfaced task #11 |
| 10 | **Wire CapabilityInvoker into the dispatch layer** (RESILIENCE-DISPATCH-GAP, from 07/C-1) | ⬜ pending — real gap the analyzer caught; behavior-changing hot-path remediation, needs a live rig gate. Own branch off master. |
| 11 | **Review Critical 1 premise** — SBR was already active on master, not NoDowning (from #9) | ⬜ pending — correct Crit-1's rationale/docs; the fix isn't harmful but "was NoDowning" is inaccurate. |

## Findings surfaced by the guards (NEW — need user decision)

> Each has a dedicated follow-up file with the full design/action:
> [`FOLLOWUP-10-resilience-dispatch-gap.md`](FOLLOWUP-10-resilience-dispatch-gap.md) ·
> [`FOLLOWUP-11-crit1-sbr-premise.md`](FOLLOWUP-11-crit1-sbr-premise.md).

- **#10 — RESILIENCE-DISPATCH-GAP (from wiring the analyzer, 07/C-1):** the Phase 6.1 `CapabilityInvoker`
  resilience pipeline (retry/breaker/bulkhead/telemetry) is constructed **only in tests** — the production
  dispatch layer (`DriverInstanceActor` + `GenericDriverNodeManager`) calls driver capability methods directly.
  The 7 sites are `#pragma`-suppressed with a greppable `RESILIENCE-DISPATCH-GAP` marker (tree stays green + the
  analyzer stays live everywhere else). Wiring the invoker in is a behavior-changing hot-path remediation that
  needs a live rig gate — a genuine High, its own branch off master.
- **#11 — Critical 1 premise inaccurate (from the #9 failover negative control):** SBR was **already active on
  master** (Akka.Cluster.Hosting's `WithClustering` applies `SplitBrainResolverOption.Default` when the typed
  option is null → enables the downing provider that reads the pre-existing akka.conf `keep-oldest` block). So the
  cluster was **not** running NoDowning; hard-crash failover already worked. Crit-1's typed option is reinforcing
  (fine to keep), but its "HOCON inert / NoDowning → no failover" justification + the Redundancy.md/Architecture.md
  wording are wrong and should be corrected. Do NOT describe Crit-1 as "fixed a NoDowning bug."

## Suggested next (from 00-INDEX execution order)

**All 4 Criticals + all 3 systemic guards (tasks 6/7/8) + the #9 failover test are DONE.** Remaining:

1. **Resolve the two surfaced findings** — task 10 (wire CapabilityInvoker into dispatch; behavior-changing,
   live-gate) and task 11 (correct Crit-1's rationale/docs).
2. **Failure-visibility Highs** — 01/S-1 (AddressSpaceApplyOutcome failure field), 06/S-1 (Galaxy fail-closed),
   03/S4 (primary-gate default-deny).
3. **P-tier** — 03/P1 surgical adds/removals (now gated by the U2 forwarding guard), 02/P batched `WriteValues`.

## Key resume facts

- **Verification bar met so far:** each fix has a deterministic unit guard AND (for SBR) an end-to-end
  config-resolution check. Follow the same pattern — the review's whole thesis is "unit-green ≠ wired".
- **The forwarding-test guard (task 6) is DONE (`a65c2ced`):** 03/P1 (surgical pure-adds) and 02/P-… batch-write
  each add new sink capability methods that ship inert unless forwarded through `DeferredAddressSpaceSink`
  (F10b/PR#423 precedent). The reflection guard now fails loudly if any such member isn't forwarded — add the
  member to the recorder-covered interface set and it's auto-checked.
- **Nothing is pushed.** 8 local branches: the 4 `fix/archreview-crit*` (crit1 now carries #9 too), the `u2`,
  `c1-wire-analyzer`, and `s1-s2-ci-coverage` guard branches, + the docs branch. The docs branch can merge first
  (plans + doc fixes are low-risk).
- **CI changes can't be run-verified without a push** — the S-1/S-2 workflow + skip-gate were validated locally
  (YAML parse + script against real/synthetic trx), but the actual CI-job run is only observable once pushed.
- **Two findings (#10, #11) need user decisions** before those areas are "closed" — see the Findings section above.
- **Plans are on the docs branch only** — see topology note above for cross-branch reads.