Files
lmxopcua/archreview/plans/STATUS.md
T

175 lines
25 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Arch-Review Remediation — Status & Handoff
**Snapshot:** 2026-07-12 · **Review baseline:** master `9cad9ed0`; **re-review baseline:** master
`f6eaa267` · this file is the single source of truth for remediation progress. Update it as work lands.
> **2026-07-12 UPDATE — everything below the topology table landed:** all remediation branches were
> **merged to master and pushed to `origin/master`** (merge commits `7834b3b7`…`b67bd9e8`; final state
> `f6eaa267`). The topology table is kept for history but is no longer live. A full **re-review** of all
> seven domains ran at `f6eaa267` — see [`../00-OVERALL.md`](../00-OVERALL.md) and the
> "Re-review 2026-07-12" section at the bottom of this file for the new findings (one High regression
> in the S7 fix, one High ResilienceConfig validation gap, and a batch of resilience-seam Mediums).
## Branch topology (HISTORICAL — as of 2026-07-08; since merged + pushed)
| Branch | Head | Contains |
|---|---|---|
| `master` | `9cad9ed0` | untouched baseline |
| `docs/archreview-plans-and-drift-fixes` | `9fadead6` | the 7 domain plans + `00-INDEX.md` + this `STATUS.md` + the two CLAUDE.md doc-drift fixes. **(the plans live ONLY here)** |
| `fix/archreview-crit1-split-brain-resolver` | `eaf78aad` | Critical 1 (03/S1) typed SBR option + guard test + Redundancy.md; **+ #9 hard-kill failover test** (`a25c9ed0`); **+ #11 premise correction** (`eaf78aad`) |
| `fix/archreview-crit2-vt-timeout` | `7fd44f0f` | Critical 2 (02/U2 + 02/U3) + tests |
| `fix/archreview-crit3-s7-reconnect` | `25c0c6f6` | Critical 3 (05/STAB-1) IS7Plc seam + lazy reconnect + unit guards + live bounce test |
| `fix/archreview-crit4-twincat-replay` | `af318fb4` | Critical 4 (05/STAB-2) native-notification replay + probe-recycle + unit guards |
| `fix/archreview-u2-deferred-sink-forwarding-guard` | `a65c2ced` | Guard 03/U2 — reflection-exhaustive deferred-sink forwarding test (+ DeferredServiceLevelPublisher) |
| `fix/archreview-c1-wire-analyzer` | `f0082af5` | Guard 07/C-1 — wire OTOPCUA0001 analyzer tree-wide + triage (surfaced the RESILIENCE-DISPATCH-GAP, task #10) |
| `fix/archreview-s1-s2-ci-coverage` | `10b89830` | Guard 07/S-1,S-2,S-4 — whole-solution CI leg + fail-on-skip gate + CLI sleep deflake |
| `fix/archreview-crit10-wire-capability-invoker` | `62556c24` | **#10 — wire CapabilityInvoker into dispatch via a Core.Abstractions seam** (off `fix/archreview-c1-wire-analyzer`, NOT master — the analyzer is the standing guard + the pragmas live there). Code-complete + unit/analyzer/context-verified; live gate pending. |
| `fix/archreview-crit13-resilience-config-artifact` | `8a8b9ec5` | **#13 — apply per-instance ResilienceConfig from the artifact** (`75403caa`, off the crit10 branch) **+ #10/#13 DI guard** (`8a8b9ec5``ResilienceInvokerFactoryRegistrationTests`, from the #12 live gate). Read-path plumbing + invalidate/respawn-on-change; deterministically verified + rig-booted. |
| `fix/docker-dev-serverhistorian-stale-keys` | `7233e2ba` | **Rig fix (off master, independent):** docker-dev compose's retired Wonderware `ServerHistorian` keys → gateway shape, disabled by default. Found during the #12 live-gate rig session; central-1 verified booting clean. |
**Decision (user):** one branch per Critical, each off `master`. **#9 (failover test) lives on the crit1 branch**
(not its own branch off master) because SBR only exists there — it is Critical 1's integration half. The plans are only on the docs branch,
so from a critical branch read them via `git show docs/archreview-plans-and-drift-fixes:archreview/plans/0X-<name>.md`.
## Completed
| Item | Finding | Branch @ commit | Verification |
|---|---|---|---|
| **Critical 1 — split-brain resolver activated** | 03/S1 | `fix/archreview-crit1-split-brain-resolver` `a81dea10` | `BuildClusterOptions` sets `KeepOldestOption{DownIfAlone=true}`; 3 unit guards (Cluster.Tests 29/29); **e2e-verified** the resolved ActorSystem config carries `SplitBrainResolverProvider` + `stable-after=15s` survives the HOCON/Hosting merge (throwaway spike, removed). HOCON cross-ref comment + Redundancy.md corrected. |
| **Critical 2a — real VT script timeout** | 02/U2 | `fix/archreview-crit2-vt-timeout` `7fd44f0f` | Routes through `TimedScriptEvaluator`; infinite-loop test (bounded by WaitAsync) proves no hang. Host.Integration VT 18/18. |
| **Critical 2b — ALC-safe compile cache** | 02/U3 | same `7fd44f0f` | Swapped raw dict → `CompiledScriptCache`; new `IScriptCacheOwner` seam; apply-boundary clear in `VirtualTagHostActor`; wiring guard test. Runtime VT 12/12. |
| **Doc drift — KNOWN LIMITATION 2 stale** | 03/U1 = 06/U-1 | `docs/…` `9fadead6` (CLAUDE.md) | historized-ref feed IS wired; rewrote to "value-capture code-complete, live-verify pending". |
| **Doc drift — ScriptAnalysis policy** | theme #5 | same `9fadead6` (CLAUDE.md) | corrected `FleetAdmin``Administrator,Designer` roles. |
| **Critical 3 — S7 lazy reconnect** | 05/STAB-1 | `fix/archreview-crit3-s7-reconnect` `25c0c6f6` | New `IS7Plc`/`IS7PlcFactory` seam + `EnsureConnectedAsync` (dispose-dead-handle → reopen on next data call); connection-fatal faults (socket/`ErrorCode.ConnectionError`) mark the handle dead, data-address errors do NOT; probe loop routes through it as a backstop. 4 unit guards + full S7 suite 234/234; **LIVE-verified** against real python-snap7 — bounced the container (`docker restart`), driver observed the outage and resumed Good reads with no redeploy (`S7_1500ReconnectTests`, double-gated on sim + `S7_RECONNECT_BOUNCE_CMD`). Also closes the S7 TEST-1 gap. |
| **Critical 4 — TwinCAT native-notification replay** | 05/STAB-2 | `fix/archreview-crit4-twincat-replay` `af318fb4` | Native ADS notifications were opaque handles (no replayable intent) → silently orphaned on a client swap. Now `NativeRegistration` (symbol/type/bit/interval/onChange + swappable handle) hangs off `DeviceState`; `EnsureConnectedAsync` split into a gate wrapper + under-gate core that **replays every intent onto a freshly-built client** (register+replay both under `ConnectGate`); probe-failure `RecycleClientAsync` disposes+nulls the wire-dead-but-port-open client so the next tick rebuilds+replays. 4 unit guards + full TwinCAT suite 174/174. **Unit-only — no TwinCAT docker fixture exists** (integration needs a real TC3 XAR), so the fake-client tests are authoritative (no live gate like S7 had). |
| **Guard — reflection-exhaustive deferred-sink forwarding** | 03/U2 | `fix/archreview-u2-deferred-sink-forwarding-guard` `a65c2ced` | New `DeferredSinkForwardingReflectionTests` (Commons.Tests) reflects over **every** method of every forwarding interface `DeferredAddressSpaceSink` implements (`IOpcUaAddressSpaceSink` + `ISurgicalAddressSpaceSink`) — plus the sibling `DeferredServiceLevelPublisher` — invokes each on the wrapper over a `DispatchProxy` recording inner, and asserts the call **reached** the inner. A `DispatchProxy` recorder auto-implements future members (zero maintenance); a guard-of-the-guard asserts the wrapper implements exactly the known forwarding-interface set, so a brand-new capability *interface* trips too. **This closes the F10b/PR#423 trap class and is the prerequisite gate for 03/P1 remove methods + 02/P `WriteValues`.** 3 new tests green (Commons.Tests 57/57); **negative control verified** — stubbing `UpdateFolderDisplayName` to a non-forwarding no-op fails the guard with the exact member-named diagnostic. NO production change. |
| **Guard — wire OTOPCUA0001 analyzer + triage** | 07/C-1 | `fix/archreview-c1-wire-analyzer` `f0082af5` | Injected the analyzer tree-wide via `Directory.Build.props` (`OutputItemType="Analyzer"`), then triaged ~280 surfaced hits: **(1) RESILIENCE-DISPATCH-GAP** — 7 production sites (`DriverInstanceActor` ×6 + `GenericDriverNodeManager`) scoped-`#pragma`'d as a **real tracked gap** (CapabilityInvoker built but never wired into dispatch; **new task #10**); **(2) driver-internal self-calls** — 3 sites (AbCip/S7 own poll/ack) `#pragma`'d as intentional; **(3) 12 wire-level test/CLI projects** → commented `NoWarn`. Full build green, 0 hits; analyzer 31/31; **negative control** — dropping one dispatch-gap pragma re-fires OTOPCUA0001 and fails the build, proving it's live tree-wide. |
| **Guard — CI to all suites + fail-on-skip + deflake** | 07/S-1,S-2,S-4 | `fix/archreview-s1-s2-ci-coverage` `10b89830` | S-1: replaced the 5-project `unit-tests` matrix with ONE whole-solution leg (self-maintaining; now covers Client's 388 + Analyzers' 31 + all drivers). S-2: `scripts/ci/assert-not-all-skipped.sh` fail-on-skip gate (turns "green==all-skipped" red) — strict on the unit leg, report-only on the fixtureless integration leg. S-4: readiness-signal (`SubscribeInvoked`/`SubscribeAlarmsInvoked`) replaces 11 fixed-sleep startup races in the newly-CI'd Client.CLI.Tests. Skip-gate proven locally (real trx OK / all-skip trx exit 1 / report-only / edge cases); Client.CLI.Tests 104/104. **CI job execution itself is verifiable only on push (nothing pushed).** |
| **#10 — wire CapabilityInvoker into dispatch (seam)** | RESILIENCE-DISPATCH-GAP (07/C-1) | `fix/archreview-crit10-wire-capability-invoker` `62556c24` | **Discovered the filed plan was infeasible** — Runtime is deliberately Polly-free (refs `Core.Abstractions`, not `Core`). Fix = a seam mirroring `IDriverFactory`: `IDriverCapabilityInvoker`(+null) + `IDriverCapabilityInvokerFactory`(+null) in Core.Abstractions; `CapabilityInvoker` implements it; `DriverCapabilityInvokerFactory` (Core) builds per-instance over the singleton builder+tracker+`GetTier`; Host `DriverFactoryBootstrap` registers all three; Runtime SCE resolves like `IDriverFactory`; `DriverInstanceActor` wraps its **6** dispatch sites; analyzer taught the interface as a wrapper home; all 6 pragmas removed. Triage corrections: the 7th site (`GenericDriverNodeManager`) is test-only scaffolding (not wired, re-annotated); **tier-defaults only** (ResilienceConfig not in the artifact → sub-gap). Added retry/breaker **logging** to the pipeline builder (the tracker has no reader — Stream E.2/E.3 never shipped — so it ran invisibly). **Verified:** negative-control (unwrap→OTOPCUA0001 build error); recording-invoker routing test; **real-`CapabilityInvoker` actor-context test** (internal `ConfigureAwait(false)` doesn't leak); analyzer + logging tests; clean solution build; Runtime.Tests 358 / Core.Tests 238 / Analyzers 32. **Live behavioral gate pending** (recipe in FOLLOWUP-10). |
| **#9 — hard-kill failover integration test** | 03/S1 | `fix/archreview-crit1-split-brain-resolver` `a25c9ed0` | Critical 1's load-bearing integration half. `TwoNodeClusterHarness.HardKillNodeAAsync` (transport-shutdown crash sim → node B sees A Unreachable, no graceful Leave) + `HardKillFailoverTests` asserting B downs the alone oldest node and takes over as `driver` role-leader. **LIVE-verified in-process** (~35s); **negative control** (force `downing-provider-class=""`) makes it correctly time out. ⚠️ The negative control also surfaced a **Crit-1 premise finding (task #11): SBR was already active on master** (Akka.Cluster.Hosting applies `SplitBrainResolverOption.Default` when the typed option is null → reads the pre-existing akka.conf keep-oldest block), so the cluster was NOT NoDowning before Crit-1 — the typed option is reinforcing, not the activator. |
| **#11 — Critical 1 premise correction** | 03/S1 (from #9) | `fix/archreview-crit1-split-brain-resolver` `eaf78aad` | Docs/comment-only fix (no code revert): corrected the "HOCON inert / cluster runs NoDowning / never fails over" framing in `BuildClusterOptions` XML comment, `akka.conf` split-brain-resolver comment, `docs/Redundancy.md`, and `SplitBrainResolverActivationTests` (summary + `ShouldNotBeNull` message + method rename). Akka.Cluster.Hosting enables an SBR provider **by default** (applies `SplitBrainResolverOption.Default` when the typed option is null → reads the pre-existing akka.conf `keep-oldest` block), so the cluster was never NoDowning and the typed `KeepOldestOption` is reinforcing/explicit-in-code, not the activator. Cluster.Tests 29/29; failover behaviour still guarded by `HardKillFailoverTests`. Architecture.md carried no such framing. |
| **#13 — apply per-instance ResilienceConfig from the artifact** | RESILIENCE-DISPATCH-GAP sub-gap (from #10) | `fix/archreview-crit13-resilience-config-artifact` `75403caa` | The `DriverInstance.ResilienceConfig` column was authored+persisted (AdminUI) and already in the artifact (the composer serializes the whole entity), but the driver-node read path dropped it → tier-defaults only. `DriverInstanceSpec` now carries it; `TryReadSpec` reads it; `Create` takes+layers it (+ logs parse diagnostics, never throws); `SpawnChild` threads it. **Invalidate-on-change:** `Create` calls `builder.Invalidate(instance)` (the cache is options-blind on a hit) so a respawn with changed options rebuilds; `DriverSpawnPlanner` respawns on a ResilienceConfig change (incl. null→json), a pure DriverConfig change stays an in-place delta. **Verified:** override reaches execution (retryCount:0 suppresses tier-A retry; a control retries), invalidate is per-instance, malformed logs+falls-back; planner+artifact tests. Core.Tests 243 (+5), Runtime.Tests 363 (+5). See [`FOLLOWUP-13`](FOLLOWUP-13-resilience-config-artifact.md). |
## Task list (TaskCreate IDs)
| # | Task | Status |
|---|---|---|
| 1 | Critical 1 — split-brain resolver | ✅ completed |
| 2 | Critical 2a — VT timeout | ✅ completed |
| 3 | Critical 2b — CompiledScriptCache | ✅ completed |
| 4 | Critical 3 — S7 reconnect (05/STAB-1) | ✅ completed |
| 5 | Critical 4 — TwinCAT ADS re-registration (05/STAB-2) | ✅ completed |
| 6 | Guard — DeferredAddressSpaceSink reflection forwarding test (03/U2) | ✅ completed — `a65c2ced`; gate for 03/P1 + 02/P now in place |
| 7 | Guard — wire OTOPCUA0001 analyzer + triage (07/C-1) | ✅ completed — `f0082af5`; surfaced task #10 |
| 8 | Guard — CI to all unit suites + fail-on-skip (07/S-1,S-2, pair S-4) | ✅ completed — `10b89830` |
| 9 | Critical 1 follow-up — hard-kill failover integration test (03/S1) | ✅ completed — `a25c9ed0`; surfaced task #11 |
| 10 | **Wire CapabilityInvoker into the dispatch layer** (RESILIENCE-DISPATCH-GAP, from 07/C-1) | ✅ code-complete + unit/analyzer/context-verified — `fix/archreview-crit10-wire-capability-invoker` `62556c24` (seam, NOT direct thread — Runtime is Polly-free). **Live behavioral gate pending** (rig recipe in FOLLOWUP-10). Surfaced a sub-gap: ResilienceConfig not in the deploy artifact → tier-defaults only. |
| 11 | **Review Critical 1 premise** — SBR was already active on master, not NoDowning (from #9) | ✅ completed — `fix/archreview-crit1-split-brain-resolver` `eaf78aad`; docs/comment-only correction (no code revert), Cluster.Tests 29/29. |
| 12 | **#10 live behavioral gate** — rig session (from #10) | ✅ ran 2026-07-08 (PARTIAL) — built the Host image from source (crit13 tip) + stood up the rig; central-1 booted healthy; **prod DI binds the real invoker factory** confirmed live + by a new deterministic guard (`ResilienceInvokerFactoryRegistrationTests` `8a8b9ec5`). Behavioral retry/breaker **log-line not observed** (disciplined stop): Modbus's cheap paths don't throw a wrapped exception (Discover offline, Subscribe lazy, reads non-wrapped) — only an idempotent write-to-dead-endpoint (or S7) does, not worth the multi-dependency chase since the retry behavior is deterministically proven. **Found + FIXED a rig bug:** docker-dev compose's retired Wonderware `ServerHistorian` keys crashed the Host at bring-up (historian-gateway cutover → reads `Endpoint`) → fixed on `fix/docker-dev-serverhistorian-stale-keys` `7233e2ba` (off master; gateway-shape keys, disabled by default; central-1 verified booting clean). See [`FOLLOWUP-10`](FOLLOWUP-10-resilience-dispatch-gap.md). |
| 13 | **Plumb per-instance `ResilienceConfig` through the deploy artifact** (from #10) | ✅ completed — `fix/archreview-crit13-resilience-config-artifact` `75403caa`; read-path plumbing (write side was already complete), invalidate-on-change + respawn-on-change; Core.Tests 243/Runtime.Tests 363. See [`FOLLOWUP-13`](FOLLOWUP-13-resilience-config-artifact.md). |
## Findings surfaced by the guards (NEW — need user decision)
> Each has a dedicated follow-up file with the full design/action:
> [`FOLLOWUP-10-resilience-dispatch-gap.md`](FOLLOWUP-10-resilience-dispatch-gap.md) ·
> [`FOLLOWUP-11-crit1-sbr-premise.md`](FOLLOWUP-11-crit1-sbr-premise.md) ·
> [`FOLLOWUP-13-resilience-config-artifact.md`](FOLLOWUP-13-resilience-config-artifact.md).
- **#10 — RESILIENCE-DISPATCH-GAP (from wiring the analyzer, 07/C-1): ✅ IMPLEMENTED** on
`fix/archreview-crit10-wire-capability-invoker` `62556c24` — code-complete + unit/analyzer/context-verified;
**live behavioral gate pending** (recipe in [`FOLLOWUP-10`](FOLLOWUP-10-resilience-dispatch-gap.md)). The filed
"thread CapabilityInvoker into the actor" was infeasible (Runtime is deliberately Polly-free); shipped a seam
mirroring `IDriverFactory` instead. Only **6** of the 7 sites were real (the 7th, `GenericDriverNodeManager`, is
test-only scaffolding). **Two residual sub-findings:** (a) ~~per-instance `ResilienceConfig` isn't in the deploy
artifact → tier-defaults only~~ **✅ RESOLVED by task #13** (`75403caa`) — the column was already in the artifact;
the read path now applies it (see [`FOLLOWUP-13`](FOLLOWUP-13-resilience-config-artifact.md)); (b) the
resilience-status tracker has **no reader** (Admin `/hosts` Stream E.2/E.3 never shipped) — added retry/breaker
**logging** as the interim observability surface + the live-verify enabler (still open as an Admin-UI follow-on).
- **#11 — Critical 1 premise inaccurate (from the #9 failover negative control): ✅ CORRECTED** on
`fix/archreview-crit1-split-brain-resolver` `eaf78aad`. SBR was **already active on master**
(Akka.Cluster.Hosting's `WithClustering` applies `SplitBrainResolverOption.Default` when the typed option is
null → enables the downing provider that reads the pre-existing akka.conf `keep-oldest` block). So the cluster
was **not** running NoDowning; hard-crash failover already worked. Docs/comment-only fix (no code revert — the
typed `KeepOldestOption` is correct belt-and-suspenders): corrected the "HOCON inert / NoDowning → no failover"
framing in `BuildClusterOptions` XML comment, `akka.conf`, `docs/Redundancy.md`, and the
`SplitBrainResolverActivationTests` summary/message (+ method rename). Cluster.Tests 29/29. (Architecture.md
carried no such framing.) See [`FOLLOWUP-11`](FOLLOWUP-11-crit1-sbr-premise.md).
## Suggested next (from 00-INDEX execution order)
**All 4 Criticals + all 3 systemic guards (tasks 6/7/8) + the #9 failover test are DONE.** Remaining:
1. **Resolve the two surfaced findings** — task 10 (wire CapabilityInvoker into dispatch; behavior-changing,
live-gate) and task 11 (correct Crit-1's rationale/docs).
2. **Failure-visibility Highs** — 01/S-1 (AddressSpaceApplyOutcome failure field), 06/S-1 (Galaxy fail-closed),
03/S4 (primary-gate default-deny).
3. **P-tier** — 03/P1 surgical adds/removals (now gated by the U2 forwarding guard), 02/P batched `WriteValues`.
## Key resume facts
- **Verification bar met so far:** each fix has a deterministic unit guard AND (for SBR) an end-to-end
config-resolution check. Follow the same pattern — the review's whole thesis is "unit-green ≠ wired".
- **The forwarding-test guard (task 6) is DONE (`a65c2ced`):** 03/P1 (surgical pure-adds) and 02/P-… batch-write
each add new sink capability methods that ship inert unless forwarded through `DeferredAddressSpaceSink`
(F10b/PR#423 precedent). The reflection guard now fails loudly if any such member isn't forwarded — add the
member to the recorder-covered interface set and it's auto-checked.
- ~~**Nothing is pushed.**~~ **Superseded 2026-07-12:** all branches merged to master and pushed to
`origin/master`; the plans now live on master too (no more cross-branch `git show` needed).
- ~~**CI changes can't be run-verified without a push**~~ — now pushed; the first real CI-job run should be
checked on origin (the re-review verified the YAML statically only).
- **Findings #10 and #11 are resolved** (see above); the open decisions now come from the 2026-07-12
re-review below.
## Re-review 2026-07-12 (master `f6eaa267`) — outcome + new findings
All seven domain reports were re-run and rewritten in place at `f6eaa267` (each carries a
`Prior-finding status (9cad9ed0 → f6eaa267)` table; synthesis in [`../00-OVERALL.md`](../00-OVERALL.md)).
Verdict on the remediation: **all 4 Criticals verifiably closed in code** (Crit-1 by premise correction),
both systemic guards live with negative controls, CI now whole-solution + fail-on-skip
(07 stability 2.5→3.5). New findings needing scheduling, ranked:
| Rank | Finding | Severity | One-line |
|---|---|---|---|
| 1 | 05/STAB-14 | **High (regression)** | S7 ensure-wrapper rethrows connect-timeout OCE → poll loop's bare `catch (OCE) { return; }` silently kills all S7 subscriptions after an unreachable-host (non-refusing) outage; live bounce test blind to it (`S7Driver.cs:488,1000,1383,1404,1206-1208`) |
| 2 | 01/S-6 | **High** | Operator-authorable `timeoutSeconds:0` in ResilienceConfig → Polly ValidationException on every capability call = permanent driver-brick; no range validation in parser or AdminUI form |
| 3 | 05/STAB-15 | **High** | S7 fatal-classification misses framing/desync + cancel-during-I/O (half-read PDU → permanent desync); probe catch never marks handle dead |
| 4 | 01/S-8 = 03/S12 | Medium | Dispatch hardcodes `isIdempotent:true`; `WriteIdempotentAttribute` has zero readers → #13-enabled write retries would retry command writes |
| 5 | 02/S13 | Medium | VT script failure/timeout leaves node stale-Good forever (timeout fix made the failure invisible) |
| 6 | 01/U-6 | Medium | Bulkhead knob parsed/authored/documented but never built into the pipeline — dead knob |
| 7 | 06/S-11 | Medium | Enabled-but-misconfigured ServerHistorian crash-loops Host with raw UriFormatException (only config side fixed in 7233e2ba) |
| 8 | 06/U-7 | Medium | HistorianGateway 0.2.0 proto3-optional DataType change silently alters GatewayTagProvisioner Boolean provisioning; live EnsureTags never run on 0.2.0 |
| 9 | 04/C-7, 04/U-5=03/U10, 01/S-7=03/S13 | Medium/Low | Resilience seam edges: lossy AdminUI form round-trip; status tracker has no reader (operators blind); respawn stale-pipeline cache race |
Carried-open Highs unchanged from round 1 (no branch targeted them): failure-visibility trio
(01/S-1, 06/S-1, 03/S4), AdminUI authz (04/C-1 — stakes rose: resilience config is on the ungated
surface), 03/S2/S3 block-bridging, 03/P1 surgical adds (guard prerequisite in place), driver-fleet batch
(05), repo hygiene (07). Full prioritized list in `00-OVERALL.md`.
**Round-2 plans WRITTEN (2026-07-12):** all 12 action items have design+implementation plans —
`R2-01``R2-12` in this directory (+ co-located `.tasks.json` each, for resumable execution). Index,
per-plan task counts, effort, and suggested execution order: [`00-INDEX.md`](00-INDEX.md) §"Round 2
plans". 193 bite-sized TDD tasks total, ~1824 dev-days end-to-end; R2-01/02/03 (the re-review's
new-and-sharp items) are ~2 days combined.
### Round-2 remediation status
- **R2-03 (02/S13 + 02/S12 + 02/P7) — FIXED ON BRANCH `r2/03-vt-failure-quality`** (off master
`1676c8f4`). All three findings closed with TDD:
- **02/S13** (Medium) — VT script failure/timeout now degrades the OPC UA node to `OpcUaQuality.Bad`
(carrying the last-known value) instead of freezing on the last Good value forever. `EvaluationResult`
gained an additive `Quality` field; the host bridge publishes it and historizes `0x80020000u`
(BadInternalError, dormant-engine parity) on Bad. Bad is published once per Good→Bad transition, gated
on all declared dependencies having arrived (inputs-ready gate — no cold-start/respawn Bad flash), and
recovery force-publishes the next Good even at the pre-failure value.
- **02/S12** (Medium) — the apply-boundary `ClearCompiledScripts` racing an in-flight evaluation into
`ObjectDisposedException` is now retried once in `RoslynVirtualTagEvaluator` (re-fetch → recompile into
the current cache generation). A concurrent-clear stress guard reproduces the pre-fix disposed failures
and is deterministically green post-fix.
- **02/P7** (Low) — the apply-boundary clear is gated on a changed deployed *expression set*
(`HashSet.SetEquals`, ordinal), so a byte-identical redeploy — or a rename/add/remove that leaves the
expression set unchanged — no longer forces full VT recompilation.
- Verified locally: `dotnet build ZB.MOM.WW.OtOpcUa.slnx` clean (0 errors); whole `Runtime.Tests`
project 372/372 green; `RoslynVirtualTagEvaluatorTests` 20/20 green. **Deferred to the serialized
heavy pass:** the full cross-suite sweep and the docker-dev live-`/run` gate (fail→Bad→recover +
P7 no-recompile spot-check).