175 lines
25 KiB
Markdown
175 lines
25 KiB
Markdown
# Arch-Review Remediation — Status & Handoff
|
||
|
||
**Snapshot:** 2026-07-12 · **Review baseline:** master `9cad9ed0`; **re-review baseline:** master
|
||
`f6eaa267` · this file is the single source of truth for remediation progress. Update it as work lands.
|
||
|
||
> **2026-07-12 UPDATE — everything below the topology table landed:** all remediation branches were
|
||
> **merged to master and pushed to `origin/master`** (merge commits `7834b3b7`…`b67bd9e8`; final state
|
||
> `f6eaa267`). The topology table is kept for history but is no longer live. A full **re-review** of all
|
||
> seven domains ran at `f6eaa267` — see [`../00-OVERALL.md`](../00-OVERALL.md) and the
|
||
> "Re-review 2026-07-12" section at the bottom of this file for the new findings (one High regression
|
||
> in the S7 fix, one High ResilienceConfig validation gap, and a batch of resilience-seam Mediums).
|
||
|
||
## Branch topology (HISTORICAL — as of 2026-07-08; since merged + pushed)
|
||
|
||
| Branch | Head | Contains |
|
||
|---|---|---|
|
||
| `master` | `9cad9ed0` | untouched baseline |
|
||
| `docs/archreview-plans-and-drift-fixes` | `9fadead6` | the 7 domain plans + `00-INDEX.md` + this `STATUS.md` + the two CLAUDE.md doc-drift fixes. **(the plans live ONLY here)** |
|
||
| `fix/archreview-crit1-split-brain-resolver` | `eaf78aad` | Critical 1 (03/S1) typed SBR option + guard test + Redundancy.md; **+ #9 hard-kill failover test** (`a25c9ed0`); **+ #11 premise correction** (`eaf78aad`) |
|
||
| `fix/archreview-crit2-vt-timeout` | `7fd44f0f` | Critical 2 (02/U2 + 02/U3) + tests |
|
||
| `fix/archreview-crit3-s7-reconnect` | `25c0c6f6` | Critical 3 (05/STAB-1) IS7Plc seam + lazy reconnect + unit guards + live bounce test |
|
||
| `fix/archreview-crit4-twincat-replay` | `af318fb4` | Critical 4 (05/STAB-2) native-notification replay + probe-recycle + unit guards |
|
||
| `fix/archreview-u2-deferred-sink-forwarding-guard` | `a65c2ced` | Guard 03/U2 — reflection-exhaustive deferred-sink forwarding test (+ DeferredServiceLevelPublisher) |
|
||
| `fix/archreview-c1-wire-analyzer` | `f0082af5` | Guard 07/C-1 — wire OTOPCUA0001 analyzer tree-wide + triage (surfaced the RESILIENCE-DISPATCH-GAP, task #10) |
|
||
| `fix/archreview-s1-s2-ci-coverage` | `10b89830` | Guard 07/S-1,S-2,S-4 — whole-solution CI leg + fail-on-skip gate + CLI sleep deflake |
|
||
| `fix/archreview-crit10-wire-capability-invoker` | `62556c24` | **#10 — wire CapabilityInvoker into dispatch via a Core.Abstractions seam** (off `fix/archreview-c1-wire-analyzer`, NOT master — the analyzer is the standing guard + the pragmas live there). Code-complete + unit/analyzer/context-verified; live gate pending. |
|
||
| `fix/archreview-crit13-resilience-config-artifact` | `8a8b9ec5` | **#13 — apply per-instance ResilienceConfig from the artifact** (`75403caa`, off the crit10 branch) **+ #10/#13 DI guard** (`8a8b9ec5` — `ResilienceInvokerFactoryRegistrationTests`, from the #12 live gate). Read-path plumbing + invalidate/respawn-on-change; deterministically verified + rig-booted. |
|
||
| `fix/docker-dev-serverhistorian-stale-keys` | `7233e2ba` | **Rig fix (off master, independent):** docker-dev compose's retired Wonderware `ServerHistorian` keys → gateway shape, disabled by default. Found during the #12 live-gate rig session; central-1 verified booting clean. |
|
||
|
||
**Decision (user):** one branch per Critical, each off `master`. **#9 (failover test) lives on the crit1 branch**
|
||
(not its own branch off master) because SBR only exists there — it is Critical 1's integration half. The plans are only on the docs branch,
|
||
so from a critical branch read them via `git show docs/archreview-plans-and-drift-fixes:archreview/plans/0X-<name>.md`.
|
||
|
||
## Completed
|
||
|
||
| Item | Finding | Branch @ commit | Verification |
|
||
|---|---|---|---|
|
||
| **Critical 1 — split-brain resolver activated** | 03/S1 | `fix/archreview-crit1-split-brain-resolver` `a81dea10` | `BuildClusterOptions` sets `KeepOldestOption{DownIfAlone=true}`; 3 unit guards (Cluster.Tests 29/29); **e2e-verified** the resolved ActorSystem config carries `SplitBrainResolverProvider` + `stable-after=15s` survives the HOCON/Hosting merge (throwaway spike, removed). HOCON cross-ref comment + Redundancy.md corrected. |
|
||
| **Critical 2a — real VT script timeout** | 02/U2 | `fix/archreview-crit2-vt-timeout` `7fd44f0f` | Routes through `TimedScriptEvaluator`; infinite-loop test (bounded by WaitAsync) proves no hang. Host.Integration VT 18/18. |
|
||
| **Critical 2b — ALC-safe compile cache** | 02/U3 | same `7fd44f0f` | Swapped raw dict → `CompiledScriptCache`; new `IScriptCacheOwner` seam; apply-boundary clear in `VirtualTagHostActor`; wiring guard test. Runtime VT 12/12. |
|
||
| **Doc drift — KNOWN LIMITATION 2 stale** | 03/U1 = 06/U-1 | `docs/…` `9fadead6` (CLAUDE.md) | historized-ref feed IS wired; rewrote to "value-capture code-complete, live-verify pending". |
|
||
| **Doc drift — ScriptAnalysis policy** | theme #5 | same `9fadead6` (CLAUDE.md) | corrected `FleetAdmin` → `Administrator,Designer` roles. |
|
||
| **Critical 3 — S7 lazy reconnect** | 05/STAB-1 | `fix/archreview-crit3-s7-reconnect` `25c0c6f6` | New `IS7Plc`/`IS7PlcFactory` seam + `EnsureConnectedAsync` (dispose-dead-handle → reopen on next data call); connection-fatal faults (socket/`ErrorCode.ConnectionError`) mark the handle dead, data-address errors do NOT; probe loop routes through it as a backstop. 4 unit guards + full S7 suite 234/234; **LIVE-verified** against real python-snap7 — bounced the container (`docker restart`), driver observed the outage and resumed Good reads with no redeploy (`S7_1500ReconnectTests`, double-gated on sim + `S7_RECONNECT_BOUNCE_CMD`). Also closes the S7 TEST-1 gap. |
|
||
| **Critical 4 — TwinCAT native-notification replay** | 05/STAB-2 | `fix/archreview-crit4-twincat-replay` `af318fb4` | Native ADS notifications were opaque handles (no replayable intent) → silently orphaned on a client swap. Now `NativeRegistration` (symbol/type/bit/interval/onChange + swappable handle) hangs off `DeviceState`; `EnsureConnectedAsync` split into a gate wrapper + under-gate core that **replays every intent onto a freshly-built client** (register+replay both under `ConnectGate`); probe-failure `RecycleClientAsync` disposes+nulls the wire-dead-but-port-open client so the next tick rebuilds+replays. 4 unit guards + full TwinCAT suite 174/174. **Unit-only — no TwinCAT docker fixture exists** (integration needs a real TC3 XAR), so the fake-client tests are authoritative (no live gate like S7 had). |
|
||
| **Guard — reflection-exhaustive deferred-sink forwarding** | 03/U2 | `fix/archreview-u2-deferred-sink-forwarding-guard` `a65c2ced` | New `DeferredSinkForwardingReflectionTests` (Commons.Tests) reflects over **every** method of every forwarding interface `DeferredAddressSpaceSink` implements (`IOpcUaAddressSpaceSink` + `ISurgicalAddressSpaceSink`) — plus the sibling `DeferredServiceLevelPublisher` — invokes each on the wrapper over a `DispatchProxy` recording inner, and asserts the call **reached** the inner. A `DispatchProxy` recorder auto-implements future members (zero maintenance); a guard-of-the-guard asserts the wrapper implements exactly the known forwarding-interface set, so a brand-new capability *interface* trips too. **This closes the F10b/PR#423 trap class and is the prerequisite gate for 03/P1 remove methods + 02/P `WriteValues`.** 3 new tests green (Commons.Tests 57/57); **negative control verified** — stubbing `UpdateFolderDisplayName` to a non-forwarding no-op fails the guard with the exact member-named diagnostic. NO production change. |
|
||
| **Guard — wire OTOPCUA0001 analyzer + triage** | 07/C-1 | `fix/archreview-c1-wire-analyzer` `f0082af5` | Injected the analyzer tree-wide via `Directory.Build.props` (`OutputItemType="Analyzer"`), then triaged ~280 surfaced hits: **(1) RESILIENCE-DISPATCH-GAP** — 7 production sites (`DriverInstanceActor` ×6 + `GenericDriverNodeManager`) scoped-`#pragma`'d as a **real tracked gap** (CapabilityInvoker built but never wired into dispatch; **new task #10**); **(2) driver-internal self-calls** — 3 sites (AbCip/S7 own poll/ack) `#pragma`'d as intentional; **(3) 12 wire-level test/CLI projects** → commented `NoWarn`. Full build green, 0 hits; analyzer 31/31; **negative control** — dropping one dispatch-gap pragma re-fires OTOPCUA0001 and fails the build, proving it's live tree-wide. |
|
||
| **Guard — CI to all suites + fail-on-skip + deflake** | 07/S-1,S-2,S-4 | `fix/archreview-s1-s2-ci-coverage` `10b89830` | S-1: replaced the 5-project `unit-tests` matrix with ONE whole-solution leg (self-maintaining; now covers Client's 388 + Analyzers' 31 + all drivers). S-2: `scripts/ci/assert-not-all-skipped.sh` fail-on-skip gate (turns "green==all-skipped" red) — strict on the unit leg, report-only on the fixtureless integration leg. S-4: readiness-signal (`SubscribeInvoked`/`SubscribeAlarmsInvoked`) replaces 11 fixed-sleep startup races in the newly-CI'd Client.CLI.Tests. Skip-gate proven locally (real trx OK / all-skip trx exit 1 / report-only / edge cases); Client.CLI.Tests 104/104. **CI job execution itself is verifiable only on push (nothing pushed).** |
|
||
| **#10 — wire CapabilityInvoker into dispatch (seam)** | RESILIENCE-DISPATCH-GAP (07/C-1) | `fix/archreview-crit10-wire-capability-invoker` `62556c24` | **Discovered the filed plan was infeasible** — Runtime is deliberately Polly-free (refs `Core.Abstractions`, not `Core`). Fix = a seam mirroring `IDriverFactory`: `IDriverCapabilityInvoker`(+null) + `IDriverCapabilityInvokerFactory`(+null) in Core.Abstractions; `CapabilityInvoker` implements it; `DriverCapabilityInvokerFactory` (Core) builds per-instance over the singleton builder+tracker+`GetTier`; Host `DriverFactoryBootstrap` registers all three; Runtime SCE resolves like `IDriverFactory`; `DriverInstanceActor` wraps its **6** dispatch sites; analyzer taught the interface as a wrapper home; all 6 pragmas removed. Triage corrections: the 7th site (`GenericDriverNodeManager`) is test-only scaffolding (not wired, re-annotated); **tier-defaults only** (ResilienceConfig not in the artifact → sub-gap). Added retry/breaker **logging** to the pipeline builder (the tracker has no reader — Stream E.2/E.3 never shipped — so it ran invisibly). **Verified:** negative-control (unwrap→OTOPCUA0001 build error); recording-invoker routing test; **real-`CapabilityInvoker` actor-context test** (internal `ConfigureAwait(false)` doesn't leak); analyzer + logging tests; clean solution build; Runtime.Tests 358 / Core.Tests 238 / Analyzers 32. **Live behavioral gate pending** (recipe in FOLLOWUP-10). |
|
||
| **#9 — hard-kill failover integration test** | 03/S1 | `fix/archreview-crit1-split-brain-resolver` `a25c9ed0` | Critical 1's load-bearing integration half. `TwoNodeClusterHarness.HardKillNodeAAsync` (transport-shutdown crash sim → node B sees A Unreachable, no graceful Leave) + `HardKillFailoverTests` asserting B downs the alone oldest node and takes over as `driver` role-leader. **LIVE-verified in-process** (~35s); **negative control** (force `downing-provider-class=""`) makes it correctly time out. ⚠️ The negative control also surfaced a **Crit-1 premise finding (task #11): SBR was already active on master** (Akka.Cluster.Hosting applies `SplitBrainResolverOption.Default` when the typed option is null → reads the pre-existing akka.conf keep-oldest block), so the cluster was NOT NoDowning before Crit-1 — the typed option is reinforcing, not the activator. |
|
||
| **#11 — Critical 1 premise correction** | 03/S1 (from #9) | `fix/archreview-crit1-split-brain-resolver` `eaf78aad` | Docs/comment-only fix (no code revert): corrected the "HOCON inert / cluster runs NoDowning / never fails over" framing in `BuildClusterOptions` XML comment, `akka.conf` split-brain-resolver comment, `docs/Redundancy.md`, and `SplitBrainResolverActivationTests` (summary + `ShouldNotBeNull` message + method rename). Akka.Cluster.Hosting enables an SBR provider **by default** (applies `SplitBrainResolverOption.Default` when the typed option is null → reads the pre-existing akka.conf `keep-oldest` block), so the cluster was never NoDowning and the typed `KeepOldestOption` is reinforcing/explicit-in-code, not the activator. Cluster.Tests 29/29; failover behaviour still guarded by `HardKillFailoverTests`. Architecture.md carried no such framing. |
|
||
| **#13 — apply per-instance ResilienceConfig from the artifact** | RESILIENCE-DISPATCH-GAP sub-gap (from #10) | `fix/archreview-crit13-resilience-config-artifact` `75403caa` | The `DriverInstance.ResilienceConfig` column was authored+persisted (AdminUI) and already in the artifact (the composer serializes the whole entity), but the driver-node read path dropped it → tier-defaults only. `DriverInstanceSpec` now carries it; `TryReadSpec` reads it; `Create` takes+layers it (+ logs parse diagnostics, never throws); `SpawnChild` threads it. **Invalidate-on-change:** `Create` calls `builder.Invalidate(instance)` (the cache is options-blind on a hit) so a respawn with changed options rebuilds; `DriverSpawnPlanner` respawns on a ResilienceConfig change (incl. null→json), a pure DriverConfig change stays an in-place delta. **Verified:** override reaches execution (retryCount:0 suppresses tier-A retry; a control retries), invalidate is per-instance, malformed logs+falls-back; planner+artifact tests. Core.Tests 243 (+5), Runtime.Tests 363 (+5). See [`FOLLOWUP-13`](FOLLOWUP-13-resilience-config-artifact.md). |
|
||
|
||
## Task list (TaskCreate IDs)
|
||
|
||
| # | Task | Status |
|
||
|---|---|---|
|
||
| 1 | Critical 1 — split-brain resolver | ✅ completed |
|
||
| 2 | Critical 2a — VT timeout | ✅ completed |
|
||
| 3 | Critical 2b — CompiledScriptCache | ✅ completed |
|
||
| 4 | Critical 3 — S7 reconnect (05/STAB-1) | ✅ completed |
|
||
| 5 | Critical 4 — TwinCAT ADS re-registration (05/STAB-2) | ✅ completed |
|
||
| 6 | Guard — DeferredAddressSpaceSink reflection forwarding test (03/U2) | ✅ completed — `a65c2ced`; gate for 03/P1 + 02/P now in place |
|
||
| 7 | Guard — wire OTOPCUA0001 analyzer + triage (07/C-1) | ✅ completed — `f0082af5`; surfaced task #10 |
|
||
| 8 | Guard — CI to all unit suites + fail-on-skip (07/S-1,S-2, pair S-4) | ✅ completed — `10b89830` |
|
||
| 9 | Critical 1 follow-up — hard-kill failover integration test (03/S1) | ✅ completed — `a25c9ed0`; surfaced task #11 |
|
||
| 10 | **Wire CapabilityInvoker into the dispatch layer** (RESILIENCE-DISPATCH-GAP, from 07/C-1) | ✅ code-complete + unit/analyzer/context-verified — `fix/archreview-crit10-wire-capability-invoker` `62556c24` (seam, NOT direct thread — Runtime is Polly-free). **Live behavioral gate pending** (rig recipe in FOLLOWUP-10). Surfaced a sub-gap: ResilienceConfig not in the deploy artifact → tier-defaults only. |
|
||
| 11 | **Review Critical 1 premise** — SBR was already active on master, not NoDowning (from #9) | ✅ completed — `fix/archreview-crit1-split-brain-resolver` `eaf78aad`; docs/comment-only correction (no code revert), Cluster.Tests 29/29. |
|
||
| 12 | **#10 live behavioral gate** — rig session (from #10) | ✅ ran 2026-07-08 (PARTIAL) — built the Host image from source (crit13 tip) + stood up the rig; central-1 booted healthy; **prod DI binds the real invoker factory** confirmed live + by a new deterministic guard (`ResilienceInvokerFactoryRegistrationTests` `8a8b9ec5`). Behavioral retry/breaker **log-line not observed** (disciplined stop): Modbus's cheap paths don't throw a wrapped exception (Discover offline, Subscribe lazy, reads non-wrapped) — only an idempotent write-to-dead-endpoint (or S7) does, not worth the multi-dependency chase since the retry behavior is deterministically proven. **Found + FIXED a rig bug:** docker-dev compose's retired Wonderware `ServerHistorian` keys crashed the Host at bring-up (historian-gateway cutover → reads `Endpoint`) → fixed on `fix/docker-dev-serverhistorian-stale-keys` `7233e2ba` (off master; gateway-shape keys, disabled by default; central-1 verified booting clean). See [`FOLLOWUP-10`](FOLLOWUP-10-resilience-dispatch-gap.md). |
|
||
| 13 | **Plumb per-instance `ResilienceConfig` through the deploy artifact** (from #10) | ✅ completed — `fix/archreview-crit13-resilience-config-artifact` `75403caa`; read-path plumbing (write side was already complete), invalidate-on-change + respawn-on-change; Core.Tests 243/Runtime.Tests 363. See [`FOLLOWUP-13`](FOLLOWUP-13-resilience-config-artifact.md). |
|
||
|
||
## Findings surfaced by the guards (NEW — need user decision)
|
||
|
||
> Each has a dedicated follow-up file with the full design/action:
|
||
> [`FOLLOWUP-10-resilience-dispatch-gap.md`](FOLLOWUP-10-resilience-dispatch-gap.md) ·
|
||
> [`FOLLOWUP-11-crit1-sbr-premise.md`](FOLLOWUP-11-crit1-sbr-premise.md) ·
|
||
> [`FOLLOWUP-13-resilience-config-artifact.md`](FOLLOWUP-13-resilience-config-artifact.md).
|
||
|
||
- **#10 — RESILIENCE-DISPATCH-GAP (from wiring the analyzer, 07/C-1): ✅ IMPLEMENTED** on
|
||
`fix/archreview-crit10-wire-capability-invoker` `62556c24` — code-complete + unit/analyzer/context-verified;
|
||
**live behavioral gate pending** (recipe in [`FOLLOWUP-10`](FOLLOWUP-10-resilience-dispatch-gap.md)). The filed
|
||
"thread CapabilityInvoker into the actor" was infeasible (Runtime is deliberately Polly-free); shipped a seam
|
||
mirroring `IDriverFactory` instead. Only **6** of the 7 sites were real (the 7th, `GenericDriverNodeManager`, is
|
||
test-only scaffolding). **Two residual sub-findings:** (a) ~~per-instance `ResilienceConfig` isn't in the deploy
|
||
artifact → tier-defaults only~~ **✅ RESOLVED by task #13** (`75403caa`) — the column was already in the artifact;
|
||
the read path now applies it (see [`FOLLOWUP-13`](FOLLOWUP-13-resilience-config-artifact.md)); (b) the
|
||
resilience-status tracker has **no reader** (Admin `/hosts` Stream E.2/E.3 never shipped) — added retry/breaker
|
||
**logging** as the interim observability surface + the live-verify enabler (still open as an Admin-UI follow-on).
|
||
- **#11 — Critical 1 premise inaccurate (from the #9 failover negative control): ✅ CORRECTED** on
|
||
`fix/archreview-crit1-split-brain-resolver` `eaf78aad`. SBR was **already active on master**
|
||
(Akka.Cluster.Hosting's `WithClustering` applies `SplitBrainResolverOption.Default` when the typed option is
|
||
null → enables the downing provider that reads the pre-existing akka.conf `keep-oldest` block). So the cluster
|
||
was **not** running NoDowning; hard-crash failover already worked. Docs/comment-only fix (no code revert — the
|
||
typed `KeepOldestOption` is correct belt-and-suspenders): corrected the "HOCON inert / NoDowning → no failover"
|
||
framing in `BuildClusterOptions` XML comment, `akka.conf`, `docs/Redundancy.md`, and the
|
||
`SplitBrainResolverActivationTests` summary/message (+ method rename). Cluster.Tests 29/29. (Architecture.md
|
||
carried no such framing.) See [`FOLLOWUP-11`](FOLLOWUP-11-crit1-sbr-premise.md).
|
||
|
||
## Suggested next (from 00-INDEX execution order)
|
||
|
||
**All 4 Criticals + all 3 systemic guards (tasks 6/7/8) + the #9 failover test are DONE.** Remaining:
|
||
|
||
1. **Resolve the two surfaced findings** — task 10 (wire CapabilityInvoker into dispatch; behavior-changing,
|
||
live-gate) and task 11 (correct Crit-1's rationale/docs).
|
||
2. **Failure-visibility Highs** — 01/S-1 (AddressSpaceApplyOutcome failure field), 06/S-1 (Galaxy fail-closed),
|
||
03/S4 (primary-gate default-deny).
|
||
3. **P-tier** — 03/P1 surgical adds/removals (now gated by the U2 forwarding guard), 02/P batched `WriteValues`.
|
||
|
||
## Key resume facts
|
||
|
||
- **Verification bar met so far:** each fix has a deterministic unit guard AND (for SBR) an end-to-end
|
||
config-resolution check. Follow the same pattern — the review's whole thesis is "unit-green ≠ wired".
|
||
- **The forwarding-test guard (task 6) is DONE (`a65c2ced`):** 03/P1 (surgical pure-adds) and 02/P-… batch-write
|
||
each add new sink capability methods that ship inert unless forwarded through `DeferredAddressSpaceSink`
|
||
(F10b/PR#423 precedent). The reflection guard now fails loudly if any such member isn't forwarded — add the
|
||
member to the recorder-covered interface set and it's auto-checked.
|
||
- ~~**Nothing is pushed.**~~ **Superseded 2026-07-12:** all branches merged to master and pushed to
|
||
`origin/master`; the plans now live on master too (no more cross-branch `git show` needed).
|
||
- ~~**CI changes can't be run-verified without a push**~~ — now pushed; the first real CI-job run should be
|
||
checked on origin (the re-review verified the YAML statically only).
|
||
- **Findings #10 and #11 are resolved** (see above); the open decisions now come from the 2026-07-12
|
||
re-review below.
|
||
|
||
## Re-review 2026-07-12 (master `f6eaa267`) — outcome + new findings
|
||
|
||
All seven domain reports were re-run and rewritten in place at `f6eaa267` (each carries a
|
||
`Prior-finding status (9cad9ed0 → f6eaa267)` table; synthesis in [`../00-OVERALL.md`](../00-OVERALL.md)).
|
||
Verdict on the remediation: **all 4 Criticals verifiably closed in code** (Crit-1 by premise correction),
|
||
both systemic guards live with negative controls, CI now whole-solution + fail-on-skip
|
||
(07 stability 2.5→3.5). New findings needing scheduling, ranked:
|
||
|
||
| Rank | Finding | Severity | One-line |
|
||
|---|---|---|---|
|
||
| 1 | 05/STAB-14 | **High (regression)** | S7 ensure-wrapper rethrows connect-timeout OCE → poll loop's bare `catch (OCE) { return; }` silently kills all S7 subscriptions after an unreachable-host (non-refusing) outage; live bounce test blind to it (`S7Driver.cs:488,1000,1383,1404,1206-1208`) |
|
||
| 2 | 01/S-6 | **High** | Operator-authorable `timeoutSeconds:0` in ResilienceConfig → Polly ValidationException on every capability call = permanent driver-brick; no range validation in parser or AdminUI form |
|
||
| 3 | 05/STAB-15 | **High** | S7 fatal-classification misses framing/desync + cancel-during-I/O (half-read PDU → permanent desync); probe catch never marks handle dead |
|
||
| 4 | 01/S-8 = 03/S12 | Medium | Dispatch hardcodes `isIdempotent:true`; `WriteIdempotentAttribute` has zero readers → #13-enabled write retries would retry command writes |
|
||
| 5 | 02/S13 | Medium | VT script failure/timeout leaves node stale-Good forever (timeout fix made the failure invisible) |
|
||
| 6 | 01/U-6 | Medium | Bulkhead knob parsed/authored/documented but never built into the pipeline — dead knob |
|
||
| 7 | 06/S-11 | Medium | Enabled-but-misconfigured ServerHistorian crash-loops Host with raw UriFormatException (only config side fixed in 7233e2ba) |
|
||
| 8 | 06/U-7 | Medium | HistorianGateway 0.2.0 proto3-optional DataType change silently alters GatewayTagProvisioner Boolean provisioning; live EnsureTags never run on 0.2.0 |
|
||
| 9 | 04/C-7, 04/U-5=03/U10, 01/S-7=03/S13 | Medium/Low | Resilience seam edges: lossy AdminUI form round-trip; status tracker has no reader (operators blind); respawn stale-pipeline cache race |
|
||
|
||
Carried-open Highs unchanged from round 1 (no branch targeted them): failure-visibility trio
|
||
(01/S-1, 06/S-1, 03/S4), AdminUI authz (04/C-1 — stakes rose: resilience config is on the ungated
|
||
surface), 03/S2/S3 block-bridging, 03/P1 surgical adds (guard prerequisite in place), driver-fleet batch
|
||
(05), repo hygiene (07). Full prioritized list in `00-OVERALL.md`.
|
||
|
||
**Round-2 plans WRITTEN (2026-07-12):** all 12 action items have design+implementation plans —
|
||
`R2-01`…`R2-12` in this directory (+ co-located `.tasks.json` each, for resumable execution). Index,
|
||
per-plan task counts, effort, and suggested execution order: [`00-INDEX.md`](00-INDEX.md) §"Round 2
|
||
plans". 193 bite-sized TDD tasks total, ~18–24 dev-days end-to-end; R2-01/02/03 (the re-review's
|
||
new-and-sharp items) are ~2 days combined.
|
||
|
||
### Round-2 remediation status
|
||
|
||
- **R2-03 (02/S13 + 02/S12 + 02/P7) — FIXED ON BRANCH `r2/03-vt-failure-quality`** (off master
|
||
`1676c8f4`). All three findings closed with TDD:
|
||
- **02/S13** (Medium) — VT script failure/timeout now degrades the OPC UA node to `OpcUaQuality.Bad`
|
||
(carrying the last-known value) instead of freezing on the last Good value forever. `EvaluationResult`
|
||
gained an additive `Quality` field; the host bridge publishes it and historizes `0x80020000u`
|
||
(BadInternalError, dormant-engine parity) on Bad. Bad is published once per Good→Bad transition, gated
|
||
on all declared dependencies having arrived (inputs-ready gate — no cold-start/respawn Bad flash), and
|
||
recovery force-publishes the next Good even at the pre-failure value.
|
||
- **02/S12** (Medium) — the apply-boundary `ClearCompiledScripts` racing an in-flight evaluation into
|
||
`ObjectDisposedException` is now retried once in `RoslynVirtualTagEvaluator` (re-fetch → recompile into
|
||
the current cache generation). A concurrent-clear stress guard reproduces the pre-fix disposed failures
|
||
and is deterministically green post-fix.
|
||
- **02/P7** (Low) — the apply-boundary clear is gated on a changed deployed *expression set*
|
||
(`HashSet.SetEquals`, ordinal), so a byte-identical redeploy — or a rename/add/remove that leaves the
|
||
expression set unchanged — no longer forces full VT recompilation.
|
||
- Verified locally: `dotnet build ZB.MOM.WW.OtOpcUa.slnx` clean (0 errors); whole `Runtime.Tests`
|
||
project 372/372 green; `RoslynVirtualTagEvaluatorTests` 20/20 green. **Deferred to the serialized
|
||
heavy pass:** the full cross-suite sweep and the docker-dev live-`/run` gate (fail→Bad→recover +
|
||
P7 no-recompile spot-check).
|