Joseph Doherty
36 KiB
Arch-Review Remediation — Status & Handoff
Snapshot: 2026-07-12 · Review baseline: master 9cad9ed0; re-review baseline: master
f6eaa267 · this file is the single source of truth for remediation progress. Update it as work lands.
2026-07-12 UPDATE — everything below the topology table landed: all remediation branches were merged to master and pushed to
origin/master(merge commits7834b3b7…b67bd9e8; final statef6eaa267). The topology table is kept for history but is no longer live. A full re-review of all seven domains ran atf6eaa267— see../00-OVERALL.mdand the "Re-review 2026-07-12" section at the bottom of this file for the new findings (one High regression in the S7 fix, one High ResilienceConfig validation gap, and a batch of resilience-seam Mediums).
Branch topology (HISTORICAL — as of 2026-07-08; since merged + pushed)
| Branch | Head | Contains |
|---|---|---|
master |
9cad9ed0 |
untouched baseline |
docs/archreview-plans-and-drift-fixes |
9fadead6 |
the 7 domain plans + 00-INDEX.md + this STATUS.md + the two CLAUDE.md doc-drift fixes. (the plans live ONLY here) |
fix/archreview-crit1-split-brain-resolver |
eaf78aad |
Critical 1 (03/S1) typed SBR option + guard test + Redundancy.md; + #9 hard-kill failover test (a25c9ed0); + #11 premise correction (eaf78aad) |
fix/archreview-crit2-vt-timeout |
7fd44f0f |
Critical 2 (02/U2 + 02/U3) + tests |
fix/archreview-crit3-s7-reconnect |
25c0c6f6 |
Critical 3 (05/STAB-1) IS7Plc seam + lazy reconnect + unit guards + live bounce test |
fix/archreview-crit4-twincat-replay |
af318fb4 |
Critical 4 (05/STAB-2) native-notification replay + probe-recycle + unit guards |
fix/archreview-u2-deferred-sink-forwarding-guard |
a65c2ced |
Guard 03/U2 — reflection-exhaustive deferred-sink forwarding test (+ DeferredServiceLevelPublisher) |
fix/archreview-c1-wire-analyzer |
f0082af5 |
Guard 07/C-1 — wire OTOPCUA0001 analyzer tree-wide + triage (surfaced the RESILIENCE-DISPATCH-GAP, task #10) |
fix/archreview-s1-s2-ci-coverage |
10b89830 |
Guard 07/S-1,S-2,S-4 — whole-solution CI leg + fail-on-skip gate + CLI sleep deflake |
fix/archreview-crit10-wire-capability-invoker |
62556c24 |
#10 — wire CapabilityInvoker into dispatch via a Core.Abstractions seam (off fix/archreview-c1-wire-analyzer, NOT master — the analyzer is the standing guard + the pragmas live there). Code-complete + unit/analyzer/context-verified; live gate pending. |
fix/archreview-crit13-resilience-config-artifact |
8a8b9ec5 |
#13 — apply per-instance ResilienceConfig from the artifact (75403caa, off the crit10 branch) + #10/#13 DI guard (8a8b9ec5 — ResilienceInvokerFactoryRegistrationTests, from the #12 live gate). Read-path plumbing + invalidate/respawn-on-change; deterministically verified + rig-booted. |
fix/docker-dev-serverhistorian-stale-keys |
7233e2ba |
Rig fix (off master, independent): docker-dev compose's retired Wonderware ServerHistorian keys → gateway shape, disabled by default. Found during the #12 live-gate rig session; central-1 verified booting clean. |
Decision (user): one branch per Critical, each off master. #9 (failover test) lives on the crit1 branch
(not its own branch off master) because SBR only exists there — it is Critical 1's integration half. The plans are only on the docs branch,
so from a critical branch read them via git show docs/archreview-plans-and-drift-fixes:archreview/plans/0X-<name>.md.
Completed
| Item | Finding | Branch @ commit | Verification |
|---|---|---|---|
| Critical 1 — split-brain resolver activated | 03/S1 | fix/archreview-crit1-split-brain-resolver a81dea10 |
BuildClusterOptions sets KeepOldestOption{DownIfAlone=true}; 3 unit guards (Cluster.Tests 29/29); e2e-verified the resolved ActorSystem config carries SplitBrainResolverProvider + stable-after=15s survives the HOCON/Hosting merge (throwaway spike, removed). HOCON cross-ref comment + Redundancy.md corrected. |
| Critical 2a — real VT script timeout | 02/U2 | fix/archreview-crit2-vt-timeout 7fd44f0f |
Routes through TimedScriptEvaluator; infinite-loop test (bounded by WaitAsync) proves no hang. Host.Integration VT 18/18. |
| Critical 2b — ALC-safe compile cache | 02/U3 | same 7fd44f0f |
Swapped raw dict → CompiledScriptCache; new IScriptCacheOwner seam; apply-boundary clear in VirtualTagHostActor; wiring guard test. Runtime VT 12/12. |
| Doc drift — KNOWN LIMITATION 2 stale | 03/U1 = 06/U-1 | docs/… 9fadead6 (CLAUDE.md) |
historized-ref feed IS wired; rewrote to "value-capture code-complete, live-verify pending". |
| Doc drift — ScriptAnalysis policy | theme #5 | same 9fadead6 (CLAUDE.md) |
corrected FleetAdmin → Administrator,Designer roles. |
| Critical 3 — S7 lazy reconnect | 05/STAB-1 | fix/archreview-crit3-s7-reconnect 25c0c6f6 |
New IS7Plc/IS7PlcFactory seam + EnsureConnectedAsync (dispose-dead-handle → reopen on next data call); connection-fatal faults (socket/ErrorCode.ConnectionError) mark the handle dead, data-address errors do NOT; probe loop routes through it as a backstop. 4 unit guards + full S7 suite 234/234; LIVE-verified against real python-snap7 — bounced the container (docker restart), driver observed the outage and resumed Good reads with no redeploy (S7_1500ReconnectTests, double-gated on sim + S7_RECONNECT_BOUNCE_CMD). Also closes the S7 TEST-1 gap. |
| Critical 4 — TwinCAT native-notification replay | 05/STAB-2 | fix/archreview-crit4-twincat-replay af318fb4 |
Native ADS notifications were opaque handles (no replayable intent) → silently orphaned on a client swap. Now NativeRegistration (symbol/type/bit/interval/onChange + swappable handle) hangs off DeviceState; EnsureConnectedAsync split into a gate wrapper + under-gate core that replays every intent onto a freshly-built client (register+replay both under ConnectGate); probe-failure RecycleClientAsync disposes+nulls the wire-dead-but-port-open client so the next tick rebuilds+replays. 4 unit guards + full TwinCAT suite 174/174. Unit-only — no TwinCAT docker fixture exists (integration needs a real TC3 XAR), so the fake-client tests are authoritative (no live gate like S7 had). |
| Guard — reflection-exhaustive deferred-sink forwarding | 03/U2 | fix/archreview-u2-deferred-sink-forwarding-guard a65c2ced |
New DeferredSinkForwardingReflectionTests (Commons.Tests) reflects over every method of every forwarding interface DeferredAddressSpaceSink implements (IOpcUaAddressSpaceSink + ISurgicalAddressSpaceSink) — plus the sibling DeferredServiceLevelPublisher — invokes each on the wrapper over a DispatchProxy recording inner, and asserts the call reached the inner. A DispatchProxy recorder auto-implements future members (zero maintenance); a guard-of-the-guard asserts the wrapper implements exactly the known forwarding-interface set, so a brand-new capability interface trips too. This closes the F10b/PR#423 trap class and is the prerequisite gate for 03/P1 remove methods + 02/P WriteValues. 3 new tests green (Commons.Tests 57/57); negative control verified — stubbing UpdateFolderDisplayName to a non-forwarding no-op fails the guard with the exact member-named diagnostic. NO production change. |
| Guard — wire OTOPCUA0001 analyzer + triage | 07/C-1 | fix/archreview-c1-wire-analyzer f0082af5 |
Injected the analyzer tree-wide via Directory.Build.props (OutputItemType="Analyzer"), then triaged ~280 surfaced hits: (1) RESILIENCE-DISPATCH-GAP — 7 production sites (DriverInstanceActor ×6 + GenericDriverNodeManager) scoped-#pragma'd as a real tracked gap (CapabilityInvoker built but never wired into dispatch; new task #10); (2) driver-internal self-calls — 3 sites (AbCip/S7 own poll/ack) #pragma'd as intentional; (3) 12 wire-level test/CLI projects → commented NoWarn. Full build green, 0 hits; analyzer 31/31; negative control — dropping one dispatch-gap pragma re-fires OTOPCUA0001 and fails the build, proving it's live tree-wide. |
| Guard — CI to all suites + fail-on-skip + deflake | 07/S-1,S-2,S-4 | fix/archreview-s1-s2-ci-coverage 10b89830 |
S-1: replaced the 5-project unit-tests matrix with ONE whole-solution leg (self-maintaining; now covers Client's 388 + Analyzers' 31 + all drivers). S-2: scripts/ci/assert-not-all-skipped.sh fail-on-skip gate (turns "green==all-skipped" red) — strict on the unit leg, report-only on the fixtureless integration leg. S-4: readiness-signal (SubscribeInvoked/SubscribeAlarmsInvoked) replaces 11 fixed-sleep startup races in the newly-CI'd Client.CLI.Tests. Skip-gate proven locally (real trx OK / all-skip trx exit 1 / report-only / edge cases); Client.CLI.Tests 104/104. CI job execution itself is verifiable only on push (nothing pushed). |
| #10 — wire CapabilityInvoker into dispatch (seam) | RESILIENCE-DISPATCH-GAP (07/C-1) | fix/archreview-crit10-wire-capability-invoker 62556c24 |
Discovered the filed plan was infeasible — Runtime is deliberately Polly-free (refs Core.Abstractions, not Core). Fix = a seam mirroring IDriverFactory: IDriverCapabilityInvoker(+null) + IDriverCapabilityInvokerFactory(+null) in Core.Abstractions; CapabilityInvoker implements it; DriverCapabilityInvokerFactory (Core) builds per-instance over the singleton builder+tracker+GetTier; Host DriverFactoryBootstrap registers all three; Runtime SCE resolves like IDriverFactory; DriverInstanceActor wraps its 6 dispatch sites; analyzer taught the interface as a wrapper home; all 6 pragmas removed. Triage corrections: the 7th site (GenericDriverNodeManager) is test-only scaffolding (not wired, re-annotated); tier-defaults only (ResilienceConfig not in the artifact → sub-gap). Added retry/breaker logging to the pipeline builder (the tracker has no reader — Stream E.2/E.3 never shipped — so it ran invisibly). Verified: negative-control (unwrap→OTOPCUA0001 build error); recording-invoker routing test; real-CapabilityInvoker actor-context test (internal ConfigureAwait(false) doesn't leak); analyzer + logging tests; clean solution build; Runtime.Tests 358 / Core.Tests 238 / Analyzers 32. Live behavioral gate pending (recipe in FOLLOWUP-10). |
| #9 — hard-kill failover integration test | 03/S1 | fix/archreview-crit1-split-brain-resolver a25c9ed0 |
Critical 1's load-bearing integration half. TwoNodeClusterHarness.HardKillNodeAAsync (transport-shutdown crash sim → node B sees A Unreachable, no graceful Leave) + HardKillFailoverTests asserting B downs the alone oldest node and takes over as driver role-leader. LIVE-verified in-process (~35s); negative control (force downing-provider-class="") makes it correctly time out. ⚠️ The negative control also surfaced a Crit-1 premise finding (task #11): SBR was already active on master (Akka.Cluster.Hosting applies SplitBrainResolverOption.Default when the typed option is null → reads the pre-existing akka.conf keep-oldest block), so the cluster was NOT NoDowning before Crit-1 — the typed option is reinforcing, not the activator. |
| #11 — Critical 1 premise correction | 03/S1 (from #9) | fix/archreview-crit1-split-brain-resolver eaf78aad |
Docs/comment-only fix (no code revert): corrected the "HOCON inert / cluster runs NoDowning / never fails over" framing in BuildClusterOptions XML comment, akka.conf split-brain-resolver comment, docs/Redundancy.md, and SplitBrainResolverActivationTests (summary + ShouldNotBeNull message + method rename). Akka.Cluster.Hosting enables an SBR provider by default (applies SplitBrainResolverOption.Default when the typed option is null → reads the pre-existing akka.conf keep-oldest block), so the cluster was never NoDowning and the typed KeepOldestOption is reinforcing/explicit-in-code, not the activator. Cluster.Tests 29/29; failover behaviour still guarded by HardKillFailoverTests. Architecture.md carried no such framing. |
| #13 — apply per-instance ResilienceConfig from the artifact | RESILIENCE-DISPATCH-GAP sub-gap (from #10) | fix/archreview-crit13-resilience-config-artifact 75403caa |
The DriverInstance.ResilienceConfig column was authored+persisted (AdminUI) and already in the artifact (the composer serializes the whole entity), but the driver-node read path dropped it → tier-defaults only. DriverInstanceSpec now carries it; TryReadSpec reads it; Create takes+layers it (+ logs parse diagnostics, never throws); SpawnChild threads it. Invalidate-on-change: Create calls builder.Invalidate(instance) (the cache is options-blind on a hit) so a respawn with changed options rebuilds; DriverSpawnPlanner respawns on a ResilienceConfig change (incl. null→json), a pure DriverConfig change stays an in-place delta. Verified: override reaches execution (retryCount:0 suppresses tier-A retry; a control retries), invalidate is per-instance, malformed logs+falls-back; planner+artifact tests. Core.Tests 243 (+5), Runtime.Tests 363 (+5). See FOLLOWUP-13. |
Task list (TaskCreate IDs)
| # | Task | Status |
|---|---|---|
| 1 | Critical 1 — split-brain resolver | ✅ completed |
| 2 | Critical 2a — VT timeout | ✅ completed |
| 3 | Critical 2b — CompiledScriptCache | ✅ completed |
| 4 | Critical 3 — S7 reconnect (05/STAB-1) | ✅ completed |
| 5 | Critical 4 — TwinCAT ADS re-registration (05/STAB-2) | ✅ completed |
| 6 | Guard — DeferredAddressSpaceSink reflection forwarding test (03/U2) | ✅ completed — a65c2ced; gate for 03/P1 + 02/P now in place |
| 7 | Guard — wire OTOPCUA0001 analyzer + triage (07/C-1) | ✅ completed — f0082af5; surfaced task #10 |
| 8 | Guard — CI to all unit suites + fail-on-skip (07/S-1,S-2, pair S-4) | ✅ completed — 10b89830 |
| 9 | Critical 1 follow-up — hard-kill failover integration test (03/S1) | ✅ completed — a25c9ed0; surfaced task #11 |
| 10 | Wire CapabilityInvoker into the dispatch layer (RESILIENCE-DISPATCH-GAP, from 07/C-1) | ✅ code-complete + unit/analyzer/context-verified — fix/archreview-crit10-wire-capability-invoker 62556c24 (seam, NOT direct thread — Runtime is Polly-free). Live behavioral gate pending (rig recipe in FOLLOWUP-10). Surfaced a sub-gap: ResilienceConfig not in the deploy artifact → tier-defaults only. |
| 11 | Review Critical 1 premise — SBR was already active on master, not NoDowning (from #9) | ✅ completed — fix/archreview-crit1-split-brain-resolver eaf78aad; docs/comment-only correction (no code revert), Cluster.Tests 29/29. |
| 12 | #10 live behavioral gate — rig session (from #10) | ✅ ran 2026-07-08 (PARTIAL) — built the Host image from source (crit13 tip) + stood up the rig; central-1 booted healthy; prod DI binds the real invoker factory confirmed live + by a new deterministic guard (ResilienceInvokerFactoryRegistrationTests 8a8b9ec5). Behavioral retry/breaker log-line not observed (disciplined stop): Modbus's cheap paths don't throw a wrapped exception (Discover offline, Subscribe lazy, reads non-wrapped) — only an idempotent write-to-dead-endpoint (or S7) does, not worth the multi-dependency chase since the retry behavior is deterministically proven. Found + FIXED a rig bug: docker-dev compose's retired Wonderware ServerHistorian keys crashed the Host at bring-up (historian-gateway cutover → reads Endpoint) → fixed on fix/docker-dev-serverhistorian-stale-keys 7233e2ba (off master; gateway-shape keys, disabled by default; central-1 verified booting clean). See FOLLOWUP-10. |
| 13 | Plumb per-instance ResilienceConfig through the deploy artifact (from #10) |
✅ completed — fix/archreview-crit13-resilience-config-artifact 75403caa; read-path plumbing (write side was already complete), invalidate-on-change + respawn-on-change; Core.Tests 243/Runtime.Tests 363. See FOLLOWUP-13. |
Findings surfaced by the guards (NEW — need user decision)
Each has a dedicated follow-up file with the full design/action:
FOLLOWUP-10-resilience-dispatch-gap.md·FOLLOWUP-11-crit1-sbr-premise.md·FOLLOWUP-13-resilience-config-artifact.md.
- #10 — RESILIENCE-DISPATCH-GAP (from wiring the analyzer, 07/C-1): ✅ IMPLEMENTED on
fix/archreview-crit10-wire-capability-invoker62556c24— code-complete + unit/analyzer/context-verified; live behavioral gate pending (recipe inFOLLOWUP-10). The filed "thread CapabilityInvoker into the actor" was infeasible (Runtime is deliberately Polly-free); shipped a seam mirroringIDriverFactoryinstead. Only 6 of the 7 sites were real (the 7th,GenericDriverNodeManager, is test-only scaffolding). Two residual sub-findings: (a)per-instance✅ RESOLVED by task #13 (ResilienceConfigisn't in the deploy artifact → tier-defaults only75403caa) — the column was already in the artifact; the read path now applies it (seeFOLLOWUP-13); (b) the resilience-status tracker has no reader (Admin/hostsStream E.2/E.3 never shipped) — added retry/breaker logging as the interim observability surface + the live-verify enabler (still open as an Admin-UI follow-on). - #11 — Critical 1 premise inaccurate (from the #9 failover negative control): ✅ CORRECTED on
fix/archreview-crit1-split-brain-resolvereaf78aad. SBR was already active on master (Akka.Cluster.Hosting'sWithClusteringappliesSplitBrainResolverOption.Defaultwhen the typed option is null → enables the downing provider that reads the pre-existing akka.confkeep-oldestblock). So the cluster was not running NoDowning; hard-crash failover already worked. Docs/comment-only fix (no code revert — the typedKeepOldestOptionis correct belt-and-suspenders): corrected the "HOCON inert / NoDowning → no failover" framing inBuildClusterOptionsXML comment,akka.conf,docs/Redundancy.md, and theSplitBrainResolverActivationTestssummary/message (+ method rename). Cluster.Tests 29/29. (Architecture.md carried no such framing.) SeeFOLLOWUP-11.
Suggested next (from 00-INDEX execution order)
All 4 Criticals + all 3 systemic guards (tasks 6/7/8) + the #9 failover test are DONE. Remaining:
- Resolve the two surfaced findings — task 10 (wire CapabilityInvoker into dispatch; behavior-changing, live-gate) and task 11 (correct Crit-1's rationale/docs).
- Failure-visibility Highs — 01/S-1 (AddressSpaceApplyOutcome failure field), 06/S-1 (Galaxy fail-closed), 03/S4 (primary-gate default-deny).
- P-tier — 03/P1 surgical adds/removals (now gated by the U2 forwarding guard), 02/P batched
WriteValues.
Key resume facts
- Verification bar met so far: each fix has a deterministic unit guard AND (for SBR) an end-to-end config-resolution check. Follow the same pattern — the review's whole thesis is "unit-green ≠ wired".
- The forwarding-test guard (task 6) is DONE (
a65c2ced): 03/P1 (surgical pure-adds) and 02/P-… batch-write each add new sink capability methods that ship inert unless forwarded throughDeferredAddressSpaceSink(F10b/PR#423 precedent). The reflection guard now fails loudly if any such member isn't forwarded — add the member to the recorder-covered interface set and it's auto-checked. Nothing is pushed.Superseded 2026-07-12: all branches merged to master and pushed toorigin/master; the plans now live on master too (no more cross-branchgit showneeded).CI changes can't be run-verified without a push— now pushed; the first real CI-job run should be checked on origin (the re-review verified the YAML statically only).- Findings #10 and #11 are resolved (see above); the open decisions now come from the 2026-07-12 re-review below.
Re-review 2026-07-12 (master f6eaa267) — outcome + new findings
All seven domain reports were re-run and rewritten in place at f6eaa267 (each carries a
Prior-finding status (9cad9ed0 → f6eaa267) table; synthesis in ../00-OVERALL.md).
Verdict on the remediation: all 4 Criticals verifiably closed in code (Crit-1 by premise correction),
both systemic guards live with negative controls, CI now whole-solution + fail-on-skip
(07 stability 2.5→3.5). New findings needing scheduling, ranked:
| Rank | Finding | Severity | One-line |
|---|---|---|---|
| 1 | 05/STAB-14 | High (regression) | S7 ensure-wrapper rethrows connect-timeout OCE → poll loop's bare catch (OCE) { return; } silently kills all S7 subscriptions after an unreachable-host (non-refusing) outage; live bounce test blind to it (S7Driver.cs:488,1000,1383,1404,1206-1208) |
| 2 | 01/S-6 | High | Operator-authorable timeoutSeconds:0 in ResilienceConfig → Polly ValidationException on every capability call = permanent driver-brick; no range validation in parser or AdminUI form |
| 3 | 05/STAB-15 | High | S7 fatal-classification misses framing/desync + cancel-during-I/O (half-read PDU → permanent desync); probe catch never marks handle dead |
| 4 | 01/S-8 = 03/S12 | Medium | Dispatch hardcodes isIdempotent:true; WriteIdempotentAttribute has zero readers → #13-enabled write retries would retry command writes |
| 5 | 02/S13 | Medium | VT script failure/timeout leaves node stale-Good forever (timeout fix made the failure invisible) |
| 6 | 01/U-6 | Medium | Bulkhead knob parsed/authored/documented but never built into the pipeline — dead knob |
| 7 | 06/S-11 | Medium | Enabled-but-misconfigured ServerHistorian crash-loops Host with raw UriFormatException (only config side fixed in 7233e2ba) |
| 8 | 06/U-7 | Medium | HistorianGateway 0.2.0 proto3-optional DataType change silently alters GatewayTagProvisioner Boolean provisioning; live EnsureTags never run on 0.2.0 |
| 9 | 04/C-7, 04/U-5=03/U10, 01/S-7=03/S13 | Medium/Low | Resilience seam edges: lossy AdminUI form round-trip; status tracker has no reader (operators blind); respawn stale-pipeline cache race |
Carried-open Highs unchanged from round 1 (no branch targeted them): failure-visibility trio
(01/S-1, 06/S-1, 03/S4), AdminUI authz (04/C-1 — stakes rose: resilience config is on the ungated
surface), 03/S2/S3 block-bridging, 03/P1 surgical adds (guard prerequisite in place), driver-fleet batch
(05), repo hygiene (07). Full prioritized list in 00-OVERALL.md.
Round-2 plans WRITTEN (2026-07-12): all 12 action items have design+implementation plans —
R2-01…R2-12 in this directory (+ co-located .tasks.json each, for resumable execution). Index,
per-plan task counts, effort, and suggested execution order: 00-INDEX.md §"Round 2
plans". 193 bite-sized TDD tasks total, ~18–24 dev-days end-to-end; R2-01/02/03 (the re-review's
new-and-sharp items) are ~2 days combined.
Round-2 EXECUTED — R2-12 Repo hygiene (2026-07-13, branch r2/12-repo-hygiene): DONE.
07/U-5 (7 lib/ AVEVA DLLs removed + gitignored, zero-reference proven), 07/U-4 (root planning
files: pending.md/current.md untracked+gitignored, looseends.md deleted, stillpending.md +
HISTORIAN-GATEWAY-INTEGRATION-ISSUES.md archived under docs/plans/, docs/Historian.md
repointed), 07/S-3 (v2-e2e.yml no-op nightly deleted), 07/P-5 (redundant integration matrix job
removed from v2-ci.yml), 07/U-3 (5 alarm operator CLI commands documented — now 13/13), 07/C-5
(StyleGuide.md ScadaBridge→OtOpcUa + minimal root .editorconfig), 07/C-4 (xunit pin 2.9.2→2.9.3
- blocker comment; upstream-blocked — no xunit.v3 Akka TestKit exists). 07/U-6 needed no action (the
Wonderware husk dirs are untracked build artifacts, absent in a fresh worktree). Whole-solution build
clean. OPEN follow-ups: (a) when a runner-reachable fleet + a real
tests/Server/…E2ETestsland, add a fresh E2E workflow WITH a zero-match guard (fail ifCategory=E2Ematches nothing); (b) S-2 option #2 — a fixture-backed integration leg (opc-plcservices:block,MIN_EXECUTED=1) covering what the unit leg cannot; (c) migrate the 3 xunit v2 holdouts once Akka ships an xunit.v3 TestKit.
Round-2 remediation landed
| Plan | Findings | Branch | Commits | Status |
|---|---|---|---|---|
| R2-06 | 06/S-11 (rank 7), 06/U-7 (rank 8), 06/C-7 | r2/06-serverhistorian-failfast (off 1676c8f4) |
3a049a13…4640ecb1 (10 task commits) |
Code-complete, offline-verified. S-11: fail-fast ServerHistorianOptionsValidator (consumer-gated on ServerHistorian:Enabled OR AlarmHistorian:Enabled, fails empty/non-http(s) Endpoint at startup) + AddValidatedOptions wiring + adapter defense-in-depth (TryCreate→named InvalidOperationException, no more raw UriFormatException crash-loop) + Validate() warning parity + alarm-only-mode startup warning. U-7: pinned Boolean→Int1 explicit-presence (HasDataType) + HISTGW_BOOL_SANDBOX_TAG live Boolean EnsureTags leg + retype probe (skip-clean offline) + upsert/retype migration note (docs/Historian.md). C-7: keep-populating decision + dead-Quality doc comments (0.2.0 C-002). Live leg operator-gated (T12) — pending a VPN run on a 0.2.0 gateway (records the observed in-place retype policy back into docs/Historian.md; also discharges 06/U-2's full documented run). |
| R2-05 | 04/C-1 (High) | r2/05-adminui-authz (off 1676c8f4) |
183b72b7…6c1f05cb (5 commits) |
Code-complete, offline-verified. All 38 routable AdminUI pages carry an explicit named-policy @attribute via new AdminUiPolicies constants (Security/Auth/). New ConfigEditor (Administrator+Designer) gates the 20-page config-authoring surface incl. the 8 driver pages that author live ResilienceConfig + /api/script-analysis/*; new AuthenticatedRead on the 16 read pages; FleetAdmin/DriverOperator unchanged (literals→constants). Anti-drift guard PageAuthorizationGuardTests (bans bare [Authorize]/Roles=/unknown-policy/unclassified) + AdminUiPoliciesTests semantics matrix. Corrections folded in: Reservations + ClusterRedundancy read-only (→AuthenticatedRead); Home had no attribute (now AuthenticatedRead). AdminUI.Tests 517/517, Security.Tests 80/80. Live /run positive pass deferred (T15) — docker-dev DisableLogin=true auto-admin can't observe deny; negative proof is CI-side (both guard tests). |
| R2-01 | 05/STAB-14 (High regression, rank 1), 05/STAB-15 (rank 3), 05/STAB-8 seam | r2/01-s7-fault-hardening (off 1676c8f4) |
fcdcf0e2…bdb7ed41 (13 task commits) |
Code-complete, offline-verified. All prod changes in S7Driver.cs. STAB-14: connect-timeout surfaces as TimeoutException (not an escaping OCE) at the EnsureConnectedAsync/InitializeAsync source + when(token.IsCancellationRequested) filters on the read/write ensure-wrappers and all three poll-loop OCE catches — subscription poll loops survive an unreachable-host outage and recover (T2 regression RED→GREEN). STAB-15: IsS7ConnectionFatal broadened to the ISO-on-TCP framing surface (TPKT/TPDU/WrongNumberOfBytes/PlcException{WrongNumberReceivedBytes}, NOT InvalidDataException); mid-PDU cancel marks handle dead; probe classifies+marks-dead under the gate. STAB-8 (S7 part): EnsureConnectedAsync documented as the R2-09 connect-throttle seam (note-only). Driver.S7.Tests 246/246. Live outage gate deferred (T11) — S7_1500ConnectTimeoutOutageTests (env S7_TIMEOUT_OUTAGE_START_CMD/STOP_CMD, SYN-blackhole) skips clean offline; the connect-timeout class the docker restart bounce can't produce. (Pre-existing unrelated CLI docs-scan test fails at base — out of scope.) |
| R2-03 | 02/S13 (rank 5), 02/S12, 02/P7 | r2/03-vt-failure-quality (off 1676c8f4) |
b9775485…6bb91e8d (12 task commits) |
Code-complete, offline-verified. S13: VT script failure/timeout degrades the node to OpcUaQuality.Bad (carrying last-known value) instead of freezing on last-Good forever — additive EvaluationResult.Quality, bridge historizes 0x80020000u BadInternalError, published once per Good→Bad transition gated on all deps arrived (inputs-ready, no cold-start flash), recovery force-publishes next Good. S12: apply-boundary ClearCompiledScripts racing an in-flight eval into ObjectDisposedException retried once in RoslynVirtualTagEvaluator (re-fetch→recompile into current generation); stress guard reproduces pre-fix, deterministically green post-fix. P7: apply-boundary clear gated on a changed deployed expression set (HashSet.SetEquals, ordinal) — identical redeploy / rename-with-same-expressions no longer forces full VT recompile. Runtime.Tests 372/372, RoslynVirtualTagEvaluatorTests 20/20. Deferred (T11/T12): full cross-suite sweep + docker-dev live-/run (fail→Bad→recover + P7 no-recompile spot-check). |
| R2-02 | 01/S-6 (High, rank 2), 01/U-6, 01/S-8=03/S12 (rank 4), 04/C-7, 01/S-7=03/S13 | r2/02-resilience-config-hardening (off 1676c8f4) |
4be13429…dd82e740 (17 task commits) |
Code-complete, offline-verified. S-6: parser clamps timeout/retry/breaker overrides (timeout≤0→tier default, breaker==1→2, caps) with diagnostics + belt-and-braces Build clamp so pipeline construction can never throw Polly ValidationException (new ResiliencePolicyRanges in Core.Abstractions); brick-repro + Build_NeverThrows_ForHostileDirectOptions + factory-seam wiring proof; AdminUI Validate() range warnings. S-8: parser forces Write/AlarmAck retry→0 (spec invariant) and dispatch flips to isIdempotent:false (RED-first on the Runtime wiring test); non-idempotent arm caches the no-retry snapshot once/invoker (01/P-4) + tracker in-flight accounting; WriteIdempotentAttribute false claim corrected; retry cells disabled. U-6: bulkhead knob deleted end-to-end (migration-free; stored keys ignored) + Options_properties_are_exactly_the_pipeline_wired_set inertness guard. C-7: ResilienceFormModel JsonObject-bag round-trip preserves unknown keys; malformed JSON → ParseFailed lock + verbatim ToJson + discard button. S-7: per-Create Interlocked options-generation in the pipeline-cache key kills the respawn stale-pipeline race. Core resilience 121/121, AdminUI form 10/10, Analyzers 32/32, Runtime 364/364; 0 production OTOPCUA0001. Deferred (T15/T18): docker-dev :9200 /run (warning banner, input lockout, unknown-key survival) + whole-solution sweep. |
| R2-10 | 03/U10=04/U-5 (Medium/Low, action #10) | r2/10-resilience-observability (off 46fedda3) |
205d3980…<docs> (11 task commits) |
Code-complete, offline-verified. The resilience status tracker now has a production reader, mirroring the driver-health flow. Core truthfulness: RecordBreakerClose + derived IsBreakerOpen (breaker-open state derivable), OnClosed records the close, and RecordSuccess wired onto the invoker success paths (ConsecutiveFailures is a true consecutive counter, not stuck-forever). Transport: DriverResilienceStatusChanged Commons contract → periodic 5 s full-snapshot publisher DriverResilienceStatusPublisherService (driver nodes, registered in AddOtOpcUaDriverFactories with wiring guard ResilienceStatusReaderRegistrationTests — closes the built-but-never-wired class) → driver-resilience-status DPS topic → per-admin-node DriverResilienceStatusBridge → IDriverResilienceStatusStore → resilience chips (breaker-OPEN / consecutive-failures / in-flight / staleness) on DriverStatusPanel.razor, read in-process (self-HubConnection ban honored). Core.Tests resilience 31/31 (incl. deterministic breaker-close via a controllable TimeProvider), Host.IntegrationTests publisher+guard 6/6, AdminUI.Tests store 5/5; full-solution build clean. Deferred (T10): docker-dev live-/run — S7 dead-endpoint breaker-open observed on BOTH centrals + recovery-clear + staleness-dim (also the first live sighting of the #10 circuit-breaker OPENED log line, closing FOLLOWUP-10's behavioral gate). |
| R2-11 | 01/C-1, 01/P-1 + 05/CONV-2, UNDER-1, UNDER-6 | r2/11-tagconfig-consolidation (off 46fedda3) | first 2e14fe1f … (24 task commits) | Code-complete, offline-verified. Part 1 (C-1/P-1): single TagConfigIntent.Parse + DeviceConfigIntent in Commons replace the four byte-parity TagConfig parse copies (composer/artifact/walker/DraftValidator) — composer now parses once per tag (P-1); a permanent Runtime.Tests/TagConfigCorpusParityTests (30-blob corpus) guards compose≡decode; grep "MUST parse identically" src/ = 0. Part 2 (CONV-2): shared strict-capable TagConfigJson readers in Core.Abstractions replace the six copy-pasted ReadEnums; all six driver parsers gain Inspect() warnings + honour the writable key (FOCAS forced read-only, 05/UNDER-1); FOCAS equipment tags run the capability-matrix pre-flight (05/UNDER-6); Modbus probe parses the factory DTO shape (timeoutMs no longer dropped). New EquipmentTagConfigInspector in ControlPlane + AdminOperationsActor deploy-gate wiring with Deployment:TagConfigValidationMode (Warn default | Error opt-in), proven through the actor (F10b consuming test). AdminUI: writable checkbox in the six typed editors. Phase-C follow-up (deferred, recorded here): flip the runtime parsers to strict (typo'd enum ⇒ BadNodeIdUnknown instead of wrong-width Good) once fleets have run Error mode clean — behavior change deliberately not shipped in this plan. Deferred (T22/T24): docker-dev /run of one typed editor's Writable checkbox + whole-solution serial sweep. |
| R2-07 (Phases 1+2) | 03/P1 (action #7, surgical pure-adds/removes) | r2/07-surgical-pure-adds (off 1a698cbb) | bb226f45… (T1–T12) | Phases 1+2 code-complete, offline-verified. New pure AddressSpaceChangeClassifier (Empty/AttributeOnly/PureAdd/PureRemove/AddRemoveMix/Rebuild; first-match-wins, default-closed Rebuild; the F10b tag/vtag surgical-eligibility predicates moved here as internal static). AddressSpaceApplier.Apply routes by kind. Phase 1 (PureAdd/AttributeOnly): skip RebuildAddressSpace (the idempotent Materialise passes create exactly the added nodes; existing subscriptions survive); adds announced via ComputeAddAnnouncements+AnnounceAddedNodes (dedup affected-parent → existing guard-covered RaiseNodesAddedModelChange), fired by OpcUaPublishActor after the 4 passes gated on !outcome.RebuildCalled; MaterialiseAlarmCondition made idempotent (skip-if-present-and-same-kind keeps the condition NodeState alive). NO new sink members in Phase 1. Phase 2 (PureRemove): 3 new ISurgicalAddressSpaceSink members (RemoveVariableNode/RemoveAlarmConditionNode/RemoveEquipmentSubtree) — guard-first with the observed negative control (DeferredSinkForwardingReflectionTests fails member-named until DeferredAddressSpaceSink forwards) + SdkAddressSpaceSink forwards + OtOpcUaNodeManager in-place teardown (scoped map cleanup, notifier demotion via RemoveRootNotifier, BuildNodesRemovedModelChange verb=NodeDeleted reported outside Lock). Applier PureRemove arm: terminal Bad / RemovedConditionState writes (closing the removed-alarm-tag today-gap) → subsumption partition (removed-equipment children folded into the subtree remove) → in-place removes → any false/throw ⇒ full-rebuild ratchet; non-surgical sink ⇒ rebuild. AddRemoveMix still full-rebuild until Phase 3. Guard stays green (Commons.Tests 17/17), OpcUaServer.Tests 334/334, Runtime.Tests 398/398. Deferred-live (T5, T6, T12): SubscriptionSurvivalTests add + remove cases (authored + compiles; NOT run — *.IntegrationTests memory constraint) and the docker-dev live-/run add + remove gates (kind=PureAdd/PureRemove, rebuild=False; survivor subscription uninterrupted; removed node → BadNodeIdUnknown) — MUST run in the serial heavy pass (F10b inertness risk). Phase 3 (mixed add+remove) still pending. |
Report anchor drifts to fold into the next report refresh (from the R2-02 verification pass): (1)
01-core-composition.md U-6 anchor DriverResilienceOptions.cs:23-25 → the two bulkhead properties sat at
:23 and :29 (now deleted); (2) 01-core-composition.md U-6 anchor
DriverResiliencePipelineBuilder.cs:99-177 → Build was :99-176. All other cited lines were exact at
f6eaa267.