lmxopcua/archreview/plans/STATUS.md

Arch-Review Remediation — Status & Handoff

Snapshot: 2026-07-12 · Review baseline: master 9cad9ed0; re-review baseline: master f6eaa267 · this file is the single source of truth for remediation progress. Update it as work lands.

2026-07-12 UPDATE — everything below the topology table landed: all remediation branches were merged to master and pushed to origin/master (merge commits 7834b3b7…b67bd9e8; final state f6eaa267). The topology table is kept for history but is no longer live. A full re-review of all seven domains ran at f6eaa267 — see ../00-OVERALL.md and the "Re-review 2026-07-12" section at the bottom of this file for the new findings (one High regression in the S7 fix, one High ResilienceConfig validation gap, and a batch of resilience-seam Mediums).

Branch topology (HISTORICAL — as of 2026-07-08; since merged + pushed)

Branch	Head	Contains
master	9cad9ed0	untouched baseline
docs/archreview-plans-and-drift-fixes	9fadead6	the 7 domain plans + 00-INDEX.md + this STATUS.md + the two CLAUDE.md doc-drift fixes. (the plans live ONLY here)
fix/archreview-crit1-split-brain-resolver	eaf78aad	Critical 1 (03/S1) typed SBR option + guard test + Redundancy.md; + #9 hard-kill failover test (a25c9ed0); + #11 premise correction (eaf78aad)
fix/archreview-crit2-vt-timeout	7fd44f0f	Critical 2 (02/U2 + 02/U3) + tests
fix/archreview-crit3-s7-reconnect	25c0c6f6	Critical 3 (05/STAB-1) IS7Plc seam + lazy reconnect + unit guards + live bounce test
fix/archreview-crit4-twincat-replay	af318fb4	Critical 4 (05/STAB-2) native-notification replay + probe-recycle + unit guards
fix/archreview-u2-deferred-sink-forwarding-guard	a65c2ced	Guard 03/U2 — reflection-exhaustive deferred-sink forwarding test (+ DeferredServiceLevelPublisher)
fix/archreview-c1-wire-analyzer	f0082af5	Guard 07/C-1 — wire OTOPCUA0001 analyzer tree-wide + triage (surfaced the RESILIENCE-DISPATCH-GAP, task #10)
fix/archreview-s1-s2-ci-coverage	10b89830	Guard 07/S-1,S-2,S-4 — whole-solution CI leg + fail-on-skip gate + CLI sleep deflake
fix/archreview-crit10-wire-capability-invoker	62556c24	#10 — wire CapabilityInvoker into dispatch via a Core.Abstractions seam (off fix/archreview-c1-wire-analyzer, NOT master — the analyzer is the standing guard + the pragmas live there). Code-complete + unit/analyzer/context-verified; live gate pending.
fix/archreview-crit13-resilience-config-artifact	8a8b9ec5	#13 — apply per-instance ResilienceConfig from the artifact (75403caa, off the crit10 branch) + #10/#13 DI guard (8a8b9ec5 — ResilienceInvokerFactoryRegistrationTests, from the #12 live gate). Read-path plumbing + invalidate/respawn-on-change; deterministically verified + rig-booted.
fix/docker-dev-serverhistorian-stale-keys	7233e2ba	Rig fix (off master, independent): docker-dev compose's retired Wonderware ServerHistorian keys → gateway shape, disabled by default. Found during the #12 live-gate rig session; central-1 verified booting clean.

Decision (user): one branch per Critical, each off master. #9 (failover test) lives on the crit1 branch (not its own branch off master) because SBR only exists there — it is Critical 1's integration half. The plans are only on the docs branch, so from a critical branch read them via git show docs/archreview-plans-and-drift-fixes:archreview/plans/0X-<name>.md.

Completed

Item	Finding	Branch @ commit	Verification
Critical 1 — split-brain resolver activated	03/S1	fix/archreview-crit1-split-brain-resolver a81dea10	BuildClusterOptions sets KeepOldestOption{DownIfAlone=true}; 3 unit guards (Cluster.Tests 29/29); e2e-verified the resolved ActorSystem config carries SplitBrainResolverProvider + stable-after=15s survives the HOCON/Hosting merge (throwaway spike, removed). HOCON cross-ref comment + Redundancy.md corrected.
Critical 2a — real VT script timeout	02/U2	fix/archreview-crit2-vt-timeout 7fd44f0f	Routes through TimedScriptEvaluator; infinite-loop test (bounded by WaitAsync) proves no hang. Host.Integration VT 18/18.
Critical 2b — ALC-safe compile cache	02/U3	same 7fd44f0f	Swapped raw dict → CompiledScriptCache; new IScriptCacheOwner seam; apply-boundary clear in VirtualTagHostActor; wiring guard test. Runtime VT 12/12.
Doc drift — KNOWN LIMITATION 2 stale	03/U1 = 06/U-1	docs/… 9fadead6 (CLAUDE.md)	historized-ref feed IS wired; rewrote to "value-capture code-complete, live-verify pending".
Doc drift — ScriptAnalysis policy	theme #5	same 9fadead6 (CLAUDE.md)	corrected FleetAdmin → Administrator,Designer roles.
Critical 3 — S7 lazy reconnect	05/STAB-1	fix/archreview-crit3-s7-reconnect 25c0c6f6	New IS7Plc/IS7PlcFactory seam + EnsureConnectedAsync (dispose-dead-handle → reopen on next data call); connection-fatal faults (socket/ErrorCode.ConnectionError) mark the handle dead, data-address errors do NOT; probe loop routes through it as a backstop. 4 unit guards + full S7 suite 234/234; LIVE-verified against real python-snap7 — bounced the container (docker restart), driver observed the outage and resumed Good reads with no redeploy (S7_1500ReconnectTests, double-gated on sim + S7_RECONNECT_BOUNCE_CMD). Also closes the S7 TEST-1 gap.
Critical 4 — TwinCAT native-notification replay	05/STAB-2	fix/archreview-crit4-twincat-replay af318fb4	Native ADS notifications were opaque handles (no replayable intent) → silently orphaned on a client swap. Now NativeRegistration (symbol/type/bit/interval/onChange + swappable handle) hangs off DeviceState; EnsureConnectedAsync split into a gate wrapper + under-gate core that replays every intent onto a freshly-built client (register+replay both under ConnectGate); probe-failure RecycleClientAsync disposes+nulls the wire-dead-but-port-open client so the next tick rebuilds+replays. 4 unit guards + full TwinCAT suite 174/174. Unit-only — no TwinCAT docker fixture exists (integration needs a real TC3 XAR), so the fake-client tests are authoritative (no live gate like S7 had).
Guard — reflection-exhaustive deferred-sink forwarding	03/U2	fix/archreview-u2-deferred-sink-forwarding-guard a65c2ced	New DeferredSinkForwardingReflectionTests (Commons.Tests) reflects over every method of every forwarding interface DeferredAddressSpaceSink implements (IOpcUaAddressSpaceSink + ISurgicalAddressSpaceSink) — plus the sibling DeferredServiceLevelPublisher — invokes each on the wrapper over a DispatchProxy recording inner, and asserts the call reached the inner. A DispatchProxy recorder auto-implements future members (zero maintenance); a guard-of-the-guard asserts the wrapper implements exactly the known forwarding-interface set, so a brand-new capability interface trips too. This closes the F10b/PR#423 trap class and is the prerequisite gate for 03/P1 remove methods + 02/P WriteValues. 3 new tests green (Commons.Tests 57/57); negative control verified — stubbing UpdateFolderDisplayName to a non-forwarding no-op fails the guard with the exact member-named diagnostic. NO production change.
Guard — wire OTOPCUA0001 analyzer + triage	07/C-1	fix/archreview-c1-wire-analyzer f0082af5	Injected the analyzer tree-wide via Directory.Build.props (OutputItemType="Analyzer"), then triaged ~280 surfaced hits: (1) RESILIENCE-DISPATCH-GAP — 7 production sites (DriverInstanceActor ×6 + GenericDriverNodeManager) scoped-#pragma'd as a real tracked gap (CapabilityInvoker built but never wired into dispatch; new task #10); (2) driver-internal self-calls — 3 sites (AbCip/S7 own poll/ack) #pragma'd as intentional; (3) 12 wire-level test/CLI projects → commented NoWarn. Full build green, 0 hits; analyzer 31/31; negative control — dropping one dispatch-gap pragma re-fires OTOPCUA0001 and fails the build, proving it's live tree-wide.
Guard — CI to all suites + fail-on-skip + deflake	07/S-1,S-2,S-4	fix/archreview-s1-s2-ci-coverage 10b89830	S-1: replaced the 5-project unit-tests matrix with ONE whole-solution leg (self-maintaining; now covers Client's 388 + Analyzers' 31 + all drivers). S-2: scripts/ci/assert-not-all-skipped.sh fail-on-skip gate (turns "green==all-skipped" red) — strict on the unit leg, report-only on the fixtureless integration leg. S-4: readiness-signal (SubscribeInvoked/SubscribeAlarmsInvoked) replaces 11 fixed-sleep startup races in the newly-CI'd Client.CLI.Tests. Skip-gate proven locally (real trx OK / all-skip trx exit 1 / report-only / edge cases); Client.CLI.Tests 104/104. CI job execution itself is verifiable only on push (nothing pushed).
#10 — wire CapabilityInvoker into dispatch (seam)	RESILIENCE-DISPATCH-GAP (07/C-1)	fix/archreview-crit10-wire-capability-invoker 62556c24	Discovered the filed plan was infeasible — Runtime is deliberately Polly-free (refs Core.Abstractions, not Core). Fix = a seam mirroring IDriverFactory: IDriverCapabilityInvoker(+null) + IDriverCapabilityInvokerFactory(+null) in Core.Abstractions; CapabilityInvoker implements it; DriverCapabilityInvokerFactory (Core) builds per-instance over the singleton builder+tracker+GetTier; Host DriverFactoryBootstrap registers all three; Runtime SCE resolves like IDriverFactory; DriverInstanceActor wraps its 6 dispatch sites; analyzer taught the interface as a wrapper home; all 6 pragmas removed. Triage corrections: the 7th site (GenericDriverNodeManager) is test-only scaffolding (not wired, re-annotated); tier-defaults only (ResilienceConfig not in the artifact → sub-gap). Added retry/breaker logging to the pipeline builder (the tracker has no reader — Stream E.2/E.3 never shipped — so it ran invisibly). Verified: negative-control (unwrap→OTOPCUA0001 build error); recording-invoker routing test; real-CapabilityInvoker actor-context test (internal ConfigureAwait(false) doesn't leak); analyzer + logging tests; clean solution build; Runtime.Tests 358 / Core.Tests 238 / Analyzers 32. Live behavioral gate pending (recipe in FOLLOWUP-10).
#9 — hard-kill failover integration test	03/S1	fix/archreview-crit1-split-brain-resolver a25c9ed0	Critical 1's load-bearing integration half. TwoNodeClusterHarness.HardKillNodeAAsync (transport-shutdown crash sim → node B sees A Unreachable, no graceful Leave) + HardKillFailoverTests asserting B downs the alone oldest node and takes over as driver role-leader. LIVE-verified in-process (~35s); negative control (force downing-provider-class="") makes it correctly time out. ⚠️ The negative control also surfaced a Crit-1 premise finding (task #11): SBR was already active on master (Akka.Cluster.Hosting applies SplitBrainResolverOption.Default when the typed option is null → reads the pre-existing akka.conf keep-oldest block), so the cluster was NOT NoDowning before Crit-1 — the typed option is reinforcing, not the activator.
#11 — Critical 1 premise correction	03/S1 (from #9)	fix/archreview-crit1-split-brain-resolver eaf78aad	Docs/comment-only fix (no code revert): corrected the "HOCON inert / cluster runs NoDowning / never fails over" framing in BuildClusterOptions XML comment, akka.conf split-brain-resolver comment, docs/Redundancy.md, and SplitBrainResolverActivationTests (summary + ShouldNotBeNull message + method rename). Akka.Cluster.Hosting enables an SBR provider by default (applies SplitBrainResolverOption.Default when the typed option is null → reads the pre-existing akka.conf keep-oldest block), so the cluster was never NoDowning and the typed KeepOldestOption is reinforcing/explicit-in-code, not the activator. Cluster.Tests 29/29; failover behaviour still guarded by HardKillFailoverTests. Architecture.md carried no such framing.
#13 — apply per-instance ResilienceConfig from the artifact	RESILIENCE-DISPATCH-GAP sub-gap (from #10)	fix/archreview-crit13-resilience-config-artifact 75403caa	The DriverInstance.ResilienceConfig column was authored+persisted (AdminUI) and already in the artifact (the composer serializes the whole entity), but the driver-node read path dropped it → tier-defaults only. DriverInstanceSpec now carries it; TryReadSpec reads it; Create takes+layers it (+ logs parse diagnostics, never throws); SpawnChild threads it. Invalidate-on-change: Create calls builder.Invalidate(instance) (the cache is options-blind on a hit) so a respawn with changed options rebuilds; DriverSpawnPlanner respawns on a ResilienceConfig change (incl. null→json), a pure DriverConfig change stays an in-place delta. Verified: override reaches execution (retryCount:0 suppresses tier-A retry; a control retries), invalidate is per-instance, malformed logs+falls-back; planner+artifact tests. Core.Tests 243 (+5), Runtime.Tests 363 (+5). See FOLLOWUP-13.

Task list (TaskCreate IDs)

#	Task	Status
1	Critical 1 — split-brain resolver	✅ completed
2	Critical 2a — VT timeout	✅ completed
3	Critical 2b — CompiledScriptCache	✅ completed
4	Critical 3 — S7 reconnect (05/STAB-1)	✅ completed
5	Critical 4 — TwinCAT ADS re-registration (05/STAB-2)	✅ completed
6	Guard — DeferredAddressSpaceSink reflection forwarding test (03/U2)	✅ completed — a65c2ced; gate for 03/P1 + 02/P now in place
7	Guard — wire OTOPCUA0001 analyzer + triage (07/C-1)	✅ completed — f0082af5; surfaced task #10
8	Guard — CI to all unit suites + fail-on-skip (07/S-1,S-2, pair S-4)	✅ completed — 10b89830
9	Critical 1 follow-up — hard-kill failover integration test (03/S1)	✅ completed — a25c9ed0; surfaced task #11
10	Wire CapabilityInvoker into the dispatch layer (RESILIENCE-DISPATCH-GAP, from 07/C-1)	✅ code-complete + unit/analyzer/context-verified — fix/archreview-crit10-wire-capability-invoker 62556c24 (seam, NOT direct thread — Runtime is Polly-free). Live behavioral gate pending (rig recipe in FOLLOWUP-10). Surfaced a sub-gap: ResilienceConfig not in the deploy artifact → tier-defaults only.
11	Review Critical 1 premise — SBR was already active on master, not NoDowning (from #9)	✅ completed — fix/archreview-crit1-split-brain-resolver eaf78aad; docs/comment-only correction (no code revert), Cluster.Tests 29/29.
12	#10 live behavioral gate — rig session (from #10)	✅ ran 2026-07-08 (PARTIAL) — built the Host image from source (crit13 tip) + stood up the rig; central-1 booted healthy; prod DI binds the real invoker factory confirmed live + by a new deterministic guard (ResilienceInvokerFactoryRegistrationTests 8a8b9ec5). Behavioral retry/breaker log-line not observed (disciplined stop): Modbus's cheap paths don't throw a wrapped exception (Discover offline, Subscribe lazy, reads non-wrapped) — only an idempotent write-to-dead-endpoint (or S7) does, not worth the multi-dependency chase since the retry behavior is deterministically proven. Found + FIXED a rig bug: docker-dev compose's retired Wonderware ServerHistorian keys crashed the Host at bring-up (historian-gateway cutover → reads Endpoint) → fixed on fix/docker-dev-serverhistorian-stale-keys 7233e2ba (off master; gateway-shape keys, disabled by default; central-1 verified booting clean). See FOLLOWUP-10.
13	Plumb per-instance ResilienceConfig through the deploy artifact (from #10)	✅ completed — fix/archreview-crit13-resilience-config-artifact 75403caa; read-path plumbing (write side was already complete), invalidate-on-change + respawn-on-change; Core.Tests 243/Runtime.Tests 363. See FOLLOWUP-13.

Findings surfaced by the guards (NEW — need user decision)

Each has a dedicated follow-up file with the full design/action: FOLLOWUP-10-resilience-dispatch-gap.md · FOLLOWUP-11-crit1-sbr-premise.md · FOLLOWUP-13-resilience-config-artifact.md.

• #10 — RESILIENCE-DISPATCH-GAP (from wiring the analyzer, 07/C-1): ✅ IMPLEMENTED on fix/archreview-crit10-wire-capability-invoker 62556c24 — code-complete + unit/analyzer/context-verified; live behavioral gate pending (recipe in FOLLOWUP-10). The filed "thread CapabilityInvoker into the actor" was infeasible (Runtime is deliberately Polly-free); shipped a seam mirroring IDriverFactory instead. Only 6 of the 7 sites were real (the 7th, GenericDriverNodeManager, is test-only scaffolding). Two residual sub-findings: (a) per-instance ResilienceConfig isn't in the deploy artifact → tier-defaults only ✅ RESOLVED by task #13 (75403caa) — the column was already in the artifact; the read path now applies it (see FOLLOWUP-13); (b) the resilience-status tracker has no reader (Admin /hosts Stream E.2/E.3 never shipped) — added retry/breaker logging as the interim observability surface + the live-verify enabler (still open as an Admin-UI follow-on).
• #11 — Critical 1 premise inaccurate (from the #9 failover negative control): ✅ CORRECTED on fix/archreview-crit1-split-brain-resolver eaf78aad. SBR was already active on master (Akka.Cluster.Hosting's WithClustering applies SplitBrainResolverOption.Default when the typed option is null → enables the downing provider that reads the pre-existing akka.conf keep-oldest block). So the cluster was not running NoDowning; hard-crash failover already worked. Docs/comment-only fix (no code revert — the typed KeepOldestOption is correct belt-and-suspenders): corrected the "HOCON inert / NoDowning → no failover" framing in BuildClusterOptions XML comment, akka.conf, docs/Redundancy.md, and the SplitBrainResolverActivationTests summary/message (+ method rename). Cluster.Tests 29/29. (Architecture.md carried no such framing.) See FOLLOWUP-11.

Suggested next (from 00-INDEX execution order)

All 4 Criticals + all 3 systemic guards (tasks 6/7/8) + the #9 failover test are DONE. Remaining:

1. Resolve the two surfaced findings — task 10 (wire CapabilityInvoker into dispatch; behavior-changing, live-gate) and task 11 (correct Crit-1's rationale/docs).
2. Failure-visibility Highs — 01/S-1 (AddressSpaceApplyOutcome failure field), 06/S-1 (Galaxy fail-closed), 03/S4 (primary-gate default-deny).
3. P-tier — 03/P1 surgical adds/removals (now gated by the U2 forwarding guard), 02/P batched WriteValues.

Key resume facts

• Verification bar met so far: each fix has a deterministic unit guard AND (for SBR) an end-to-end config-resolution check. Follow the same pattern — the review's whole thesis is "unit-green ≠ wired".
• The forwarding-test guard (task 6) is DONE (a65c2ced): 03/P1 (surgical pure-adds) and 02/P-… batch-write each add new sink capability methods that ship inert unless forwarded through DeferredAddressSpaceSink (F10b/PR#423 precedent). The reflection guard now fails loudly if any such member isn't forwarded — add the member to the recorder-covered interface set and it's auto-checked.
• Nothing is pushed. Superseded 2026-07-12: all branches merged to master and pushed to origin/master; the plans now live on master too (no more cross-branch git show needed).
• CI changes can't be run-verified without a push — now pushed; the first real CI-job run should be checked on origin (the re-review verified the YAML statically only).
• Findings #10 and #11 are resolved (see above); the open decisions now come from the 2026-07-12 re-review below.

Re-review 2026-07-12 (master f6eaa267) — outcome + new findings

All seven domain reports were re-run and rewritten in place at f6eaa267 (each carries a Prior-finding status (9cad9ed0 → f6eaa267) table; synthesis in ../00-OVERALL.md). Verdict on the remediation: all 4 Criticals verifiably closed in code (Crit-1 by premise correction), both systemic guards live with negative controls, CI now whole-solution + fail-on-skip (07 stability 2.5→3.5). New findings needing scheduling, ranked:

Rank	Finding	Severity	One-line
1	05/STAB-14	High (regression)	S7 ensure-wrapper rethrows connect-timeout OCE → poll loop's bare catch (OCE) { return; } silently kills all S7 subscriptions after an unreachable-host (non-refusing) outage; live bounce test blind to it (S7Driver.cs:488,1000,1383,1404,1206-1208)
2	01/S-6	High	Operator-authorable timeoutSeconds:0 in ResilienceConfig → Polly ValidationException on every capability call = permanent driver-brick; no range validation in parser or AdminUI form
3	05/STAB-15	High	S7 fatal-classification misses framing/desync + cancel-during-I/O (half-read PDU → permanent desync); probe catch never marks handle dead
4	01/S-8 = 03/S12	Medium	Dispatch hardcodes isIdempotent:true; WriteIdempotentAttribute has zero readers → #13-enabled write retries would retry command writes
5	02/S13	Medium	VT script failure/timeout leaves node stale-Good forever (timeout fix made the failure invisible)
6	01/U-6	Medium	Bulkhead knob parsed/authored/documented but never built into the pipeline — dead knob
7	06/S-11	Medium	Enabled-but-misconfigured ServerHistorian crash-loops Host with raw UriFormatException (only config side fixed in 7233e2ba)
8	06/U-7	Medium	HistorianGateway 0.2.0 proto3-optional DataType change silently alters GatewayTagProvisioner Boolean provisioning; live EnsureTags never run on 0.2.0
9	04/C-7, 04/U-5=03/U10, 01/S-7=03/S13	Medium/Low	Resilience seam edges: lossy AdminUI form round-trip; status tracker has no reader (operators blind); respawn stale-pipeline cache race

Carried-open Highs unchanged from round 1 (no branch targeted them): failure-visibility trio (01/S-1, 06/S-1, 03/S4), AdminUI authz (04/C-1 — stakes rose: resilience config is on the ungated surface), 03/S2/S3 block-bridging, 03/P1 surgical adds (guard prerequisite in place), driver-fleet batch (05), repo hygiene (07). Full prioritized list in 00-OVERALL.md.

Round-2 plans WRITTEN (2026-07-12): all 12 action items have design+implementation plans — R2-01…R2-12 in this directory (+ co-located .tasks.json each, for resumable execution). Index, per-plan task counts, effort, and suggested execution order: 00-INDEX.md §"Round 2 plans". 193 bite-sized TDD tasks total, ~18–24 dev-days end-to-end; R2-01/02/03 (the re-review's new-and-sharp items) are ~2 days combined.

Round-2 remediation landed

Plan	Findings	Branch	Commits	Status
R2-06	06/S-11 (rank 7), 06/U-7 (rank 8), 06/C-7	r2/06-serverhistorian-failfast (off 1676c8f4)	3a049a13…4640ecb1 (10 task commits)	Code-complete, offline-verified. S-11: fail-fast ServerHistorianOptionsValidator (consumer-gated on ServerHistorian:Enabled OR AlarmHistorian:Enabled, fails empty/non-http(s) Endpoint at startup) + AddValidatedOptions wiring + adapter defense-in-depth (TryCreate→named InvalidOperationException, no more raw UriFormatException crash-loop) + Validate() warning parity + alarm-only-mode startup warning. U-7: pinned Boolean→Int1 explicit-presence (HasDataType) + HISTGW_BOOL_SANDBOX_TAG live Boolean EnsureTags leg + retype probe (skip-clean offline) + upsert/retype migration note (docs/Historian.md). C-7: keep-populating decision + dead-Quality doc comments (0.2.0 C-002). Live leg operator-gated (T12) — pending a VPN run on a 0.2.0 gateway (records the observed in-place retype policy back into docs/Historian.md; also discharges 06/U-2's full documented run).