lmxopcua/archreview/plans/STATUS.md

Arch-Review Remediation — Status & Handoff

Snapshot: 2026-07-12 · Review baseline: master 9cad9ed0; re-review baseline: master f6eaa267 · this file is the single source of truth for remediation progress. Update it as work lands.

2026-07-12 UPDATE — everything below the topology table landed: all remediation branches were merged to master and pushed to origin/master (merge commits 7834b3b7…b67bd9e8; final state f6eaa267). The topology table is kept for history but is no longer live. A full re-review of all seven domains ran at f6eaa267 — see ../00-OVERALL.md and the "Re-review 2026-07-12" section at the bottom of this file for the new findings (one High regression in the S7 fix, one High ResilienceConfig validation gap, and a batch of resilience-seam Mediums).

Branch topology (HISTORICAL — as of 2026-07-08; since merged + pushed)

Branch	Head	Contains
master	9cad9ed0	untouched baseline
docs/archreview-plans-and-drift-fixes	9fadead6	the 7 domain plans + 00-INDEX.md + this STATUS.md + the two CLAUDE.md doc-drift fixes. (the plans live ONLY here)
fix/archreview-crit1-split-brain-resolver	eaf78aad	Critical 1 (03/S1) typed SBR option + guard test + Redundancy.md; + #9 hard-kill failover test (a25c9ed0); + #11 premise correction (eaf78aad)
fix/archreview-crit2-vt-timeout	7fd44f0f	Critical 2 (02/U2 + 02/U3) + tests
fix/archreview-crit3-s7-reconnect	25c0c6f6	Critical 3 (05/STAB-1) IS7Plc seam + lazy reconnect + unit guards + live bounce test
fix/archreview-crit4-twincat-replay	af318fb4	Critical 4 (05/STAB-2) native-notification replay + probe-recycle + unit guards
fix/archreview-u2-deferred-sink-forwarding-guard	a65c2ced	Guard 03/U2 — reflection-exhaustive deferred-sink forwarding test (+ DeferredServiceLevelPublisher)
fix/archreview-c1-wire-analyzer	f0082af5	Guard 07/C-1 — wire OTOPCUA0001 analyzer tree-wide + triage (surfaced the RESILIENCE-DISPATCH-GAP, task #10)
fix/archreview-s1-s2-ci-coverage	10b89830	Guard 07/S-1,S-2,S-4 — whole-solution CI leg + fail-on-skip gate + CLI sleep deflake
fix/archreview-crit10-wire-capability-invoker	62556c24	#10 — wire CapabilityInvoker into dispatch via a Core.Abstractions seam (off fix/archreview-c1-wire-analyzer, NOT master — the analyzer is the standing guard + the pragmas live there). Code-complete + unit/analyzer/context-verified; live gate pending.
fix/archreview-crit13-resilience-config-artifact	8a8b9ec5	#13 — apply per-instance ResilienceConfig from the artifact (75403caa, off the crit10 branch) + #10/#13 DI guard (8a8b9ec5 — ResilienceInvokerFactoryRegistrationTests, from the #12 live gate). Read-path plumbing + invalidate/respawn-on-change; deterministically verified + rig-booted.
fix/docker-dev-serverhistorian-stale-keys	7233e2ba	Rig fix (off master, independent): docker-dev compose's retired Wonderware ServerHistorian keys → gateway shape, disabled by default. Found during the #12 live-gate rig session; central-1 verified booting clean.

Decision (user): one branch per Critical, each off master. #9 (failover test) lives on the crit1 branch (not its own branch off master) because SBR only exists there — it is Critical 1's integration half. The plans are only on the docs branch, so from a critical branch read them via git show docs/archreview-plans-and-drift-fixes:archreview/plans/0X-<name>.md.

Completed

Item	Finding	Branch @ commit	Verification
Critical 1 — split-brain resolver activated	03/S1	fix/archreview-crit1-split-brain-resolver a81dea10	BuildClusterOptions sets KeepOldestOption{DownIfAlone=true}; 3 unit guards (Cluster.Tests 29/29); e2e-verified the resolved ActorSystem config carries SplitBrainResolverProvider + stable-after=15s survives the HOCON/Hosting merge (throwaway spike, removed). HOCON cross-ref comment + Redundancy.md corrected.
Critical 2a — real VT script timeout	02/U2	fix/archreview-crit2-vt-timeout 7fd44f0f	Routes through TimedScriptEvaluator; infinite-loop test (bounded by WaitAsync) proves no hang. Host.Integration VT 18/18.
Critical 2b — ALC-safe compile cache	02/U3	same 7fd44f0f	Swapped raw dict → CompiledScriptCache; new IScriptCacheOwner seam; apply-boundary clear in VirtualTagHostActor; wiring guard test. Runtime VT 12/12.
Doc drift — KNOWN LIMITATION 2 stale	03/U1 = 06/U-1	docs/… 9fadead6 (CLAUDE.md)	historized-ref feed IS wired; rewrote to "value-capture code-complete, live-verify pending".
Doc drift — ScriptAnalysis policy	theme #5	same 9fadead6 (CLAUDE.md)	corrected FleetAdmin → Administrator,Designer roles.
Critical 3 — S7 lazy reconnect	05/STAB-1	fix/archreview-crit3-s7-reconnect 25c0c6f6	New IS7Plc/IS7PlcFactory seam + EnsureConnectedAsync (dispose-dead-handle → reopen on next data call); connection-fatal faults (socket/ErrorCode.ConnectionError) mark the handle dead, data-address errors do NOT; probe loop routes through it as a backstop. 4 unit guards + full S7 suite 234/234; LIVE-verified against real python-snap7 — bounced the container (docker restart), driver observed the outage and resumed Good reads with no redeploy (S7_1500ReconnectTests, double-gated on sim + S7_RECONNECT_BOUNCE_CMD). Also closes the S7 TEST-1 gap.
Critical 4 — TwinCAT native-notification replay	05/STAB-2	fix/archreview-crit4-twincat-replay af318fb4	Native ADS notifications were opaque handles (no replayable intent) → silently orphaned on a client swap. Now NativeRegistration (symbol/type/bit/interval/onChange + swappable handle) hangs off DeviceState; EnsureConnectedAsync split into a gate wrapper + under-gate core that replays every intent onto a freshly-built client (register+replay both under ConnectGate); probe-failure RecycleClientAsync disposes+nulls the wire-dead-but-port-open client so the next tick rebuilds+replays. 4 unit guards + full TwinCAT suite 174/174. Unit-only — no TwinCAT docker fixture exists (integration needs a real TC3 XAR), so the fake-client tests are authoritative (no live gate like S7 had).
Guard — reflection-exhaustive deferred-sink forwarding	03/U2	fix/archreview-u2-deferred-sink-forwarding-guard a65c2ced	New DeferredSinkForwardingReflectionTests (Commons.Tests) reflects over every method of every forwarding interface DeferredAddressSpaceSink implements (IOpcUaAddressSpaceSink + ISurgicalAddressSpaceSink) — plus the sibling DeferredServiceLevelPublisher — invokes each on the wrapper over a DispatchProxy recording inner, and asserts the call reached the inner. A DispatchProxy recorder auto-implements future members (zero maintenance); a guard-of-the-guard asserts the wrapper implements exactly the known forwarding-interface set, so a brand-new capability interface trips too. This closes the F10b/PR#423 trap class and is the prerequisite gate for 03/P1 remove methods + 02/P WriteValues. 3 new tests green (Commons.Tests 57/57); negative control verified — stubbing UpdateFolderDisplayName to a non-forwarding no-op fails the guard with the exact member-named diagnostic. NO production change.
Guard — wire OTOPCUA0001 analyzer + triage	07/C-1	fix/archreview-c1-wire-analyzer f0082af5	Injected the analyzer tree-wide via Directory.Build.props (OutputItemType="Analyzer"), then triaged ~280 surfaced hits: (1) RESILIENCE-DISPATCH-GAP — 7 production sites (DriverInstanceActor ×6 + GenericDriverNodeManager) scoped-#pragma'd as a real tracked gap (CapabilityInvoker built but never wired into dispatch; new task #10); (2) driver-internal self-calls — 3 sites (AbCip/S7 own poll/ack) #pragma'd as intentional; (3) 12 wire-level test/CLI projects → commented NoWarn. Full build green, 0 hits; analyzer 31/31; negative control — dropping one dispatch-gap pragma re-fires OTOPCUA0001 and fails the build, proving it's live tree-wide.
Guard — CI to all suites + fail-on-skip + deflake	07/S-1,S-2,S-4	fix/archreview-s1-s2-ci-coverage 10b89830	S-1: replaced the 5-project unit-tests matrix with ONE whole-solution leg (self-maintaining; now covers Client's 388 + Analyzers' 31 + all drivers). S-2: scripts/ci/assert-not-all-skipped.sh fail-on-skip gate (turns "green==all-skipped" red) — strict on the unit leg, report-only on the fixtureless integration leg. S-4: readiness-signal (SubscribeInvoked/SubscribeAlarmsInvoked) replaces 11 fixed-sleep startup races in the newly-CI'd Client.CLI.Tests. Skip-gate proven locally (real trx OK / all-skip trx exit 1 / report-only / edge cases); Client.CLI.Tests 104/104. CI job execution itself is verifiable only on push (nothing pushed).
#10 — wire CapabilityInvoker into dispatch (seam)	RESILIENCE-DISPATCH-GAP (07/C-1)	fix/archreview-crit10-wire-capability-invoker 62556c24	Discovered the filed plan was infeasible — Runtime is deliberately Polly-free (refs Core.Abstractions, not Core). Fix = a seam mirroring IDriverFactory: IDriverCapabilityInvoker(+null) + IDriverCapabilityInvokerFactory(+null) in Core.Abstractions; CapabilityInvoker implements it; DriverCapabilityInvokerFactory (Core) builds per-instance over the singleton builder+tracker+GetTier; Host DriverFactoryBootstrap registers all three; Runtime SCE resolves like IDriverFactory; DriverInstanceActor wraps its 6 dispatch sites; analyzer taught the interface as a wrapper home; all 6 pragmas removed. Triage corrections: the 7th site (GenericDriverNodeManager) is test-only scaffolding (not wired, re-annotated); tier-defaults only (ResilienceConfig not in the artifact → sub-gap). Added retry/breaker logging to the pipeline builder (the tracker has no reader — Stream E.2/E.3 never shipped — so it ran invisibly). Verified: negative-control (unwrap→OTOPCUA0001 build error); recording-invoker routing test; real-CapabilityInvoker actor-context test (internal ConfigureAwait(false) doesn't leak); analyzer + logging tests; clean solution build; Runtime.Tests 358 / Core.Tests 238 / Analyzers 32. Live behavioral gate pending (recipe in FOLLOWUP-10).
#9 — hard-kill failover integration test	03/S1	fix/archreview-crit1-split-brain-resolver a25c9ed0	Critical 1's load-bearing integration half. TwoNodeClusterHarness.HardKillNodeAAsync (transport-shutdown crash sim → node B sees A Unreachable, no graceful Leave) + HardKillFailoverTests asserting B downs the alone oldest node and takes over as driver role-leader. LIVE-verified in-process (~35s); negative control (force downing-provider-class="") makes it correctly time out. ⚠️ The negative control also surfaced a Crit-1 premise finding (task #11): SBR was already active on master (Akka.Cluster.Hosting applies SplitBrainResolverOption.Default when the typed option is null → reads the pre-existing akka.conf keep-oldest block), so the cluster was NOT NoDowning before Crit-1 — the typed option is reinforcing, not the activator.
#11 — Critical 1 premise correction	03/S1 (from #9)	fix/archreview-crit1-split-brain-resolver eaf78aad	Docs/comment-only fix (no code revert): corrected the "HOCON inert / cluster runs NoDowning / never fails over" framing in BuildClusterOptions XML comment, akka.conf split-brain-resolver comment, docs/Redundancy.md, and SplitBrainResolverActivationTests (summary + ShouldNotBeNull message + method rename). Akka.Cluster.Hosting enables an SBR provider by default (applies SplitBrainResolverOption.Default when the typed option is null → reads the pre-existing akka.conf keep-oldest block), so the cluster was never NoDowning and the typed KeepOldestOption is reinforcing/explicit-in-code, not the activator. Cluster.Tests 29/29; failover behaviour still guarded by HardKillFailoverTests. Architecture.md carried no such framing.
#13 — apply per-instance ResilienceConfig from the artifact	RESILIENCE-DISPATCH-GAP sub-gap (from #10)	fix/archreview-crit13-resilience-config-artifact 75403caa	The DriverInstance.ResilienceConfig column was authored+persisted (AdminUI) and already in the artifact (the composer serializes the whole entity), but the driver-node read path dropped it → tier-defaults only. DriverInstanceSpec now carries it; TryReadSpec reads it; Create takes+layers it (+ logs parse diagnostics, never throws); SpawnChild threads it. Invalidate-on-change: Create calls builder.Invalidate(instance) (the cache is options-blind on a hit) so a respawn with changed options rebuilds; DriverSpawnPlanner respawns on a ResilienceConfig change (incl. null→json), a pure DriverConfig change stays an in-place delta. Verified: override reaches execution (retryCount:0 suppresses tier-A retry; a control retries), invalidate is per-instance, malformed logs+falls-back; planner+artifact tests. Core.Tests 243 (+5), Runtime.Tests 363 (+5). See FOLLOWUP-13.

Task list (TaskCreate IDs)

#	Task	Status
1	Critical 1 — split-brain resolver	✅ completed
2	Critical 2a — VT timeout	✅ completed
3	Critical 2b — CompiledScriptCache	✅ completed
4	Critical 3 — S7 reconnect (05/STAB-1)	✅ completed
5	Critical 4 — TwinCAT ADS re-registration (05/STAB-2)	✅ completed
6	Guard — DeferredAddressSpaceSink reflection forwarding test (03/U2)	✅ completed — a65c2ced; gate for 03/P1 + 02/P now in place
7	Guard — wire OTOPCUA0001 analyzer + triage (07/C-1)	✅ completed — f0082af5; surfaced task #10
8	Guard — CI to all unit suites + fail-on-skip (07/S-1,S-2, pair S-4)	✅ completed — 10b89830
9	Critical 1 follow-up — hard-kill failover integration test (03/S1)	✅ completed — a25c9ed0; surfaced task #11
10	Wire CapabilityInvoker into the dispatch layer (RESILIENCE-DISPATCH-GAP, from 07/C-1)	✅ code-complete + unit/analyzer/context-verified — fix/archreview-crit10-wire-capability-invoker 62556c24 (seam, NOT direct thread — Runtime is Polly-free). Live behavioral gate pending (rig recipe in FOLLOWUP-10). Surfaced a sub-gap: ResilienceConfig not in the deploy artifact → tier-defaults only.
11	Review Critical 1 premise — SBR was already active on master, not NoDowning (from #9)	✅ completed — fix/archreview-crit1-split-brain-resolver eaf78aad; docs/comment-only correction (no code revert), Cluster.Tests 29/29.
12	#10 live behavioral gate — rig session (from #10)	✅ ran 2026-07-08 (PARTIAL) — built the Host image from source (crit13 tip) + stood up the rig; central-1 booted healthy; prod DI binds the real invoker factory confirmed live + by a new deterministic guard (ResilienceInvokerFactoryRegistrationTests 8a8b9ec5). Behavioral retry/breaker log-line not observed (disciplined stop): Modbus's cheap paths don't throw a wrapped exception (Discover offline, Subscribe lazy, reads non-wrapped) — only an idempotent write-to-dead-endpoint (or S7) does, not worth the multi-dependency chase since the retry behavior is deterministically proven. Found + FIXED a rig bug: docker-dev compose's retired Wonderware ServerHistorian keys crashed the Host at bring-up (historian-gateway cutover → reads Endpoint) → fixed on fix/docker-dev-serverhistorian-stale-keys 7233e2ba (off master; gateway-shape keys, disabled by default; central-1 verified booting clean). See FOLLOWUP-10.
13	Plumb per-instance ResilienceConfig through the deploy artifact (from #10)	✅ completed — fix/archreview-crit13-resilience-config-artifact 75403caa; read-path plumbing (write side was already complete), invalidate-on-change + respawn-on-change; Core.Tests 243/Runtime.Tests 363. See FOLLOWUP-13.

Findings surfaced by the guards (NEW — need user decision)

Each has a dedicated follow-up file with the full design/action: FOLLOWUP-10-resilience-dispatch-gap.md · FOLLOWUP-11-crit1-sbr-premise.md · FOLLOWUP-13-resilience-config-artifact.md.

• #10 — RESILIENCE-DISPATCH-GAP (from wiring the analyzer, 07/C-1): ✅ IMPLEMENTED on fix/archreview-crit10-wire-capability-invoker 62556c24 — code-complete + unit/analyzer/context-verified; live behavioral gate pending (recipe in FOLLOWUP-10). The filed "thread CapabilityInvoker into the actor" was infeasible (Runtime is deliberately Polly-free); shipped a seam mirroring IDriverFactory instead. Only 6 of the 7 sites were real (the 7th, GenericDriverNodeManager, is test-only scaffolding). Two residual sub-findings: (a) per-instance ResilienceConfig isn't in the deploy artifact → tier-defaults only ✅ RESOLVED by task #13 (75403caa) — the column was already in the artifact; the read path now applies it (see FOLLOWUP-13); (b) the resilience-status tracker has no reader (Admin /hosts Stream E.2/E.3 never shipped) — added retry/breaker logging as the interim observability surface + the live-verify enabler (still open as an Admin-UI follow-on).
• #11 — Critical 1 premise inaccurate (from the #9 failover negative control): ✅ CORRECTED on fix/archreview-crit1-split-brain-resolver eaf78aad. SBR was already active on master (Akka.Cluster.Hosting's WithClustering applies SplitBrainResolverOption.Default when the typed option is null → enables the downing provider that reads the pre-existing akka.conf keep-oldest block). So the cluster was not running NoDowning; hard-crash failover already worked. Docs/comment-only fix (no code revert — the typed KeepOldestOption is correct belt-and-suspenders): corrected the "HOCON inert / NoDowning → no failover" framing in BuildClusterOptions XML comment, akka.conf, docs/Redundancy.md, and the SplitBrainResolverActivationTests summary/message (+ method rename). Cluster.Tests 29/29. (Architecture.md carried no such framing.) See FOLLOWUP-11.

Suggested next (from 00-INDEX execution order)

All 4 Criticals + all 3 systemic guards (tasks 6/7/8) + the #9 failover test are DONE. Remaining:

1. Resolve the two surfaced findings — task 10 (wire CapabilityInvoker into dispatch; behavior-changing, live-gate) and task 11 (correct Crit-1's rationale/docs).
2. Failure-visibility Highs — 01/S-1 (AddressSpaceApplyOutcome failure field), 06/S-1 (Galaxy fail-closed), 03/S4 (primary-gate default-deny).
3. P-tier — 03/P1 surgical adds/removals (now gated by the U2 forwarding guard), 02/P batched WriteValues.

Key resume facts

• Verification bar met so far: each fix has a deterministic unit guard AND (for SBR) an end-to-end config-resolution check. Follow the same pattern — the review's whole thesis is "unit-green ≠ wired".
• The forwarding-test guard (task 6) is DONE (a65c2ced): 03/P1 (surgical pure-adds) and 02/P-… batch-write each add new sink capability methods that ship inert unless forwarded through DeferredAddressSpaceSink (F10b/PR#423 precedent). The reflection guard now fails loudly if any such member isn't forwarded — add the member to the recorder-covered interface set and it's auto-checked.
• Nothing is pushed. Superseded 2026-07-12: all branches merged to master and pushed to origin/master; the plans now live on master too (no more cross-branch git show needed).
• CI changes can't be run-verified without a push — now pushed; the first real CI-job run should be checked on origin (the re-review verified the YAML statically only).
• Findings #10 and #11 are resolved (see above); the open decisions now come from the 2026-07-12 re-review below.

Re-review 2026-07-12 (master f6eaa267) — outcome + new findings

All seven domain reports were re-run and rewritten in place at f6eaa267 (each carries a Prior-finding status (9cad9ed0 → f6eaa267) table; synthesis in ../00-OVERALL.md). Verdict on the remediation: all 4 Criticals verifiably closed in code (Crit-1 by premise correction), both systemic guards live with negative controls, CI now whole-solution + fail-on-skip (07 stability 2.5→3.5). New findings needing scheduling, ranked:

Rank	Finding	Severity	One-line
1	05/STAB-14	High (regression)	S7 ensure-wrapper rethrows connect-timeout OCE → poll loop's bare catch (OCE) { return; } silently kills all S7 subscriptions after an unreachable-host (non-refusing) outage; live bounce test blind to it (S7Driver.cs:488,1000,1383,1404,1206-1208)
2	01/S-6	High	Operator-authorable timeoutSeconds:0 in ResilienceConfig → Polly ValidationException on every capability call = permanent driver-brick; no range validation in parser or AdminUI form
3	05/STAB-15	High	S7 fatal-classification misses framing/desync + cancel-during-I/O (half-read PDU → permanent desync); probe catch never marks handle dead
4	01/S-8 = 03/S12	Medium	Dispatch hardcodes isIdempotent:true; WriteIdempotentAttribute has zero readers → #13-enabled write retries would retry command writes
5	02/S13	Medium	VT script failure/timeout leaves node stale-Good forever (timeout fix made the failure invisible)
6	01/U-6	Medium	Bulkhead knob parsed/authored/documented but never built into the pipeline — dead knob
7	06/S-11	Medium	Enabled-but-misconfigured ServerHistorian crash-loops Host with raw UriFormatException (only config side fixed in 7233e2ba)
8	06/U-7	Medium	HistorianGateway 0.2.0 proto3-optional DataType change silently alters GatewayTagProvisioner Boolean provisioning; live EnsureTags never run on 0.2.0
9	04/C-7, 04/U-5=03/U10, 01/S-7=03/S13	Medium/Low	Resilience seam edges: lossy AdminUI form round-trip; status tracker has no reader (operators blind); respawn stale-pipeline cache race

Carried-open Highs unchanged from round 1 (no branch targeted them): failure-visibility trio (01/S-1, 06/S-1, 03/S4), AdminUI authz (04/C-1 — stakes rose: resilience config is on the ungated surface), 03/S2/S3 block-bridging, 03/P1 surgical adds (guard prerequisite in place), driver-fleet batch (05), repo hygiene (07). Full prioritized list in 00-OVERALL.md.

Round-2 plans WRITTEN (2026-07-12): all 12 action items have design+implementation plans — R2-01…R2-12 in this directory (+ co-located .tasks.json each, for resumable execution). Index, per-plan task counts, effort, and suggested execution order: 00-INDEX.md §"Round 2 plans". 193 bite-sized TDD tasks total, ~18–24 dev-days end-to-end; R2-01/02/03 (the re-review's new-and-sharp items) are ~2 days combined.

Round-2 EXECUTED — R2-12 Repo hygiene (2026-07-13, branch r2/12-repo-hygiene): DONE. 07/U-5 (7 lib/ AVEVA DLLs removed + gitignored, zero-reference proven), 07/U-4 (root planning files: pending.md/current.md untracked+gitignored, looseends.md deleted, stillpending.md + HISTORIAN-GATEWAY-INTEGRATION-ISSUES.md archived under docs/plans/, docs/Historian.md repointed), 07/S-3 (v2-e2e.yml no-op nightly deleted), 07/P-5 (redundant integration matrix job removed from v2-ci.yml), 07/U-3 (5 alarm operator CLI commands documented — now 13/13), 07/C-5 (StyleGuide.md ScadaBridge→OtOpcUa + minimal root .editorconfig), 07/C-4 (xunit pin 2.9.2→2.9.3

• blocker comment; upstream-blocked — no xunit.v3 Akka TestKit exists). 07/U-6 needed no action (the Wonderware husk dirs are untracked build artifacts, absent in a fresh worktree). Whole-solution build clean. OPEN follow-ups: (a) when a runner-reachable fleet + a real tests/Server/…E2ETests land, add a fresh E2E workflow WITH a zero-match guard (fail if Category=E2E matches nothing); (b) S-2 option #2 — a fixture-backed integration leg (opc-plc services: block, MIN_EXECUTED=1) covering what the unit leg cannot; (c) migrate the 3 xunit v2 holdouts once Akka ships an xunit.v3 TestKit.

Round-2 remediation landed

Plan	Findings	Branch	Commits	Status
R2-06	06/S-11 (rank 7), 06/U-7 (rank 8), 06/C-7	r2/06-serverhistorian-failfast (off 1676c8f4)	3a049a13…4640ecb1 (10 task commits)	Code-complete, offline-verified. S-11: fail-fast ServerHistorianOptionsValidator (consumer-gated on ServerHistorian:Enabled OR AlarmHistorian:Enabled, fails empty/non-http(s) Endpoint at startup) + AddValidatedOptions wiring + adapter defense-in-depth (TryCreate→named InvalidOperationException, no more raw UriFormatException crash-loop) + Validate() warning parity + alarm-only-mode startup warning. U-7: pinned Boolean→Int1 explicit-presence (HasDataType) + HISTGW_BOOL_SANDBOX_TAG live Boolean EnsureTags leg + retype probe (skip-clean offline) + upsert/retype migration note (docs/Historian.md). C-7: keep-populating decision + dead-Quality doc comments (0.2.0 C-002). Live leg operator-gated (T12) — pending a VPN run on a 0.2.0 gateway (records the observed in-place retype policy back into docs/Historian.md; also discharges 06/U-2's full documented run).
R2-05	04/C-1 (High)	r2/05-adminui-authz (off 1676c8f4)	183b72b7…6c1f05cb (5 commits)	Code-complete, offline-verified. All 38 routable AdminUI pages carry an explicit named-policy @attribute via new AdminUiPolicies constants (Security/Auth/). New ConfigEditor (Administrator+Designer) gates the 20-page config-authoring surface incl. the 8 driver pages that author live ResilienceConfig + /api/script-analysis/*; new AuthenticatedRead on the 16 read pages; FleetAdmin/DriverOperator unchanged (literals→constants). Anti-drift guard PageAuthorizationGuardTests (bans bare [Authorize]/Roles=/unknown-policy/unclassified) + AdminUiPoliciesTests semantics matrix. Corrections folded in: Reservations + ClusterRedundancy read-only (→AuthenticatedRead); Home had no attribute (now AuthenticatedRead). AdminUI.Tests 517/517, Security.Tests 80/80. Live /run positive pass deferred (T15) — docker-dev DisableLogin=true auto-admin can't observe deny; negative proof is CI-side (both guard tests).
R2-01	05/STAB-14 (High regression, rank 1), 05/STAB-15 (rank 3), 05/STAB-8 seam	r2/01-s7-fault-hardening (off 1676c8f4)	fcdcf0e2…bdb7ed41 (13 task commits)	Code-complete, offline-verified. All prod changes in S7Driver.cs. STAB-14: connect-timeout surfaces as TimeoutException (not an escaping OCE) at the EnsureConnectedAsync/InitializeAsync source + when(token.IsCancellationRequested) filters on the read/write ensure-wrappers and all three poll-loop OCE catches — subscription poll loops survive an unreachable-host outage and recover (T2 regression RED→GREEN). STAB-15: IsS7ConnectionFatal broadened to the ISO-on-TCP framing surface (TPKT/TPDU/WrongNumberOfBytes/PlcException{WrongNumberReceivedBytes}, NOT InvalidDataException); mid-PDU cancel marks handle dead; probe classifies+marks-dead under the gate. STAB-8 (S7 part): EnsureConnectedAsync documented as the R2-09 connect-throttle seam (note-only). Driver.S7.Tests 246/246. Live outage gate deferred (T11) — S7_1500ConnectTimeoutOutageTests (env S7_TIMEOUT_OUTAGE_START_CMD/STOP_CMD, SYN-blackhole) skips clean offline; the connect-timeout class the docker restart bounce can't produce. (Pre-existing unrelated CLI docs-scan test fails at base — out of scope.)
R2-03	02/S13 (rank 5), 02/S12, 02/P7	r2/03-vt-failure-quality (off 1676c8f4)	b9775485…6bb91e8d (12 task commits)	Code-complete, offline-verified. S13: VT script failure/timeout degrades the node to OpcUaQuality.Bad (carrying last-known value) instead of freezing on last-Good forever — additive EvaluationResult.Quality, bridge historizes 0x80020000u BadInternalError, published once per Good→Bad transition gated on all deps arrived (inputs-ready, no cold-start flash), recovery force-publishes next Good. S12: apply-boundary ClearCompiledScripts racing an in-flight eval into ObjectDisposedException retried once in RoslynVirtualTagEvaluator (re-fetch→recompile into current generation); stress guard reproduces pre-fix, deterministically green post-fix. P7: apply-boundary clear gated on a changed deployed expression set (HashSet.SetEquals, ordinal) — identical redeploy / rename-with-same-expressions no longer forces full VT recompile. Runtime.Tests 372/372, RoslynVirtualTagEvaluatorTests 20/20. Deferred (T11/T12): full cross-suite sweep + docker-dev live-/run (fail→Bad→recover + P7 no-recompile spot-check).
R2-02	01/S-6 (High, rank 2), 01/U-6, 01/S-8=03/S12 (rank 4), 04/C-7, 01/S-7=03/S13	r2/02-resilience-config-hardening (off 1676c8f4)	4be13429…dd82e740 (17 task commits)	Code-complete, offline-verified. S-6: parser clamps timeout/retry/breaker overrides (timeout≤0→tier default, breaker==1→2, caps) with diagnostics + belt-and-braces Build clamp so pipeline construction can never throw Polly ValidationException (new ResiliencePolicyRanges in Core.Abstractions); brick-repro + Build_NeverThrows_ForHostileDirectOptions + factory-seam wiring proof; AdminUI Validate() range warnings. S-8: parser forces Write/AlarmAck retry→0 (spec invariant) and dispatch flips to isIdempotent:false (RED-first on the Runtime wiring test); non-idempotent arm caches the no-retry snapshot once/invoker (01/P-4) + tracker in-flight accounting; WriteIdempotentAttribute false claim corrected; retry cells disabled. U-6: bulkhead knob deleted end-to-end (migration-free; stored keys ignored) + Options_properties_are_exactly_the_pipeline_wired_set inertness guard. C-7: ResilienceFormModel JsonObject-bag round-trip preserves unknown keys; malformed JSON → ParseFailed lock + verbatim ToJson + discard button. S-7: per-Create Interlocked options-generation in the pipeline-cache key kills the respawn stale-pipeline race. Core resilience 121/121, AdminUI form 10/10, Analyzers 32/32, Runtime 364/364; 0 production OTOPCUA0001. Deferred (T15/T18): docker-dev :9200 /run (warning banner, input lockout, unknown-key survival) + whole-solution sweep.
R2-10	03/U10=04/U-5 (Medium/Low, action #10)	r2/10-resilience-observability (off 46fedda3)	205d3980…<docs> (11 task commits)	Code-complete, offline-verified. The resilience status tracker now has a production reader, mirroring the driver-health flow. Core truthfulness: RecordBreakerClose + derived IsBreakerOpen (breaker-open state derivable), OnClosed records the close, and RecordSuccess wired onto the invoker success paths (ConsecutiveFailures is a true consecutive counter, not stuck-forever). Transport: DriverResilienceStatusChanged Commons contract → periodic 5 s full-snapshot publisher DriverResilienceStatusPublisherService (driver nodes, registered in AddOtOpcUaDriverFactories with wiring guard ResilienceStatusReaderRegistrationTests — closes the built-but-never-wired class) → driver-resilience-status DPS topic → per-admin-node DriverResilienceStatusBridge → IDriverResilienceStatusStore → resilience chips (breaker-OPEN / consecutive-failures / in-flight / staleness) on DriverStatusPanel.razor, read in-process (self-HubConnection ban honored). Core.Tests resilience 31/31 (incl. deterministic breaker-close via a controllable TimeProvider), Host.IntegrationTests publisher+guard 6/6, AdminUI.Tests store 5/5; full-solution build clean. Deferred (T10): docker-dev live-/run — S7 dead-endpoint breaker-open observed on BOTH centrals + recovery-clear + staleness-dim (also the first live sighting of the #10 circuit-breaker OPENED log line, closing FOLLOWUP-10's behavioral gate).

| R2-08 | 03/S2 (High, action #8), 03/S3 (High) | r2/08-ldap-historyread-async (off 1a698cbb) | ce6d9e21…<docs> (14 task commits) | Code-complete, offline-verified. S2 (LDAP async/timeout): the SDK raises the impersonation callback synchronously under a SessionManager-wide m_eventLock (verified vs. 1.5.378 — a hung bind serializes ALL activations), so the fix is entirely at the LdapOpcUaUserAuthenticator boundary: five new Security:Ldap keys (ConnectionTimeoutMs=10000 projected into the shared lib, AuthTimeoutMs=15000 hard whole-flow WaitAsync bound → fail-closed "timed out", MaxConcurrentAuthentications=8 SemaphoreSlim with release-on-core-completion so orphaned binds still count, OutageFailureThreshold=3/OutageCooldownSeconds=15 directory-outage circuit → fast-deny "unavailable", half-open probe) + LdapAuthResult.IsSystemFailure seam (mapped from ServiceAccountBindFailed) distinguishing outage from credential-deny + LdapOptionsValidator rules + HandleImpersonation contract doc + docs/security.md. Stall-repro RED→GREEN; offline outage-sim through the real HandleImpersonation against an unroutable host. S3 (HistoryRead fan-out): all four HistoryRead arms async-ified (ServeNodeAsync/ServeRawPagedAsync/ServeEventsAsync, OCE→BadTimeout) then bounded per-node fan-out (RunBounded semaphore + single arm-boundary bridge) + per-request deadline CTS (HistoryReadDeadline=60s) + process-wide lazy batch limiter (MaxConcurrentHistoryReadBatches=16 → flood degrades to BadTooManyOperations); 16×4=64 in-flight cap; three new ServerHistorian keys + node-manager props + hosted-service wiring + docs/Historian.md/CLAUDE.md tables. Sequential-serving RED→GREEN. Host.IntegrationTests (LDAP resilience) 13/13, Security.Tests 11/11, Runtime.Tests options 11/11, OpcUaServer.Tests HistoryRead 30/30. Deferred (live): S2 docker-dev unreachable-LDAP + GLAuth outage/recovery cycle; S3 Category=LiveIntegration gateway soak. | | R2-11 | 01/C-1, 01/P-1 + 05/CONV-2, UNDER-1, UNDER-6 | r2/11-tagconfig-consolidation (off 46fedda3) | first 2e14fe1f … (24 task commits) | Code-complete, offline-verified. Part 1 (C-1/P-1): single TagConfigIntent.Parse + DeviceConfigIntent in Commons replace the four byte-parity TagConfig parse copies (composer/artifact/walker/DraftValidator) — composer now parses once per tag (P-1); a permanent Runtime.Tests/TagConfigCorpusParityTests (30-blob corpus) guards compose≡decode; grep "MUST parse identically" src/ = 0. Part 2 (CONV-2): shared strict-capable TagConfigJson readers in Core.Abstractions replace the six copy-pasted ReadEnums; all six driver parsers gain Inspect() warnings + honour the writable key (FOCAS forced read-only, 05/UNDER-1); FOCAS equipment tags run the capability-matrix pre-flight (05/UNDER-6); Modbus probe parses the factory DTO shape (timeoutMs no longer dropped). New EquipmentTagConfigInspector in ControlPlane + AdminOperationsActor deploy-gate wiring with Deployment:TagConfigValidationMode (Warn default | Error opt-in), proven through the actor (F10b consuming test). AdminUI: writable checkbox in the six typed editors. Phase-C follow-up (deferred, recorded here): flip the runtime parsers to strict (typo'd enum ⇒ BadNodeIdUnknown instead of wrong-width Good) once fleets have run Error mode clean — behavior change deliberately not shipped in this plan. Deferred (T22/T24): docker-dev /run of one typed editor's Writable checkbox + whole-solution serial sweep. |

| R2-07 (Phases 1+2+3) | 03/P1 (action #7, surgical pure-adds/removes/mixed) | r2/07-surgical-pure-adds (off 1a698cbb) | bb226f45…e2b47638 (T1–T14) | Phases 1+2 code-complete, offline-verified. New pure AddressSpaceChangeClassifier (Empty/AttributeOnly/PureAdd/PureRemove/AddRemoveMix/Rebuild; first-match-wins, default-closed Rebuild; the F10b tag/vtag surgical-eligibility predicates moved here as internal static). AddressSpaceApplier.Apply routes by kind. Phase 1 (PureAdd/AttributeOnly): skip RebuildAddressSpace (the idempotent Materialise passes create exactly the added nodes; existing subscriptions survive); adds announced via ComputeAddAnnouncements+AnnounceAddedNodes (dedup affected-parent → existing guard-covered RaiseNodesAddedModelChange), fired by OpcUaPublishActor after the 4 passes gated on !outcome.RebuildCalled; MaterialiseAlarmCondition made idempotent (skip-if-present-and-same-kind keeps the condition NodeState alive). NO new sink members in Phase 1. Phase 2 (PureRemove): 3 new ISurgicalAddressSpaceSink members (RemoveVariableNode/RemoveAlarmConditionNode/RemoveEquipmentSubtree) — guard-first with the observed negative control (DeferredSinkForwardingReflectionTests fails member-named until DeferredAddressSpaceSink forwards) + SdkAddressSpaceSink forwards + OtOpcUaNodeManager in-place teardown (scoped map cleanup, notifier demotion via RemoveRootNotifier, BuildNodesRemovedModelChange verb=NodeDeleted reported outside Lock). Applier PureRemove arm: terminal Bad / RemovedConditionState writes (closing the removed-alarm-tag today-gap) → subsumption partition (removed-equipment children folded into the subtree remove) → in-place removes → any false/throw ⇒ full-rebuild ratchet; non-surgical sink ⇒ rebuild. Phase 3 (AddRemoveMix): removes-then-adds within one apply — the remove pass runs in Apply (no rebuild), then the publish actor's Materialise passes create the adds + AnnounceAddedNodes announces them; an id reused across the remove+add sets is remove-then-recreated as a fresh NodeState (correct — a different tag). Guard stays green (Commons.Tests 17/17), OpcUaServer.Tests 336/336, Runtime.Tests 398/398. 03/P1 marked REMEDIATED in archreview/03-server-runtime.md. Deferred-live (T5, T6, T12, T14): SubscriptionSurvivalTests add + remove + mixed cases (authored + compile-green; NOT run — *.IntegrationTests memory constraint) and the docker-dev live-/run add/remove/mixed gates (kind=PureAdd/PureRemove/AddRemoveMix, rebuild=False; survivor subscription uninterrupted; removed node → BadNodeIdUnknown) — MUST run in the serial heavy pass (F10b inertness risk). |

Report anchor drifts to fold into the next report refresh (from the R2-02 verification pass): (1) 01-core-composition.md U-6 anchor DriverResilienceOptions.cs:23-25 → the two bulkhead properties sat at :23 and :29 (now deleted); (2) 01-core-composition.md U-6 anchor DriverResiliencePipelineBuilder.cs:99-177 → Build was :99-176. All other cited lines were exact at f6eaa267.

✅ Round-2 COMPLETE — all 12 plans merged (2026-07-13)

All 12 round-2 plans were executed (isolated worktree per plan, per-task TDD, pathspec-scoped commits) and merged to master + pushed to origin/master in 3 dependency-ordered waves, each plan landing as its own Gitea PR (closed-as-merged) behind a dotnet build barrier. Final state: master 31c98200.

Wave	Plans (PR#)
A (no cross-plan deps)	R2-12 #426 · R2-06 #427 · R2-05 #428 · R2-01 #429 · R2-03 #430 · R2-02 #431
B (parse-site barrier)	R2-10 #432 · R2-04 #433 · R2-11 #434
C (largest, depend on earlier)	R2-08 #435 · R2-07 #436 · R2-09 #437

Cross-plan mutexes honored: R2-01 landed before R2-09 (PollGroupEngine v2 / S7 fork retirement); R2-11's TagConfigIntent.Parse consolidation landed before R2-09's driver parse edits; the three plans that all edit AddressSpaceApplier.cs + DriverHostActor.cs + OtOpcUaNodeManager.cs (R2-04/R2-07/R2-08/R2-10/R2-11) were merged sequentially with the shared-file auto-merges verified by targeted unit runs each time.

Consolidated unit sweep on merged master 31c98200 (integration suites excluded — ~16 GB/run leak)

Solution dotnet build clean (0 errors). All R2-relevant unit suites GREEN, incl.: OpcUaServer 341, Runtime 398, Commons(+forwarding guard) 122, Core 276, Core.Abstractions 128, Core.Scripting 137, Core.VirtualTags 59, Cluster 29, Security 84, ControlPlane 78, Analyzers 32, AdminUI 537, Client.CLI 104, Client.Shared 182, Client.UI 126, Core.AlarmHistorian 29, Core.ScriptedAlarms 71, and all drivers (S7 254, Modbus 312+161, AbCip 336, AbLegacy 209, TwinCAT 189, FOCAS 265, Galaxy 304, Galaxy.Browser 21, Historian.Gateway 102, OpcUaClient 131).

3 failures, all confirmed PRE-EXISTING / environmental (git diff 1676c8f4..HEAD empty on their paths — R2 never touched them):

1. Core.Abstractions.Tests.InterfaceIndependenceTests.AllPublicTypes_LiveInRootNamespace — flags the pre-R2 Core.Abstractions.Historian.* continuous-historization types living in a sub-namespace. Pre-existing convention debt — needs-owner-decision (move the types to root, or relax the test to allow the intentional Historian grouping).
2. Driver.S7.Cli.Tests.SubscribeCommandConsoleHandlerCommentTests... — pre-existing docs-scan test looking for a "CliFx console" phrase removed by 9cad9ed0. Follow-up: repoint or delete the phrase-scan.
3. Driver.OpcUaClient.Browser.Tests (3) — Connection refused; need the live opc-plc endpoint (10.100.0.35:50000), unreachable offline. Environmental — pass with the fixture up.

Deferred heavy / live-/run pass (serialized — single-tenant rig; controller-owned)

Every plan's integration-suite and live-/run gates were deferred (not skipped) so nothing OOMs the box and the shared docker-dev rig / VPN infra isn't contended. To run one-at-a-time:

• Full integration-suite sweep (10 *.IntegrationTests projects, one at a time): discharges R2-02 T18, R2-03 T11, R2-07 T5/T6, R2-08 T14, R2-11 T24.
• docker-dev :9200 live-/run (rebuild BOTH centrals): R2-02 T15 (ResilienceConfig warning banner / input lockout / unknown-key survival), R2-03 T12 (VT fail→Bad→recover + P7 no-recompile), R2-10 T10 (breaker-open on both centrals + recovery + FOLLOWUP-10 log line), R2-11 T22 (typed-editor Writable checkbox), R2-07 T12 (surgical add/remove/mixed — F10b inertness risk, MUST run), R2-05 T15 (over-gating regression only — deny unobservable under DisableLogin).
• 2-node redundancy rig: R2-04 T13/T15 — behavior-affecting S4 primary-gate default-deny; MUST run before trusting the failover-role change.
• Genuinely external / needs-user: R2-06 T12 (VPN to a 0.2.0 HistorianGateway for the live EnsureTags retype probe), R2-08 live legs (real GLAuth outage/recovery cycle; Category=LiveIntegration gateway soak), R2-01 T11 (S7 SYN-blackhole outage on the 10.100.0.35 s7_1500 fixture).

Deferred behavior-change follow-up (deliberately not shipped)

R2-11 Phase C — flip the runtime driver parsers to strict (typo'd enum ⇒ BadNodeIdUnknown instead of a wrong-width Good). Land only after fleets have run Deployment:TagConfigValidationMode=Error clean.

docker-dev live-/run pass (2026-07-13) — CONCLUDED; 5 gates LIVE-confirmed on the built-from-source 6aeb53a9 rig

Bottom line: R2-02, R2-11, R2-07 (both surgical phases), R2-04 (achievable failover), and R2-10 (resilience pipeline in dispatch) verified live on the docker-dev central cluster. R2-05 partial-by-design; R2-03 unit-verified (rig has no driver data). Everything still open needs infra this box lacks: the *.IntegrationTests sweep needs the 10.100.0.35 driver fixtures + ephemeral LDAP up; R2-06 needs VPN to a 0.2.0 gateway; R2-08 a live GLAuth outage cycle; R2-01 the S7 blackhole fixture. Those are the only Round-2 items not yet exercised.

Brought up the 2-node central cluster (central-1+central-2, SQL, traefik) from source at merged master 6aeb53a9 (single-tenant rig; drove the :9200 AdminUI via browser + Client.CLI on :4840/:4841).

• R2-02 (S-6 High) ✅ LIVE — on the Modbus driver's Resilience overrides, entering timeoutSeconds:0 renders the exact range-validation banner: "Out-of-range values will be clamped to a safe value on deploy: Read timeout must be between 1 and 86400 seconds (got 0)." S-8 ✅ — the Write and AlarmAcknowledge RETRIES cells are disabled ("never").
• R2-11 (T22) ✅ LIVE — the typed Modbus tag editor shows the Writable toggle; saving persists writable: yes.
• R2-07 (P1 — F10b MUST) ✅ LIVE — central-1 boot apply logged AddressSpaceApplier: applied plan (kind=PureAdd, added=91, removed=0, changed=0, rebuild=False); deleting a UNS virtual-tag + redeploy logged (kind=PureRemove, added=0, removed=1, changed=0, rebuild=False) and the node vanished from the address space (capper-03 lost its production-count child). Surgical add AND remove both execute in prod with no full rebuild — the remove path's 3 new sink members are wired, not inert.
• R2-04 (S4 MUST) ✅ LIVE (achievable guarantee) — younger-crash (kill central-2): SBR took decision DownUnreachable and is downing [central-2], Member removed, central-1 stays Up, AdminUI serving, ServiceLevel 250 (sole primary, no split-brain). Oldest-crash (kill central-1) reproduced the documented keep-oldest asymmetry live: the survivor took SBR … DownReachable … including myself and CoordinatedShutdown (ExitCode 3) — i.e. crashing the oldest node does not fail over. This is the pre-registered SBR topology gap (a deferred USER strategy decision), not an R2 regression; R2-04's PrimaryGatePolicy write-gate change is orthogonal + unit-verified.
• R2-05 (T15) ✅ (partial, as expected) — every config/read page renders under the all-roles multi-role-test session (named-policy gating didn't over-gate a legitimate admin); deny is unobservable under docker-dev DisableLogin=true, as documented.
• R2-10 (T10) ✅ pipeline LIVE (breaker-OPEN state not forced). The /hosts page shows the driver-health flow live (MAIN-galaxy-eq HEALTHY; MAIN-modbus-eq FAULTED "Connection refused"; MAIN-opcua-eq FAULTED "failed to connect to any endpoints"). The failing drivers' subscribe/discover exceptions propagate through the full Polly resilience pipeline — central-1 stack traces show Polly.CircuitBreaker → Polly.Retry → Polly.Timeout → CapabilityInvoker.ExecuteAsync (CapabilityInvoker.cs:84), i.e. the #10 dispatch-gap fix + R2-10's tracker data-source are live in prod. The explicit breaker OPENED log line + the active resilience chip were NOT triggered: R2-09's ConnectionBackoff throttles retries, so consecutive failures don't accumulate fast enough to open the Polly breaker within its window (the documented FOLLOWUP-10 difficulty — the reliable trigger is a rapid idempotent write-to-dead-endpoint). Reader/store/publisher/bridge are unit-verified (Core resilience 128, publisher+guard 6, store 8).
• R2-03 (T12) — unit-verified; live-blocked on this data-less rig. Every materialised node reads 0x80320000 Bad_WaitingForInitialData because no live driver data flows (Modbus 10.100.0.35:5020 unreachable in-rig; VTs depend on driver data), so a VT can't be staged Good→Bad; and the Client.CLI read surfaces any Bad StatusCode as a FetchValueAsync exception rather than printing it. Good→Bad-with-last-value transition is covered by unit suites (Runtime 372, RoslynVTEvaluator 20). Would be live-observable with a reachable driver fixture feeding the VT.

Rig note: the pass mutated the persisted ConfigDb (added a drv-modbus-line3-01 Modbus driver on MAIN; removed capper-03's production-count virtual tag for the PureRemove test). Reset with docker compose -f docker-dev/docker-compose.yml down -v. The remaining integration-suite sweep + genuinely-external gates (R2-06 VPN, R2-08 live LDAP, R2-01 S7 blackhole) still need their infra.