lmxopcua/archreview/plans/STATUS.md

Arch-Review Remediation — Status & Handoff

Snapshot: 2026-07-12 · Review baseline: master 9cad9ed0; re-review baseline: master f6eaa267 · this file is the single source of truth for remediation progress. Update it as work lands.

2026-07-12 UPDATE — everything below the topology table landed: all remediation branches were merged to master and pushed to origin/master (merge commits 7834b3b7…b67bd9e8; final state f6eaa267). The topology table is kept for history but is no longer live. A full re-review of all seven domains ran at f6eaa267 — see ../00-OVERALL.md and the "Re-review 2026-07-12" section at the bottom of this file for the new findings (one High regression in the S7 fix, one High ResilienceConfig validation gap, and a batch of resilience-seam Mediums).

Branch topology (HISTORICAL — as of 2026-07-08; since merged + pushed)

Branch	Head	Contains
master	9cad9ed0	untouched baseline
docs/archreview-plans-and-drift-fixes	9fadead6	the 7 domain plans + 00-INDEX.md + this STATUS.md + the two CLAUDE.md doc-drift fixes. (the plans live ONLY here)
fix/archreview-crit1-split-brain-resolver	eaf78aad	Critical 1 (03/S1) typed SBR option + guard test + Redundancy.md; + #9 hard-kill failover test (a25c9ed0); + #11 premise correction (eaf78aad)
fix/archreview-crit2-vt-timeout	7fd44f0f	Critical 2 (02/U2 + 02/U3) + tests
fix/archreview-crit3-s7-reconnect	25c0c6f6	Critical 3 (05/STAB-1) IS7Plc seam + lazy reconnect + unit guards + live bounce test
fix/archreview-crit4-twincat-replay	af318fb4	Critical 4 (05/STAB-2) native-notification replay + probe-recycle + unit guards
fix/archreview-u2-deferred-sink-forwarding-guard	a65c2ced	Guard 03/U2 — reflection-exhaustive deferred-sink forwarding test (+ DeferredServiceLevelPublisher)
fix/archreview-c1-wire-analyzer	f0082af5	Guard 07/C-1 — wire OTOPCUA0001 analyzer tree-wide + triage (surfaced the RESILIENCE-DISPATCH-GAP, task #10)
fix/archreview-s1-s2-ci-coverage	10b89830	Guard 07/S-1,S-2,S-4 — whole-solution CI leg + fail-on-skip gate + CLI sleep deflake
fix/archreview-crit10-wire-capability-invoker	62556c24	#10 — wire CapabilityInvoker into dispatch via a Core.Abstractions seam (off fix/archreview-c1-wire-analyzer, NOT master — the analyzer is the standing guard + the pragmas live there). Code-complete + unit/analyzer/context-verified; live gate pending.
fix/archreview-crit13-resilience-config-artifact	8a8b9ec5	#13 — apply per-instance ResilienceConfig from the artifact (75403caa, off the crit10 branch) + #10/#13 DI guard (8a8b9ec5 — ResilienceInvokerFactoryRegistrationTests, from the #12 live gate). Read-path plumbing + invalidate/respawn-on-change; deterministically verified + rig-booted.
fix/docker-dev-serverhistorian-stale-keys	7233e2ba	Rig fix (off master, independent): docker-dev compose's retired Wonderware ServerHistorian keys → gateway shape, disabled by default. Found during the #12 live-gate rig session; central-1 verified booting clean.

Decision (user): one branch per Critical, each off master. #9 (failover test) lives on the crit1 branch (not its own branch off master) because SBR only exists there — it is Critical 1's integration half. The plans are only on the docs branch, so from a critical branch read them via git show docs/archreview-plans-and-drift-fixes:archreview/plans/0X-<name>.md.

Completed

Item	Finding	Branch @ commit	Verification
Critical 1 — split-brain resolver activated	03/S1	fix/archreview-crit1-split-brain-resolver a81dea10	BuildClusterOptions sets KeepOldestOption{DownIfAlone=true}; 3 unit guards (Cluster.Tests 29/29); e2e-verified the resolved ActorSystem config carries SplitBrainResolverProvider + stable-after=15s survives the HOCON/Hosting merge (throwaway spike, removed). HOCON cross-ref comment + Redundancy.md corrected.
Critical 2a — real VT script timeout	02/U2	fix/archreview-crit2-vt-timeout 7fd44f0f	Routes through TimedScriptEvaluator; infinite-loop test (bounded by WaitAsync) proves no hang. Host.Integration VT 18/18.
Critical 2b — ALC-safe compile cache	02/U3	same 7fd44f0f	Swapped raw dict → CompiledScriptCache; new IScriptCacheOwner seam; apply-boundary clear in VirtualTagHostActor; wiring guard test. Runtime VT 12/12.
Doc drift — KNOWN LIMITATION 2 stale	03/U1 = 06/U-1	docs/… 9fadead6 (CLAUDE.md)	historized-ref feed IS wired; rewrote to "value-capture code-complete, live-verify pending".
Doc drift — ScriptAnalysis policy	theme #5	same 9fadead6 (CLAUDE.md)	corrected FleetAdmin → Administrator,Designer roles.
Critical 3 — S7 lazy reconnect	05/STAB-1	fix/archreview-crit3-s7-reconnect 25c0c6f6	New IS7Plc/IS7PlcFactory seam + EnsureConnectedAsync (dispose-dead-handle → reopen on next data call); connection-fatal faults (socket/ErrorCode.ConnectionError) mark the handle dead, data-address errors do NOT; probe loop routes through it as a backstop. 4 unit guards + full S7 suite 234/234; LIVE-verified against real python-snap7 — bounced the container (docker restart), driver observed the outage and resumed Good reads with no redeploy (S7_1500ReconnectTests, double-gated on sim + S7_RECONNECT_BOUNCE_CMD). Also closes the S7 TEST-1 gap.
Critical 4 — TwinCAT native-notification replay	05/STAB-2	fix/archreview-crit4-twincat-replay af318fb4	Native ADS notifications were opaque handles (no replayable intent) → silently orphaned on a client swap. Now NativeRegistration (symbol/type/bit/interval/onChange + swappable handle) hangs off DeviceState; EnsureConnectedAsync split into a gate wrapper + under-gate core that replays every intent onto a freshly-built client (register+replay both under ConnectGate); probe-failure RecycleClientAsync disposes+nulls the wire-dead-but-port-open client so the next tick rebuilds+replays. 4 unit guards + full TwinCAT suite 174/174. Unit-only — no TwinCAT docker fixture exists (integration needs a real TC3 XAR), so the fake-client tests are authoritative (no live gate like S7 had).
Guard — reflection-exhaustive deferred-sink forwarding	03/U2	fix/archreview-u2-deferred-sink-forwarding-guard a65c2ced	New DeferredSinkForwardingReflectionTests (Commons.Tests) reflects over every method of every forwarding interface DeferredAddressSpaceSink implements (IOpcUaAddressSpaceSink + ISurgicalAddressSpaceSink) — plus the sibling DeferredServiceLevelPublisher — invokes each on the wrapper over a DispatchProxy recording inner, and asserts the call reached the inner. A DispatchProxy recorder auto-implements future members (zero maintenance); a guard-of-the-guard asserts the wrapper implements exactly the known forwarding-interface set, so a brand-new capability interface trips too. This closes the F10b/PR#423 trap class and is the prerequisite gate for 03/P1 remove methods + 02/P WriteValues. 3 new tests green (Commons.Tests 57/57); negative control verified — stubbing UpdateFolderDisplayName to a non-forwarding no-op fails the guard with the exact member-named diagnostic. NO production change.
Guard — wire OTOPCUA0001 analyzer + triage	07/C-1	fix/archreview-c1-wire-analyzer f0082af5	Injected the analyzer tree-wide via Directory.Build.props (OutputItemType="Analyzer"), then triaged ~280 surfaced hits: (1) RESILIENCE-DISPATCH-GAP — 7 production sites (DriverInstanceActor ×6 + GenericDriverNodeManager) scoped-#pragma'd as a real tracked gap (CapabilityInvoker built but never wired into dispatch; new task #10); (2) driver-internal self-calls — 3 sites (AbCip/S7 own poll/ack) #pragma'd as intentional; (3) 12 wire-level test/CLI projects → commented NoWarn. Full build green, 0 hits; analyzer 31/31; negative control — dropping one dispatch-gap pragma re-fires OTOPCUA0001 and fails the build, proving it's live tree-wide.
Guard — CI to all suites + fail-on-skip + deflake	07/S-1,S-2,S-4	fix/archreview-s1-s2-ci-coverage 10b89830	S-1: replaced the 5-project unit-tests matrix with ONE whole-solution leg (self-maintaining; now covers Client's 388 + Analyzers' 31 + all drivers). S-2: scripts/ci/assert-not-all-skipped.sh fail-on-skip gate (turns "green==all-skipped" red) — strict on the unit leg, report-only on the fixtureless integration leg. S-4: readiness-signal (SubscribeInvoked/SubscribeAlarmsInvoked) replaces 11 fixed-sleep startup races in the newly-CI'd Client.CLI.Tests. Skip-gate proven locally (real trx OK / all-skip trx exit 1 / report-only / edge cases); Client.CLI.Tests 104/104. CI job execution itself is verifiable only on push (nothing pushed).
#10 — wire CapabilityInvoker into dispatch (seam)	RESILIENCE-DISPATCH-GAP (07/C-1)	fix/archreview-crit10-wire-capability-invoker 62556c24	Discovered the filed plan was infeasible — Runtime is deliberately Polly-free (refs Core.Abstractions, not Core). Fix = a seam mirroring IDriverFactory: IDriverCapabilityInvoker(+null) + IDriverCapabilityInvokerFactory(+null) in Core.Abstractions; CapabilityInvoker implements it; DriverCapabilityInvokerFactory (Core) builds per-instance over the singleton builder+tracker+GetTier; Host DriverFactoryBootstrap registers all three; Runtime SCE resolves like IDriverFactory; DriverInstanceActor wraps its 6 dispatch sites; analyzer taught the interface as a wrapper home; all 6 pragmas removed. Triage corrections: the 7th site (GenericDriverNodeManager) is test-only scaffolding (not wired, re-annotated); tier-defaults only (ResilienceConfig not in the artifact → sub-gap). Added retry/breaker logging to the pipeline builder (the tracker has no reader — Stream E.2/E.3 never shipped — so it ran invisibly). Verified: negative-control (unwrap→OTOPCUA0001 build error); recording-invoker routing test; real-CapabilityInvoker actor-context test (internal ConfigureAwait(false) doesn't leak); analyzer + logging tests; clean solution build; Runtime.Tests 358 / Core.Tests 238 / Analyzers 32. Live behavioral gate pending (recipe in FOLLOWUP-10).
#9 — hard-kill failover integration test	03/S1	fix/archreview-crit1-split-brain-resolver a25c9ed0	Critical 1's load-bearing integration half. TwoNodeClusterHarness.HardKillNodeAAsync (transport-shutdown crash sim → node B sees A Unreachable, no graceful Leave) + HardKillFailoverTests asserting B downs the alone oldest node and takes over as driver role-leader. LIVE-verified in-process (~35s); negative control (force downing-provider-class="") makes it correctly time out. ⚠️ The negative control also surfaced a Crit-1 premise finding (task #11): SBR was already active on master (Akka.Cluster.Hosting applies SplitBrainResolverOption.Default when the typed option is null → reads the pre-existing akka.conf keep-oldest block), so the cluster was NOT NoDowning before Crit-1 — the typed option is reinforcing, not the activator.
#11 — Critical 1 premise correction	03/S1 (from #9)	fix/archreview-crit1-split-brain-resolver eaf78aad	Docs/comment-only fix (no code revert): corrected the "HOCON inert / cluster runs NoDowning / never fails over" framing in BuildClusterOptions XML comment, akka.conf split-brain-resolver comment, docs/Redundancy.md, and SplitBrainResolverActivationTests (summary + ShouldNotBeNull message + method rename). Akka.Cluster.Hosting enables an SBR provider by default (applies SplitBrainResolverOption.Default when the typed option is null → reads the pre-existing akka.conf keep-oldest block), so the cluster was never NoDowning and the typed KeepOldestOption is reinforcing/explicit-in-code, not the activator. Cluster.Tests 29/29; failover behaviour still guarded by HardKillFailoverTests. Architecture.md carried no such framing.
#13 — apply per-instance ResilienceConfig from the artifact	RESILIENCE-DISPATCH-GAP sub-gap (from #10)	fix/archreview-crit13-resilience-config-artifact 75403caa	The DriverInstance.ResilienceConfig column was authored+persisted (AdminUI) and already in the artifact (the composer serializes the whole entity), but the driver-node read path dropped it → tier-defaults only. DriverInstanceSpec now carries it; TryReadSpec reads it; Create takes+layers it (+ logs parse diagnostics, never throws); SpawnChild threads it. Invalidate-on-change: Create calls builder.Invalidate(instance) (the cache is options-blind on a hit) so a respawn with changed options rebuilds; DriverSpawnPlanner respawns on a ResilienceConfig change (incl. null→json), a pure DriverConfig change stays an in-place delta. Verified: override reaches execution (retryCount:0 suppresses tier-A retry; a control retries), invalidate is per-instance, malformed logs+falls-back; planner+artifact tests. Core.Tests 243 (+5), Runtime.Tests 363 (+5). See FOLLOWUP-13.

Task list (TaskCreate IDs)

#	Task	Status
1	Critical 1 — split-brain resolver	✅ completed
2	Critical 2a — VT timeout	✅ completed
3	Critical 2b — CompiledScriptCache	✅ completed
4	Critical 3 — S7 reconnect (05/STAB-1)	✅ completed
5	Critical 4 — TwinCAT ADS re-registration (05/STAB-2)	✅ completed
6	Guard — DeferredAddressSpaceSink reflection forwarding test (03/U2)	✅ completed — a65c2ced; gate for 03/P1 + 02/P now in place
7	Guard — wire OTOPCUA0001 analyzer + triage (07/C-1)	✅ completed — f0082af5; surfaced task #10
8	Guard — CI to all unit suites + fail-on-skip (07/S-1,S-2, pair S-4)	✅ completed — 10b89830
9	Critical 1 follow-up — hard-kill failover integration test (03/S1)	✅ completed — a25c9ed0; surfaced task #11
10	Wire CapabilityInvoker into the dispatch layer (RESILIENCE-DISPATCH-GAP, from 07/C-1)	✅ code-complete + unit/analyzer/context-verified — fix/archreview-crit10-wire-capability-invoker 62556c24 (seam, NOT direct thread — Runtime is Polly-free). Live behavioral gate pending (rig recipe in FOLLOWUP-10). Surfaced a sub-gap: ResilienceConfig not in the deploy artifact → tier-defaults only.
11	Review Critical 1 premise — SBR was already active on master, not NoDowning (from #9)	✅ completed — fix/archreview-crit1-split-brain-resolver eaf78aad; docs/comment-only correction (no code revert), Cluster.Tests 29/29.
12	#10 live behavioral gate — rig session (from #10)	✅ ran 2026-07-08 (PARTIAL) — built the Host image from source (crit13 tip) + stood up the rig; central-1 booted healthy; prod DI binds the real invoker factory confirmed live + by a new deterministic guard (ResilienceInvokerFactoryRegistrationTests 8a8b9ec5). Behavioral retry/breaker log-line not observed (disciplined stop): Modbus's cheap paths don't throw a wrapped exception (Discover offline, Subscribe lazy, reads non-wrapped) — only an idempotent write-to-dead-endpoint (or S7) does, not worth the multi-dependency chase since the retry behavior is deterministically proven. Found + FIXED a rig bug: docker-dev compose's retired Wonderware ServerHistorian keys crashed the Host at bring-up (historian-gateway cutover → reads Endpoint) → fixed on fix/docker-dev-serverhistorian-stale-keys 7233e2ba (off master; gateway-shape keys, disabled by default; central-1 verified booting clean). See FOLLOWUP-10.
13	Plumb per-instance ResilienceConfig through the deploy artifact (from #10)	✅ completed — fix/archreview-crit13-resilience-config-artifact 75403caa; read-path plumbing (write side was already complete), invalidate-on-change + respawn-on-change; Core.Tests 243/Runtime.Tests 363. See FOLLOWUP-13.

Findings surfaced by the guards (NEW — need user decision)

Each has a dedicated follow-up file with the full design/action: FOLLOWUP-10-resilience-dispatch-gap.md · FOLLOWUP-11-crit1-sbr-premise.md · FOLLOWUP-13-resilience-config-artifact.md.

• #10 — RESILIENCE-DISPATCH-GAP (from wiring the analyzer, 07/C-1): ✅ IMPLEMENTED on fix/archreview-crit10-wire-capability-invoker 62556c24 — code-complete + unit/analyzer/context-verified; live behavioral gate pending (recipe in FOLLOWUP-10). The filed "thread CapabilityInvoker into the actor" was infeasible (Runtime is deliberately Polly-free); shipped a seam mirroring IDriverFactory instead. Only 6 of the 7 sites were real (the 7th, GenericDriverNodeManager, is test-only scaffolding). Two residual sub-findings: (a) per-instance ResilienceConfig isn't in the deploy artifact → tier-defaults only ✅ RESOLVED by task #13 (75403caa) — the column was already in the artifact; the read path now applies it (see FOLLOWUP-13); (b) the resilience-status tracker has no reader (Admin /hosts Stream E.2/E.3 never shipped) — added retry/breaker logging as the interim observability surface + the live-verify enabler (still open as an Admin-UI follow-on).
• #11 — Critical 1 premise inaccurate (from the #9 failover negative control): ✅ CORRECTED on fix/archreview-crit1-split-brain-resolver eaf78aad. SBR was already active on master (Akka.Cluster.Hosting's WithClustering applies SplitBrainResolverOption.Default when the typed option is null → enables the downing provider that reads the pre-existing akka.conf keep-oldest block). So the cluster was not running NoDowning; hard-crash failover already worked. Docs/comment-only fix (no code revert — the typed KeepOldestOption is correct belt-and-suspenders): corrected the "HOCON inert / NoDowning → no failover" framing in BuildClusterOptions XML comment, akka.conf, docs/Redundancy.md, and the SplitBrainResolverActivationTests summary/message (+ method rename). Cluster.Tests 29/29. (Architecture.md carried no such framing.) See FOLLOWUP-11.

Suggested next (from 00-INDEX execution order)

All 4 Criticals + all 3 systemic guards (tasks 6/7/8) + the #9 failover test are DONE. Remaining:

1. Resolve the two surfaced findings — task 10 (wire CapabilityInvoker into dispatch; behavior-changing, live-gate) and task 11 (correct Crit-1's rationale/docs).
2. Failure-visibility Highs — 01/S-1 (AddressSpaceApplyOutcome failure field), 06/S-1 (Galaxy fail-closed), 03/S4 (primary-gate default-deny).
3. P-tier — 03/P1 surgical adds/removals (now gated by the U2 forwarding guard), 02/P batched WriteValues.

Key resume facts

• Verification bar met so far: each fix has a deterministic unit guard AND (for SBR) an end-to-end config-resolution check. Follow the same pattern — the review's whole thesis is "unit-green ≠ wired".
• The forwarding-test guard (task 6) is DONE (a65c2ced): 03/P1 (surgical pure-adds) and 02/P-… batch-write each add new sink capability methods that ship inert unless forwarded through DeferredAddressSpaceSink (F10b/PR#423 precedent). The reflection guard now fails loudly if any such member isn't forwarded — add the member to the recorder-covered interface set and it's auto-checked.
• Nothing is pushed. Superseded 2026-07-12: all branches merged to master and pushed to origin/master; the plans now live on master too (no more cross-branch git show needed).
• CI changes can't be run-verified without a push — now pushed; the first real CI-job run should be checked on origin (the re-review verified the YAML statically only).
• Findings #10 and #11 are resolved (see above); the open decisions now come from the 2026-07-12 re-review below.

Re-review 2026-07-12 (master f6eaa267) — outcome + new findings

All seven domain reports were re-run and rewritten in place at f6eaa267 (each carries a Prior-finding status (9cad9ed0 → f6eaa267) table; synthesis in ../00-OVERALL.md). Verdict on the remediation: all 4 Criticals verifiably closed in code (Crit-1 by premise correction), both systemic guards live with negative controls, CI now whole-solution + fail-on-skip (07 stability 2.5→3.5). New findings needing scheduling, ranked:

Rank	Finding	Severity	One-line
1	05/STAB-14	High (regression)	S7 ensure-wrapper rethrows connect-timeout OCE → poll loop's bare catch (OCE) { return; } silently kills all S7 subscriptions after an unreachable-host (non-refusing) outage; live bounce test blind to it (S7Driver.cs:488,1000,1383,1404,1206-1208)
2	01/S-6	High	Operator-authorable timeoutSeconds:0 in ResilienceConfig → Polly ValidationException on every capability call = permanent driver-brick; no range validation in parser or AdminUI form
3	05/STAB-15	High	S7 fatal-classification misses framing/desync + cancel-during-I/O (half-read PDU → permanent desync); probe catch never marks handle dead
4	01/S-8 = 03/S12	Medium	Dispatch hardcodes isIdempotent:true; WriteIdempotentAttribute has zero readers → #13-enabled write retries would retry command writes
5	02/S13	Medium	VT script failure/timeout leaves node stale-Good forever (timeout fix made the failure invisible)
6	01/U-6	Medium	Bulkhead knob parsed/authored/documented but never built into the pipeline — dead knob
7	06/S-11	Medium	Enabled-but-misconfigured ServerHistorian crash-loops Host with raw UriFormatException (only config side fixed in 7233e2ba)
8	06/U-7	Medium	HistorianGateway 0.2.0 proto3-optional DataType change silently alters GatewayTagProvisioner Boolean provisioning; live EnsureTags never run on 0.2.0
9	04/C-7, 04/U-5=03/U10, 01/S-7=03/S13	Medium/Low	Resilience seam edges: lossy AdminUI form round-trip; status tracker has no reader (operators blind); respawn stale-pipeline cache race

Carried-open Highs unchanged from round 1 (no branch targeted them): failure-visibility trio (01/S-1, 06/S-1, 03/S4), AdminUI authz (04/C-1 — stakes rose: resilience config is on the ungated surface), 03/S2/S3 block-bridging, 03/P1 surgical adds (guard prerequisite in place), driver-fleet batch (05), repo hygiene (07). Full prioritized list in 00-OVERALL.md.

Round-2 plans WRITTEN (2026-07-12): all 12 action items have design+implementation plans — R2-01…R2-12 in this directory (+ co-located .tasks.json each, for resumable execution). Index, per-plan task counts, effort, and suggested execution order: 00-INDEX.md §"Round 2 plans". 193 bite-sized TDD tasks total, ~18–24 dev-days end-to-end; R2-01/02/03 (the re-review's new-and-sharp items) are ~2 days combined.

Round-2 EXECUTED — R2-12 Repo hygiene (2026-07-13, branch r2/12-repo-hygiene): DONE. 07/U-5 (7 lib/ AVEVA DLLs removed + gitignored, zero-reference proven), 07/U-4 (root planning files: pending.md/current.md untracked+gitignored, looseends.md deleted, stillpending.md + HISTORIAN-GATEWAY-INTEGRATION-ISSUES.md archived under docs/plans/, docs/Historian.md repointed), 07/S-3 (v2-e2e.yml no-op nightly deleted), 07/P-5 (redundant integration matrix job removed from v2-ci.yml), 07/U-3 (5 alarm operator CLI commands documented — now 13/13), 07/C-5 (StyleGuide.md ScadaBridge→OtOpcUa + minimal root .editorconfig), 07/C-4 (xunit pin 2.9.2→2.9.3

• blocker comment; upstream-blocked — no xunit.v3 Akka TestKit exists). 07/U-6 needed no action (the Wonderware husk dirs are untracked build artifacts, absent in a fresh worktree). Whole-solution build clean. OPEN follow-ups: (a) when a runner-reachable fleet + a real tests/Server/…E2ETests land, add a fresh E2E workflow WITH a zero-match guard (fail if Category=E2E matches nothing); (b) S-2 option #2 — a fixture-backed integration leg (opc-plc services: block, MIN_EXECUTED=1) covering what the unit leg cannot; (c) migrate the 3 xunit v2 holdouts once Akka ships an xunit.v3 TestKit.

Round-2 remediation landed

Plan	Findings	Branch	Commits	Status
R2-06	06/S-11 (rank 7), 06/U-7 (rank 8), 06/C-7	r2/06-serverhistorian-failfast (off 1676c8f4)	3a049a13…4640ecb1 (10 task commits)	Code-complete, offline-verified. S-11: fail-fast ServerHistorianOptionsValidator (consumer-gated on ServerHistorian:Enabled OR AlarmHistorian:Enabled, fails empty/non-http(s) Endpoint at startup) + AddValidatedOptions wiring + adapter defense-in-depth (TryCreate→named InvalidOperationException, no more raw UriFormatException crash-loop) + Validate() warning parity + alarm-only-mode startup warning. U-7: pinned Boolean→Int1 explicit-presence (HasDataType) + HISTGW_BOOL_SANDBOX_TAG live Boolean EnsureTags leg + retype probe (skip-clean offline) + upsert/retype migration note (docs/Historian.md). C-7: keep-populating decision + dead-Quality doc comments (0.2.0 C-002). Live leg operator-gated (T12) — pending a VPN run on a 0.2.0 gateway (records the observed in-place retype policy back into docs/Historian.md; also discharges 06/U-2's full documented run).
R2-05	04/C-1 (High)	r2/05-adminui-authz (off 1676c8f4)	183b72b7…6c1f05cb (5 commits)	Code-complete, offline-verified. All 38 routable AdminUI pages carry an explicit named-policy @attribute via new AdminUiPolicies constants (Security/Auth/). New ConfigEditor (Administrator+Designer) gates the 20-page config-authoring surface incl. the 8 driver pages that author live ResilienceConfig + /api/script-analysis/*; new AuthenticatedRead on the 16 read pages; FleetAdmin/DriverOperator unchanged (literals→constants). Anti-drift guard PageAuthorizationGuardTests (bans bare [Authorize]/Roles=/unknown-policy/unclassified) + AdminUiPoliciesTests semantics matrix. Corrections folded in: Reservations + ClusterRedundancy read-only (→AuthenticatedRead); Home had no attribute (now AuthenticatedRead). AdminUI.Tests 517/517, Security.Tests 80/80. Live /run positive pass deferred (T15) — docker-dev DisableLogin=true auto-admin can't observe deny; negative proof is CI-side (both guard tests).
R2-01	05/STAB-14 (High regression, rank 1), 05/STAB-15 (rank 3), 05/STAB-8 seam	r2/01-s7-fault-hardening (off 1676c8f4)	fcdcf0e2…bdb7ed41 (13 task commits)	Code-complete, offline-verified. All prod changes in S7Driver.cs. STAB-14: connect-timeout surfaces as TimeoutException (not an escaping OCE) at the EnsureConnectedAsync/InitializeAsync source + when(token.IsCancellationRequested) filters on the read/write ensure-wrappers and all three poll-loop OCE catches — subscription poll loops survive an unreachable-host outage and recover (T2 regression RED→GREEN). STAB-15: IsS7ConnectionFatal broadened to the ISO-on-TCP framing surface (TPKT/TPDU/WrongNumberOfBytes/PlcException{WrongNumberReceivedBytes}, NOT InvalidDataException); mid-PDU cancel marks handle dead; probe classifies+marks-dead under the gate. STAB-8 (S7 part): EnsureConnectedAsync documented as the R2-09 connect-throttle seam (note-only). Driver.S7.Tests 246/246. Live outage gate CLOSED 2026-07-15 (T11) — and it uncovered + fixed a real prod gap (PR #453, master 4dc937fc). Running S7_1500ConnectTimeoutOutageTests against the live s7_1500 sim (blackholed via docker pause, the ACK-less-established shape) exposed that STAB-14 fixed only the connect leg: S7.Net's ReadTimeout/WriteTimeout map to the TcpClient's synchronous receive/send timeout, which async reads ignore, so a read on an established-but-frozen peer blocked for minutes — the poll loop wedged (no Bad tick, no reconnect), which offline fakes (open-hang only) couldn't surface. Fix = new S7OperationDeadline wall-clock ceiling on every data-plane wire op in S7PlcAdapter (→ TimeoutException) + IsS7ConnectionFatal now classifies TimeoutException fatal → mark-dead → reopen (the read-leg sibling of the connect-timeout fix). Driver.S7.Tests 246→260 (6 new: 5 deadline + 1 driver-reaction); live gate now GREEN (8s: baseline Good → pause blackhole → Bad tick → unpause → recovered Good). (Pre-existing unrelated CLI docs-scan test fails at base — out of scope.)
R2-03	02/S13 (rank 5), 02/S12, 02/P7	r2/03-vt-failure-quality (off 1676c8f4)	b9775485…6bb91e8d (12 task commits)	Code-complete, offline-verified. S13: VT script failure/timeout degrades the node to OpcUaQuality.Bad (carrying last-known value) instead of freezing on last-Good forever — additive EvaluationResult.Quality, bridge historizes 0x80020000u BadInternalError, published once per Good→Bad transition gated on all deps arrived (inputs-ready, no cold-start flash), recovery force-publishes next Good. S12: apply-boundary ClearCompiledScripts racing an in-flight eval into ObjectDisposedException retried once in RoslynVirtualTagEvaluator (re-fetch→recompile into current generation); stress guard reproduces pre-fix, deterministically green post-fix. P7: apply-boundary clear gated on a changed deployed expression set (HashSet.SetEquals, ordinal) — identical redeploy / rename-with-same-expressions no longer forces full VT recompile. Runtime.Tests 372/372, RoslynVirtualTagEvaluatorTests 20/20. Deferred (T11/T12): full cross-suite sweep + docker-dev live-/run (fail→Bad→recover + P7 no-recompile spot-check).
R2-02	01/S-6 (High, rank 2), 01/U-6, 01/S-8=03/S12 (rank 4), 04/C-7, 01/S-7=03/S13	r2/02-resilience-config-hardening (off 1676c8f4)	4be13429…dd82e740 (17 task commits)	Code-complete, offline-verified. S-6: parser clamps timeout/retry/breaker overrides (timeout≤0→tier default, breaker==1→2, caps) with diagnostics + belt-and-braces Build clamp so pipeline construction can never throw Polly ValidationException (new ResiliencePolicyRanges in Core.Abstractions); brick-repro + Build_NeverThrows_ForHostileDirectOptions + factory-seam wiring proof; AdminUI Validate() range warnings. S-8: parser forces Write/AlarmAck retry→0 (spec invariant) and dispatch flips to isIdempotent:false (RED-first on the Runtime wiring test); non-idempotent arm caches the no-retry snapshot once/invoker (01/P-4) + tracker in-flight accounting; WriteIdempotentAttribute false claim corrected; retry cells disabled. U-6: bulkhead knob deleted end-to-end (migration-free; stored keys ignored) + Options_properties_are_exactly_the_pipeline_wired_set inertness guard. C-7: ResilienceFormModel JsonObject-bag round-trip preserves unknown keys; malformed JSON → ParseFailed lock + verbatim ToJson + discard button. S-7: per-Create Interlocked options-generation in the pipeline-cache key kills the respawn stale-pipeline race. Core resilience 121/121, AdminUI form 10/10, Analyzers 32/32, Runtime 364/364; 0 production OTOPCUA0001. Deferred (T15/T18): docker-dev :9200 /run (warning banner, input lockout, unknown-key survival) + whole-solution sweep.
R2-10	03/U10=04/U-5 (Medium/Low, action #10)	r2/10-resilience-observability (off 46fedda3)	205d3980…<docs> (11 task commits)	Code-complete, offline-verified. The resilience status tracker now has a production reader, mirroring the driver-health flow. Core truthfulness: RecordBreakerClose + derived IsBreakerOpen (breaker-open state derivable), OnClosed records the close, and RecordSuccess wired onto the invoker success paths (ConsecutiveFailures is a true consecutive counter, not stuck-forever). Transport: DriverResilienceStatusChanged Commons contract → periodic 5 s full-snapshot publisher DriverResilienceStatusPublisherService (driver nodes, registered in AddOtOpcUaDriverFactories with wiring guard ResilienceStatusReaderRegistrationTests — closes the built-but-never-wired class) → driver-resilience-status DPS topic → per-admin-node DriverResilienceStatusBridge → IDriverResilienceStatusStore → resilience chips (breaker-OPEN / consecutive-failures / in-flight / staleness) on DriverStatusPanel.razor, read in-process (self-HubConnection ban honored). Core.Tests resilience 31/31 (incl. deterministic breaker-close via a controllable TimeProvider), Host.IntegrationTests publisher+guard 6/6, AdminUI.Tests store 5/5; full-solution build clean. Deferred (T10): docker-dev live-/run — S7 dead-endpoint breaker-open observed on BOTH centrals + recovery-clear + staleness-dim (also the first live sighting of the #10 circuit-breaker OPENED log line, closing FOLLOWUP-10's behavioral gate).

| R2-08 | 03/S2 (High, action #8), 03/S3 (High) | r2/08-ldap-historyread-async (off 1a698cbb) | ce6d9e21…<docs> (14 task commits) | Code-complete, offline-verified. S2 (LDAP async/timeout): the SDK raises the impersonation callback synchronously under a SessionManager-wide m_eventLock (verified vs. 1.5.378 — a hung bind serializes ALL activations), so the fix is entirely at the LdapOpcUaUserAuthenticator boundary: five new Security:Ldap keys (ConnectionTimeoutMs=10000 projected into the shared lib, AuthTimeoutMs=15000 hard whole-flow WaitAsync bound → fail-closed "timed out", MaxConcurrentAuthentications=8 SemaphoreSlim with release-on-core-completion so orphaned binds still count, OutageFailureThreshold=3/OutageCooldownSeconds=15 directory-outage circuit → fast-deny "unavailable", half-open probe) + LdapAuthResult.IsSystemFailure seam (mapped from ServiceAccountBindFailed) distinguishing outage from credential-deny + LdapOptionsValidator rules + HandleImpersonation contract doc + docs/security.md. Stall-repro RED→GREEN; offline outage-sim through the real HandleImpersonation against an unroutable host. S3 (HistoryRead fan-out): all four HistoryRead arms async-ified (ServeNodeAsync/ServeRawPagedAsync/ServeEventsAsync, OCE→BadTimeout) then bounded per-node fan-out (RunBounded semaphore + single arm-boundary bridge) + per-request deadline CTS (HistoryReadDeadline=60s) + process-wide lazy batch limiter (MaxConcurrentHistoryReadBatches=16 → flood degrades to BadTooManyOperations); 16×4=64 in-flight cap; three new ServerHistorian keys + node-manager props + hosted-service wiring + docs/Historian.md/CLAUDE.md tables. Sequential-serving RED→GREEN. Host.IntegrationTests (LDAP resilience) 13/13, Security.Tests 11/11, Runtime.Tests options 11/11, OpcUaServer.Tests HistoryRead 30/30. S2 GLAuth outage/recovery LIVE gate CLOSED 2026-07-15 (PR #455, master 5f12636c) — new env-gated LdapGlauthOutageLiveTests drives the circuit through the production path (OtOpcUaLdapAuthService+LdapOpcUaUserAuthenticator) against a REAL GLAuth on 10.100.0.35:3894 (paused/unpaused via docker pause); GREEN (12s): healthy alice/bob binds w/ mapped roles → pause → 3 bounded failures → circuit opens → sub-second "unavailable" → unpause → half-open close → recover. Also the FIRST end-to-end proof the PR #451 GLAuth swap binds correctly. Skips clean offline (1 skipped, 8ms). S3 Category=LiveIntegration gateway soak — GREEN 2026-07-15 (6/6) against the running local 0.2.0 gateway (http://localhost:5221 h2c) + the real AVEVA historian over VPN; the Galaxy read round-trip exercises the GatewayHistorianDataSource read path S3 hardened. R2-08 fully live-verified. | | R2-11 | 01/C-1, 01/P-1 + 05/CONV-2, UNDER-1, UNDER-6 | r2/11-tagconfig-consolidation (off 46fedda3) | first 2e14fe1f … (24 task commits) | Code-complete, offline-verified. Part 1 (C-1/P-1): single TagConfigIntent.Parse + DeviceConfigIntent in Commons replace the four byte-parity TagConfig parse copies (composer/artifact/walker/DraftValidator) — composer now parses once per tag (P-1); a permanent Runtime.Tests/TagConfigCorpusParityTests (30-blob corpus) guards compose≡decode; grep "MUST parse identically" src/ = 0. Part 2 (CONV-2): shared strict-capable TagConfigJson readers in Core.Abstractions replace the six copy-pasted ReadEnums; all six driver parsers gain Inspect() warnings + honour the writable key (FOCAS forced read-only, 05/UNDER-1); FOCAS equipment tags run the capability-matrix pre-flight (05/UNDER-6); Modbus probe parses the factory DTO shape (timeoutMs no longer dropped). New EquipmentTagConfigInspector in ControlPlane + AdminOperationsActor deploy-gate wiring with Deployment:TagConfigValidationMode (Warn default | Error opt-in), proven through the actor (F10b consuming test). AdminUI: writable checkbox in the six typed editors. Phase-C follow-up — SHIPPED 2026-07-15 (Gitea #457): the runtime parsers are now strict (typo'd enum ⇒ BadNodeIdUnknown instead of wrong-width Good) via TagConfigJson.TryReadEnumStrict across all six drivers; six strictness suites inverted + a driver-level ReadAsync→BadNodeIdUnknown end-to-end proof. See the "Deferred behavior-change follow-up" section below. Deferred (T22/T24): docker-dev /run of one typed editor's Writable checkbox + whole-solution serial sweep. |

| R2-07 (Phases 1+2+3) | 03/P1 (action #7, surgical pure-adds/removes/mixed) | r2/07-surgical-pure-adds (off 1a698cbb) | bb226f45…e2b47638 (T1–T14) | Phases 1+2 code-complete, offline-verified. New pure AddressSpaceChangeClassifier (Empty/AttributeOnly/PureAdd/PureRemove/AddRemoveMix/Rebuild; first-match-wins, default-closed Rebuild; the F10b tag/vtag surgical-eligibility predicates moved here as internal static). AddressSpaceApplier.Apply routes by kind. Phase 1 (PureAdd/AttributeOnly): skip RebuildAddressSpace (the idempotent Materialise passes create exactly the added nodes; existing subscriptions survive); adds announced via ComputeAddAnnouncements+AnnounceAddedNodes (dedup affected-parent → existing guard-covered RaiseNodesAddedModelChange), fired by OpcUaPublishActor after the 4 passes gated on !outcome.RebuildCalled; MaterialiseAlarmCondition made idempotent (skip-if-present-and-same-kind keeps the condition NodeState alive). NO new sink members in Phase 1. Phase 2 (PureRemove): 3 new ISurgicalAddressSpaceSink members (RemoveVariableNode/RemoveAlarmConditionNode/RemoveEquipmentSubtree) — guard-first with the observed negative control (DeferredSinkForwardingReflectionTests fails member-named until DeferredAddressSpaceSink forwards) + SdkAddressSpaceSink forwards + OtOpcUaNodeManager in-place teardown (scoped map cleanup, notifier demotion via RemoveRootNotifier, BuildNodesRemovedModelChange verb=NodeDeleted reported outside Lock). Applier PureRemove arm: terminal Bad / RemovedConditionState writes (closing the removed-alarm-tag today-gap) → subsumption partition (removed-equipment children folded into the subtree remove) → in-place removes → any false/throw ⇒ full-rebuild ratchet; non-surgical sink ⇒ rebuild. Phase 3 (AddRemoveMix): removes-then-adds within one apply — the remove pass runs in Apply (no rebuild), then the publish actor's Materialise passes create the adds + AnnounceAddedNodes announces them; an id reused across the remove+add sets is remove-then-recreated as a fresh NodeState (correct — a different tag). Guard stays green (Commons.Tests 17/17), OpcUaServer.Tests 336/336, Runtime.Tests 398/398. 03/P1 marked REMEDIATED in archreview/03-server-runtime.md. Deferred-live (T5, T6, T12, T14): SubscriptionSurvivalTests add + remove + mixed cases (authored + compile-green; NOT run — *.IntegrationTests memory constraint) and the docker-dev live-/run add/remove/mixed gates (kind=PureAdd/PureRemove/AddRemoveMix, rebuild=False; survivor subscription uninterrupted; removed node → BadNodeIdUnknown) — MUST run in the serial heavy pass (F10b inertness risk). |

Report anchor drifts to fold into the next report refresh (from the R2-02 verification pass): (1) 01-core-composition.md U-6 anchor DriverResilienceOptions.cs:23-25 → the two bulkhead properties sat at :23 and :29 (now deleted); (2) 01-core-composition.md U-6 anchor DriverResiliencePipelineBuilder.cs:99-177 → Build was :99-176. All other cited lines were exact at f6eaa267.

✅ Round-2 COMPLETE — all 12 plans merged (2026-07-13)

All 12 round-2 plans were executed (isolated worktree per plan, per-task TDD, pathspec-scoped commits) and merged to master + pushed to origin/master in 3 dependency-ordered waves, each plan landing as its own Gitea PR (closed-as-merged) behind a dotnet build barrier. Final state: master 31c98200.

Wave	Plans (PR#)
A (no cross-plan deps)	R2-12 #426 · R2-06 #427 · R2-05 #428 · R2-01 #429 · R2-03 #430 · R2-02 #431
B (parse-site barrier)	R2-10 #432 · R2-04 #433 · R2-11 #434
C (largest, depend on earlier)	R2-08 #435 · R2-07 #436 · R2-09 #437

Cross-plan mutexes honored: R2-01 landed before R2-09 (PollGroupEngine v2 / S7 fork retirement); R2-11's TagConfigIntent.Parse consolidation landed before R2-09's driver parse edits; the three plans that all edit AddressSpaceApplier.cs + DriverHostActor.cs + OtOpcUaNodeManager.cs (R2-04/R2-07/R2-08/R2-10/R2-11) were merged sequentially with the shared-file auto-merges verified by targeted unit runs each time.

Consolidated unit sweep on merged master 31c98200 (integration suites excluded — ~16 GB/run leak)

Solution dotnet build clean (0 errors). All R2-relevant unit suites GREEN, incl.: OpcUaServer 341, Runtime 398, Commons(+forwarding guard) 122, Core 276, Core.Abstractions 128, Core.Scripting 137, Core.VirtualTags 59, Cluster 29, Security 84, ControlPlane 78, Analyzers 32, AdminUI 537, Client.CLI 104, Client.Shared 182, Client.UI 126, Core.AlarmHistorian 29, Core.ScriptedAlarms 71, and all drivers (S7 254, Modbus 312+161, AbCip 336, AbLegacy 209, TwinCAT 189, FOCAS 265, Galaxy 304, Galaxy.Browser 21, Historian.Gateway 102, OpcUaClient 131).

3 failures, all confirmed PRE-EXISTING / environmental (git diff 1676c8f4..HEAD empty on their paths — R2 never touched them):

1. Core.Abstractions.Tests.InterfaceIndependenceTests.AllPublicTypes_LiveInRootNamespace — flags the pre-R2 Core.Abstractions.Historian.* continuous-historization types living in a sub-namespace. Pre-existing convention debt — needs-owner-decision (move the types to root, or relax the test to allow the intentional Historian grouping).
2. Driver.S7.Cli.Tests.SubscribeCommandConsoleHandlerCommentTests... — pre-existing docs-scan test looking for a "CliFx console" phrase removed by 9cad9ed0. Follow-up: repoint or delete the phrase-scan.
3. Driver.OpcUaClient.Browser.Tests (3) — Connection refused; need the live opc-plc endpoint (10.100.0.35:50000), unreachable offline. Environmental — pass with the fixture up.

Deferred heavy / live-/run pass (serialized — single-tenant rig; controller-owned)

Every plan's integration-suite and live-/run gates were deferred (not skipped) so nothing OOMs the box and the shared docker-dev rig / VPN infra isn't contended. To run one-at-a-time:

• Full integration-suite sweep (10 *.IntegrationTests projects, one at a time): discharges R2-02 T18, R2-03 T11, R2-07 T5/T6, R2-08 T14, R2-11 T24. RAN 2026-07-14 — see INTEGRATION-SWEEP-STATUS.md. Outcome: OpcUaClient(+Browser)/S7/Modbus-standard green, Host 105/117, TwinCAT clean-skip; the rest fail on pre-existing test/fixture bit-rot (AbCip static-init bug, Modbus pymodbus-4.x sim drift, FOCAS 127.0.0.1-mock topology, AbLegacy no fixture, OpcUaServer cert-store config, Host openldap:2.6 image gone). No OtOpcUa regression. 7 follow-ups in that file.
• docker-dev :9200 live-/run (rebuild BOTH centrals): R2-02 T15 (ResilienceConfig warning banner / input lockout / unknown-key survival), R2-03 T12 (VT fail→Bad→recover + P7 no-recompile), R2-10 T10 (breaker-open on both centrals + recovery + FOLLOWUP-10 log line), R2-11 T22 (typed-editor Writable checkbox), R2-07 T12 (surgical add/remove/mixed — F10b inertness risk, MUST run), R2-05 T15 (over-gating regression only — deny unobservable under DisableLogin).
• 2-node redundancy rig: R2-04 T13/T15 BOTH ✅ PASSED 2026-07-14 — T15 live docker-dev gate + T13 in-process delivery-path test (see below).
• Genuinely external / needs-user: ALL CLOSED as of 2026-07-15. R2-06 T12 (6/6 on the VPN, 2026-07-13). R2-08 S3 Category=LiveIntegration gateway soak CLOSED 2026-07-15 (6/6 vs. the running local 0.2.0 gateway + real historian over VPN). R2-01 T11 CLOSED 2026-07-15 (S7 blackhole via docker pause; found + fixed the read-leg wall-clock gap, PR #453). R2-08 GLAuth outage cycle CLOSED 2026-07-15 (LdapGlauthOutageLiveTests vs. the harness GLAuth paused via docker pause, PR #455). Every Round-2 live gate is now GREEN.

Deferred behavior-change follow-up — SHIPPED 2026-07-15 (Gitea #457)

R2-11 Phase C — flip the runtime driver parsers to strict (typo'd enum ⇒ BadNodeIdUnknown instead of a wrong-width Good). DONE. All six equipment-tag parsers ({Modbus,S7,AbCip,AbLegacy,TwinCAT,Focas}EquipmentTagParser.TryParse) now read enums via the new TagConfigJson.TryReadEnumStrict (absent ⇒ fallback; valid ⇒ parsed; present-but-invalid ⇒ TryParse returns false ⇒ EquipmentTagRefResolver.TryResolve false ⇒ driver surfaces BadNodeIdUnknown). Modbus flips all three enum fields (region/dataType/byteOrder); the other five flip dataType. Regression coverage: the six *EquipmentTagParserStrictnessTests inverted from Freeze_typo_* (lenient) to Typo_*_rejects_the_tag + Valid_*_still_parses; a TagConfigJsonTests.TryReadEnumStrict_rejects_only_invalid matrix; and a driver-level end-to-end proof (AbCipEquipmentTagTests.Driver_read_of_a_typod_dataType_ref_surfaces_BadNodeIdUnknown) driving the real AbCipDriver.ReadAsync through resolve→status. The deploy-time Deployment:TagConfigValidationMode=Error gate (unit-proven in AdminOperationsActorTagConfigGateTests) remains the operator's pre-flight — a fleet that deploys clean under Error has, by construction, no typo'd enums and is safe on the flip. The golden parity corpus carries no typo'd enums, so the flip is a no-op there.

docker-dev live-/run pass (2026-07-13) — CONCLUDED; 5 gates LIVE-confirmed on the built-from-source 6aeb53a9 rig

Bottom line: R2-02, R2-11, R2-07 (both surgical phases), R2-04 (achievable failover), and R2-10 (resilience pipeline in dispatch) verified live on the docker-dev central cluster. R2-05 partial-by-design; R2-03 unit-verified (rig has no driver data). R2-01's S7 blackhole gate is CLOSED (2026-07-15) — run via docker pause on the 10.100.0.35 s7_1500 fixture; it found + fixed the read-leg wall-clock gap (PR #453). R2-08's GLAuth outage-cycle leg is CLOSED (2026-07-15, PR #455) — LdapGlauthOutageLiveTests vs. the harness GLAuth on 10.100.0.35. R2-08 S3 Category=LiveIntegration gateway soak is CLOSED (2026-07-15, 6/6) and R2-06 (6/6, 2026-07-13) — both vs. the local 0.2.0 gateway + real historian over VPN. Every Round-2 live gate is now GREEN — no Round-2 items remain unexercised.

Residual follow-ups — tracked in Gitea (2026-07-15)

The remaining non-blocking items are filed as issues (none gate the remediation, which is complete):

Issue	Item	Nature
#456	#10 — observe retry/breaker firing in live dispatch; thread ResilienceConfig into the deploy artifact (tier-defaults only today)	niche / sub-gap
#457	R2-11 Phase C — flip runtime parsers to strict, after fleets run TagConfigValidationMode=Error clean	deliberately deferred (operational rollout)
#458	Remove the NuGetAuditSuppress for CVE-2025-6965 once a patched SQLitePCLRaw bundle ships	upstream-gated
#459	SBR topology — crashing the OLDEST Akka node does not fail over	pre-registered user-deferred design gap

Brought up the 2-node central cluster (central-1+central-2, SQL, traefik) from source at merged master 6aeb53a9 (single-tenant rig; drove the :9200 AdminUI via browser + Client.CLI on :4840/:4841).

• R2-02 (S-6 High) ✅ LIVE — on the Modbus driver's Resilience overrides, entering timeoutSeconds:0 renders the exact range-validation banner: "Out-of-range values will be clamped to a safe value on deploy: Read timeout must be between 1 and 86400 seconds (got 0)." S-8 ✅ — the Write and AlarmAcknowledge RETRIES cells are disabled ("never").
• R2-11 (T22) ✅ LIVE — the typed Modbus tag editor shows the Writable toggle; saving persists writable: yes.
• R2-07 (P1 — F10b MUST) ✅ LIVE — central-1 boot apply logged AddressSpaceApplier: applied plan (kind=PureAdd, added=91, removed=0, changed=0, rebuild=False); deleting a UNS virtual-tag + redeploy logged (kind=PureRemove, added=0, removed=1, changed=0, rebuild=False) and the node vanished from the address space (capper-03 lost its production-count child). Surgical add AND remove both execute in prod with no full rebuild — the remove path's 3 new sink members are wired, not inert.
• R2-04 (S4 MUST) ✅ LIVE (achievable guarantee) — younger-crash (kill central-2): SBR took decision DownUnreachable and is downing [central-2], Member removed, central-1 stays Up, AdminUI serving, ServiceLevel 250 (sole primary, no split-brain). Oldest-crash (kill central-1) reproduced the documented keep-oldest asymmetry live: the survivor took SBR … DownReachable … including myself and CoordinatedShutdown (ExitCode 3) — i.e. crashing the oldest node does not fail over. This is the pre-registered SBR topology gap (a deferred USER strategy decision), not an R2 regression; R2-04's PrimaryGatePolicy write-gate change is orthogonal + unit-verified.
• R2-05 (T15) ✅ (partial, as expected) — every config/read page renders under the all-roles multi-role-test session (named-policy gating didn't over-gate a legitimate admin); deny is unobservable under docker-dev DisableLogin=true, as documented.
• R2-10 (T10) ✅ pipeline LIVE (breaker-OPEN state not forced). The /hosts page shows the driver-health flow live (MAIN-galaxy-eq HEALTHY; MAIN-modbus-eq FAULTED "Connection refused"; MAIN-opcua-eq FAULTED "failed to connect to any endpoints"). The failing drivers' subscribe/discover exceptions propagate through the full Polly resilience pipeline — central-1 stack traces show Polly.CircuitBreaker → Polly.Retry → Polly.Timeout → CapabilityInvoker.ExecuteAsync (CapabilityInvoker.cs:84), i.e. the #10 dispatch-gap fix + R2-10's tracker data-source are live in prod. The explicit breaker OPENED log line + the active resilience chip were NOT triggered: R2-09's ConnectionBackoff throttles retries, so consecutive failures don't accumulate fast enough to open the Polly breaker within its window (the documented FOLLOWUP-10 difficulty — the reliable trigger is a rapid idempotent write-to-dead-endpoint). Reader/store/publisher/bridge are unit-verified (Core resilience 128, publisher+guard 6, store 8).
• R2-03 (T12) — unit-verified; live-blocked on this data-less rig. Every materialised node reads 0x80320000 Bad_WaitingForInitialData because no live driver data flows (Modbus 10.100.0.35:5020 unreachable in-rig; VTs depend on driver data), so a VT can't be staged Good→Bad; and the Client.CLI read surfaces any Bad StatusCode as a FetchValueAsync exception rather than printing it. Good→Bad-with-last-value transition is covered by unit suites (Runtime 372, RoslynVTEvaluator 20). Would be live-observable with a reachable driver fixture feeding the VT.

Rig note: the pass mutated the persisted ConfigDb (added a drv-modbus-line3-01 Modbus driver on MAIN; removed capper-03's production-count virtual tag for the PureRemove test). Reset with docker compose -f docker-dev/docker-compose.yml down -v. R2-01's S7 blackhole gate is now CLOSED (2026-07-15, PR #453); all Round-2 live gates are now GREEN (R2-06 + R2-08 S3 verified on the VPN 2026-07-15; R2-01 + R2-08 GLAuth self-serviced on 10.100.0.35).

---

R2-04 T15 — 2-node primary-gate LIVE GATE ✅ PASSED (2026-07-14)

Ran the S4 primary-gate live recipe on the docker-dev 2-node central cluster rebuilt from master 365a90e6 (up -d --build central-1 central-2). Brought up the Modbus sim (otopcua-pymodbus-standard on 10.100.0.35:5020) so MAIN's Modbus equipment tags read Good (a data-less rig materialises every node Bad_WaitingForInitialData, and the CLI write reads-first — so a live driver is required to have a writable-Good node to drive the gate). Made EQ-55297329838d/ModbusHR200 writable (TagConfig.writable=true via SQL + POST :9200/api/deployments). Data-plane writes bound multi-role/password (carries the WriteOperate LDAP role).

All core assertions PASS:

• Step 1 — ServiceLevel split: Server_ServiceLevel (i=2267) = 250 on :4840 (primary) / 240 on :4841 (secondary) — differentiated, no split-brain. (Plan said 240/100; the deployment's healthy values are 250/240 — the invariant "primary > secondary" holds.)
• Step 2 — boot-window default-DENY (the NEW S4 behavior): restarting both centrals (to widen the role-null-with-≥2-members window — a single-node restart's window is sub-second because cluster-join and the first redundancy snapshot land near-simultaneously) caught, ~5 s into boot: WRN Operator write to EQ-55297329838d/ModbusHR200 rejected: not primary (role unknown). Meter otopcua_redundancy_primary_gate_denied_total{reason="role-unknown",site="write"} = 1.
• Step 3 — steady-state secondary DENY + #5 revert: write 9999 to :4841 → CLI reports optimistic Write successful (synchronous, by design) but read-back = 1234 (the primary's value, reverted — NOT 9999); central-2 logs rejected: not primary; meter reason="secondary",site="write" = 1.
• Step 3b — primary write reaches the device: write 1234 to :4840 → Write successful and read-back on :4841 = 1234 (the Modbus driver wrote through).
• T4 sanity — clean apply: both centrals logged AddressSpaceApplier: applied plan (kind=PureAdd … failed=0) / materialised (… failed=0); no [ERR] / apply.failed / degraded-apply line; the apply_failed meter is absent (never incremented).
• Step 4 — /alerts: endpoint 200, 0 rows — no active alarm on the data-less rig, so trivially no dual-primary double-emission (the double-row regression is only observable with a firing alarm, which needs live driver data this rig lacks; the alerts-emit gate shares the same PrimaryGatePolicy and is unit-covered).

Net: the R2-04 PrimaryGatePolicy membership-aware default-DENY-on-unknown is confirmed live end-to-end — both denial reasons (role-unknown, secondary) fire the log + the new meter, the #5 optimistic-write revert corrects the node, and the primary path is unaffected. The behavior-affecting S4 slice is trusted.

R2-04 T13 — in-process delivery-path test ✅ PASSED (2026-07-14). dotnet test …Host.IntegrationTests --filter FullyQualifiedName~PrimaryGateFailoverTests → 1/1, 3 s. Ran without SQL (OTOPCUA_HARNESS_USE_SQL unset) — the gate test needs only the in-process 2-node Akka cluster + the DPS-delivered redundancy snapshot, not a ConfigDb deploy. The TwoNodeClusterHarness formed the cluster and the real DistributedPubSub-delivered snapshot promoted exactly one node to Primary (Secondary's DriverHostActor.RouteNodeWrite → not primary; Primary passed the gate, rejecting only with a mapping reason). Its built-in negative control (a broken redundancy-topic subscribe leaves both stuck role unknown → 45 s timeout) means the PASS proves genuine snapshot delivery drives the gate — the half the pure-TestKit unit guards can't cover.

Rig mutations (this pass): ModbusHR200.TagConfig.writable=true; Modbus sim brought up on the shared docker host (10.100.0.35). Reset the central rig with docker compose -f docker-dev/docker-compose.yml down -v; the sim with ssh dohertj2@10.100.0.35 'cd /opt/otopcua-modbus && docker compose --profile standard down'.

---

Post-round-2 loose-end closure (2026-07-13)

• Pre-existing red unit test closed (theme-#4 "CI to all suites" hygiene; predates round 2, not an R2 plan). InterfaceIndependenceTests.AllPublicTypes_LiveInRootNamespace was red on master because four continuous-historization types (IHistorizationOutbox, HistorizationOutboxEntry, IHistorianValueWriter / HistorizationValue, HistorizationCommitMode) had drifted into a ZB.MOM.WW.OtOpcUa.Core.Abstractions.Historian sub-namespace while the other six files in the same folder use the root namespace — violating decision #59's flat-public-surface rule. Fixed by moving all four to the root namespace and dropping/retargeting the redundant using …Core.Abstractions.Historian; in ~11 consumers (no behavioral change). Gitea PR #438 → master 50e04707. Verified: Core.Abstractions 129/129 (was 128/129), Runtime historian+DI 72/72, Gateway driver unit 102/102, full solution build 0 errors.

---

R2-06 live gate (T12) — 5/6 green on the VPN + a real provisioning bug fixed (2026-07-13)

The R2-06 Category=LiveIntegration gate ran against a real 0.2.0 ZB.MOM.WW.HistorianGateway deployed locally in Docker (~/Desktop/HistorianGateway/docker, republished at 0.2.0) pointed at the real AVEVA Historian on wonder-sql-vd03.zmr.zimmer.com:32565 + Galaxy SQL on wonder-app-vd03 + the Runtime SQL DB, over the corporate VPN. OtOpcUa's GatewayLiveFixture connected h2c to the local gateway (http://localhost:5221, HISTGW_GATEWAY_ALLOW_UNTRUSTED=true since h2c ⇒ no cert validation; gateway ApiKeys:Mode=Disabled, placeholder key). See [[reference_histgw_local_deploy]] for the full recipe.

Three deployment fixes (sibling HistorianGateway/docker/, local-only):

1. DNS — base compose set the short Historian__Host: wonder-sql-vd03, which doesn't resolve in the container; overrode to the FQDN.
2. Kerberos/NTLM libs — the historian v6 session path (reads + EnsureTags) + the SQL path authenticate the Windows account NAM\dohertj2 via .NET Negotiate/GSSAPI, which aspnet:10.0 lacks → every read/tag op failed InvalidOperationException (Cannot load library libgssapi_krb5.so.2). Added libgssapi-krb5-2 gss-ntlmssp to docker/Dockerfile. (The v8 event path, SendEvent, uses ECDH — worked without it.)
3. RuntimeDb SQL auth — the Runtime DB SQL login had to be a SQL login (not the Windows account, which a Linux container can't SQL-auth). The user supplied wonderapp (SQL_LOGIN with INSERT on Runtime.dbo.History
   • SELECT on dbo.Events); wiring it made WriteLiveValues + SQL ReadEvents work.

🐞 Real OtOpcUa bug found + fixed (PR #439, master 666908d7): GatewayTagProvisioner built HistorianTagDefinition without StorageRateMs, so the proto uint32 defaulted to 0; the gateway maps it straight to the AVEVA SDK's storageRateMs, which **throws ArgumentOutOfRangeException ("Storage rate must be

0 ms")** on 0. So every deploy-time historized-tag auto-provision (IHistorianProvisioning.EnsureTagsAsync) was silently failing at the gateway (failed=N). Fixed by stamping DefaultStorageRateMs=1000 (gateway convention) + a regression unit test + StorageRateMs on the live-suite inline defs. Gateway driver unit 103/103.

Result — 5/6 PASS (was 3/6 before the fixes):

• ✅ Galaxy_tag_read_round_trip — historian gRPC read path.

• ✅ Write_then_read_on_sandbox_tag — EnsureTags(Float) + WriteLiveValues (SQL) + read-back.

• ✅ Alarm_SendEvent_is_acked + ✅ Alarm_SendEvent_then_ReadEvents — alarm SendEvent + SQL ReadEvents.

• ✅ EnsureTags_type_change_outcome_is_observable — now observes a real retype outcome (not a storage-rate error).

• ⚠️ EnsureTags_boolean_sandbox_provisions_as_Int1 (as originally written) — Int1 (Boolean) tag creation is not supported by the gateway's histsdk on this historian: EnsureTags(Int1) returns ProtocolEvidenceMissingException (the vendored client's "no capture evidence for this type" guard → gRPC Unimplemented), both fresh and as a retype target. A type sweep confirmed it is Int1-specific: Int2/UInt2/Int4/UInt4/Float/Double all provision success=True; only Int1 fails. Root cause: histsdk HistorianTagWriteProtocol.GetAnalogDataTypeCode has no captured CTagMetadata wire code for Int1 — a gateway/histsdk type-support gap, not an OtOpcUa defect (filed as historiangw#11, which routed the decision back as issue #441).

  ✅ RESOLVED (issue #441, 2026-07-14): HistorianTypeMapper now maps Boolean → Int2 (0/1 integer) instead of the degenerate Int1. Int2 is a proven-creatable analog type on this historian, and the value-write path already sends every value as a double on WriteLiveValues regardless of tag type, so Boolean 0.0/1.0 round-trips cleanly. The live gate test is renamed EnsureTags_boolean_sandbox_provisions_as_Int2 and now passes → 6/6. Unit pins updated (HistorianTypeMapperTests, GatewayTagProvisionerTests.Boolean_definition_carries_explicit_Int2_presence); gateway driver unit 103/103.

✅ U-7 retype policy — ANSWERED (via grpcurl against the real historian):

• EnsureTags is a real create-or-update upsert — re-EnsureTags of an existing tag at the same type → success.
• An in-place retype between SUPPORTED types is ACCEPTED — Float→Double on an existing tag returns success (of the three 06/U-7 hypotheses: accepted-retyped, not rejected).
• The specific Int1 case can't retype — Int1 isn't a creatable type here. Rather than accept a broken Boolean path, OtOpcUa remapped Boolean → Int2 (issue #441, above), which retypes cleanly from a pre-existing Float Boolean tag (both are supported analog types) and never hits the Int1 gap.

The R2-06 code deliverables (S-11 fail-fast + ServerHistorianOptionsValidator; Boolean provisioning mapping + upsert; C-7 doc) were already unit-covered + merged; this live gate additionally found + fixed the StorageRateMs provisioning bug, answered the U-7 retype policy, and surfaced the Int1 histsdk gap that drove the Boolean→Int2 remap (issue #441) — closing the last red test to reach 6/6.