# Arch-Review Remediation — Status & Handoff **Snapshot:** 2026-07-12 · **Review baseline:** master `9cad9ed0`; **re-review baseline:** master `f6eaa267` · this file is the single source of truth for remediation progress. Update it as work lands. > **2026-07-12 UPDATE — everything below the topology table landed:** all remediation branches were > **merged to master and pushed to `origin/master`** (merge commits `7834b3b7`…`b67bd9e8`; final state > `f6eaa267`). The topology table is kept for history but is no longer live. A full **re-review** of all > seven domains ran at `f6eaa267` — see [`../00-OVERALL.md`](../00-OVERALL.md) and the > "Re-review 2026-07-12" section at the bottom of this file for the new findings (one High regression > in the S7 fix, one High ResilienceConfig validation gap, and a batch of resilience-seam Mediums). ## Branch topology (HISTORICAL — as of 2026-07-08; since merged + pushed) | Branch | Head | Contains | |---|---|---| | `master` | `9cad9ed0` | untouched baseline | | `docs/archreview-plans-and-drift-fixes` | `9fadead6` | the 7 domain plans + `00-INDEX.md` + this `STATUS.md` + the two CLAUDE.md doc-drift fixes. **(the plans live ONLY here)** | | `fix/archreview-crit1-split-brain-resolver` | `eaf78aad` | Critical 1 (03/S1) typed SBR option + guard test + Redundancy.md; **+ #9 hard-kill failover test** (`a25c9ed0`); **+ #11 premise correction** (`eaf78aad`) | | `fix/archreview-crit2-vt-timeout` | `7fd44f0f` | Critical 2 (02/U2 + 02/U3) + tests | | `fix/archreview-crit3-s7-reconnect` | `25c0c6f6` | Critical 3 (05/STAB-1) IS7Plc seam + lazy reconnect + unit guards + live bounce test | | `fix/archreview-crit4-twincat-replay` | `af318fb4` | Critical 4 (05/STAB-2) native-notification replay + probe-recycle + unit guards | | `fix/archreview-u2-deferred-sink-forwarding-guard` | `a65c2ced` | Guard 03/U2 — reflection-exhaustive deferred-sink forwarding test (+ DeferredServiceLevelPublisher) | | `fix/archreview-c1-wire-analyzer` | `f0082af5` | Guard 07/C-1 — wire OTOPCUA0001 analyzer tree-wide + triage (surfaced the RESILIENCE-DISPATCH-GAP, task #10) | | `fix/archreview-s1-s2-ci-coverage` | `10b89830` | Guard 07/S-1,S-2,S-4 — whole-solution CI leg + fail-on-skip gate + CLI sleep deflake | | `fix/archreview-crit10-wire-capability-invoker` | `62556c24` | **#10 — wire CapabilityInvoker into dispatch via a Core.Abstractions seam** (off `fix/archreview-c1-wire-analyzer`, NOT master — the analyzer is the standing guard + the pragmas live there). Code-complete + unit/analyzer/context-verified; live gate pending. | | `fix/archreview-crit13-resilience-config-artifact` | `8a8b9ec5` | **#13 — apply per-instance ResilienceConfig from the artifact** (`75403caa`, off the crit10 branch) **+ #10/#13 DI guard** (`8a8b9ec5` — `ResilienceInvokerFactoryRegistrationTests`, from the #12 live gate). Read-path plumbing + invalidate/respawn-on-change; deterministically verified + rig-booted. | | `fix/docker-dev-serverhistorian-stale-keys` | `7233e2ba` | **Rig fix (off master, independent):** docker-dev compose's retired Wonderware `ServerHistorian` keys → gateway shape, disabled by default. Found during the #12 live-gate rig session; central-1 verified booting clean. | **Decision (user):** one branch per Critical, each off `master`. **#9 (failover test) lives on the crit1 branch** (not its own branch off master) because SBR only exists there — it is Critical 1's integration half. The plans are only on the docs branch, so from a critical branch read them via `git show docs/archreview-plans-and-drift-fixes:archreview/plans/0X-.md`. ## Completed | Item | Finding | Branch @ commit | Verification | |---|---|---|---| | **Critical 1 — split-brain resolver activated** | 03/S1 | `fix/archreview-crit1-split-brain-resolver` `a81dea10` | `BuildClusterOptions` sets `KeepOldestOption{DownIfAlone=true}`; 3 unit guards (Cluster.Tests 29/29); **e2e-verified** the resolved ActorSystem config carries `SplitBrainResolverProvider` + `stable-after=15s` survives the HOCON/Hosting merge (throwaway spike, removed). HOCON cross-ref comment + Redundancy.md corrected. | | **Critical 2a — real VT script timeout** | 02/U2 | `fix/archreview-crit2-vt-timeout` `7fd44f0f` | Routes through `TimedScriptEvaluator`; infinite-loop test (bounded by WaitAsync) proves no hang. Host.Integration VT 18/18. | | **Critical 2b — ALC-safe compile cache** | 02/U3 | same `7fd44f0f` | Swapped raw dict → `CompiledScriptCache`; new `IScriptCacheOwner` seam; apply-boundary clear in `VirtualTagHostActor`; wiring guard test. Runtime VT 12/12. | | **Doc drift — KNOWN LIMITATION 2 stale** | 03/U1 = 06/U-1 | `docs/…` `9fadead6` (CLAUDE.md) | historized-ref feed IS wired; rewrote to "value-capture code-complete, live-verify pending". | | **Doc drift — ScriptAnalysis policy** | theme #5 | same `9fadead6` (CLAUDE.md) | corrected `FleetAdmin` → `Administrator,Designer` roles. | | **Critical 3 — S7 lazy reconnect** | 05/STAB-1 | `fix/archreview-crit3-s7-reconnect` `25c0c6f6` | New `IS7Plc`/`IS7PlcFactory` seam + `EnsureConnectedAsync` (dispose-dead-handle → reopen on next data call); connection-fatal faults (socket/`ErrorCode.ConnectionError`) mark the handle dead, data-address errors do NOT; probe loop routes through it as a backstop. 4 unit guards + full S7 suite 234/234; **LIVE-verified** against real python-snap7 — bounced the container (`docker restart`), driver observed the outage and resumed Good reads with no redeploy (`S7_1500ReconnectTests`, double-gated on sim + `S7_RECONNECT_BOUNCE_CMD`). Also closes the S7 TEST-1 gap. | | **Critical 4 — TwinCAT native-notification replay** | 05/STAB-2 | `fix/archreview-crit4-twincat-replay` `af318fb4` | Native ADS notifications were opaque handles (no replayable intent) → silently orphaned on a client swap. Now `NativeRegistration` (symbol/type/bit/interval/onChange + swappable handle) hangs off `DeviceState`; `EnsureConnectedAsync` split into a gate wrapper + under-gate core that **replays every intent onto a freshly-built client** (register+replay both under `ConnectGate`); probe-failure `RecycleClientAsync` disposes+nulls the wire-dead-but-port-open client so the next tick rebuilds+replays. 4 unit guards + full TwinCAT suite 174/174. **Unit-only — no TwinCAT docker fixture exists** (integration needs a real TC3 XAR), so the fake-client tests are authoritative (no live gate like S7 had). | | **Guard — reflection-exhaustive deferred-sink forwarding** | 03/U2 | `fix/archreview-u2-deferred-sink-forwarding-guard` `a65c2ced` | New `DeferredSinkForwardingReflectionTests` (Commons.Tests) reflects over **every** method of every forwarding interface `DeferredAddressSpaceSink` implements (`IOpcUaAddressSpaceSink` + `ISurgicalAddressSpaceSink`) — plus the sibling `DeferredServiceLevelPublisher` — invokes each on the wrapper over a `DispatchProxy` recording inner, and asserts the call **reached** the inner. A `DispatchProxy` recorder auto-implements future members (zero maintenance); a guard-of-the-guard asserts the wrapper implements exactly the known forwarding-interface set, so a brand-new capability *interface* trips too. **This closes the F10b/PR#423 trap class and is the prerequisite gate for 03/P1 remove methods + 02/P `WriteValues`.** 3 new tests green (Commons.Tests 57/57); **negative control verified** — stubbing `UpdateFolderDisplayName` to a non-forwarding no-op fails the guard with the exact member-named diagnostic. NO production change. | | **Guard — wire OTOPCUA0001 analyzer + triage** | 07/C-1 | `fix/archreview-c1-wire-analyzer` `f0082af5` | Injected the analyzer tree-wide via `Directory.Build.props` (`OutputItemType="Analyzer"`), then triaged ~280 surfaced hits: **(1) RESILIENCE-DISPATCH-GAP** — 7 production sites (`DriverInstanceActor` ×6 + `GenericDriverNodeManager`) scoped-`#pragma`'d as a **real tracked gap** (CapabilityInvoker built but never wired into dispatch; **new task #10**); **(2) driver-internal self-calls** — 3 sites (AbCip/S7 own poll/ack) `#pragma`'d as intentional; **(3) 12 wire-level test/CLI projects** → commented `NoWarn`. Full build green, 0 hits; analyzer 31/31; **negative control** — dropping one dispatch-gap pragma re-fires OTOPCUA0001 and fails the build, proving it's live tree-wide. | | **Guard — CI to all suites + fail-on-skip + deflake** | 07/S-1,S-2,S-4 | `fix/archreview-s1-s2-ci-coverage` `10b89830` | S-1: replaced the 5-project `unit-tests` matrix with ONE whole-solution leg (self-maintaining; now covers Client's 388 + Analyzers' 31 + all drivers). S-2: `scripts/ci/assert-not-all-skipped.sh` fail-on-skip gate (turns "green==all-skipped" red) — strict on the unit leg, report-only on the fixtureless integration leg. S-4: readiness-signal (`SubscribeInvoked`/`SubscribeAlarmsInvoked`) replaces 11 fixed-sleep startup races in the newly-CI'd Client.CLI.Tests. Skip-gate proven locally (real trx OK / all-skip trx exit 1 / report-only / edge cases); Client.CLI.Tests 104/104. **CI job execution itself is verifiable only on push (nothing pushed).** | | **#10 — wire CapabilityInvoker into dispatch (seam)** | RESILIENCE-DISPATCH-GAP (07/C-1) | `fix/archreview-crit10-wire-capability-invoker` `62556c24` | **Discovered the filed plan was infeasible** — Runtime is deliberately Polly-free (refs `Core.Abstractions`, not `Core`). Fix = a seam mirroring `IDriverFactory`: `IDriverCapabilityInvoker`(+null) + `IDriverCapabilityInvokerFactory`(+null) in Core.Abstractions; `CapabilityInvoker` implements it; `DriverCapabilityInvokerFactory` (Core) builds per-instance over the singleton builder+tracker+`GetTier`; Host `DriverFactoryBootstrap` registers all three; Runtime SCE resolves like `IDriverFactory`; `DriverInstanceActor` wraps its **6** dispatch sites; analyzer taught the interface as a wrapper home; all 6 pragmas removed. Triage corrections: the 7th site (`GenericDriverNodeManager`) is test-only scaffolding (not wired, re-annotated); **tier-defaults only** (ResilienceConfig not in the artifact → sub-gap). Added retry/breaker **logging** to the pipeline builder (the tracker has no reader — Stream E.2/E.3 never shipped — so it ran invisibly). **Verified:** negative-control (unwrap→OTOPCUA0001 build error); recording-invoker routing test; **real-`CapabilityInvoker` actor-context test** (internal `ConfigureAwait(false)` doesn't leak); analyzer + logging tests; clean solution build; Runtime.Tests 358 / Core.Tests 238 / Analyzers 32. **Live behavioral gate pending** (recipe in FOLLOWUP-10). | | **#9 — hard-kill failover integration test** | 03/S1 | `fix/archreview-crit1-split-brain-resolver` `a25c9ed0` | Critical 1's load-bearing integration half. `TwoNodeClusterHarness.HardKillNodeAAsync` (transport-shutdown crash sim → node B sees A Unreachable, no graceful Leave) + `HardKillFailoverTests` asserting B downs the alone oldest node and takes over as `driver` role-leader. **LIVE-verified in-process** (~35s); **negative control** (force `downing-provider-class=""`) makes it correctly time out. ⚠️ The negative control also surfaced a **Crit-1 premise finding (task #11): SBR was already active on master** (Akka.Cluster.Hosting applies `SplitBrainResolverOption.Default` when the typed option is null → reads the pre-existing akka.conf keep-oldest block), so the cluster was NOT NoDowning before Crit-1 — the typed option is reinforcing, not the activator. | | **#11 — Critical 1 premise correction** | 03/S1 (from #9) | `fix/archreview-crit1-split-brain-resolver` `eaf78aad` | Docs/comment-only fix (no code revert): corrected the "HOCON inert / cluster runs NoDowning / never fails over" framing in `BuildClusterOptions` XML comment, `akka.conf` split-brain-resolver comment, `docs/Redundancy.md`, and `SplitBrainResolverActivationTests` (summary + `ShouldNotBeNull` message + method rename). Akka.Cluster.Hosting enables an SBR provider **by default** (applies `SplitBrainResolverOption.Default` when the typed option is null → reads the pre-existing akka.conf `keep-oldest` block), so the cluster was never NoDowning and the typed `KeepOldestOption` is reinforcing/explicit-in-code, not the activator. Cluster.Tests 29/29; failover behaviour still guarded by `HardKillFailoverTests`. Architecture.md carried no such framing. | | **#13 — apply per-instance ResilienceConfig from the artifact** | RESILIENCE-DISPATCH-GAP sub-gap (from #10) | `fix/archreview-crit13-resilience-config-artifact` `75403caa` | The `DriverInstance.ResilienceConfig` column was authored+persisted (AdminUI) and already in the artifact (the composer serializes the whole entity), but the driver-node read path dropped it → tier-defaults only. `DriverInstanceSpec` now carries it; `TryReadSpec` reads it; `Create` takes+layers it (+ logs parse diagnostics, never throws); `SpawnChild` threads it. **Invalidate-on-change:** `Create` calls `builder.Invalidate(instance)` (the cache is options-blind on a hit) so a respawn with changed options rebuilds; `DriverSpawnPlanner` respawns on a ResilienceConfig change (incl. null→json), a pure DriverConfig change stays an in-place delta. **Verified:** override reaches execution (retryCount:0 suppresses tier-A retry; a control retries), invalidate is per-instance, malformed logs+falls-back; planner+artifact tests. Core.Tests 243 (+5), Runtime.Tests 363 (+5). See [`FOLLOWUP-13`](FOLLOWUP-13-resilience-config-artifact.md). | ## Task list (TaskCreate IDs) | # | Task | Status | |---|---|---| | 1 | Critical 1 — split-brain resolver | ✅ completed | | 2 | Critical 2a — VT timeout | ✅ completed | | 3 | Critical 2b — CompiledScriptCache | ✅ completed | | 4 | Critical 3 — S7 reconnect (05/STAB-1) | ✅ completed | | 5 | Critical 4 — TwinCAT ADS re-registration (05/STAB-2) | ✅ completed | | 6 | Guard — DeferredAddressSpaceSink reflection forwarding test (03/U2) | ✅ completed — `a65c2ced`; gate for 03/P1 + 02/P now in place | | 7 | Guard — wire OTOPCUA0001 analyzer + triage (07/C-1) | ✅ completed — `f0082af5`; surfaced task #10 | | 8 | Guard — CI to all unit suites + fail-on-skip (07/S-1,S-2, pair S-4) | ✅ completed — `10b89830` | | 9 | Critical 1 follow-up — hard-kill failover integration test (03/S1) | ✅ completed — `a25c9ed0`; surfaced task #11 | | 10 | **Wire CapabilityInvoker into the dispatch layer** (RESILIENCE-DISPATCH-GAP, from 07/C-1) | ✅ code-complete + unit/analyzer/context-verified — `fix/archreview-crit10-wire-capability-invoker` `62556c24` (seam, NOT direct thread — Runtime is Polly-free). **Live behavioral gate pending** (rig recipe in FOLLOWUP-10). Surfaced a sub-gap: ResilienceConfig not in the deploy artifact → tier-defaults only. | | 11 | **Review Critical 1 premise** — SBR was already active on master, not NoDowning (from #9) | ✅ completed — `fix/archreview-crit1-split-brain-resolver` `eaf78aad`; docs/comment-only correction (no code revert), Cluster.Tests 29/29. | | 12 | **#10 live behavioral gate** — rig session (from #10) | ✅ ran 2026-07-08 (PARTIAL) — built the Host image from source (crit13 tip) + stood up the rig; central-1 booted healthy; **prod DI binds the real invoker factory** confirmed live + by a new deterministic guard (`ResilienceInvokerFactoryRegistrationTests` `8a8b9ec5`). Behavioral retry/breaker **log-line not observed** (disciplined stop): Modbus's cheap paths don't throw a wrapped exception (Discover offline, Subscribe lazy, reads non-wrapped) — only an idempotent write-to-dead-endpoint (or S7) does, not worth the multi-dependency chase since the retry behavior is deterministically proven. **Found + FIXED a rig bug:** docker-dev compose's retired Wonderware `ServerHistorian` keys crashed the Host at bring-up (historian-gateway cutover → reads `Endpoint`) → fixed on `fix/docker-dev-serverhistorian-stale-keys` `7233e2ba` (off master; gateway-shape keys, disabled by default; central-1 verified booting clean). See [`FOLLOWUP-10`](FOLLOWUP-10-resilience-dispatch-gap.md). | | 13 | **Plumb per-instance `ResilienceConfig` through the deploy artifact** (from #10) | ✅ completed — `fix/archreview-crit13-resilience-config-artifact` `75403caa`; read-path plumbing (write side was already complete), invalidate-on-change + respawn-on-change; Core.Tests 243/Runtime.Tests 363. See [`FOLLOWUP-13`](FOLLOWUP-13-resilience-config-artifact.md). | ## Findings surfaced by the guards (NEW — need user decision) > Each has a dedicated follow-up file with the full design/action: > [`FOLLOWUP-10-resilience-dispatch-gap.md`](FOLLOWUP-10-resilience-dispatch-gap.md) · > [`FOLLOWUP-11-crit1-sbr-premise.md`](FOLLOWUP-11-crit1-sbr-premise.md) · > [`FOLLOWUP-13-resilience-config-artifact.md`](FOLLOWUP-13-resilience-config-artifact.md). - **#10 — RESILIENCE-DISPATCH-GAP (from wiring the analyzer, 07/C-1): ✅ IMPLEMENTED** on `fix/archreview-crit10-wire-capability-invoker` `62556c24` — code-complete + unit/analyzer/context-verified; **live behavioral gate pending** (recipe in [`FOLLOWUP-10`](FOLLOWUP-10-resilience-dispatch-gap.md)). The filed "thread CapabilityInvoker into the actor" was infeasible (Runtime is deliberately Polly-free); shipped a seam mirroring `IDriverFactory` instead. Only **6** of the 7 sites were real (the 7th, `GenericDriverNodeManager`, is test-only scaffolding). **Two residual sub-findings:** (a) ~~per-instance `ResilienceConfig` isn't in the deploy artifact → tier-defaults only~~ **✅ RESOLVED by task #13** (`75403caa`) — the column was already in the artifact; the read path now applies it (see [`FOLLOWUP-13`](FOLLOWUP-13-resilience-config-artifact.md)); (b) the resilience-status tracker has **no reader** (Admin `/hosts` Stream E.2/E.3 never shipped) — added retry/breaker **logging** as the interim observability surface + the live-verify enabler (still open as an Admin-UI follow-on). - **#11 — Critical 1 premise inaccurate (from the #9 failover negative control): ✅ CORRECTED** on `fix/archreview-crit1-split-brain-resolver` `eaf78aad`. SBR was **already active on master** (Akka.Cluster.Hosting's `WithClustering` applies `SplitBrainResolverOption.Default` when the typed option is null → enables the downing provider that reads the pre-existing akka.conf `keep-oldest` block). So the cluster was **not** running NoDowning; hard-crash failover already worked. Docs/comment-only fix (no code revert — the typed `KeepOldestOption` is correct belt-and-suspenders): corrected the "HOCON inert / NoDowning → no failover" framing in `BuildClusterOptions` XML comment, `akka.conf`, `docs/Redundancy.md`, and the `SplitBrainResolverActivationTests` summary/message (+ method rename). Cluster.Tests 29/29. (Architecture.md carried no such framing.) See [`FOLLOWUP-11`](FOLLOWUP-11-crit1-sbr-premise.md). ## Suggested next (from 00-INDEX execution order) **All 4 Criticals + all 3 systemic guards (tasks 6/7/8) + the #9 failover test are DONE.** Remaining: 1. **Resolve the two surfaced findings** — task 10 (wire CapabilityInvoker into dispatch; behavior-changing, live-gate) and task 11 (correct Crit-1's rationale/docs). 2. **Failure-visibility Highs** — 01/S-1 (AddressSpaceApplyOutcome failure field), 06/S-1 (Galaxy fail-closed), 03/S4 (primary-gate default-deny). 3. **P-tier** — 03/P1 surgical adds/removals (now gated by the U2 forwarding guard), 02/P batched `WriteValues`. ## Key resume facts - **Verification bar met so far:** each fix has a deterministic unit guard AND (for SBR) an end-to-end config-resolution check. Follow the same pattern — the review's whole thesis is "unit-green ≠ wired". - **The forwarding-test guard (task 6) is DONE (`a65c2ced`):** 03/P1 (surgical pure-adds) and 02/P-… batch-write each add new sink capability methods that ship inert unless forwarded through `DeferredAddressSpaceSink` (F10b/PR#423 precedent). The reflection guard now fails loudly if any such member isn't forwarded — add the member to the recorder-covered interface set and it's auto-checked. - ~~**Nothing is pushed.**~~ **Superseded 2026-07-12:** all branches merged to master and pushed to `origin/master`; the plans now live on master too (no more cross-branch `git show` needed). - ~~**CI changes can't be run-verified without a push**~~ — now pushed; the first real CI-job run should be checked on origin (the re-review verified the YAML statically only). - **Findings #10 and #11 are resolved** (see above); the open decisions now come from the 2026-07-12 re-review below. ## Re-review 2026-07-12 (master `f6eaa267`) — outcome + new findings All seven domain reports were re-run and rewritten in place at `f6eaa267` (each carries a `Prior-finding status (9cad9ed0 → f6eaa267)` table; synthesis in [`../00-OVERALL.md`](../00-OVERALL.md)). Verdict on the remediation: **all 4 Criticals verifiably closed in code** (Crit-1 by premise correction), both systemic guards live with negative controls, CI now whole-solution + fail-on-skip (07 stability 2.5→3.5). New findings needing scheduling, ranked: | Rank | Finding | Severity | One-line | |---|---|---|---| | 1 | 05/STAB-14 | **High (regression)** | S7 ensure-wrapper rethrows connect-timeout OCE → poll loop's bare `catch (OCE) { return; }` silently kills all S7 subscriptions after an unreachable-host (non-refusing) outage; live bounce test blind to it (`S7Driver.cs:488,1000,1383,1404,1206-1208`) | | 2 | 01/S-6 | **High** | Operator-authorable `timeoutSeconds:0` in ResilienceConfig → Polly ValidationException on every capability call = permanent driver-brick; no range validation in parser or AdminUI form | | 3 | 05/STAB-15 | **High** | S7 fatal-classification misses framing/desync + cancel-during-I/O (half-read PDU → permanent desync); probe catch never marks handle dead | | 4 | 01/S-8 = 03/S12 | Medium | Dispatch hardcodes `isIdempotent:true`; `WriteIdempotentAttribute` has zero readers → #13-enabled write retries would retry command writes | | 5 | 02/S13 | Medium | VT script failure/timeout leaves node stale-Good forever (timeout fix made the failure invisible) | | 6 | 01/U-6 | Medium | Bulkhead knob parsed/authored/documented but never built into the pipeline — dead knob | | 7 | 06/S-11 | Medium | Enabled-but-misconfigured ServerHistorian crash-loops Host with raw UriFormatException (only config side fixed in 7233e2ba) | | 8 | 06/U-7 | Medium | HistorianGateway 0.2.0 proto3-optional DataType change silently alters GatewayTagProvisioner Boolean provisioning; live EnsureTags never run on 0.2.0 | | 9 | 04/C-7, 04/U-5=03/U10, 01/S-7=03/S13 | Medium/Low | Resilience seam edges: lossy AdminUI form round-trip; status tracker has no reader (operators blind); respawn stale-pipeline cache race | Carried-open Highs unchanged from round 1 (no branch targeted them): failure-visibility trio (01/S-1, 06/S-1, 03/S4), AdminUI authz (04/C-1 — stakes rose: resilience config is on the ungated surface), 03/S2/S3 block-bridging, 03/P1 surgical adds (guard prerequisite in place), driver-fleet batch (05), repo hygiene (07). Full prioritized list in `00-OVERALL.md`. **Round-2 plans WRITTEN (2026-07-12):** all 12 action items have design+implementation plans — `R2-01`…`R2-12` in this directory (+ co-located `.tasks.json` each, for resumable execution). Index, per-plan task counts, effort, and suggested execution order: [`00-INDEX.md`](00-INDEX.md) §"Round 2 plans". 193 bite-sized TDD tasks total, ~18–24 dev-days end-to-end; R2-01/02/03 (the re-review's new-and-sharp items) are ~2 days combined. **Round-2 EXECUTED — R2-12 Repo hygiene (2026-07-13, branch `r2/12-repo-hygiene`):** DONE. 07/U-5 (7 `lib/` AVEVA DLLs removed + gitignored, zero-reference proven), 07/U-4 (root planning files: `pending.md`/`current.md` untracked+gitignored, `looseends.md` deleted, `stillpending.md` + `HISTORIAN-GATEWAY-INTEGRATION-ISSUES.md` archived under `docs/plans/`, `docs/Historian.md` repointed), 07/S-3 (`v2-e2e.yml` no-op nightly deleted), 07/P-5 (redundant `integration` matrix job removed from `v2-ci.yml`), 07/U-3 (5 alarm operator CLI commands documented — now 13/13), 07/C-5 (`StyleGuide.md` ScadaBridge→OtOpcUa + minimal root `.editorconfig`), 07/C-4 (xunit pin 2.9.2→2.9.3 + blocker comment; upstream-blocked — no xunit.v3 Akka TestKit exists). 07/U-6 needed no action (the Wonderware husk dirs are untracked build artifacts, absent in a fresh worktree). Whole-solution build clean. **OPEN follow-ups:** (a) when a runner-reachable fleet + a real `tests/Server/…E2ETests` land, add a fresh E2E workflow WITH a zero-match guard (fail if `Category=E2E` matches nothing); (b) S-2 option #2 — a fixture-backed integration leg (opc-plc `services:` block, `MIN_EXECUTED=1`) covering what the unit leg cannot; (c) migrate the 3 xunit v2 holdouts once Akka ships an xunit.v3 TestKit. ### Round-2 remediation landed | Plan | Findings | Branch | Commits | Status | |---|---|---|---|---| | **R2-06** | 06/S-11 (rank 7), 06/U-7 (rank 8), 06/C-7 | `r2/06-serverhistorian-failfast` (off `1676c8f4`) | `3a049a13`…`4640ecb1` (10 task commits) | **Code-complete, offline-verified.** S-11: fail-fast `ServerHistorianOptionsValidator` (consumer-gated on `ServerHistorian:Enabled` OR `AlarmHistorian:Enabled`, fails empty/non-http(s) Endpoint at startup) + `AddValidatedOptions` wiring + adapter defense-in-depth (`TryCreate`→named `InvalidOperationException`, no more raw `UriFormatException` crash-loop) + `Validate()` warning parity + alarm-only-mode startup warning. U-7: pinned Boolean→Int1 explicit-presence (`HasDataType`) + `HISTGW_BOOL_SANDBOX_TAG` live Boolean EnsureTags leg + retype probe (skip-clean offline) + upsert/retype migration note (`docs/Historian.md`). C-7: keep-populating decision + dead-`Quality` doc comments (0.2.0 C-002). **Live leg operator-gated (T12) — pending a VPN run on a 0.2.0 gateway** (records the observed in-place retype policy back into `docs/Historian.md`; also discharges 06/U-2's full documented run). | | **R2-05** | 04/C-1 (High) | `r2/05-adminui-authz` (off `1676c8f4`) | `183b72b7`…`6c1f05cb` (5 commits) | **Code-complete, offline-verified.** All 38 routable AdminUI pages carry an explicit named-policy `@attribute` via new `AdminUiPolicies` constants (`Security/Auth/`). New `ConfigEditor` (Administrator+Designer) gates the 20-page config-authoring surface incl. the 8 driver pages that author live `ResilienceConfig` + `/api/script-analysis/*`; new `AuthenticatedRead` on the 16 read pages; `FleetAdmin`/`DriverOperator` unchanged (literals→constants). Anti-drift guard `PageAuthorizationGuardTests` (bans bare `[Authorize]`/`Roles=`/unknown-policy/unclassified) + `AdminUiPoliciesTests` semantics matrix. Corrections folded in: Reservations + ClusterRedundancy read-only (→`AuthenticatedRead`); Home had no attribute (now `AuthenticatedRead`). AdminUI.Tests 517/517, Security.Tests 80/80. **Live `/run` positive pass deferred (T15)** — docker-dev `DisableLogin=true` auto-admin can't observe deny; negative proof is CI-side (both guard tests). | | **R2-01** | 05/STAB-14 (High regression, rank 1), 05/STAB-15 (rank 3), 05/STAB-8 seam | `r2/01-s7-fault-hardening` (off `1676c8f4`) | `fcdcf0e2`…`bdb7ed41` (13 task commits) | **Code-complete, offline-verified.** All prod changes in `S7Driver.cs`. STAB-14: connect-timeout surfaces as `TimeoutException` (not an escaping OCE) at the `EnsureConnectedAsync`/`InitializeAsync` source + `when(token.IsCancellationRequested)` filters on the read/write ensure-wrappers and all three poll-loop OCE catches — subscription poll loops survive an unreachable-host outage and recover (T2 regression RED→GREEN). STAB-15: `IsS7ConnectionFatal` broadened to the ISO-on-TCP framing surface (TPKT/TPDU/WrongNumberOfBytes/`PlcException{WrongNumberReceivedBytes}`, NOT `InvalidDataException`); mid-PDU cancel marks handle dead; probe classifies+marks-dead under the gate. STAB-8 (S7 part): `EnsureConnectedAsync` documented as the R2-09 connect-throttle seam (note-only). Driver.S7.Tests 246/246. **Live outage gate deferred (T11)** — `S7_1500ConnectTimeoutOutageTests` (env `S7_TIMEOUT_OUTAGE_START_CMD`/`STOP_CMD`, SYN-blackhole) skips clean offline; the connect-*timeout* class the `docker restart` bounce can't produce. (Pre-existing unrelated CLI docs-scan test fails at base — out of scope.) | | **R2-03** | 02/S13 (rank 5), 02/S12, 02/P7 | `r2/03-vt-failure-quality` (off `1676c8f4`) | `b9775485`…`6bb91e8d` (12 task commits) | **Code-complete, offline-verified.** S13: VT script failure/timeout degrades the node to `OpcUaQuality.Bad` (carrying last-known value) instead of freezing on last-Good forever — additive `EvaluationResult.Quality`, bridge historizes `0x80020000u` BadInternalError, published once per Good→Bad transition gated on all deps arrived (inputs-ready, no cold-start flash), recovery force-publishes next Good. S12: apply-boundary `ClearCompiledScripts` racing an in-flight eval into ObjectDisposedException retried once in `RoslynVirtualTagEvaluator` (re-fetch→recompile into current generation); stress guard reproduces pre-fix, deterministically green post-fix. P7: apply-boundary clear gated on a changed deployed expression set (`HashSet.SetEquals`, ordinal) — identical redeploy / rename-with-same-expressions no longer forces full VT recompile. Runtime.Tests 372/372, RoslynVirtualTagEvaluatorTests 20/20. **Deferred (T11/T12):** full cross-suite sweep + docker-dev live-`/run` (fail→Bad→recover + P7 no-recompile spot-check). | | **R2-02** | 01/S-6 (High, rank 2), 01/U-6, 01/S-8=03/S12 (rank 4), 04/C-7, 01/S-7=03/S13 | `r2/02-resilience-config-hardening` (off `1676c8f4`) | `4be13429`…`dd82e740` (17 task commits) | **Code-complete, offline-verified.** **S-6:** parser clamps timeout/retry/breaker overrides (timeout≤0→tier default, breaker==1→2, caps) with diagnostics + belt-and-braces `Build` clamp so pipeline construction can never throw Polly `ValidationException` (new `ResiliencePolicyRanges` in Core.Abstractions); brick-repro + `Build_NeverThrows_ForHostileDirectOptions` + factory-seam wiring proof; AdminUI `Validate()` range warnings. **S-8:** parser forces Write/AlarmAck retry→0 (spec invariant) **and** dispatch flips to `isIdempotent:false` (RED-first on the Runtime wiring test); non-idempotent arm caches the no-retry snapshot once/invoker (01/P-4) + tracker in-flight accounting; `WriteIdempotentAttribute` false claim corrected; retry cells disabled. **U-6:** bulkhead knob deleted end-to-end (migration-free; stored keys ignored) + `Options_properties_are_exactly_the_pipeline_wired_set` inertness guard. **C-7:** `ResilienceFormModel` JsonObject-bag round-trip preserves unknown keys; malformed JSON → `ParseFailed` lock + verbatim `ToJson` + discard button. **S-7:** per-`Create` `Interlocked` options-generation in the pipeline-cache key kills the respawn stale-pipeline race. Core resilience 121/121, AdminUI form 10/10, Analyzers 32/32, Runtime 364/364; 0 production OTOPCUA0001. **Deferred (T15/T18):** docker-dev `:9200` `/run` (warning banner, input lockout, unknown-key survival) + whole-solution sweep. | | **R2-10** | 03/U10=04/U-5 (Medium/Low, action #10) | `r2/10-resilience-observability` (off `46fedda3`) | `205d3980`…`` (11 task commits) | **Code-complete, offline-verified.** The resilience status tracker now has a production reader, mirroring the driver-health flow. Core truthfulness: `RecordBreakerClose` + derived `IsBreakerOpen` (breaker-open state derivable), `OnClosed` records the close, and `RecordSuccess` wired onto the invoker success paths (`ConsecutiveFailures` is a true consecutive counter, not stuck-forever). Transport: `DriverResilienceStatusChanged` Commons contract → periodic 5 s full-snapshot publisher `DriverResilienceStatusPublisherService` (driver nodes, registered in `AddOtOpcUaDriverFactories` with wiring guard `ResilienceStatusReaderRegistrationTests` — closes the built-but-never-wired class) → `driver-resilience-status` DPS topic → per-admin-node `DriverResilienceStatusBridge` → `IDriverResilienceStatusStore` → resilience chips (breaker-OPEN / consecutive-failures / in-flight / staleness) on `DriverStatusPanel.razor`, read in-process (self-HubConnection ban honored). Core.Tests resilience 31/31 (incl. deterministic breaker-close via a controllable TimeProvider), Host.IntegrationTests publisher+guard 6/6, AdminUI.Tests store 5/5; full-solution build clean. **Deferred (T10):** docker-dev live-`/run` — S7 dead-endpoint breaker-open observed on BOTH centrals + recovery-clear + staleness-dim (also the first live sighting of the #10 `circuit-breaker OPENED` log line, closing FOLLOWUP-10's behavioral gate). | | **R2-08** | 03/S2 (High, action #8), 03/S3 (High) | `r2/08-ldap-historyread-async` (off `1a698cbb`) | `ce6d9e21`…`` (14 task commits) | **Code-complete, offline-verified.** **S2 (LDAP async/timeout):** the SDK raises the impersonation callback synchronously under a `SessionManager`-wide `m_eventLock` (verified vs. 1.5.378 — a hung bind serializes ALL activations), so the fix is entirely at the `LdapOpcUaUserAuthenticator` boundary: five new `Security:Ldap` keys (`ConnectionTimeoutMs`=10000 projected into the shared lib, `AuthTimeoutMs`=15000 hard whole-flow `WaitAsync` bound → fail-closed "timed out", `MaxConcurrentAuthentications`=8 `SemaphoreSlim` with release-on-core-completion so orphaned binds still count, `OutageFailureThreshold`=3/`OutageCooldownSeconds`=15 directory-outage circuit → fast-deny "unavailable", half-open probe) + `LdapAuthResult.IsSystemFailure` seam (mapped from `ServiceAccountBindFailed`) distinguishing outage from credential-deny + `LdapOptionsValidator` rules + `HandleImpersonation` contract doc + `docs/security.md`. Stall-repro RED→GREEN; offline outage-sim through the real `HandleImpersonation` against an unroutable host. **S3 (HistoryRead fan-out):** all four HistoryRead arms async-ified (`ServeNodeAsync`/`ServeRawPagedAsync`/`ServeEventsAsync`, OCE→`BadTimeout`) then bounded per-node fan-out (`RunBounded` semaphore + single arm-boundary bridge) + per-request deadline CTS (`HistoryReadDeadline`=60s) + process-wide lazy batch limiter (`MaxConcurrentHistoryReadBatches`=16 → flood degrades to `BadTooManyOperations`); 16×4=64 in-flight cap; three new `ServerHistorian` keys + node-manager props + hosted-service wiring + `docs/Historian.md`/CLAUDE.md tables. Sequential-serving RED→GREEN. Host.IntegrationTests (LDAP resilience) 13/13, Security.Tests 11/11, Runtime.Tests options 11/11, OpcUaServer.Tests HistoryRead 30/30. **Deferred (live):** S2 docker-dev unreachable-LDAP + GLAuth outage/recovery cycle; S3 `Category=LiveIntegration` gateway soak. | | **R2-11** | 01/C-1, 01/P-1 + 05/CONV-2, UNDER-1, UNDER-6 | `r2/11-tagconfig-consolidation` (off `46fedda3`) | first `2e14fe1f` … (24 task commits) | **Code-complete, offline-verified.** **Part 1 (C-1/P-1):** single `TagConfigIntent.Parse` + `DeviceConfigIntent` in Commons replace the four byte-parity TagConfig parse copies (composer/artifact/walker/DraftValidator) — composer now parses once per tag (P-1); a permanent `Runtime.Tests/TagConfigCorpusParityTests` (30-blob corpus) guards compose≡decode; `grep "MUST parse identically" src/` = 0. **Part 2 (CONV-2):** shared strict-capable `TagConfigJson` readers in Core.Abstractions replace the six copy-pasted `ReadEnum`s; all six driver parsers gain `Inspect()` warnings + honour the `writable` key (FOCAS **forced read-only**, 05/UNDER-1); FOCAS equipment tags run the capability-matrix pre-flight (05/UNDER-6); Modbus probe parses the factory DTO shape (`timeoutMs` no longer dropped). New `EquipmentTagConfigInspector` in ControlPlane + `AdminOperationsActor` deploy-gate wiring with `Deployment:TagConfigValidationMode` (**Warn default | Error opt-in**), proven through the actor (F10b consuming test). AdminUI: `writable` checkbox in the six typed editors. **Phase-C follow-up (deferred, recorded here):** flip the runtime parsers to strict (typo'd enum ⇒ `BadNodeIdUnknown` instead of wrong-width `Good`) once fleets have run `Error` mode clean — behavior change deliberately not shipped in this plan. **Deferred (T22/T24):** docker-dev `/run` of one typed editor's Writable checkbox + whole-solution serial sweep. | | **R2-07 (Phases 1+2+3)** | 03/P1 (action #7, surgical pure-adds/removes/mixed) | `r2/07-surgical-pure-adds` (off `1a698cbb`) | `bb226f45`…`e2b47638` (T1–T14) | **Phases 1+2 code-complete, offline-verified.** New pure `AddressSpaceChangeClassifier` (Empty/AttributeOnly/PureAdd/PureRemove/AddRemoveMix/Rebuild; first-match-wins, default-closed Rebuild; the F10b tag/vtag surgical-eligibility predicates moved here as `internal static`). `AddressSpaceApplier.Apply` routes by kind. **Phase 1 (PureAdd/AttributeOnly):** skip `RebuildAddressSpace` (the idempotent Materialise passes create exactly the added nodes; existing subscriptions survive); adds announced via `ComputeAddAnnouncements`+`AnnounceAddedNodes` (dedup affected-parent → existing guard-covered `RaiseNodesAddedModelChange`), fired by `OpcUaPublishActor` after the 4 passes gated on `!outcome.RebuildCalled`; `MaterialiseAlarmCondition` made idempotent (skip-if-present-and-same-kind keeps the condition NodeState alive). **NO new sink members in Phase 1.** **Phase 2 (PureRemove):** 3 new `ISurgicalAddressSpaceSink` members (`RemoveVariableNode`/`RemoveAlarmConditionNode`/`RemoveEquipmentSubtree`) — guard-first with the **observed negative control** (`DeferredSinkForwardingReflectionTests` fails member-named until `DeferredAddressSpaceSink` forwards) + `SdkAddressSpaceSink` forwards + `OtOpcUaNodeManager` in-place teardown (scoped map cleanup, notifier demotion via `RemoveRootNotifier`, `BuildNodesRemovedModelChange` verb=NodeDeleted reported outside `Lock`). Applier `PureRemove` arm: terminal Bad / `RemovedConditionState` writes (closing the removed-alarm-tag today-gap) → subsumption partition (removed-equipment children folded into the subtree remove) → in-place removes → any false/throw ⇒ full-rebuild ratchet; non-surgical sink ⇒ rebuild. **Phase 3 (AddRemoveMix):** removes-then-adds within one apply — the remove pass runs in `Apply` (no rebuild), then the publish actor's Materialise passes create the adds + `AnnounceAddedNodes` announces them; an id reused across the remove+add sets is remove-then-recreated as a fresh NodeState (correct — a different tag). **Guard stays green (Commons.Tests 17/17), OpcUaServer.Tests 336/336, Runtime.Tests 398/398.** **03/P1 marked REMEDIATED in `archreview/03-server-runtime.md`.** **Deferred-live (T5, T6, T12, T14):** `SubscriptionSurvivalTests` add + remove + mixed cases (authored + compile-green; NOT run — `*.IntegrationTests` memory constraint) and the docker-dev live-`/run` add/remove/mixed gates (`kind=PureAdd/PureRemove/AddRemoveMix, rebuild=False`; survivor subscription uninterrupted; removed node → `BadNodeIdUnknown`) — MUST run in the serial heavy pass (F10b inertness risk). | **Report anchor drifts to fold into the next report refresh** (from the R2-02 verification pass): (1) `01-core-composition.md` U-6 anchor `DriverResilienceOptions.cs:23-25` → the two bulkhead properties sat at `:23` and `:29` (now deleted); (2) `01-core-composition.md` U-6 anchor `DriverResiliencePipelineBuilder.cs:99-177` → `Build` was `:99-176`. All other cited lines were exact at `f6eaa267`. ## ✅ Round-2 COMPLETE — all 12 plans merged (2026-07-13) All 12 round-2 plans were executed (isolated worktree per plan, per-task TDD, pathspec-scoped commits) and **merged to `master` + pushed to `origin/master`** in 3 dependency-ordered waves, each plan landing as its own Gitea PR (closed-as-merged) behind a `dotnet build` barrier. **Final state: master `31c98200`.** | Wave | Plans (PR#) | |---|---| | A (no cross-plan deps) | R2-12 #426 · R2-06 #427 · R2-05 #428 · R2-01 #429 · R2-03 #430 · R2-02 #431 | | B (parse-site barrier) | R2-10 #432 · R2-04 #433 · R2-11 #434 | | C (largest, depend on earlier) | R2-08 #435 · R2-07 #436 · R2-09 #437 | Cross-plan mutexes honored: R2-01 landed before R2-09 (PollGroupEngine v2 / S7 fork retirement); R2-11's `TagConfigIntent.Parse` consolidation landed before R2-09's driver parse edits; the three plans that all edit `AddressSpaceApplier.cs` + `DriverHostActor.cs` + `OtOpcUaNodeManager.cs` (R2-04/R2-07/R2-08/R2-10/R2-11) were merged sequentially with the shared-file auto-merges verified by targeted unit runs each time. ### Consolidated unit sweep on merged `master` `31c98200` (integration suites excluded — ~16 GB/run leak) Solution `dotnet build` clean (0 errors). All R2-relevant unit suites GREEN, incl.: OpcUaServer 341, Runtime 398, Commons(+forwarding guard) 122, Core 276, Core.Abstractions 128, Core.Scripting 137, Core.VirtualTags 59, Cluster 29, Security 84, ControlPlane 78, Analyzers 32, AdminUI 537, Client.CLI 104, Client.Shared 182, Client.UI 126, Core.AlarmHistorian 29, Core.ScriptedAlarms 71, and all drivers (S7 254, Modbus 312+161, AbCip 336, AbLegacy 209, TwinCAT 189, FOCAS 265, Galaxy 304, Galaxy.Browser 21, Historian.Gateway 102, OpcUaClient 131). **3 failures, all confirmed PRE-EXISTING / environmental (git diff `1676c8f4..HEAD` empty on their paths — R2 never touched them):** 1. `Core.Abstractions.Tests.InterfaceIndependenceTests.AllPublicTypes_LiveInRootNamespace` — flags the pre-R2 `Core.Abstractions.Historian.*` continuous-historization types living in a sub-namespace. **Pre-existing convention debt — needs-owner-decision** (move the types to root, or relax the test to allow the intentional Historian grouping). 2. `Driver.S7.Cli.Tests.SubscribeCommandConsoleHandlerCommentTests...` — pre-existing docs-scan test looking for a `"CliFx console"` phrase removed by `9cad9ed0`. **Follow-up: repoint or delete the phrase-scan.** 3. `Driver.OpcUaClient.Browser.Tests` (3) — `Connection refused`; need the live opc-plc endpoint (`10.100.0.35:50000`), unreachable offline. **Environmental — pass with the fixture up.** ### Deferred heavy / live-`/run` pass (serialized — single-tenant rig; controller-owned) Every plan's integration-suite and live-`/run` gates were deferred (not skipped) so nothing OOMs the box and the shared docker-dev rig / VPN infra isn't contended. To run one-at-a-time: - **Full integration-suite sweep** (10 `*.IntegrationTests` projects, one at a time): discharges R2-02 T18, R2-03 T11, R2-07 T5/T6, R2-08 T14, R2-11 T24. - **docker-dev `:9200` live-`/run`** (rebuild BOTH centrals): R2-02 T15 (ResilienceConfig warning banner / input lockout / unknown-key survival), R2-03 T12 (VT fail→Bad→recover + P7 no-recompile), R2-10 T10 (breaker-open on both centrals + recovery + FOLLOWUP-10 log line), R2-11 T22 (typed-editor Writable checkbox), **R2-07 T12 (surgical add/remove/mixed — F10b inertness risk, MUST run)**, R2-05 T15 (over-gating regression only — deny unobservable under `DisableLogin`). - **2-node redundancy rig:** **R2-04 T13/T15 — behavior-affecting S4 primary-gate default-deny; MUST run before trusting the failover-role change.** - **Genuinely external / needs-user:** R2-06 T12 (VPN to a 0.2.0 HistorianGateway for the live EnsureTags retype probe), R2-08 live legs (real GLAuth outage/recovery cycle; `Category=LiveIntegration` gateway soak), R2-01 T11 (S7 SYN-blackhole outage on the `10.100.0.35` s7_1500 fixture). ### Deferred behavior-change follow-up (deliberately not shipped) **R2-11 Phase C** — flip the runtime driver parsers to strict (typo'd enum ⇒ `BadNodeIdUnknown` instead of a wrong-width `Good`). Land only after fleets have run `Deployment:TagConfigValidationMode=Error` clean. ## docker-dev live-`/run` pass (2026-07-13) — CONCLUDED; 5 gates LIVE-confirmed on the built-from-source `6aeb53a9` rig **Bottom line:** R2-02, R2-11, R2-07 (both surgical phases), R2-04 (achievable failover), and R2-10 (resilience pipeline in dispatch) verified live on the docker-dev central cluster. R2-05 partial-by-design; R2-03 unit-verified (rig has no driver data). Everything still open needs infra this box lacks: the `*.IntegrationTests` sweep needs the `10.100.0.35` driver fixtures + ephemeral LDAP up; R2-06 needs VPN to a 0.2.0 gateway; R2-08 a live GLAuth outage cycle; R2-01 the S7 blackhole fixture. Those are the only Round-2 items not yet exercised. Brought up the 2-node central cluster (`central-1`+`central-2`, SQL, traefik) from source at merged master `6aeb53a9` (single-tenant rig; drove the `:9200` AdminUI via browser + Client.CLI on `:4840`/`:4841`). - **R2-02 (S-6 High) ✅ LIVE** — on the Modbus driver's *Resilience overrides*, entering `timeoutSeconds:0` renders the exact range-validation banner: *"Out-of-range values will be clamped to a safe value on deploy: Read timeout must be between 1 and 86400 seconds (got 0)."* **S-8 ✅** — the Write and AlarmAcknowledge RETRIES cells are disabled ("never"). - **R2-11 (T22) ✅ LIVE** — the typed Modbus tag editor shows the **Writable** toggle; saving persists `writable: yes`. - **R2-07 (P1 — F10b MUST) ✅ LIVE** — central-1 boot apply logged `AddressSpaceApplier: applied plan (kind=PureAdd, added=91, removed=0, changed=0, rebuild=False)`; deleting a UNS virtual-tag + redeploy logged `(kind=PureRemove, added=0, removed=1, changed=0, rebuild=False)` and the node vanished from the address space (`capper-03` lost its `production-count` child). **Surgical add AND remove both execute in prod with no full rebuild — the remove path's 3 new sink members are wired, not inert.** - **R2-04 (S4 MUST) ✅ LIVE (achievable guarantee)** — **younger-crash** (kill `central-2`): `SBR took decision DownUnreachable and is downing [central-2]`, `Member removed`, `central-1` stays Up, AdminUI serving, ServiceLevel **250** (sole primary, no split-brain). **Oldest-crash** (kill `central-1`) reproduced the **documented keep-oldest asymmetry** live: the survivor took `SBR … DownReachable … including myself` and CoordinatedShutdown (ExitCode 3) — i.e. crashing the *oldest* node does **not** fail over. This is the pre-registered SBR topology gap (a deferred USER strategy decision), **not** an R2 regression; R2-04's `PrimaryGatePolicy` write-gate change is orthogonal + unit-verified. - **R2-05 (T15) ✅ (partial, as expected)** — every config/read page renders under the all-roles `multi-role-test` session (named-policy gating didn't over-gate a legitimate admin); deny is unobservable under docker-dev `DisableLogin=true`, as documented. - **R2-10 (T10) ✅ pipeline LIVE (breaker-OPEN state not forced).** The `/hosts` page shows the driver-health flow live (MAIN-galaxy-eq HEALTHY; MAIN-modbus-eq FAULTED "Connection refused"; MAIN-opcua-eq FAULTED "failed to connect to any endpoints"). The failing drivers' subscribe/discover exceptions propagate through the **full Polly resilience pipeline** — central-1 stack traces show `Polly.CircuitBreaker` → `Polly.Retry` → `Polly.Timeout` → `CapabilityInvoker.ExecuteAsync (CapabilityInvoker.cs:84)`, i.e. **the #10 dispatch-gap fix + R2-10's tracker data-source are live in prod.** The explicit `breaker OPENED` log line + the active resilience chip were NOT triggered: R2-09's ConnectionBackoff throttles retries, so consecutive failures don't accumulate fast enough to open the Polly breaker within its window (the documented FOLLOWUP-10 difficulty — the reliable trigger is a rapid idempotent write-to-dead-endpoint). Reader/store/publisher/bridge are unit-verified (Core resilience 128, publisher+guard 6, store 8). - **R2-03 (T12) — unit-verified; live-blocked on this data-less rig.** Every materialised node reads `0x80320000 Bad_WaitingForInitialData` because no live driver data flows (Modbus `10.100.0.35:5020` unreachable in-rig; VTs depend on driver data), so a VT can't be staged Good→Bad; and the Client.CLI `read` surfaces any Bad StatusCode as a `FetchValueAsync` exception rather than printing it. Good→Bad-with-last-value transition is covered by unit suites (Runtime 372, RoslynVTEvaluator 20). Would be live-observable with a reachable driver fixture feeding the VT. **Rig note:** the pass mutated the persisted ConfigDb (added a `drv-modbus-line3-01` Modbus driver on MAIN; removed `capper-03`'s `production-count` virtual tag for the PureRemove test). Reset with `docker compose -f docker-dev/docker-compose.yml down -v`. The remaining integration-suite sweep + genuinely-external gates (R2-06 VPN, R2-08 live LDAP, R2-01 S7 blackhole) still need their infra.