lmxopcua/archreview/plans/STATUS.md

# Arch-Review Remediation — Status & Handoff

**Snapshot:** 2026-07-12 · **Review baseline:** master `9cad9ed0`; **re-review baseline:** master
`f6eaa267` · this file is the single source of truth for remediation progress. Update it as work lands.

> **2026-07-12 UPDATE — everything below the topology table landed:** all remediation branches were
> **merged to master and pushed to `origin/master`** (merge commits `7834b3b7`…`b67bd9e8`; final state
> `f6eaa267`). The topology table is kept for history but is no longer live. A full **re-review** of all
> seven domains ran at `f6eaa267` — see [`../00-OVERALL.md`](../00-OVERALL.md) and the
> "Re-review 2026-07-12" section at the bottom of this file for the new findings (one High regression
> in the S7 fix, one High ResilienceConfig validation gap, and a batch of resilience-seam Mediums).

## Branch topology (HISTORICAL — as of 2026-07-08; since merged + pushed)

| Branch | Head | Contains |
|---|---|---|
| `master` | `9cad9ed0` | untouched baseline |
| `docs/archreview-plans-and-drift-fixes` | `9fadead6` | the 7 domain plans + `00-INDEX.md` + this `STATUS.md` + the two CLAUDE.md doc-drift fixes. **(the plans live ONLY here)** |
| `fix/archreview-crit1-split-brain-resolver` | `eaf78aad` | Critical 1 (03/S1) typed SBR option + guard test + Redundancy.md; **+ #9 hard-kill failover test** (`a25c9ed0`); **+ #11 premise correction** (`eaf78aad`) |
| `fix/archreview-crit2-vt-timeout` | `7fd44f0f` | Critical 2 (02/U2 + 02/U3) + tests |
| `fix/archreview-crit3-s7-reconnect` | `25c0c6f6` | Critical 3 (05/STAB-1) IS7Plc seam + lazy reconnect + unit guards + live bounce test |
| `fix/archreview-crit4-twincat-replay` | `af318fb4` | Critical 4 (05/STAB-2) native-notification replay + probe-recycle + unit guards |
| `fix/archreview-u2-deferred-sink-forwarding-guard` | `a65c2ced` | Guard 03/U2 — reflection-exhaustive deferred-sink forwarding test (+ DeferredServiceLevelPublisher) |
| `fix/archreview-c1-wire-analyzer` | `f0082af5` | Guard 07/C-1 — wire OTOPCUA0001 analyzer tree-wide + triage (surfaced the RESILIENCE-DISPATCH-GAP, task #10) |
| `fix/archreview-s1-s2-ci-coverage` | `10b89830` | Guard 07/S-1,S-2,S-4 — whole-solution CI leg + fail-on-skip gate + CLI sleep deflake |
| `fix/archreview-crit10-wire-capability-invoker` | `62556c24` | **#10 — wire CapabilityInvoker into dispatch via a Core.Abstractions seam** (off `fix/archreview-c1-wire-analyzer`, NOT master — the analyzer is the standing guard + the pragmas live there). Code-complete + unit/analyzer/context-verified; live gate pending. |
| `fix/archreview-crit13-resilience-config-artifact` | `8a8b9ec5` | **#13 — apply per-instance ResilienceConfig from the artifact** (`75403caa`, off the crit10 branch) **+ #10/#13 DI guard** (`8a8b9ec5` — `ResilienceInvokerFactoryRegistrationTests`, from the #12 live gate). Read-path plumbing + invalidate/respawn-on-change; deterministically verified + rig-booted. |
| `fix/docker-dev-serverhistorian-stale-keys` | `7233e2ba` | **Rig fix (off master, independent):** docker-dev compose's retired Wonderware `ServerHistorian` keys → gateway shape, disabled by default. Found during the #12 live-gate rig session; central-1 verified booting clean. |

**Decision (user):** one branch per Critical, each off `master`. **#9 (failover test) lives on the crit1 branch**
(not its own branch off master) because SBR only exists there — it is Critical 1's integration half. The plans are only on the docs branch,
so from a critical branch read them via `git show docs/archreview-plans-and-drift-fixes:archreview/plans/0X-<name>.md`.

## Completed

| Item | Finding | Branch @ commit | Verification |
|---|---|---|---|
| **Critical 1 — split-brain resolver activated** | 03/S1 | `fix/archreview-crit1-split-brain-resolver` `a81dea10` | `BuildClusterOptions` sets `KeepOldestOption{DownIfAlone=true}`; 3 unit guards (Cluster.Tests 29/29); **e2e-verified** the resolved ActorSystem config carries `SplitBrainResolverProvider` + `stable-after=15s` survives the HOCON/Hosting merge (throwaway spike, removed). HOCON cross-ref comment + Redundancy.md corrected. |
| **Critical 2a — real VT script timeout** | 02/U2 | `fix/archreview-crit2-vt-timeout` `7fd44f0f` | Routes through `TimedScriptEvaluator`; infinite-loop test (bounded by WaitAsync) proves no hang. Host.Integration VT 18/18. |
| **Critical 2b — ALC-safe compile cache** | 02/U3 | same `7fd44f0f` | Swapped raw dict → `CompiledScriptCache`; new `IScriptCacheOwner` seam; apply-boundary clear in `VirtualTagHostActor`; wiring guard test. Runtime VT 12/12. |
| **Doc drift — KNOWN LIMITATION 2 stale** | 03/U1 = 06/U-1 | `docs/…` `9fadead6` (CLAUDE.md) | historized-ref feed IS wired; rewrote to "value-capture code-complete, live-verify pending". |
| **Doc drift — ScriptAnalysis policy** | theme #5 | same `9fadead6` (CLAUDE.md) | corrected `FleetAdmin` → `Administrator,Designer` roles. |
| **Critical 3 — S7 lazy reconnect** | 05/STAB-1 | `fix/archreview-crit3-s7-reconnect` `25c0c6f6` | New `IS7Plc`/`IS7PlcFactory` seam + `EnsureConnectedAsync` (dispose-dead-handle → reopen on next data call); connection-fatal faults (socket/`ErrorCode.ConnectionError`) mark the handle dead, data-address errors do NOT; probe loop routes through it as a backstop. 4 unit guards + full S7 suite 234/234; **LIVE-verified** against real python-snap7 — bounced the container (`docker restart`), driver observed the outage and resumed Good reads with no redeploy (`S7_1500ReconnectTests`, double-gated on sim + `S7_RECONNECT_BOUNCE_CMD`). Also closes the S7 TEST-1 gap. |
| **Critical 4 — TwinCAT native-notification replay** | 05/STAB-2 | `fix/archreview-crit4-twincat-replay` `af318fb4` | Native ADS notifications were opaque handles (no replayable intent) → silently orphaned on a client swap. Now `NativeRegistration` (symbol/type/bit/interval/onChange + swappable handle) hangs off `DeviceState`; `EnsureConnectedAsync` split into a gate wrapper + under-gate core that **replays every intent onto a freshly-built client** (register+replay both under `ConnectGate`); probe-failure `RecycleClientAsync` disposes+nulls the wire-dead-but-port-open client so the next tick rebuilds+replays. 4 unit guards + full TwinCAT suite 174/174. **Unit-only — no TwinCAT docker fixture exists** (integration needs a real TC3 XAR), so the fake-client tests are authoritative (no live gate like S7 had). |
| **Guard — reflection-exhaustive deferred-sink forwarding** | 03/U2 | `fix/archreview-u2-deferred-sink-forwarding-guard` `a65c2ced` | New `DeferredSinkForwardingReflectionTests` (Commons.Tests) reflects over **every** method of every forwarding interface `DeferredAddressSpaceSink` implements (`IOpcUaAddressSpaceSink` + `ISurgicalAddressSpaceSink`) — plus the sibling `DeferredServiceLevelPublisher` — invokes each on the wrapper over a `DispatchProxy` recording inner, and asserts the call **reached** the inner. A `DispatchProxy` recorder auto-implements future members (zero maintenance); a guard-of-the-guard asserts the wrapper implements exactly the known forwarding-interface set, so a brand-new capability *interface* trips too. **This closes the F10b/PR#423 trap class and is the prerequisite gate for 03/P1 remove methods + 02/P `WriteValues`.** 3 new tests green (Commons.Tests 57/57); **negative control verified** — stubbing `UpdateFolderDisplayName` to a non-forwarding no-op fails the guard with the exact member-named diagnostic. NO production change. |
| **Guard — wire OTOPCUA0001 analyzer + triage** | 07/C-1 | `fix/archreview-c1-wire-analyzer` `f0082af5` | Injected the analyzer tree-wide via `Directory.Build.props` (`OutputItemType="Analyzer"`), then triaged ~280 surfaced hits: **(1) RESILIENCE-DISPATCH-GAP** — 7 production sites (`DriverInstanceActor` ×6 + `GenericDriverNodeManager`) scoped-`#pragma`'d as a **real tracked gap** (CapabilityInvoker built but never wired into dispatch; **new task #10**); **(2) driver-internal self-calls** — 3 sites (AbCip/S7 own poll/ack) `#pragma`'d as intentional; **(3) 12 wire-level test/CLI projects** → commented `NoWarn`. Full build green, 0 hits; analyzer 31/31; **negative control** — dropping one dispatch-gap pragma re-fires OTOPCUA0001 and fails the build, proving it's live tree-wide. |
| **Guard — CI to all suites + fail-on-skip + deflake** | 07/S-1,S-2,S-4 | `fix/archreview-s1-s2-ci-coverage` `10b89830` | S-1: replaced the 5-project `unit-tests` matrix with ONE whole-solution leg (self-maintaining; now covers Client's 388 + Analyzers' 31 + all drivers). S-2: `scripts/ci/assert-not-all-skipped.sh` fail-on-skip gate (turns "green==all-skipped" red) — strict on the unit leg, report-only on the fixtureless integration leg. S-4: readiness-signal (`SubscribeInvoked`/`SubscribeAlarmsInvoked`) replaces 11 fixed-sleep startup races in the newly-CI'd Client.CLI.Tests. Skip-gate proven locally (real trx OK / all-skip trx exit 1 / report-only / edge cases); Client.CLI.Tests 104/104. **CI job execution itself is verifiable only on push (nothing pushed).** |
| **#10 — wire CapabilityInvoker into dispatch (seam)** | RESILIENCE-DISPATCH-GAP (07/C-1) | `fix/archreview-crit10-wire-capability-invoker` `62556c24` | **Discovered the filed plan was infeasible** — Runtime is deliberately Polly-free (refs `Core.Abstractions`, not `Core`). Fix = a seam mirroring `IDriverFactory`: `IDriverCapabilityInvoker`(+null) + `IDriverCapabilityInvokerFactory`(+null) in Core.Abstractions; `CapabilityInvoker` implements it; `DriverCapabilityInvokerFactory` (Core) builds per-instance over the singleton builder+tracker+`GetTier`; Host `DriverFactoryBootstrap` registers all three; Runtime SCE resolves like `IDriverFactory`; `DriverInstanceActor` wraps its **6** dispatch sites; analyzer taught the interface as a wrapper home; all 6 pragmas removed. Triage corrections: the 7th site (`GenericDriverNodeManager`) is test-only scaffolding (not wired, re-annotated); **tier-defaults only** (ResilienceConfig not in the artifact → sub-gap). Added retry/breaker **logging** to the pipeline builder (the tracker has no reader — Stream E.2/E.3 never shipped — so it ran invisibly). **Verified:** negative-control (unwrap→OTOPCUA0001 build error); recording-invoker routing test; **real-`CapabilityInvoker` actor-context test** (internal `ConfigureAwait(false)` doesn't leak); analyzer + logging tests; clean solution build; Runtime.Tests 358 / Core.Tests 238 / Analyzers 32. **Live behavioral gate pending** (recipe in FOLLOWUP-10). |
| **#9 — hard-kill failover integration test** | 03/S1 | `fix/archreview-crit1-split-brain-resolver` `a25c9ed0` | Critical 1's load-bearing integration half. `TwoNodeClusterHarness.HardKillNodeAAsync` (transport-shutdown crash sim → node B sees A Unreachable, no graceful Leave) + `HardKillFailoverTests` asserting B downs the alone oldest node and takes over as `driver` role-leader. **LIVE-verified in-process** (~35s); **negative control** (force `downing-provider-class=""`) makes it correctly time out. ⚠️ The negative control also surfaced a **Crit-1 premise finding (task #11): SBR was already active on master** (Akka.Cluster.Hosting applies `SplitBrainResolverOption.Default` when the typed option is null → reads the pre-existing akka.conf keep-oldest block), so the cluster was NOT NoDowning before Crit-1 — the typed option is reinforcing, not the activator. |
| **#11 — Critical 1 premise correction** | 03/S1 (from #9) | `fix/archreview-crit1-split-brain-resolver` `eaf78aad` | Docs/comment-only fix (no code revert): corrected the "HOCON inert / cluster runs NoDowning / never fails over" framing in `BuildClusterOptions` XML comment, `akka.conf` split-brain-resolver comment, `docs/Redundancy.md`, and `SplitBrainResolverActivationTests` (summary + `ShouldNotBeNull` message + method rename). Akka.Cluster.Hosting enables an SBR provider **by default** (applies `SplitBrainResolverOption.Default` when the typed option is null → reads the pre-existing akka.conf `keep-oldest` block), so the cluster was never NoDowning and the typed `KeepOldestOption` is reinforcing/explicit-in-code, not the activator. Cluster.Tests 29/29; failover behaviour still guarded by `HardKillFailoverTests`. Architecture.md carried no such framing. |
| **#13 — apply per-instance ResilienceConfig from the artifact** | RESILIENCE-DISPATCH-GAP sub-gap (from #10) | `fix/archreview-crit13-resilience-config-artifact` `75403caa` | The `DriverInstance.ResilienceConfig` column was authored+persisted (AdminUI) and already in the artifact (the composer serializes the whole entity), but the driver-node read path dropped it → tier-defaults only. `DriverInstanceSpec` now carries it; `TryReadSpec` reads it; `Create` takes+layers it (+ logs parse diagnostics, never throws); `SpawnChild` threads it. **Invalidate-on-change:** `Create` calls `builder.Invalidate(instance)` (the cache is options-blind on a hit) so a respawn with changed options rebuilds; `DriverSpawnPlanner` respawns on a ResilienceConfig change (incl. null→json), a pure DriverConfig change stays an in-place delta. **Verified:** override reaches execution (retryCount:0 suppresses tier-A retry; a control retries), invalidate is per-instance, malformed logs+falls-back; planner+artifact tests. Core.Tests 243 (+5), Runtime.Tests 363 (+5). See [`FOLLOWUP-13`](FOLLOWUP-13-resilience-config-artifact.md). |

## Task list (TaskCreate IDs)

| # | Task | Status |
|---|---|---|
| 1 | Critical 1 — split-brain resolver | ✅ completed |
| 2 | Critical 2a — VT timeout | ✅ completed |
| 3 | Critical 2b — CompiledScriptCache | ✅ completed |
| 4 | Critical 3 — S7 reconnect (05/STAB-1) | ✅ completed |
| 5 | Critical 4 — TwinCAT ADS re-registration (05/STAB-2) | ✅ completed |
| 6 | Guard — DeferredAddressSpaceSink reflection forwarding test (03/U2) | ✅ completed — `a65c2ced`; gate for 03/P1 + 02/P now in place |
| 7 | Guard — wire OTOPCUA0001 analyzer + triage (07/C-1) | ✅ completed — `f0082af5`; surfaced task #10 |
| 8 | Guard — CI to all unit suites + fail-on-skip (07/S-1,S-2, pair S-4) | ✅ completed — `10b89830` |
| 9 | Critical 1 follow-up — hard-kill failover integration test (03/S1) | ✅ completed — `a25c9ed0`; surfaced task #11 |
| 10 | **Wire CapabilityInvoker into the dispatch layer** (RESILIENCE-DISPATCH-GAP, from 07/C-1) | ✅ code-complete + unit/analyzer/context-verified — `fix/archreview-crit10-wire-capability-invoker` `62556c24` (seam, NOT direct thread — Runtime is Polly-free). **Live behavioral gate pending** (rig recipe in FOLLOWUP-10). Surfaced a sub-gap: ResilienceConfig not in the deploy artifact → tier-defaults only. |
| 11 | **Review Critical 1 premise** — SBR was already active on master, not NoDowning (from #9) | ✅ completed — `fix/archreview-crit1-split-brain-resolver` `eaf78aad`; docs/comment-only correction (no code revert), Cluster.Tests 29/29. |
| 12 | **#10 live behavioral gate** — rig session (from #10) | ✅ ran 2026-07-08 (PARTIAL) — built the Host image from source (crit13 tip) + stood up the rig; central-1 booted healthy; **prod DI binds the real invoker factory** confirmed live + by a new deterministic guard (`ResilienceInvokerFactoryRegistrationTests` `8a8b9ec5`). Behavioral retry/breaker **log-line not observed** (disciplined stop): Modbus's cheap paths don't throw a wrapped exception (Discover offline, Subscribe lazy, reads non-wrapped) — only an idempotent write-to-dead-endpoint (or S7) does, not worth the multi-dependency chase since the retry behavior is deterministically proven. **Found + FIXED a rig bug:** docker-dev compose's retired Wonderware `ServerHistorian` keys crashed the Host at bring-up (historian-gateway cutover → reads `Endpoint`) → fixed on `fix/docker-dev-serverhistorian-stale-keys` `7233e2ba` (off master; gateway-shape keys, disabled by default; central-1 verified booting clean). See [`FOLLOWUP-10`](FOLLOWUP-10-resilience-dispatch-gap.md). |
| 13 | **Plumb per-instance `ResilienceConfig` through the deploy artifact** (from #10) | ✅ completed — `fix/archreview-crit13-resilience-config-artifact` `75403caa`; read-path plumbing (write side was already complete), invalidate-on-change + respawn-on-change; Core.Tests 243/Runtime.Tests 363. See [`FOLLOWUP-13`](FOLLOWUP-13-resilience-config-artifact.md). |

## Findings surfaced by the guards (NEW — need user decision)

> Each has a dedicated follow-up file with the full design/action:
> [`FOLLOWUP-10-resilience-dispatch-gap.md`](FOLLOWUP-10-resilience-dispatch-gap.md) ·
> [`FOLLOWUP-11-crit1-sbr-premise.md`](FOLLOWUP-11-crit1-sbr-premise.md) ·
> [`FOLLOWUP-13-resilience-config-artifact.md`](FOLLOWUP-13-resilience-config-artifact.md).

- **#10 — RESILIENCE-DISPATCH-GAP (from wiring the analyzer, 07/C-1): ✅ IMPLEMENTED** on
  `fix/archreview-crit10-wire-capability-invoker` `62556c24` — code-complete + unit/analyzer/context-verified;
  **live behavioral gate pending** (recipe in [`FOLLOWUP-10`](FOLLOWUP-10-resilience-dispatch-gap.md)). The filed
  "thread CapabilityInvoker into the actor" was infeasible (Runtime is deliberately Polly-free); shipped a seam
  mirroring `IDriverFactory` instead. Only **6** of the 7 sites were real (the 7th, `GenericDriverNodeManager`, is
  test-only scaffolding). **Two residual sub-findings:** (a) ~~per-instance `ResilienceConfig` isn't in the deploy
  artifact → tier-defaults only~~ **✅ RESOLVED by task #13** (`75403caa`) — the column was already in the artifact;
  the read path now applies it (see [`FOLLOWUP-13`](FOLLOWUP-13-resilience-config-artifact.md)); (b) the
  resilience-status tracker has **no reader** (Admin `/hosts` Stream E.2/E.3 never shipped) — added retry/breaker
  **logging** as the interim observability surface + the live-verify enabler (still open as an Admin-UI follow-on).
- **#11 — Critical 1 premise inaccurate (from the #9 failover negative control): ✅ CORRECTED** on
  `fix/archreview-crit1-split-brain-resolver` `eaf78aad`. SBR was **already active on master**
  (Akka.Cluster.Hosting's `WithClustering` applies `SplitBrainResolverOption.Default` when the typed option is
  null → enables the downing provider that reads the pre-existing akka.conf `keep-oldest` block). So the cluster
  was **not** running NoDowning; hard-crash failover already worked. Docs/comment-only fix (no code revert — the
  typed `KeepOldestOption` is correct belt-and-suspenders): corrected the "HOCON inert / NoDowning → no failover"
  framing in `BuildClusterOptions` XML comment, `akka.conf`, `docs/Redundancy.md`, and the
  `SplitBrainResolverActivationTests` summary/message (+ method rename). Cluster.Tests 29/29. (Architecture.md
  carried no such framing.) See [`FOLLOWUP-11`](FOLLOWUP-11-crit1-sbr-premise.md).

## Suggested next (from 00-INDEX execution order)

**All 4 Criticals + all 3 systemic guards (tasks 6/7/8) + the #9 failover test are DONE.** Remaining:

1. **Resolve the two surfaced findings** — task 10 (wire CapabilityInvoker into dispatch; behavior-changing,
   live-gate) and task 11 (correct Crit-1's rationale/docs).
2. **Failure-visibility Highs** — 01/S-1 (AddressSpaceApplyOutcome failure field), 06/S-1 (Galaxy fail-closed),
   03/S4 (primary-gate default-deny).
3. **P-tier** — 03/P1 surgical adds/removals (now gated by the U2 forwarding guard), 02/P batched `WriteValues`.

## Key resume facts

- **Verification bar met so far:** each fix has a deterministic unit guard AND (for SBR) an end-to-end
  config-resolution check. Follow the same pattern — the review's whole thesis is "unit-green ≠ wired".
- **The forwarding-test guard (task 6) is DONE (`a65c2ced`):** 03/P1 (surgical pure-adds) and 02/P-… batch-write
  each add new sink capability methods that ship inert unless forwarded through `DeferredAddressSpaceSink`
  (F10b/PR#423 precedent). The reflection guard now fails loudly if any such member isn't forwarded — add the
  member to the recorder-covered interface set and it's auto-checked.
- ~~**Nothing is pushed.**~~ **Superseded 2026-07-12:** all branches merged to master and pushed to
  `origin/master`; the plans now live on master too (no more cross-branch `git show` needed).
- ~~**CI changes can't be run-verified without a push**~~ — now pushed; the first real CI-job run should be
  checked on origin (the re-review verified the YAML statically only).
- **Findings #10 and #11 are resolved** (see above); the open decisions now come from the 2026-07-12
  re-review below.

## Re-review 2026-07-12 (master `f6eaa267`) — outcome + new findings

All seven domain reports were re-run and rewritten in place at `f6eaa267` (each carries a
`Prior-finding status (9cad9ed0 → f6eaa267)` table; synthesis in [`../00-OVERALL.md`](../00-OVERALL.md)).
Verdict on the remediation: **all 4 Criticals verifiably closed in code** (Crit-1 by premise correction),
both systemic guards live with negative controls, CI now whole-solution + fail-on-skip
(07 stability 2.5→3.5). New findings needing scheduling, ranked:

| Rank | Finding | Severity | One-line |
|---|---|---|---|
| 1 | 05/STAB-14 | **High (regression)** | S7 ensure-wrapper rethrows connect-timeout OCE → poll loop's bare `catch (OCE) { return; }` silently kills all S7 subscriptions after an unreachable-host (non-refusing) outage; live bounce test blind to it (`S7Driver.cs:488,1000,1383,1404,1206-1208`) |
| 2 | 01/S-6 | **High** | Operator-authorable `timeoutSeconds:0` in ResilienceConfig → Polly ValidationException on every capability call = permanent driver-brick; no range validation in parser or AdminUI form |
| 3 | 05/STAB-15 | **High** | S7 fatal-classification misses framing/desync + cancel-during-I/O (half-read PDU → permanent desync); probe catch never marks handle dead |
| 4 | 01/S-8 = 03/S12 | Medium | Dispatch hardcodes `isIdempotent:true`; `WriteIdempotentAttribute` has zero readers → #13-enabled write retries would retry command writes |
| 5 | 02/S13 | Medium | VT script failure/timeout leaves node stale-Good forever (timeout fix made the failure invisible) |
| 6 | 01/U-6 | Medium | Bulkhead knob parsed/authored/documented but never built into the pipeline — dead knob |
| 7 | 06/S-11 | Medium | Enabled-but-misconfigured ServerHistorian crash-loops Host with raw UriFormatException (only config side fixed in 7233e2ba) |
| 8 | 06/U-7 | Medium | HistorianGateway 0.2.0 proto3-optional DataType change silently alters GatewayTagProvisioner Boolean provisioning; live EnsureTags never run on 0.2.0 |
| 9 | 04/C-7, 04/U-5=03/U10, 01/S-7=03/S13 | Medium/Low | Resilience seam edges: lossy AdminUI form round-trip; status tracker has no reader (operators blind); respawn stale-pipeline cache race |

Carried-open Highs unchanged from round 1 (no branch targeted them): failure-visibility trio
(01/S-1, 06/S-1, 03/S4), AdminUI authz (04/C-1 — stakes rose: resilience config is on the ungated
surface), 03/S2/S3 block-bridging, 03/P1 surgical adds (guard prerequisite in place), driver-fleet batch
(05), repo hygiene (07). Full prioritized list in `00-OVERALL.md`.

**Round-2 plans WRITTEN (2026-07-12):** all 12 action items have design+implementation plans —
`R2-01`…`R2-12` in this directory (+ co-located `.tasks.json` each, for resumable execution). Index,
per-plan task counts, effort, and suggested execution order: [`00-INDEX.md`](00-INDEX.md) §"Round 2
plans". 193 bite-sized TDD tasks total, ~18–24 dev-days end-to-end; R2-01/02/03 (the re-review's
new-and-sharp items) are ~2 days combined.

**Round-2 EXECUTED — R2-12 Repo hygiene (2026-07-13, branch `r2/12-repo-hygiene`):** DONE.
07/U-5 (7 `lib/` AVEVA DLLs removed + gitignored, zero-reference proven), 07/U-4 (root planning
files: `pending.md`/`current.md` untracked+gitignored, `looseends.md` deleted, `stillpending.md` +
`HISTORIAN-GATEWAY-INTEGRATION-ISSUES.md` archived under `docs/plans/`, `docs/Historian.md`
repointed), 07/S-3 (`v2-e2e.yml` no-op nightly deleted), 07/P-5 (redundant `integration` matrix job
removed from `v2-ci.yml`), 07/U-3 (5 alarm operator CLI commands documented — now 13/13), 07/C-5
(`StyleGuide.md` ScadaBridge→OtOpcUa + minimal root `.editorconfig`), 07/C-4 (xunit pin 2.9.2→2.9.3
+ blocker comment; upstream-blocked — no xunit.v3 Akka TestKit exists). 07/U-6 needed no action (the
Wonderware husk dirs are untracked build artifacts, absent in a fresh worktree). Whole-solution build
clean. **OPEN follow-ups:** (a) when a runner-reachable fleet + a real `tests/Server/…E2ETests` land,
add a fresh E2E workflow WITH a zero-match guard (fail if `Category=E2E` matches nothing); (b) S-2
option #2 — a fixture-backed integration leg (opc-plc `services:` block, `MIN_EXECUTED=1`) covering
what the unit leg cannot; (c) migrate the 3 xunit v2 holdouts once Akka ships an xunit.v3 TestKit.

### Round-2 remediation landed

| Plan | Findings | Branch | Commits | Status |
|---|---|---|---|---|
| **R2-06** | 06/S-11 (rank 7), 06/U-7 (rank 8), 06/C-7 | `r2/06-serverhistorian-failfast` (off `1676c8f4`) | `3a049a13`…`4640ecb1` (10 task commits) | **Code-complete, offline-verified.** S-11: fail-fast `ServerHistorianOptionsValidator` (consumer-gated on `ServerHistorian:Enabled` OR `AlarmHistorian:Enabled`, fails empty/non-http(s) Endpoint at startup) + `AddValidatedOptions` wiring + adapter defense-in-depth (`TryCreate`→named `InvalidOperationException`, no more raw `UriFormatException` crash-loop) + `Validate()` warning parity + alarm-only-mode startup warning. U-7: pinned Boolean→Int1 explicit-presence (`HasDataType`) + `HISTGW_BOOL_SANDBOX_TAG` live Boolean EnsureTags leg + retype probe (skip-clean offline) + upsert/retype migration note (`docs/Historian.md`). C-7: keep-populating decision + dead-`Quality` doc comments (0.2.0 C-002). **Live leg operator-gated (T12) — pending a VPN run on a 0.2.0 gateway** (records the observed in-place retype policy back into `docs/Historian.md`; also discharges 06/U-2's full documented run). |
| **R2-05** | 04/C-1 (High) | `r2/05-adminui-authz` (off `1676c8f4`) | `183b72b7`…`6c1f05cb` (5 commits) | **Code-complete, offline-verified.** All 38 routable AdminUI pages carry an explicit named-policy `@attribute` via new `AdminUiPolicies` constants (`Security/Auth/`). New `ConfigEditor` (Administrator+Designer) gates the 20-page config-authoring surface incl. the 8 driver pages that author live `ResilienceConfig` + `/api/script-analysis/*`; new `AuthenticatedRead` on the 16 read pages; `FleetAdmin`/`DriverOperator` unchanged (literals→constants). Anti-drift guard `PageAuthorizationGuardTests` (bans bare `[Authorize]`/`Roles=`/unknown-policy/unclassified) + `AdminUiPoliciesTests` semantics matrix. Corrections folded in: Reservations + ClusterRedundancy read-only (→`AuthenticatedRead`); Home had no attribute (now `AuthenticatedRead`). AdminUI.Tests 517/517, Security.Tests 80/80. **Live `/run` positive pass deferred (T15)** — docker-dev `DisableLogin=true` auto-admin can't observe deny; negative proof is CI-side (both guard tests). |
| **R2-01** | 05/STAB-14 (High regression, rank 1), 05/STAB-15 (rank 3), 05/STAB-8 seam | `r2/01-s7-fault-hardening` (off `1676c8f4`) | `fcdcf0e2`…`bdb7ed41` (13 task commits) | **Code-complete, offline-verified.** All prod changes in `S7Driver.cs`. STAB-14: connect-timeout surfaces as `TimeoutException` (not an escaping OCE) at the `EnsureConnectedAsync`/`InitializeAsync` source + `when(token.IsCancellationRequested)` filters on the read/write ensure-wrappers and all three poll-loop OCE catches — subscription poll loops survive an unreachable-host outage and recover (T2 regression RED→GREEN). STAB-15: `IsS7ConnectionFatal` broadened to the ISO-on-TCP framing surface (TPKT/TPDU/WrongNumberOfBytes/`PlcException{WrongNumberReceivedBytes}`, NOT `InvalidDataException`); mid-PDU cancel marks handle dead; probe classifies+marks-dead under the gate. STAB-8 (S7 part): `EnsureConnectedAsync` documented as the R2-09 connect-throttle seam (note-only). Driver.S7.Tests 246/246. **Live outage gate CLOSED 2026-07-15 (T11) — and it uncovered + fixed a real prod gap (PR #453, master `4dc937fc`).** Running `S7_1500ConnectTimeoutOutageTests` against the live s7_1500 sim (blackholed via `docker pause`, the ACK-less-established shape) exposed that STAB-14 fixed only the *connect* leg: S7.Net's `ReadTimeout`/`WriteTimeout` map to the TcpClient's *synchronous* receive/send timeout, which async reads ignore, so a read on an established-but-frozen peer blocked for minutes — the poll loop wedged (no Bad tick, no reconnect), which offline fakes (open-hang only) couldn't surface. Fix = new `S7OperationDeadline` wall-clock ceiling on every data-plane wire op in `S7PlcAdapter` (→ `TimeoutException`) + `IsS7ConnectionFatal` now classifies `TimeoutException` fatal → mark-dead → reopen (the read-leg sibling of the connect-timeout fix). Driver.S7.Tests 246→260 (6 new: 5 deadline + 1 driver-reaction); **live gate now GREEN** (8s: baseline Good → pause blackhole → Bad tick → unpause → recovered Good). (Pre-existing unrelated CLI docs-scan test fails at base — out of scope.) |
| **R2-03** | 02/S13 (rank 5), 02/S12, 02/P7 | `r2/03-vt-failure-quality` (off `1676c8f4`) | `b9775485`…`6bb91e8d` (12 task commits) | **Code-complete, offline-verified.** S13: VT script failure/timeout degrades the node to `OpcUaQuality.Bad` (carrying last-known value) instead of freezing on last-Good forever — additive `EvaluationResult.Quality`, bridge historizes `0x80020000u` BadInternalError, published once per Good→Bad transition gated on all deps arrived (inputs-ready, no cold-start flash), recovery force-publishes next Good. S12: apply-boundary `ClearCompiledScripts` racing an in-flight eval into ObjectDisposedException retried once in `RoslynVirtualTagEvaluator` (re-fetch→recompile into current generation); stress guard reproduces pre-fix, deterministically green post-fix. P7: apply-boundary clear gated on a changed deployed expression set (`HashSet.SetEquals`, ordinal) — identical redeploy / rename-with-same-expressions no longer forces full VT recompile. Runtime.Tests 372/372, RoslynVirtualTagEvaluatorTests 20/20. **Deferred (T11/T12):** full cross-suite sweep + docker-dev live-`/run` (fail→Bad→recover + P7 no-recompile spot-check). |
| **R2-02** | 01/S-6 (High, rank 2), 01/U-6, 01/S-8=03/S12 (rank 4), 04/C-7, 01/S-7=03/S13 | `r2/02-resilience-config-hardening` (off `1676c8f4`) | `4be13429`…`dd82e740` (17 task commits) | **Code-complete, offline-verified.** **S-6:** parser clamps timeout/retry/breaker overrides (timeout≤0→tier default, breaker==1→2, caps) with diagnostics + belt-and-braces `Build` clamp so pipeline construction can never throw Polly `ValidationException` (new `ResiliencePolicyRanges` in Core.Abstractions); brick-repro + `Build_NeverThrows_ForHostileDirectOptions` + factory-seam wiring proof; AdminUI `Validate()` range warnings. **S-8:** parser forces Write/AlarmAck retry→0 (spec invariant) **and** dispatch flips to `isIdempotent:false` (RED-first on the Runtime wiring test); non-idempotent arm caches the no-retry snapshot once/invoker (01/P-4) + tracker in-flight accounting; `WriteIdempotentAttribute` false claim corrected; retry cells disabled. **U-6:** bulkhead knob deleted end-to-end (migration-free; stored keys ignored) + `Options_properties_are_exactly_the_pipeline_wired_set` inertness guard. **C-7:** `ResilienceFormModel` JsonObject-bag round-trip preserves unknown keys; malformed JSON → `ParseFailed` lock + verbatim `ToJson` + discard button. **S-7:** per-`Create` `Interlocked` options-generation in the pipeline-cache key kills the respawn stale-pipeline race. Core resilience 121/121, AdminUI form 10/10, Analyzers 32/32, Runtime 364/364; 0 production OTOPCUA0001. **Deferred (T15/T18):** docker-dev `:9200` `/run` (warning banner, input lockout, unknown-key survival) + whole-solution sweep. |
| **R2-10** | 03/U10=04/U-5 (Medium/Low, action #10) | `r2/10-resilience-observability` (off `46fedda3`) | `205d3980`…`<docs>` (11 task commits) | **Code-complete, offline-verified.** The resilience status tracker now has a production reader, mirroring the driver-health flow. Core truthfulness: `RecordBreakerClose` + derived `IsBreakerOpen` (breaker-open state derivable), `OnClosed` records the close, and `RecordSuccess` wired onto the invoker success paths (`ConsecutiveFailures` is a true consecutive counter, not stuck-forever). Transport: `DriverResilienceStatusChanged` Commons contract → periodic 5 s full-snapshot publisher `DriverResilienceStatusPublisherService` (driver nodes, registered in `AddOtOpcUaDriverFactories` with wiring guard `ResilienceStatusReaderRegistrationTests` — closes the built-but-never-wired class) → `driver-resilience-status` DPS topic → per-admin-node `DriverResilienceStatusBridge` → `IDriverResilienceStatusStore` → resilience chips (breaker-OPEN / consecutive-failures / in-flight / staleness) on `DriverStatusPanel.razor`, read in-process (self-HubConnection ban honored). Core.Tests resilience 31/31 (incl. deterministic breaker-close via a controllable TimeProvider), Host.IntegrationTests publisher+guard 6/6, AdminUI.Tests store 5/5; full-solution build clean. **Deferred (T10):** docker-dev live-`/run` — S7 dead-endpoint breaker-open observed on BOTH centrals + recovery-clear + staleness-dim (also the first live sighting of the #10 `circuit-breaker OPENED` log line, closing FOLLOWUP-10's behavioral gate). |

| **R2-08** | 03/S2 (High, action #8), 03/S3 (High) | `r2/08-ldap-historyread-async` (off `1a698cbb`) | `ce6d9e21`…`<docs>` (14 task commits) | **Code-complete, offline-verified.** **S2 (LDAP async/timeout):** the SDK raises the impersonation callback synchronously under a `SessionManager`-wide `m_eventLock` (verified vs. 1.5.378 — a hung bind serializes ALL activations), so the fix is entirely at the `LdapOpcUaUserAuthenticator` boundary: five new `Security:Ldap` keys (`ConnectionTimeoutMs`=10000 projected into the shared lib, `AuthTimeoutMs`=15000 hard whole-flow `WaitAsync` bound → fail-closed "timed out", `MaxConcurrentAuthentications`=8 `SemaphoreSlim` with release-on-core-completion so orphaned binds still count, `OutageFailureThreshold`=3/`OutageCooldownSeconds`=15 directory-outage circuit → fast-deny "unavailable", half-open probe) + `LdapAuthResult.IsSystemFailure` seam (mapped from `ServiceAccountBindFailed`) distinguishing outage from credential-deny + `LdapOptionsValidator` rules + `HandleImpersonation` contract doc + `docs/security.md`. Stall-repro RED→GREEN; offline outage-sim through the real `HandleImpersonation` against an unroutable host. **S3 (HistoryRead fan-out):** all four HistoryRead arms async-ified (`ServeNodeAsync`/`ServeRawPagedAsync`/`ServeEventsAsync`, OCE→`BadTimeout`) then bounded per-node fan-out (`RunBounded` semaphore + single arm-boundary bridge) + per-request deadline CTS (`HistoryReadDeadline`=60s) + process-wide lazy batch limiter (`MaxConcurrentHistoryReadBatches`=16 → flood degrades to `BadTooManyOperations`); 16×4=64 in-flight cap; three new `ServerHistorian` keys + node-manager props + hosted-service wiring + `docs/Historian.md`/CLAUDE.md tables. Sequential-serving RED→GREEN. Host.IntegrationTests (LDAP resilience) 13/13, Security.Tests 11/11, Runtime.Tests options 11/11, OpcUaServer.Tests HistoryRead 30/30. **S2 GLAuth outage/recovery LIVE gate CLOSED 2026-07-15 (PR #455, master `5f12636c`)** — new env-gated `LdapGlauthOutageLiveTests` drives the circuit through the production path (`OtOpcUaLdapAuthService`+`LdapOpcUaUserAuthenticator`) against a REAL GLAuth on `10.100.0.35:3894` (paused/unpaused via `docker pause`); GREEN (12s): healthy alice/bob binds w/ mapped roles → pause → 3 bounded failures → circuit opens → sub-second "unavailable" → unpause → half-open close → recover. Also the FIRST end-to-end proof the PR #451 GLAuth swap binds correctly. Skips clean offline (1 skipped, 8ms). **Still deferred (live):** S3 `Category=LiveIntegration` gateway soak (needs VPN to a real HistorianGateway). |
| **R2-11** | 01/C-1, 01/P-1 + 05/CONV-2, UNDER-1, UNDER-6 | `r2/11-tagconfig-consolidation` (off `46fedda3`) | first `2e14fe1f` … (24 task commits) | **Code-complete, offline-verified.** **Part 1 (C-1/P-1):** single `TagConfigIntent.Parse` + `DeviceConfigIntent` in Commons replace the four byte-parity TagConfig parse copies (composer/artifact/walker/DraftValidator) — composer now parses once per tag (P-1); a permanent `Runtime.Tests/TagConfigCorpusParityTests` (30-blob corpus) guards compose≡decode; `grep "MUST parse identically" src/` = 0. **Part 2 (CONV-2):** shared strict-capable `TagConfigJson` readers in Core.Abstractions replace the six copy-pasted `ReadEnum`s; all six driver parsers gain `Inspect()` warnings + honour the `writable` key (FOCAS **forced read-only**, 05/UNDER-1); FOCAS equipment tags run the capability-matrix pre-flight (05/UNDER-6); Modbus probe parses the factory DTO shape (`timeoutMs` no longer dropped). New `EquipmentTagConfigInspector` in ControlPlane + `AdminOperationsActor` deploy-gate wiring with `Deployment:TagConfigValidationMode` (**Warn default | Error opt-in**), proven through the actor (F10b consuming test). AdminUI: `writable` checkbox in the six typed editors. **Phase-C follow-up (deferred, recorded here):** flip the runtime parsers to strict (typo'd enum ⇒ `BadNodeIdUnknown` instead of wrong-width `Good`) once fleets have run `Error` mode clean — behavior change deliberately not shipped in this plan. **Deferred (T22/T24):** docker-dev `/run` of one typed editor's Writable checkbox + whole-solution serial sweep. |

| **R2-07 (Phases 1+2+3)** | 03/P1 (action #7, surgical pure-adds/removes/mixed) | `r2/07-surgical-pure-adds` (off `1a698cbb`) | `bb226f45`…`e2b47638` (T1–T14) | **Phases 1+2 code-complete, offline-verified.** New pure `AddressSpaceChangeClassifier` (Empty/AttributeOnly/PureAdd/PureRemove/AddRemoveMix/Rebuild; first-match-wins, default-closed Rebuild; the F10b tag/vtag surgical-eligibility predicates moved here as `internal static`). `AddressSpaceApplier.Apply` routes by kind. **Phase 1 (PureAdd/AttributeOnly):** skip `RebuildAddressSpace` (the idempotent Materialise passes create exactly the added nodes; existing subscriptions survive); adds announced via `ComputeAddAnnouncements`+`AnnounceAddedNodes` (dedup affected-parent → existing guard-covered `RaiseNodesAddedModelChange`), fired by `OpcUaPublishActor` after the 4 passes gated on `!outcome.RebuildCalled`; `MaterialiseAlarmCondition` made idempotent (skip-if-present-and-same-kind keeps the condition NodeState alive). **NO new sink members in Phase 1.** **Phase 2 (PureRemove):** 3 new `ISurgicalAddressSpaceSink` members (`RemoveVariableNode`/`RemoveAlarmConditionNode`/`RemoveEquipmentSubtree`) — guard-first with the **observed negative control** (`DeferredSinkForwardingReflectionTests` fails member-named until `DeferredAddressSpaceSink` forwards) + `SdkAddressSpaceSink` forwards + `OtOpcUaNodeManager` in-place teardown (scoped map cleanup, notifier demotion via `RemoveRootNotifier`, `BuildNodesRemovedModelChange` verb=NodeDeleted reported outside `Lock`). Applier `PureRemove` arm: terminal Bad / `RemovedConditionState` writes (closing the removed-alarm-tag today-gap) → subsumption partition (removed-equipment children folded into the subtree remove) → in-place removes → any false/throw ⇒ full-rebuild ratchet; non-surgical sink ⇒ rebuild. **Phase 3 (AddRemoveMix):** removes-then-adds within one apply — the remove pass runs in `Apply` (no rebuild), then the publish actor's Materialise passes create the adds + `AnnounceAddedNodes` announces them; an id reused across the remove+add sets is remove-then-recreated as a fresh NodeState (correct — a different tag). **Guard stays green (Commons.Tests 17/17), OpcUaServer.Tests 336/336, Runtime.Tests 398/398.** **03/P1 marked REMEDIATED in `archreview/03-server-runtime.md`.** **Deferred-live (T5, T6, T12, T14):** `SubscriptionSurvivalTests` add + remove + mixed cases (authored + compile-green; NOT run — `*.IntegrationTests` memory constraint) and the docker-dev live-`/run` add/remove/mixed gates (`kind=PureAdd/PureRemove/AddRemoveMix, rebuild=False`; survivor subscription uninterrupted; removed node → `BadNodeIdUnknown`) — MUST run in the serial heavy pass (F10b inertness risk). |

**Report anchor drifts to fold into the next report refresh** (from the R2-02 verification pass): (1)
`01-core-composition.md` U-6 anchor `DriverResilienceOptions.cs:23-25` → the two bulkhead properties sat at
`:23` and `:29` (now deleted); (2) `01-core-composition.md` U-6 anchor
`DriverResiliencePipelineBuilder.cs:99-177` → `Build` was `:99-176`. All other cited lines were exact at
`f6eaa267`.

## ✅ Round-2 COMPLETE — all 12 plans merged (2026-07-13)

All 12 round-2 plans were executed (isolated worktree per plan, per-task TDD, pathspec-scoped commits) and
**merged to `master` + pushed to `origin/master`** in 3 dependency-ordered waves, each plan landing as its own
Gitea PR (closed-as-merged) behind a `dotnet build` barrier. **Final state: master `31c98200`.**

| Wave | Plans (PR#) |
|---|---|
| A (no cross-plan deps) | R2-12 #426 · R2-06 #427 · R2-05 #428 · R2-01 #429 · R2-03 #430 · R2-02 #431 |
| B (parse-site barrier) | R2-10 #432 · R2-04 #433 · R2-11 #434 |
| C (largest, depend on earlier) | R2-08 #435 · R2-07 #436 · R2-09 #437 |

Cross-plan mutexes honored: R2-01 landed before R2-09 (PollGroupEngine v2 / S7 fork retirement); R2-11's
`TagConfigIntent.Parse` consolidation landed before R2-09's driver parse edits; the three plans that all edit
`AddressSpaceApplier.cs` + `DriverHostActor.cs` + `OtOpcUaNodeManager.cs` (R2-04/R2-07/R2-08/R2-10/R2-11) were
merged sequentially with the shared-file auto-merges verified by targeted unit runs each time.

### Consolidated unit sweep on merged `master` `31c98200` (integration suites excluded — ~16 GB/run leak)
Solution `dotnet build` clean (0 errors). All R2-relevant unit suites GREEN, incl.: OpcUaServer 341, Runtime
398, Commons(+forwarding guard) 122, Core 276, Core.Abstractions 128, Core.Scripting 137, Core.VirtualTags 59,
Cluster 29, Security 84, ControlPlane 78, Analyzers 32, AdminUI 537, Client.CLI 104, Client.Shared 182,
Client.UI 126, Core.AlarmHistorian 29, Core.ScriptedAlarms 71, and all drivers (S7 254, Modbus 312+161, AbCip
336, AbLegacy 209, TwinCAT 189, FOCAS 265, Galaxy 304, Galaxy.Browser 21, Historian.Gateway 102, OpcUaClient 131).

**3 failures, all confirmed PRE-EXISTING / environmental (git diff `1676c8f4..HEAD` empty on their paths — R2 never touched them):**
1. `Core.Abstractions.Tests.InterfaceIndependenceTests.AllPublicTypes_LiveInRootNamespace` — flags the pre-R2
   `Core.Abstractions.Historian.*` continuous-historization types living in a sub-namespace. **Pre-existing convention debt — needs-owner-decision** (move the types to root, or relax the test to allow the intentional Historian grouping).
2. `Driver.S7.Cli.Tests.SubscribeCommandConsoleHandlerCommentTests...` — pre-existing docs-scan test looking for a `"CliFx console"` phrase removed by `9cad9ed0`. **Follow-up: repoint or delete the phrase-scan.**
3. `Driver.OpcUaClient.Browser.Tests` (3) — `Connection refused`; need the live opc-plc endpoint (`10.100.0.35:50000`), unreachable offline. **Environmental — pass with the fixture up.**

### Deferred heavy / live-`/run` pass (serialized — single-tenant rig; controller-owned)
Every plan's integration-suite and live-`/run` gates were deferred (not skipped) so nothing OOMs the box and the
shared docker-dev rig / VPN infra isn't contended. To run one-at-a-time:

- **Full integration-suite sweep** (10 `*.IntegrationTests` projects, one at a time): discharges R2-02 T18, R2-03 T11, R2-07 T5/T6, R2-08 T14, R2-11 T24. **RAN 2026-07-14 — see [`INTEGRATION-SWEEP-STATUS.md`](INTEGRATION-SWEEP-STATUS.md).** Outcome: OpcUaClient(+Browser)/S7/Modbus-standard green, Host 105/117, TwinCAT clean-skip; the rest fail on **pre-existing test/fixture bit-rot** (AbCip static-init bug, Modbus pymodbus-4.x sim drift, FOCAS 127.0.0.1-mock topology, AbLegacy no fixture, OpcUaServer cert-store config, Host openldap:2.6 image gone). **No OtOpcUa regression.** 7 follow-ups in that file.
- **docker-dev `:9200` live-`/run`** (rebuild BOTH centrals): R2-02 T15 (ResilienceConfig warning banner / input lockout / unknown-key survival), R2-03 T12 (VT fail→Bad→recover + P7 no-recompile), R2-10 T10 (breaker-open on both centrals + recovery + FOLLOWUP-10 log line), R2-11 T22 (typed-editor Writable checkbox), **R2-07 T12 (surgical add/remove/mixed — F10b inertness risk, MUST run)**, R2-05 T15 (over-gating regression only — deny unobservable under `DisableLogin`).
- **2-node redundancy rig:** ~~R2-04 T13/T15~~ **BOTH ✅ PASSED 2026-07-14** — T15 live docker-dev gate + T13 in-process delivery-path test (see below).
- **Genuinely external / needs-user:** R2-06 T12 (VPN to a 0.2.0 HistorianGateway for the live EnsureTags retype probe), R2-08 S3 `Category=LiveIntegration` gateway soak (VPN to a real HistorianGateway). ~~R2-01 T11~~ **CLOSED 2026-07-15** (S7 blackhole via `docker pause` on the `10.100.0.35` s7_1500 fixture; found + fixed the read-leg wall-clock gap, PR #453). ~~R2-08 GLAuth outage cycle~~ **CLOSED 2026-07-15** (self-serviceable on `10.100.0.35` — `LdapGlauthOutageLiveTests` vs. the harness GLAuth paused via `docker pause`, PR #455).

### Deferred behavior-change follow-up (deliberately not shipped)
**R2-11 Phase C** — flip the runtime driver parsers to strict (typo'd enum ⇒ `BadNodeIdUnknown` instead of a
wrong-width `Good`). Land only after fleets have run `Deployment:TagConfigValidationMode=Error` clean.

## docker-dev live-`/run` pass (2026-07-13) — CONCLUDED; 5 gates LIVE-confirmed on the built-from-source `6aeb53a9` rig

**Bottom line:** R2-02, R2-11, R2-07 (both surgical phases), R2-04 (achievable failover), and R2-10 (resilience pipeline in
dispatch) verified live on the docker-dev central cluster. R2-05 partial-by-design; R2-03 unit-verified (rig has no driver
data). R2-01's S7 blackhole gate is **CLOSED (2026-07-15)** — run via `docker pause` on the `10.100.0.35` s7_1500 fixture;
it found + fixed the read-leg wall-clock gap (PR #453). R2-08's GLAuth outage-cycle leg is **CLOSED (2026-07-15, PR #455)** —
`LdapGlauthOutageLiveTests` vs. the harness GLAuth on `10.100.0.35`. Still open: R2-06 needs VPN to a 0.2.0 gateway; R2-08's
S3 `Category=LiveIntegration` gateway soak needs VPN to a real HistorianGateway. Those are the only Round-2 items not yet exercised.

Brought up the 2-node central cluster (`central-1`+`central-2`, SQL, traefik) from source at merged master `6aeb53a9`
(single-tenant rig; drove the `:9200` AdminUI via browser + Client.CLI on `:4840`/`:4841`).

- **R2-02 (S-6 High) ✅ LIVE** — on the Modbus driver's *Resilience overrides*, entering `timeoutSeconds:0` renders the
  exact range-validation banner: *"Out-of-range values will be clamped to a safe value on deploy: Read timeout must be
  between 1 and 86400 seconds (got 0)."* **S-8 ✅** — the Write and AlarmAcknowledge RETRIES cells are disabled ("never").
- **R2-11 (T22) ✅ LIVE** — the typed Modbus tag editor shows the **Writable** toggle; saving persists `writable: yes`.
- **R2-07 (P1 — F10b MUST) ✅ LIVE** — central-1 boot apply logged `AddressSpaceApplier: applied plan
  (kind=PureAdd, added=91, removed=0, changed=0, rebuild=False)`; deleting a UNS virtual-tag + redeploy logged
  `(kind=PureRemove, added=0, removed=1, changed=0, rebuild=False)` and the node vanished from the address space
  (`capper-03` lost its `production-count` child). **Surgical add AND remove both execute in prod with no full rebuild —
  the remove path's 3 new sink members are wired, not inert.**
- **R2-04 (S4 MUST) ✅ LIVE (achievable guarantee)** — **younger-crash** (kill `central-2`): `SBR took decision
  DownUnreachable and is downing [central-2]`, `Member removed`, `central-1` stays Up, AdminUI serving, ServiceLevel
  **250** (sole primary, no split-brain). **Oldest-crash** (kill `central-1`) reproduced the **documented keep-oldest
  asymmetry** live: the survivor took `SBR … DownReachable … including myself` and CoordinatedShutdown (ExitCode 3) —
  i.e. crashing the *oldest* node does **not** fail over. This is the pre-registered SBR topology gap (a deferred USER
  strategy decision), **not** an R2 regression; R2-04's `PrimaryGatePolicy` write-gate change is orthogonal + unit-verified.
- **R2-05 (T15) ✅ (partial, as expected)** — every config/read page renders under the all-roles `multi-role-test` session
  (named-policy gating didn't over-gate a legitimate admin); deny is unobservable under docker-dev `DisableLogin=true`, as documented.
- **R2-10 (T10) ✅ pipeline LIVE (breaker-OPEN state not forced).** The `/hosts` page shows the driver-health flow live
  (MAIN-galaxy-eq HEALTHY; MAIN-modbus-eq FAULTED "Connection refused"; MAIN-opcua-eq FAULTED "failed to connect to any
  endpoints"). The failing drivers' subscribe/discover exceptions propagate through the **full Polly resilience pipeline** —
  central-1 stack traces show `Polly.CircuitBreaker` → `Polly.Retry` → `Polly.Timeout` → `CapabilityInvoker.ExecuteAsync
  (CapabilityInvoker.cs:84)`, i.e. **the #10 dispatch-gap fix + R2-10's tracker data-source are live in prod.** The explicit
  `breaker OPENED` log line + the active resilience chip were NOT triggered: R2-09's ConnectionBackoff throttles retries, so
  consecutive failures don't accumulate fast enough to open the Polly breaker within its window (the documented FOLLOWUP-10
  difficulty — the reliable trigger is a rapid idempotent write-to-dead-endpoint). Reader/store/publisher/bridge are
  unit-verified (Core resilience 128, publisher+guard 6, store 8).
- **R2-03 (T12) — unit-verified; live-blocked on this data-less rig.** Every materialised node reads
  `0x80320000 Bad_WaitingForInitialData` because no live driver data flows (Modbus `10.100.0.35:5020` unreachable in-rig;
  VTs depend on driver data), so a VT can't be staged Good→Bad; and the Client.CLI `read` surfaces any Bad StatusCode as a
  `FetchValueAsync` exception rather than printing it. Good→Bad-with-last-value transition is covered by unit suites
  (Runtime 372, RoslynVTEvaluator 20). Would be live-observable with a reachable driver fixture feeding the VT.

**Rig note:** the pass mutated the persisted ConfigDb (added a `drv-modbus-line3-01` Modbus driver on MAIN; removed
`capper-03`'s `production-count` virtual tag for the PureRemove test). Reset with
`docker compose -f docker-dev/docker-compose.yml down -v`. R2-01's S7 blackhole gate is now CLOSED (2026-07-15, PR #453);
the remaining genuinely-external gates (R2-06 VPN, R2-08 S3 `LiveIntegration` gateway soak) still need VPN to a real gateway.

---

## R2-04 T15 — 2-node primary-gate LIVE GATE ✅ PASSED (2026-07-14)

Ran the S4 primary-gate live recipe on the docker-dev 2-node central cluster **rebuilt from master `365a90e6`**
(`up -d --build central-1 central-2`). Brought up the **Modbus sim** (`otopcua-pymodbus-standard` on
`10.100.0.35:5020`) so `MAIN`'s Modbus equipment tags read **Good** (a data-less rig materialises every node
`Bad_WaitingForInitialData`, and the CLI write reads-first — so a live driver is required to have a writable-Good
node to drive the gate). Made `EQ-55297329838d/ModbusHR200` writable (`TagConfig.writable=true` via SQL +
`POST :9200/api/deployments`). Data-plane writes bound `multi-role`/`password` (carries the `WriteOperate` LDAP role).

**All core assertions PASS:**
- **Step 1 — ServiceLevel split:** `Server_ServiceLevel` (`i=2267`) = **250 on `:4840` (primary) / 240 on `:4841`
  (secondary)** — differentiated, no split-brain. (Plan said 240/100; the deployment's healthy values are 250/240 —
  the invariant "primary > secondary" holds.)
- **Step 2 — boot-window default-DENY (the NEW S4 behavior):** restarting **both** centrals (to widen the
  role-null-with-≥2-members window — a single-node restart's window is sub-second because cluster-join and the first
  redundancy snapshot land near-simultaneously) caught, ~5 s into boot:
  `WRN Operator write to EQ-55297329838d/ModbusHR200 rejected: not primary (role unknown)`. Meter
  `otopcua_redundancy_primary_gate_denied_total{reason="role-unknown",site="write"} = 1`.
- **Step 3 — steady-state secondary DENY + #5 revert:** write `9999` to `:4841` → CLI reports optimistic
  `Write successful` (synchronous, by design) but read-back = **1234 (the primary's value, reverted — NOT 9999)**;
  central-2 logs `rejected: not primary`; meter `reason="secondary",site="write" = 1`.
- **Step 3b — primary write reaches the device:** write `1234` to `:4840` → `Write successful` and read-back on
  `:4841` = `1234` (the Modbus driver wrote through).
- **T4 sanity — clean apply:** both centrals logged `AddressSpaceApplier: applied plan (kind=PureAdd … failed=0)` /
  `materialised (… failed=0)`; **no `[ERR]` / `apply.failed` / degraded-apply line**; the `apply_failed` meter is
  absent (never incremented).
- **Step 4 — `/alerts`:** endpoint 200, **0 rows** — no active alarm on the data-less rig, so trivially no
  dual-primary double-emission (the double-row regression is only observable with a firing alarm, which needs live
  driver data this rig lacks; the alerts-emit gate shares the same `PrimaryGatePolicy` and is unit-covered).

**Net:** the R2-04 `PrimaryGatePolicy` membership-aware **default-DENY-on-unknown** is confirmed live end-to-end —
both denial reasons (`role-unknown`, `secondary`) fire the log + the new meter, the #5 optimistic-write revert
corrects the node, and the primary path is unaffected. The behavior-affecting S4 slice is trusted.

**R2-04 T13 — in-process delivery-path test ✅ PASSED (2026-07-14).**
`dotnet test …Host.IntegrationTests --filter FullyQualifiedName~PrimaryGateFailoverTests` → **1/1, 3 s**. Ran
**without SQL** (`OTOPCUA_HARNESS_USE_SQL` unset) — the gate test needs only the in-process 2-node Akka cluster +
the DPS-delivered redundancy snapshot, not a ConfigDb deploy. The `TwoNodeClusterHarness` formed the cluster and
the **real DistributedPubSub-delivered** snapshot promoted exactly one node to Primary (Secondary's
`DriverHostActor.RouteNodeWrite` → `not primary`; Primary passed the gate, rejecting only with a mapping reason).
Its built-in negative control (a broken redundancy-topic subscribe leaves both stuck `role unknown` → 45 s timeout)
means the PASS proves genuine snapshot delivery drives the gate — the half the pure-TestKit unit guards can't cover.

**Rig mutations (this pass):** `ModbusHR200.TagConfig.writable=true`; Modbus sim brought up on the shared docker host
(`10.100.0.35`). Reset the central rig with `docker compose -f docker-dev/docker-compose.yml down -v`; the sim with
`ssh dohertj2@10.100.0.35 'cd /opt/otopcua-modbus && docker compose --profile standard down'`.

---

## Post-round-2 loose-end closure (2026-07-13)

- **Pre-existing red unit test closed** (theme-#4 "CI to all suites" hygiene; predates round 2, not an R2 plan).
  `InterfaceIndependenceTests.AllPublicTypes_LiveInRootNamespace` was red on master because four
  continuous-historization types (`IHistorizationOutbox`, `HistorizationOutboxEntry`, `IHistorianValueWriter` /
  `HistorizationValue`, `HistorizationCommitMode`) had drifted into a `ZB.MOM.WW.OtOpcUa.Core.Abstractions.Historian`
  sub-namespace while the other six files in the same folder use the root namespace — violating decision #59's
  flat-public-surface rule. Fixed by moving all four to the root namespace and dropping/retargeting the redundant
  `using …Core.Abstractions.Historian;` in ~11 consumers (no behavioral change). **Gitea PR #438 → master `50e04707`.**
  Verified: Core.Abstractions `129/129` (was 128/129), Runtime historian+DI `72/72`, Gateway driver unit `102/102`,
  full solution build `0 errors`.

---

## R2-06 live gate (T12) — 5/6 green on the VPN + a real provisioning bug fixed (2026-07-13)

The R2-06 `Category=LiveIntegration` gate ran against a **real 0.2.0 `ZB.MOM.WW.HistorianGateway`** deployed
locally in Docker (`~/Desktop/HistorianGateway/docker`, republished at 0.2.0) pointed at the **real AVEVA
Historian** on `wonder-sql-vd03.zmr.zimmer.com:32565` + Galaxy SQL on `wonder-app-vd03` + the `Runtime` SQL DB,
over the corporate VPN. OtOpcUa's `GatewayLiveFixture` connected h2c to the local gateway
(`http://localhost:5221`, `HISTGW_GATEWAY_ALLOW_UNTRUSTED=true` since h2c ⇒ no cert validation; gateway
`ApiKeys:Mode=Disabled`, placeholder key). See `[[reference_histgw_local_deploy]]` for the full recipe.

**Three deployment fixes (sibling `HistorianGateway/docker/`, local-only):**
1. **DNS** — base compose set the *short* `Historian__Host: wonder-sql-vd03`, which doesn't resolve in the
   container; overrode to the FQDN.
2. **Kerberos/NTLM libs** — the historian **v6 session path** (reads + `EnsureTags`) + the SQL path authenticate
   the Windows account `NAM\dohertj2` via .NET Negotiate/GSSAPI, which `aspnet:10.0` lacks → every read/tag op
   failed `InvalidOperationException` (`Cannot load library libgssapi_krb5.so.2`). Added `libgssapi-krb5-2
   gss-ntlmssp` to `docker/Dockerfile`. (The **v8 event path**, `SendEvent`, uses ECDH — worked without it.)
3. **RuntimeDb SQL auth** — the `Runtime` DB SQL login had to be a **SQL login** (not the Windows account, which
   a Linux container can't SQL-auth). The user supplied `wonderapp` (SQL_LOGIN with INSERT on `Runtime.dbo.History`
   + SELECT on `dbo.Events`); wiring it made `WriteLiveValues` + SQL `ReadEvents` work.

**🐞 Real OtOpcUa bug found + fixed (PR #439, master `666908d7`):** `GatewayTagProvisioner` built
`HistorianTagDefinition` **without `StorageRateMs`**, so the proto `uint32` defaulted to `0`; the gateway maps it
straight to the AVEVA SDK's `storageRateMs`, which **throws `ArgumentOutOfRangeException` ("Storage rate must be
> 0 ms")** on `0`. So **every** deploy-time historized-tag auto-provision (`IHistorianProvisioning.EnsureTagsAsync`)
was silently failing at the gateway (`failed=N`). Fixed by stamping `DefaultStorageRateMs=1000` (gateway
convention) + a regression unit test + `StorageRateMs` on the live-suite inline defs. Gateway driver unit
`103/103`.

**Result — 5/6 PASS** (was 3/6 before the fixes):
- ✅ `Galaxy_tag_read_round_trip` — historian gRPC read path.
- ✅ `Write_then_read_on_sandbox_tag` — `EnsureTags(Float)` + `WriteLiveValues` (SQL) + read-back.
- ✅ `Alarm_SendEvent_is_acked` + ✅ `Alarm_SendEvent_then_ReadEvents` — alarm `SendEvent` + SQL `ReadEvents`.
- ✅ `EnsureTags_type_change_outcome_is_observable` — now observes a **real** retype outcome (not a storage-rate error).
- ⚠️ `EnsureTags_boolean_sandbox_provisions_as_Int1` (as originally written) — **Int1 (Boolean) tag creation is not
  supported by the gateway's histsdk on this historian**: `EnsureTags(Int1)` returns `ProtocolEvidenceMissingException`
  (the vendored client's "no capture evidence for this type" guard → gRPC `Unimplemented`), both fresh and as a retype
  target. A **type sweep** confirmed it is Int1-specific: **Int2/UInt2/Int4/UInt4/Float/Double all provision
  `success=True`; only Int1 fails.** Root cause: `histsdk` `HistorianTagWriteProtocol.GetAnalogDataTypeCode` has no
  captured CTagMetadata wire code for Int1 — a **gateway/histsdk type-support gap**, not an OtOpcUa defect
  (**filed as [historiangw#11](https://gitea.dohertylan.com/dohertj2/historiangw/issues/11)**, which routed the
  decision back as **[issue #441](https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/441)**).

  **✅ RESOLVED (issue #441, 2026-07-14):** `HistorianTypeMapper` now maps **`Boolean → Int2`** (0/1 integer) instead
  of the degenerate `Int1`. Int2 is a proven-creatable analog type on this historian, and the value-write path already
  sends every value as a `double` on `WriteLiveValues` regardless of tag type, so Boolean `0.0/1.0` round-trips
  cleanly. The live gate test is renamed `EnsureTags_boolean_sandbox_provisions_as_Int2` and now passes → **6/6**. Unit
  pins updated (`HistorianTypeMapperTests`, `GatewayTagProvisionerTests.Boolean_definition_carries_explicit_Int2_presence`);
  gateway driver unit `103/103`.

**✅ U-7 retype policy — ANSWERED (via `grpcurl` against the real historian):**
- `EnsureTags` is a **real create-or-update upsert** — re-`EnsureTags` of an existing tag at the same type → `success`.
- An **in-place retype between SUPPORTED types is ACCEPTED** — `Float→Double` on an existing tag returns `success`
  (of the three 06/U-7 hypotheses: **accepted-retyped**, not rejected).
- **The specific `Int1` case can't retype** — Int1 isn't a creatable type here. Rather than accept a broken Boolean
  path, OtOpcUa **remapped `Boolean → Int2`** (issue #441, above), which retypes cleanly from a pre-existing Float
  Boolean tag (both are supported analog types) and never hits the Int1 gap.

The R2-06 **code** deliverables (S-11 fail-fast + `ServerHistorianOptionsValidator`; Boolean provisioning mapping +
upsert; C-7 doc) were already unit-covered + merged; this live gate additionally **found + fixed the StorageRateMs
provisioning bug**, **answered the U-7 retype policy**, and surfaced the Int1 histsdk gap that drove the
**Boolean→Int2 remap (issue #441)** — closing the last red test to reach **6/6**.