e4c7c8ab6d
R2-10: driver failures propagate through the full Polly pipeline (CircuitBreaker-> Retry->Timeout->CapabilityInvoker.cs:84) live — the #10 dispatch-gap fix + tracker source confirmed in prod; breaker-OPEN not forced (backoff-throttled, per FOLLOWUP-10). R2-03: rig has no driver data (all nodes Bad_WaitingForInitialData), so VT Good->Bad can't be staged; unit-verified. 5 gates live-confirmed total; remaining items need external infra (integration fixtures, VPN gateway, live LDAP, S7 blackhole).
280 lines
48 KiB
Markdown
280 lines
48 KiB
Markdown
# Arch-Review Remediation — Status & Handoff
|
||
|
||
**Snapshot:** 2026-07-12 · **Review baseline:** master `9cad9ed0`; **re-review baseline:** master
|
||
`f6eaa267` · this file is the single source of truth for remediation progress. Update it as work lands.
|
||
|
||
> **2026-07-12 UPDATE — everything below the topology table landed:** all remediation branches were
|
||
> **merged to master and pushed to `origin/master`** (merge commits `7834b3b7`…`b67bd9e8`; final state
|
||
> `f6eaa267`). The topology table is kept for history but is no longer live. A full **re-review** of all
|
||
> seven domains ran at `f6eaa267` — see [`../00-OVERALL.md`](../00-OVERALL.md) and the
|
||
> "Re-review 2026-07-12" section at the bottom of this file for the new findings (one High regression
|
||
> in the S7 fix, one High ResilienceConfig validation gap, and a batch of resilience-seam Mediums).
|
||
|
||
## Branch topology (HISTORICAL — as of 2026-07-08; since merged + pushed)
|
||
|
||
| Branch | Head | Contains |
|
||
|---|---|---|
|
||
| `master` | `9cad9ed0` | untouched baseline |
|
||
| `docs/archreview-plans-and-drift-fixes` | `9fadead6` | the 7 domain plans + `00-INDEX.md` + this `STATUS.md` + the two CLAUDE.md doc-drift fixes. **(the plans live ONLY here)** |
|
||
| `fix/archreview-crit1-split-brain-resolver` | `eaf78aad` | Critical 1 (03/S1) typed SBR option + guard test + Redundancy.md; **+ #9 hard-kill failover test** (`a25c9ed0`); **+ #11 premise correction** (`eaf78aad`) |
|
||
| `fix/archreview-crit2-vt-timeout` | `7fd44f0f` | Critical 2 (02/U2 + 02/U3) + tests |
|
||
| `fix/archreview-crit3-s7-reconnect` | `25c0c6f6` | Critical 3 (05/STAB-1) IS7Plc seam + lazy reconnect + unit guards + live bounce test |
|
||
| `fix/archreview-crit4-twincat-replay` | `af318fb4` | Critical 4 (05/STAB-2) native-notification replay + probe-recycle + unit guards |
|
||
| `fix/archreview-u2-deferred-sink-forwarding-guard` | `a65c2ced` | Guard 03/U2 — reflection-exhaustive deferred-sink forwarding test (+ DeferredServiceLevelPublisher) |
|
||
| `fix/archreview-c1-wire-analyzer` | `f0082af5` | Guard 07/C-1 — wire OTOPCUA0001 analyzer tree-wide + triage (surfaced the RESILIENCE-DISPATCH-GAP, task #10) |
|
||
| `fix/archreview-s1-s2-ci-coverage` | `10b89830` | Guard 07/S-1,S-2,S-4 — whole-solution CI leg + fail-on-skip gate + CLI sleep deflake |
|
||
| `fix/archreview-crit10-wire-capability-invoker` | `62556c24` | **#10 — wire CapabilityInvoker into dispatch via a Core.Abstractions seam** (off `fix/archreview-c1-wire-analyzer`, NOT master — the analyzer is the standing guard + the pragmas live there). Code-complete + unit/analyzer/context-verified; live gate pending. |
|
||
| `fix/archreview-crit13-resilience-config-artifact` | `8a8b9ec5` | **#13 — apply per-instance ResilienceConfig from the artifact** (`75403caa`, off the crit10 branch) **+ #10/#13 DI guard** (`8a8b9ec5` — `ResilienceInvokerFactoryRegistrationTests`, from the #12 live gate). Read-path plumbing + invalidate/respawn-on-change; deterministically verified + rig-booted. |
|
||
| `fix/docker-dev-serverhistorian-stale-keys` | `7233e2ba` | **Rig fix (off master, independent):** docker-dev compose's retired Wonderware `ServerHistorian` keys → gateway shape, disabled by default. Found during the #12 live-gate rig session; central-1 verified booting clean. |
|
||
|
||
**Decision (user):** one branch per Critical, each off `master`. **#9 (failover test) lives on the crit1 branch**
|
||
(not its own branch off master) because SBR only exists there — it is Critical 1's integration half. The plans are only on the docs branch,
|
||
so from a critical branch read them via `git show docs/archreview-plans-and-drift-fixes:archreview/plans/0X-<name>.md`.
|
||
|
||
## Completed
|
||
|
||
| Item | Finding | Branch @ commit | Verification |
|
||
|---|---|---|---|
|
||
| **Critical 1 — split-brain resolver activated** | 03/S1 | `fix/archreview-crit1-split-brain-resolver` `a81dea10` | `BuildClusterOptions` sets `KeepOldestOption{DownIfAlone=true}`; 3 unit guards (Cluster.Tests 29/29); **e2e-verified** the resolved ActorSystem config carries `SplitBrainResolverProvider` + `stable-after=15s` survives the HOCON/Hosting merge (throwaway spike, removed). HOCON cross-ref comment + Redundancy.md corrected. |
|
||
| **Critical 2a — real VT script timeout** | 02/U2 | `fix/archreview-crit2-vt-timeout` `7fd44f0f` | Routes through `TimedScriptEvaluator`; infinite-loop test (bounded by WaitAsync) proves no hang. Host.Integration VT 18/18. |
|
||
| **Critical 2b — ALC-safe compile cache** | 02/U3 | same `7fd44f0f` | Swapped raw dict → `CompiledScriptCache`; new `IScriptCacheOwner` seam; apply-boundary clear in `VirtualTagHostActor`; wiring guard test. Runtime VT 12/12. |
|
||
| **Doc drift — KNOWN LIMITATION 2 stale** | 03/U1 = 06/U-1 | `docs/…` `9fadead6` (CLAUDE.md) | historized-ref feed IS wired; rewrote to "value-capture code-complete, live-verify pending". |
|
||
| **Doc drift — ScriptAnalysis policy** | theme #5 | same `9fadead6` (CLAUDE.md) | corrected `FleetAdmin` → `Administrator,Designer` roles. |
|
||
| **Critical 3 — S7 lazy reconnect** | 05/STAB-1 | `fix/archreview-crit3-s7-reconnect` `25c0c6f6` | New `IS7Plc`/`IS7PlcFactory` seam + `EnsureConnectedAsync` (dispose-dead-handle → reopen on next data call); connection-fatal faults (socket/`ErrorCode.ConnectionError`) mark the handle dead, data-address errors do NOT; probe loop routes through it as a backstop. 4 unit guards + full S7 suite 234/234; **LIVE-verified** against real python-snap7 — bounced the container (`docker restart`), driver observed the outage and resumed Good reads with no redeploy (`S7_1500ReconnectTests`, double-gated on sim + `S7_RECONNECT_BOUNCE_CMD`). Also closes the S7 TEST-1 gap. |
|
||
| **Critical 4 — TwinCAT native-notification replay** | 05/STAB-2 | `fix/archreview-crit4-twincat-replay` `af318fb4` | Native ADS notifications were opaque handles (no replayable intent) → silently orphaned on a client swap. Now `NativeRegistration` (symbol/type/bit/interval/onChange + swappable handle) hangs off `DeviceState`; `EnsureConnectedAsync` split into a gate wrapper + under-gate core that **replays every intent onto a freshly-built client** (register+replay both under `ConnectGate`); probe-failure `RecycleClientAsync` disposes+nulls the wire-dead-but-port-open client so the next tick rebuilds+replays. 4 unit guards + full TwinCAT suite 174/174. **Unit-only — no TwinCAT docker fixture exists** (integration needs a real TC3 XAR), so the fake-client tests are authoritative (no live gate like S7 had). |
|
||
| **Guard — reflection-exhaustive deferred-sink forwarding** | 03/U2 | `fix/archreview-u2-deferred-sink-forwarding-guard` `a65c2ced` | New `DeferredSinkForwardingReflectionTests` (Commons.Tests) reflects over **every** method of every forwarding interface `DeferredAddressSpaceSink` implements (`IOpcUaAddressSpaceSink` + `ISurgicalAddressSpaceSink`) — plus the sibling `DeferredServiceLevelPublisher` — invokes each on the wrapper over a `DispatchProxy` recording inner, and asserts the call **reached** the inner. A `DispatchProxy` recorder auto-implements future members (zero maintenance); a guard-of-the-guard asserts the wrapper implements exactly the known forwarding-interface set, so a brand-new capability *interface* trips too. **This closes the F10b/PR#423 trap class and is the prerequisite gate for 03/P1 remove methods + 02/P `WriteValues`.** 3 new tests green (Commons.Tests 57/57); **negative control verified** — stubbing `UpdateFolderDisplayName` to a non-forwarding no-op fails the guard with the exact member-named diagnostic. NO production change. |
|
||
| **Guard — wire OTOPCUA0001 analyzer + triage** | 07/C-1 | `fix/archreview-c1-wire-analyzer` `f0082af5` | Injected the analyzer tree-wide via `Directory.Build.props` (`OutputItemType="Analyzer"`), then triaged ~280 surfaced hits: **(1) RESILIENCE-DISPATCH-GAP** — 7 production sites (`DriverInstanceActor` ×6 + `GenericDriverNodeManager`) scoped-`#pragma`'d as a **real tracked gap** (CapabilityInvoker built but never wired into dispatch; **new task #10**); **(2) driver-internal self-calls** — 3 sites (AbCip/S7 own poll/ack) `#pragma`'d as intentional; **(3) 12 wire-level test/CLI projects** → commented `NoWarn`. Full build green, 0 hits; analyzer 31/31; **negative control** — dropping one dispatch-gap pragma re-fires OTOPCUA0001 and fails the build, proving it's live tree-wide. |
|
||
| **Guard — CI to all suites + fail-on-skip + deflake** | 07/S-1,S-2,S-4 | `fix/archreview-s1-s2-ci-coverage` `10b89830` | S-1: replaced the 5-project `unit-tests` matrix with ONE whole-solution leg (self-maintaining; now covers Client's 388 + Analyzers' 31 + all drivers). S-2: `scripts/ci/assert-not-all-skipped.sh` fail-on-skip gate (turns "green==all-skipped" red) — strict on the unit leg, report-only on the fixtureless integration leg. S-4: readiness-signal (`SubscribeInvoked`/`SubscribeAlarmsInvoked`) replaces 11 fixed-sleep startup races in the newly-CI'd Client.CLI.Tests. Skip-gate proven locally (real trx OK / all-skip trx exit 1 / report-only / edge cases); Client.CLI.Tests 104/104. **CI job execution itself is verifiable only on push (nothing pushed).** |
|
||
| **#10 — wire CapabilityInvoker into dispatch (seam)** | RESILIENCE-DISPATCH-GAP (07/C-1) | `fix/archreview-crit10-wire-capability-invoker` `62556c24` | **Discovered the filed plan was infeasible** — Runtime is deliberately Polly-free (refs `Core.Abstractions`, not `Core`). Fix = a seam mirroring `IDriverFactory`: `IDriverCapabilityInvoker`(+null) + `IDriverCapabilityInvokerFactory`(+null) in Core.Abstractions; `CapabilityInvoker` implements it; `DriverCapabilityInvokerFactory` (Core) builds per-instance over the singleton builder+tracker+`GetTier`; Host `DriverFactoryBootstrap` registers all three; Runtime SCE resolves like `IDriverFactory`; `DriverInstanceActor` wraps its **6** dispatch sites; analyzer taught the interface as a wrapper home; all 6 pragmas removed. Triage corrections: the 7th site (`GenericDriverNodeManager`) is test-only scaffolding (not wired, re-annotated); **tier-defaults only** (ResilienceConfig not in the artifact → sub-gap). Added retry/breaker **logging** to the pipeline builder (the tracker has no reader — Stream E.2/E.3 never shipped — so it ran invisibly). **Verified:** negative-control (unwrap→OTOPCUA0001 build error); recording-invoker routing test; **real-`CapabilityInvoker` actor-context test** (internal `ConfigureAwait(false)` doesn't leak); analyzer + logging tests; clean solution build; Runtime.Tests 358 / Core.Tests 238 / Analyzers 32. **Live behavioral gate pending** (recipe in FOLLOWUP-10). |
|
||
| **#9 — hard-kill failover integration test** | 03/S1 | `fix/archreview-crit1-split-brain-resolver` `a25c9ed0` | Critical 1's load-bearing integration half. `TwoNodeClusterHarness.HardKillNodeAAsync` (transport-shutdown crash sim → node B sees A Unreachable, no graceful Leave) + `HardKillFailoverTests` asserting B downs the alone oldest node and takes over as `driver` role-leader. **LIVE-verified in-process** (~35s); **negative control** (force `downing-provider-class=""`) makes it correctly time out. ⚠️ The negative control also surfaced a **Crit-1 premise finding (task #11): SBR was already active on master** (Akka.Cluster.Hosting applies `SplitBrainResolverOption.Default` when the typed option is null → reads the pre-existing akka.conf keep-oldest block), so the cluster was NOT NoDowning before Crit-1 — the typed option is reinforcing, not the activator. |
|
||
| **#11 — Critical 1 premise correction** | 03/S1 (from #9) | `fix/archreview-crit1-split-brain-resolver` `eaf78aad` | Docs/comment-only fix (no code revert): corrected the "HOCON inert / cluster runs NoDowning / never fails over" framing in `BuildClusterOptions` XML comment, `akka.conf` split-brain-resolver comment, `docs/Redundancy.md`, and `SplitBrainResolverActivationTests` (summary + `ShouldNotBeNull` message + method rename). Akka.Cluster.Hosting enables an SBR provider **by default** (applies `SplitBrainResolverOption.Default` when the typed option is null → reads the pre-existing akka.conf `keep-oldest` block), so the cluster was never NoDowning and the typed `KeepOldestOption` is reinforcing/explicit-in-code, not the activator. Cluster.Tests 29/29; failover behaviour still guarded by `HardKillFailoverTests`. Architecture.md carried no such framing. |
|
||
| **#13 — apply per-instance ResilienceConfig from the artifact** | RESILIENCE-DISPATCH-GAP sub-gap (from #10) | `fix/archreview-crit13-resilience-config-artifact` `75403caa` | The `DriverInstance.ResilienceConfig` column was authored+persisted (AdminUI) and already in the artifact (the composer serializes the whole entity), but the driver-node read path dropped it → tier-defaults only. `DriverInstanceSpec` now carries it; `TryReadSpec` reads it; `Create` takes+layers it (+ logs parse diagnostics, never throws); `SpawnChild` threads it. **Invalidate-on-change:** `Create` calls `builder.Invalidate(instance)` (the cache is options-blind on a hit) so a respawn with changed options rebuilds; `DriverSpawnPlanner` respawns on a ResilienceConfig change (incl. null→json), a pure DriverConfig change stays an in-place delta. **Verified:** override reaches execution (retryCount:0 suppresses tier-A retry; a control retries), invalidate is per-instance, malformed logs+falls-back; planner+artifact tests. Core.Tests 243 (+5), Runtime.Tests 363 (+5). See [`FOLLOWUP-13`](FOLLOWUP-13-resilience-config-artifact.md). |
|
||
|
||
## Task list (TaskCreate IDs)
|
||
|
||
| # | Task | Status |
|
||
|---|---|---|
|
||
| 1 | Critical 1 — split-brain resolver | ✅ completed |
|
||
| 2 | Critical 2a — VT timeout | ✅ completed |
|
||
| 3 | Critical 2b — CompiledScriptCache | ✅ completed |
|
||
| 4 | Critical 3 — S7 reconnect (05/STAB-1) | ✅ completed |
|
||
| 5 | Critical 4 — TwinCAT ADS re-registration (05/STAB-2) | ✅ completed |
|
||
| 6 | Guard — DeferredAddressSpaceSink reflection forwarding test (03/U2) | ✅ completed — `a65c2ced`; gate for 03/P1 + 02/P now in place |
|
||
| 7 | Guard — wire OTOPCUA0001 analyzer + triage (07/C-1) | ✅ completed — `f0082af5`; surfaced task #10 |
|
||
| 8 | Guard — CI to all unit suites + fail-on-skip (07/S-1,S-2, pair S-4) | ✅ completed — `10b89830` |
|
||
| 9 | Critical 1 follow-up — hard-kill failover integration test (03/S1) | ✅ completed — `a25c9ed0`; surfaced task #11 |
|
||
| 10 | **Wire CapabilityInvoker into the dispatch layer** (RESILIENCE-DISPATCH-GAP, from 07/C-1) | ✅ code-complete + unit/analyzer/context-verified — `fix/archreview-crit10-wire-capability-invoker` `62556c24` (seam, NOT direct thread — Runtime is Polly-free). **Live behavioral gate pending** (rig recipe in FOLLOWUP-10). Surfaced a sub-gap: ResilienceConfig not in the deploy artifact → tier-defaults only. |
|
||
| 11 | **Review Critical 1 premise** — SBR was already active on master, not NoDowning (from #9) | ✅ completed — `fix/archreview-crit1-split-brain-resolver` `eaf78aad`; docs/comment-only correction (no code revert), Cluster.Tests 29/29. |
|
||
| 12 | **#10 live behavioral gate** — rig session (from #10) | ✅ ran 2026-07-08 (PARTIAL) — built the Host image from source (crit13 tip) + stood up the rig; central-1 booted healthy; **prod DI binds the real invoker factory** confirmed live + by a new deterministic guard (`ResilienceInvokerFactoryRegistrationTests` `8a8b9ec5`). Behavioral retry/breaker **log-line not observed** (disciplined stop): Modbus's cheap paths don't throw a wrapped exception (Discover offline, Subscribe lazy, reads non-wrapped) — only an idempotent write-to-dead-endpoint (or S7) does, not worth the multi-dependency chase since the retry behavior is deterministically proven. **Found + FIXED a rig bug:** docker-dev compose's retired Wonderware `ServerHistorian` keys crashed the Host at bring-up (historian-gateway cutover → reads `Endpoint`) → fixed on `fix/docker-dev-serverhistorian-stale-keys` `7233e2ba` (off master; gateway-shape keys, disabled by default; central-1 verified booting clean). See [`FOLLOWUP-10`](FOLLOWUP-10-resilience-dispatch-gap.md). |
|
||
| 13 | **Plumb per-instance `ResilienceConfig` through the deploy artifact** (from #10) | ✅ completed — `fix/archreview-crit13-resilience-config-artifact` `75403caa`; read-path plumbing (write side was already complete), invalidate-on-change + respawn-on-change; Core.Tests 243/Runtime.Tests 363. See [`FOLLOWUP-13`](FOLLOWUP-13-resilience-config-artifact.md). |
|
||
|
||
## Findings surfaced by the guards (NEW — need user decision)
|
||
|
||
> Each has a dedicated follow-up file with the full design/action:
|
||
> [`FOLLOWUP-10-resilience-dispatch-gap.md`](FOLLOWUP-10-resilience-dispatch-gap.md) ·
|
||
> [`FOLLOWUP-11-crit1-sbr-premise.md`](FOLLOWUP-11-crit1-sbr-premise.md) ·
|
||
> [`FOLLOWUP-13-resilience-config-artifact.md`](FOLLOWUP-13-resilience-config-artifact.md).
|
||
|
||
- **#10 — RESILIENCE-DISPATCH-GAP (from wiring the analyzer, 07/C-1): ✅ IMPLEMENTED** on
|
||
`fix/archreview-crit10-wire-capability-invoker` `62556c24` — code-complete + unit/analyzer/context-verified;
|
||
**live behavioral gate pending** (recipe in [`FOLLOWUP-10`](FOLLOWUP-10-resilience-dispatch-gap.md)). The filed
|
||
"thread CapabilityInvoker into the actor" was infeasible (Runtime is deliberately Polly-free); shipped a seam
|
||
mirroring `IDriverFactory` instead. Only **6** of the 7 sites were real (the 7th, `GenericDriverNodeManager`, is
|
||
test-only scaffolding). **Two residual sub-findings:** (a) ~~per-instance `ResilienceConfig` isn't in the deploy
|
||
artifact → tier-defaults only~~ **✅ RESOLVED by task #13** (`75403caa`) — the column was already in the artifact;
|
||
the read path now applies it (see [`FOLLOWUP-13`](FOLLOWUP-13-resilience-config-artifact.md)); (b) the
|
||
resilience-status tracker has **no reader** (Admin `/hosts` Stream E.2/E.3 never shipped) — added retry/breaker
|
||
**logging** as the interim observability surface + the live-verify enabler (still open as an Admin-UI follow-on).
|
||
- **#11 — Critical 1 premise inaccurate (from the #9 failover negative control): ✅ CORRECTED** on
|
||
`fix/archreview-crit1-split-brain-resolver` `eaf78aad`. SBR was **already active on master**
|
||
(Akka.Cluster.Hosting's `WithClustering` applies `SplitBrainResolverOption.Default` when the typed option is
|
||
null → enables the downing provider that reads the pre-existing akka.conf `keep-oldest` block). So the cluster
|
||
was **not** running NoDowning; hard-crash failover already worked. Docs/comment-only fix (no code revert — the
|
||
typed `KeepOldestOption` is correct belt-and-suspenders): corrected the "HOCON inert / NoDowning → no failover"
|
||
framing in `BuildClusterOptions` XML comment, `akka.conf`, `docs/Redundancy.md`, and the
|
||
`SplitBrainResolverActivationTests` summary/message (+ method rename). Cluster.Tests 29/29. (Architecture.md
|
||
carried no such framing.) See [`FOLLOWUP-11`](FOLLOWUP-11-crit1-sbr-premise.md).
|
||
|
||
## Suggested next (from 00-INDEX execution order)
|
||
|
||
**All 4 Criticals + all 3 systemic guards (tasks 6/7/8) + the #9 failover test are DONE.** Remaining:
|
||
|
||
1. **Resolve the two surfaced findings** — task 10 (wire CapabilityInvoker into dispatch; behavior-changing,
|
||
live-gate) and task 11 (correct Crit-1's rationale/docs).
|
||
2. **Failure-visibility Highs** — 01/S-1 (AddressSpaceApplyOutcome failure field), 06/S-1 (Galaxy fail-closed),
|
||
03/S4 (primary-gate default-deny).
|
||
3. **P-tier** — 03/P1 surgical adds/removals (now gated by the U2 forwarding guard), 02/P batched `WriteValues`.
|
||
|
||
## Key resume facts
|
||
|
||
- **Verification bar met so far:** each fix has a deterministic unit guard AND (for SBR) an end-to-end
|
||
config-resolution check. Follow the same pattern — the review's whole thesis is "unit-green ≠ wired".
|
||
- **The forwarding-test guard (task 6) is DONE (`a65c2ced`):** 03/P1 (surgical pure-adds) and 02/P-… batch-write
|
||
each add new sink capability methods that ship inert unless forwarded through `DeferredAddressSpaceSink`
|
||
(F10b/PR#423 precedent). The reflection guard now fails loudly if any such member isn't forwarded — add the
|
||
member to the recorder-covered interface set and it's auto-checked.
|
||
- ~~**Nothing is pushed.**~~ **Superseded 2026-07-12:** all branches merged to master and pushed to
|
||
`origin/master`; the plans now live on master too (no more cross-branch `git show` needed).
|
||
- ~~**CI changes can't be run-verified without a push**~~ — now pushed; the first real CI-job run should be
|
||
checked on origin (the re-review verified the YAML statically only).
|
||
- **Findings #10 and #11 are resolved** (see above); the open decisions now come from the 2026-07-12
|
||
re-review below.
|
||
|
||
## Re-review 2026-07-12 (master `f6eaa267`) — outcome + new findings
|
||
|
||
All seven domain reports were re-run and rewritten in place at `f6eaa267` (each carries a
|
||
`Prior-finding status (9cad9ed0 → f6eaa267)` table; synthesis in [`../00-OVERALL.md`](../00-OVERALL.md)).
|
||
Verdict on the remediation: **all 4 Criticals verifiably closed in code** (Crit-1 by premise correction),
|
||
both systemic guards live with negative controls, CI now whole-solution + fail-on-skip
|
||
(07 stability 2.5→3.5). New findings needing scheduling, ranked:
|
||
|
||
| Rank | Finding | Severity | One-line |
|
||
|---|---|---|---|
|
||
| 1 | 05/STAB-14 | **High (regression)** | S7 ensure-wrapper rethrows connect-timeout OCE → poll loop's bare `catch (OCE) { return; }` silently kills all S7 subscriptions after an unreachable-host (non-refusing) outage; live bounce test blind to it (`S7Driver.cs:488,1000,1383,1404,1206-1208`) |
|
||
| 2 | 01/S-6 | **High** | Operator-authorable `timeoutSeconds:0` in ResilienceConfig → Polly ValidationException on every capability call = permanent driver-brick; no range validation in parser or AdminUI form |
|
||
| 3 | 05/STAB-15 | **High** | S7 fatal-classification misses framing/desync + cancel-during-I/O (half-read PDU → permanent desync); probe catch never marks handle dead |
|
||
| 4 | 01/S-8 = 03/S12 | Medium | Dispatch hardcodes `isIdempotent:true`; `WriteIdempotentAttribute` has zero readers → #13-enabled write retries would retry command writes |
|
||
| 5 | 02/S13 | Medium | VT script failure/timeout leaves node stale-Good forever (timeout fix made the failure invisible) |
|
||
| 6 | 01/U-6 | Medium | Bulkhead knob parsed/authored/documented but never built into the pipeline — dead knob |
|
||
| 7 | 06/S-11 | Medium | Enabled-but-misconfigured ServerHistorian crash-loops Host with raw UriFormatException (only config side fixed in 7233e2ba) |
|
||
| 8 | 06/U-7 | Medium | HistorianGateway 0.2.0 proto3-optional DataType change silently alters GatewayTagProvisioner Boolean provisioning; live EnsureTags never run on 0.2.0 |
|
||
| 9 | 04/C-7, 04/U-5=03/U10, 01/S-7=03/S13 | Medium/Low | Resilience seam edges: lossy AdminUI form round-trip; status tracker has no reader (operators blind); respawn stale-pipeline cache race |
|
||
|
||
Carried-open Highs unchanged from round 1 (no branch targeted them): failure-visibility trio
|
||
(01/S-1, 06/S-1, 03/S4), AdminUI authz (04/C-1 — stakes rose: resilience config is on the ungated
|
||
surface), 03/S2/S3 block-bridging, 03/P1 surgical adds (guard prerequisite in place), driver-fleet batch
|
||
(05), repo hygiene (07). Full prioritized list in `00-OVERALL.md`.
|
||
|
||
**Round-2 plans WRITTEN (2026-07-12):** all 12 action items have design+implementation plans —
|
||
`R2-01`…`R2-12` in this directory (+ co-located `.tasks.json` each, for resumable execution). Index,
|
||
per-plan task counts, effort, and suggested execution order: [`00-INDEX.md`](00-INDEX.md) §"Round 2
|
||
plans". 193 bite-sized TDD tasks total, ~18–24 dev-days end-to-end; R2-01/02/03 (the re-review's
|
||
new-and-sharp items) are ~2 days combined.
|
||
|
||
**Round-2 EXECUTED — R2-12 Repo hygiene (2026-07-13, branch `r2/12-repo-hygiene`):** DONE.
|
||
07/U-5 (7 `lib/` AVEVA DLLs removed + gitignored, zero-reference proven), 07/U-4 (root planning
|
||
files: `pending.md`/`current.md` untracked+gitignored, `looseends.md` deleted, `stillpending.md` +
|
||
`HISTORIAN-GATEWAY-INTEGRATION-ISSUES.md` archived under `docs/plans/`, `docs/Historian.md`
|
||
repointed), 07/S-3 (`v2-e2e.yml` no-op nightly deleted), 07/P-5 (redundant `integration` matrix job
|
||
removed from `v2-ci.yml`), 07/U-3 (5 alarm operator CLI commands documented — now 13/13), 07/C-5
|
||
(`StyleGuide.md` ScadaBridge→OtOpcUa + minimal root `.editorconfig`), 07/C-4 (xunit pin 2.9.2→2.9.3
|
||
+ blocker comment; upstream-blocked — no xunit.v3 Akka TestKit exists). 07/U-6 needed no action (the
|
||
Wonderware husk dirs are untracked build artifacts, absent in a fresh worktree). Whole-solution build
|
||
clean. **OPEN follow-ups:** (a) when a runner-reachable fleet + a real `tests/Server/…E2ETests` land,
|
||
add a fresh E2E workflow WITH a zero-match guard (fail if `Category=E2E` matches nothing); (b) S-2
|
||
option #2 — a fixture-backed integration leg (opc-plc `services:` block, `MIN_EXECUTED=1`) covering
|
||
what the unit leg cannot; (c) migrate the 3 xunit v2 holdouts once Akka ships an xunit.v3 TestKit.
|
||
|
||
### Round-2 remediation landed
|
||
|
||
| Plan | Findings | Branch | Commits | Status |
|
||
|---|---|---|---|---|
|
||
| **R2-06** | 06/S-11 (rank 7), 06/U-7 (rank 8), 06/C-7 | `r2/06-serverhistorian-failfast` (off `1676c8f4`) | `3a049a13`…`4640ecb1` (10 task commits) | **Code-complete, offline-verified.** S-11: fail-fast `ServerHistorianOptionsValidator` (consumer-gated on `ServerHistorian:Enabled` OR `AlarmHistorian:Enabled`, fails empty/non-http(s) Endpoint at startup) + `AddValidatedOptions` wiring + adapter defense-in-depth (`TryCreate`→named `InvalidOperationException`, no more raw `UriFormatException` crash-loop) + `Validate()` warning parity + alarm-only-mode startup warning. U-7: pinned Boolean→Int1 explicit-presence (`HasDataType`) + `HISTGW_BOOL_SANDBOX_TAG` live Boolean EnsureTags leg + retype probe (skip-clean offline) + upsert/retype migration note (`docs/Historian.md`). C-7: keep-populating decision + dead-`Quality` doc comments (0.2.0 C-002). **Live leg operator-gated (T12) — pending a VPN run on a 0.2.0 gateway** (records the observed in-place retype policy back into `docs/Historian.md`; also discharges 06/U-2's full documented run). |
|
||
| **R2-05** | 04/C-1 (High) | `r2/05-adminui-authz` (off `1676c8f4`) | `183b72b7`…`6c1f05cb` (5 commits) | **Code-complete, offline-verified.** All 38 routable AdminUI pages carry an explicit named-policy `@attribute` via new `AdminUiPolicies` constants (`Security/Auth/`). New `ConfigEditor` (Administrator+Designer) gates the 20-page config-authoring surface incl. the 8 driver pages that author live `ResilienceConfig` + `/api/script-analysis/*`; new `AuthenticatedRead` on the 16 read pages; `FleetAdmin`/`DriverOperator` unchanged (literals→constants). Anti-drift guard `PageAuthorizationGuardTests` (bans bare `[Authorize]`/`Roles=`/unknown-policy/unclassified) + `AdminUiPoliciesTests` semantics matrix. Corrections folded in: Reservations + ClusterRedundancy read-only (→`AuthenticatedRead`); Home had no attribute (now `AuthenticatedRead`). AdminUI.Tests 517/517, Security.Tests 80/80. **Live `/run` positive pass deferred (T15)** — docker-dev `DisableLogin=true` auto-admin can't observe deny; negative proof is CI-side (both guard tests). |
|
||
| **R2-01** | 05/STAB-14 (High regression, rank 1), 05/STAB-15 (rank 3), 05/STAB-8 seam | `r2/01-s7-fault-hardening` (off `1676c8f4`) | `fcdcf0e2`…`bdb7ed41` (13 task commits) | **Code-complete, offline-verified.** All prod changes in `S7Driver.cs`. STAB-14: connect-timeout surfaces as `TimeoutException` (not an escaping OCE) at the `EnsureConnectedAsync`/`InitializeAsync` source + `when(token.IsCancellationRequested)` filters on the read/write ensure-wrappers and all three poll-loop OCE catches — subscription poll loops survive an unreachable-host outage and recover (T2 regression RED→GREEN). STAB-15: `IsS7ConnectionFatal` broadened to the ISO-on-TCP framing surface (TPKT/TPDU/WrongNumberOfBytes/`PlcException{WrongNumberReceivedBytes}`, NOT `InvalidDataException`); mid-PDU cancel marks handle dead; probe classifies+marks-dead under the gate. STAB-8 (S7 part): `EnsureConnectedAsync` documented as the R2-09 connect-throttle seam (note-only). Driver.S7.Tests 246/246. **Live outage gate deferred (T11)** — `S7_1500ConnectTimeoutOutageTests` (env `S7_TIMEOUT_OUTAGE_START_CMD`/`STOP_CMD`, SYN-blackhole) skips clean offline; the connect-*timeout* class the `docker restart` bounce can't produce. (Pre-existing unrelated CLI docs-scan test fails at base — out of scope.) |
|
||
| **R2-03** | 02/S13 (rank 5), 02/S12, 02/P7 | `r2/03-vt-failure-quality` (off `1676c8f4`) | `b9775485`…`6bb91e8d` (12 task commits) | **Code-complete, offline-verified.** S13: VT script failure/timeout degrades the node to `OpcUaQuality.Bad` (carrying last-known value) instead of freezing on last-Good forever — additive `EvaluationResult.Quality`, bridge historizes `0x80020000u` BadInternalError, published once per Good→Bad transition gated on all deps arrived (inputs-ready, no cold-start flash), recovery force-publishes next Good. S12: apply-boundary `ClearCompiledScripts` racing an in-flight eval into ObjectDisposedException retried once in `RoslynVirtualTagEvaluator` (re-fetch→recompile into current generation); stress guard reproduces pre-fix, deterministically green post-fix. P7: apply-boundary clear gated on a changed deployed expression set (`HashSet.SetEquals`, ordinal) — identical redeploy / rename-with-same-expressions no longer forces full VT recompile. Runtime.Tests 372/372, RoslynVirtualTagEvaluatorTests 20/20. **Deferred (T11/T12):** full cross-suite sweep + docker-dev live-`/run` (fail→Bad→recover + P7 no-recompile spot-check). |
|
||
| **R2-02** | 01/S-6 (High, rank 2), 01/U-6, 01/S-8=03/S12 (rank 4), 04/C-7, 01/S-7=03/S13 | `r2/02-resilience-config-hardening` (off `1676c8f4`) | `4be13429`…`dd82e740` (17 task commits) | **Code-complete, offline-verified.** **S-6:** parser clamps timeout/retry/breaker overrides (timeout≤0→tier default, breaker==1→2, caps) with diagnostics + belt-and-braces `Build` clamp so pipeline construction can never throw Polly `ValidationException` (new `ResiliencePolicyRanges` in Core.Abstractions); brick-repro + `Build_NeverThrows_ForHostileDirectOptions` + factory-seam wiring proof; AdminUI `Validate()` range warnings. **S-8:** parser forces Write/AlarmAck retry→0 (spec invariant) **and** dispatch flips to `isIdempotent:false` (RED-first on the Runtime wiring test); non-idempotent arm caches the no-retry snapshot once/invoker (01/P-4) + tracker in-flight accounting; `WriteIdempotentAttribute` false claim corrected; retry cells disabled. **U-6:** bulkhead knob deleted end-to-end (migration-free; stored keys ignored) + `Options_properties_are_exactly_the_pipeline_wired_set` inertness guard. **C-7:** `ResilienceFormModel` JsonObject-bag round-trip preserves unknown keys; malformed JSON → `ParseFailed` lock + verbatim `ToJson` + discard button. **S-7:** per-`Create` `Interlocked` options-generation in the pipeline-cache key kills the respawn stale-pipeline race. Core resilience 121/121, AdminUI form 10/10, Analyzers 32/32, Runtime 364/364; 0 production OTOPCUA0001. **Deferred (T15/T18):** docker-dev `:9200` `/run` (warning banner, input lockout, unknown-key survival) + whole-solution sweep. |
|
||
| **R2-10** | 03/U10=04/U-5 (Medium/Low, action #10) | `r2/10-resilience-observability` (off `46fedda3`) | `205d3980`…`<docs>` (11 task commits) | **Code-complete, offline-verified.** The resilience status tracker now has a production reader, mirroring the driver-health flow. Core truthfulness: `RecordBreakerClose` + derived `IsBreakerOpen` (breaker-open state derivable), `OnClosed` records the close, and `RecordSuccess` wired onto the invoker success paths (`ConsecutiveFailures` is a true consecutive counter, not stuck-forever). Transport: `DriverResilienceStatusChanged` Commons contract → periodic 5 s full-snapshot publisher `DriverResilienceStatusPublisherService` (driver nodes, registered in `AddOtOpcUaDriverFactories` with wiring guard `ResilienceStatusReaderRegistrationTests` — closes the built-but-never-wired class) → `driver-resilience-status` DPS topic → per-admin-node `DriverResilienceStatusBridge` → `IDriverResilienceStatusStore` → resilience chips (breaker-OPEN / consecutive-failures / in-flight / staleness) on `DriverStatusPanel.razor`, read in-process (self-HubConnection ban honored). Core.Tests resilience 31/31 (incl. deterministic breaker-close via a controllable TimeProvider), Host.IntegrationTests publisher+guard 6/6, AdminUI.Tests store 5/5; full-solution build clean. **Deferred (T10):** docker-dev live-`/run` — S7 dead-endpoint breaker-open observed on BOTH centrals + recovery-clear + staleness-dim (also the first live sighting of the #10 `circuit-breaker OPENED` log line, closing FOLLOWUP-10's behavioral gate). |
|
||
|
||
| **R2-08** | 03/S2 (High, action #8), 03/S3 (High) | `r2/08-ldap-historyread-async` (off `1a698cbb`) | `ce6d9e21`…`<docs>` (14 task commits) | **Code-complete, offline-verified.** **S2 (LDAP async/timeout):** the SDK raises the impersonation callback synchronously under a `SessionManager`-wide `m_eventLock` (verified vs. 1.5.378 — a hung bind serializes ALL activations), so the fix is entirely at the `LdapOpcUaUserAuthenticator` boundary: five new `Security:Ldap` keys (`ConnectionTimeoutMs`=10000 projected into the shared lib, `AuthTimeoutMs`=15000 hard whole-flow `WaitAsync` bound → fail-closed "timed out", `MaxConcurrentAuthentications`=8 `SemaphoreSlim` with release-on-core-completion so orphaned binds still count, `OutageFailureThreshold`=3/`OutageCooldownSeconds`=15 directory-outage circuit → fast-deny "unavailable", half-open probe) + `LdapAuthResult.IsSystemFailure` seam (mapped from `ServiceAccountBindFailed`) distinguishing outage from credential-deny + `LdapOptionsValidator` rules + `HandleImpersonation` contract doc + `docs/security.md`. Stall-repro RED→GREEN; offline outage-sim through the real `HandleImpersonation` against an unroutable host. **S3 (HistoryRead fan-out):** all four HistoryRead arms async-ified (`ServeNodeAsync`/`ServeRawPagedAsync`/`ServeEventsAsync`, OCE→`BadTimeout`) then bounded per-node fan-out (`RunBounded` semaphore + single arm-boundary bridge) + per-request deadline CTS (`HistoryReadDeadline`=60s) + process-wide lazy batch limiter (`MaxConcurrentHistoryReadBatches`=16 → flood degrades to `BadTooManyOperations`); 16×4=64 in-flight cap; three new `ServerHistorian` keys + node-manager props + hosted-service wiring + `docs/Historian.md`/CLAUDE.md tables. Sequential-serving RED→GREEN. Host.IntegrationTests (LDAP resilience) 13/13, Security.Tests 11/11, Runtime.Tests options 11/11, OpcUaServer.Tests HistoryRead 30/30. **Deferred (live):** S2 docker-dev unreachable-LDAP + GLAuth outage/recovery cycle; S3 `Category=LiveIntegration` gateway soak. |
|
||
| **R2-11** | 01/C-1, 01/P-1 + 05/CONV-2, UNDER-1, UNDER-6 | `r2/11-tagconfig-consolidation` (off `46fedda3`) | first `2e14fe1f` … (24 task commits) | **Code-complete, offline-verified.** **Part 1 (C-1/P-1):** single `TagConfigIntent.Parse` + `DeviceConfigIntent` in Commons replace the four byte-parity TagConfig parse copies (composer/artifact/walker/DraftValidator) — composer now parses once per tag (P-1); a permanent `Runtime.Tests/TagConfigCorpusParityTests` (30-blob corpus) guards compose≡decode; `grep "MUST parse identically" src/` = 0. **Part 2 (CONV-2):** shared strict-capable `TagConfigJson` readers in Core.Abstractions replace the six copy-pasted `ReadEnum`s; all six driver parsers gain `Inspect()` warnings + honour the `writable` key (FOCAS **forced read-only**, 05/UNDER-1); FOCAS equipment tags run the capability-matrix pre-flight (05/UNDER-6); Modbus probe parses the factory DTO shape (`timeoutMs` no longer dropped). New `EquipmentTagConfigInspector` in ControlPlane + `AdminOperationsActor` deploy-gate wiring with `Deployment:TagConfigValidationMode` (**Warn default | Error opt-in**), proven through the actor (F10b consuming test). AdminUI: `writable` checkbox in the six typed editors. **Phase-C follow-up (deferred, recorded here):** flip the runtime parsers to strict (typo'd enum ⇒ `BadNodeIdUnknown` instead of wrong-width `Good`) once fleets have run `Error` mode clean — behavior change deliberately not shipped in this plan. **Deferred (T22/T24):** docker-dev `/run` of one typed editor's Writable checkbox + whole-solution serial sweep. |
|
||
|
||
| **R2-07 (Phases 1+2+3)** | 03/P1 (action #7, surgical pure-adds/removes/mixed) | `r2/07-surgical-pure-adds` (off `1a698cbb`) | `bb226f45`…`e2b47638` (T1–T14) | **Phases 1+2 code-complete, offline-verified.** New pure `AddressSpaceChangeClassifier` (Empty/AttributeOnly/PureAdd/PureRemove/AddRemoveMix/Rebuild; first-match-wins, default-closed Rebuild; the F10b tag/vtag surgical-eligibility predicates moved here as `internal static`). `AddressSpaceApplier.Apply` routes by kind. **Phase 1 (PureAdd/AttributeOnly):** skip `RebuildAddressSpace` (the idempotent Materialise passes create exactly the added nodes; existing subscriptions survive); adds announced via `ComputeAddAnnouncements`+`AnnounceAddedNodes` (dedup affected-parent → existing guard-covered `RaiseNodesAddedModelChange`), fired by `OpcUaPublishActor` after the 4 passes gated on `!outcome.RebuildCalled`; `MaterialiseAlarmCondition` made idempotent (skip-if-present-and-same-kind keeps the condition NodeState alive). **NO new sink members in Phase 1.** **Phase 2 (PureRemove):** 3 new `ISurgicalAddressSpaceSink` members (`RemoveVariableNode`/`RemoveAlarmConditionNode`/`RemoveEquipmentSubtree`) — guard-first with the **observed negative control** (`DeferredSinkForwardingReflectionTests` fails member-named until `DeferredAddressSpaceSink` forwards) + `SdkAddressSpaceSink` forwards + `OtOpcUaNodeManager` in-place teardown (scoped map cleanup, notifier demotion via `RemoveRootNotifier`, `BuildNodesRemovedModelChange` verb=NodeDeleted reported outside `Lock`). Applier `PureRemove` arm: terminal Bad / `RemovedConditionState` writes (closing the removed-alarm-tag today-gap) → subsumption partition (removed-equipment children folded into the subtree remove) → in-place removes → any false/throw ⇒ full-rebuild ratchet; non-surgical sink ⇒ rebuild. **Phase 3 (AddRemoveMix):** removes-then-adds within one apply — the remove pass runs in `Apply` (no rebuild), then the publish actor's Materialise passes create the adds + `AnnounceAddedNodes` announces them; an id reused across the remove+add sets is remove-then-recreated as a fresh NodeState (correct — a different tag). **Guard stays green (Commons.Tests 17/17), OpcUaServer.Tests 336/336, Runtime.Tests 398/398.** **03/P1 marked REMEDIATED in `archreview/03-server-runtime.md`.** **Deferred-live (T5, T6, T12, T14):** `SubscriptionSurvivalTests` add + remove + mixed cases (authored + compile-green; NOT run — `*.IntegrationTests` memory constraint) and the docker-dev live-`/run` add/remove/mixed gates (`kind=PureAdd/PureRemove/AddRemoveMix, rebuild=False`; survivor subscription uninterrupted; removed node → `BadNodeIdUnknown`) — MUST run in the serial heavy pass (F10b inertness risk). |
|
||
|
||
**Report anchor drifts to fold into the next report refresh** (from the R2-02 verification pass): (1)
|
||
`01-core-composition.md` U-6 anchor `DriverResilienceOptions.cs:23-25` → the two bulkhead properties sat at
|
||
`:23` and `:29` (now deleted); (2) `01-core-composition.md` U-6 anchor
|
||
`DriverResiliencePipelineBuilder.cs:99-177` → `Build` was `:99-176`. All other cited lines were exact at
|
||
`f6eaa267`.
|
||
|
||
## ✅ Round-2 COMPLETE — all 12 plans merged (2026-07-13)
|
||
|
||
All 12 round-2 plans were executed (isolated worktree per plan, per-task TDD, pathspec-scoped commits) and
|
||
**merged to `master` + pushed to `origin/master`** in 3 dependency-ordered waves, each plan landing as its own
|
||
Gitea PR (closed-as-merged) behind a `dotnet build` barrier. **Final state: master `31c98200`.**
|
||
|
||
| Wave | Plans (PR#) |
|
||
|---|---|
|
||
| A (no cross-plan deps) | R2-12 #426 · R2-06 #427 · R2-05 #428 · R2-01 #429 · R2-03 #430 · R2-02 #431 |
|
||
| B (parse-site barrier) | R2-10 #432 · R2-04 #433 · R2-11 #434 |
|
||
| C (largest, depend on earlier) | R2-08 #435 · R2-07 #436 · R2-09 #437 |
|
||
|
||
Cross-plan mutexes honored: R2-01 landed before R2-09 (PollGroupEngine v2 / S7 fork retirement); R2-11's
|
||
`TagConfigIntent.Parse` consolidation landed before R2-09's driver parse edits; the three plans that all edit
|
||
`AddressSpaceApplier.cs` + `DriverHostActor.cs` + `OtOpcUaNodeManager.cs` (R2-04/R2-07/R2-08/R2-10/R2-11) were
|
||
merged sequentially with the shared-file auto-merges verified by targeted unit runs each time.
|
||
|
||
### Consolidated unit sweep on merged `master` `31c98200` (integration suites excluded — ~16 GB/run leak)
|
||
Solution `dotnet build` clean (0 errors). All R2-relevant unit suites GREEN, incl.: OpcUaServer 341, Runtime
|
||
398, Commons(+forwarding guard) 122, Core 276, Core.Abstractions 128, Core.Scripting 137, Core.VirtualTags 59,
|
||
Cluster 29, Security 84, ControlPlane 78, Analyzers 32, AdminUI 537, Client.CLI 104, Client.Shared 182,
|
||
Client.UI 126, Core.AlarmHistorian 29, Core.ScriptedAlarms 71, and all drivers (S7 254, Modbus 312+161, AbCip
|
||
336, AbLegacy 209, TwinCAT 189, FOCAS 265, Galaxy 304, Galaxy.Browser 21, Historian.Gateway 102, OpcUaClient 131).
|
||
|
||
**3 failures, all confirmed PRE-EXISTING / environmental (git diff `1676c8f4..HEAD` empty on their paths — R2 never touched them):**
|
||
1. `Core.Abstractions.Tests.InterfaceIndependenceTests.AllPublicTypes_LiveInRootNamespace` — flags the pre-R2
|
||
`Core.Abstractions.Historian.*` continuous-historization types living in a sub-namespace. **Pre-existing convention debt — needs-owner-decision** (move the types to root, or relax the test to allow the intentional Historian grouping).
|
||
2. `Driver.S7.Cli.Tests.SubscribeCommandConsoleHandlerCommentTests...` — pre-existing docs-scan test looking for a `"CliFx console"` phrase removed by `9cad9ed0`. **Follow-up: repoint or delete the phrase-scan.**
|
||
3. `Driver.OpcUaClient.Browser.Tests` (3) — `Connection refused`; need the live opc-plc endpoint (`10.100.0.35:50000`), unreachable offline. **Environmental — pass with the fixture up.**
|
||
|
||
### Deferred heavy / live-`/run` pass (serialized — single-tenant rig; controller-owned)
|
||
Every plan's integration-suite and live-`/run` gates were deferred (not skipped) so nothing OOMs the box and the
|
||
shared docker-dev rig / VPN infra isn't contended. To run one-at-a-time:
|
||
|
||
- **Full integration-suite sweep** (10 `*.IntegrationTests` projects, one at a time): discharges R2-02 T18, R2-03 T11, R2-07 T5/T6, R2-08 T14, R2-11 T24.
|
||
- **docker-dev `:9200` live-`/run`** (rebuild BOTH centrals): R2-02 T15 (ResilienceConfig warning banner / input lockout / unknown-key survival), R2-03 T12 (VT fail→Bad→recover + P7 no-recompile), R2-10 T10 (breaker-open on both centrals + recovery + FOLLOWUP-10 log line), R2-11 T22 (typed-editor Writable checkbox), **R2-07 T12 (surgical add/remove/mixed — F10b inertness risk, MUST run)**, R2-05 T15 (over-gating regression only — deny unobservable under `DisableLogin`).
|
||
- **2-node redundancy rig:** **R2-04 T13/T15 — behavior-affecting S4 primary-gate default-deny; MUST run before trusting the failover-role change.**
|
||
- **Genuinely external / needs-user:** R2-06 T12 (VPN to a 0.2.0 HistorianGateway for the live EnsureTags retype probe), R2-08 live legs (real GLAuth outage/recovery cycle; `Category=LiveIntegration` gateway soak), R2-01 T11 (S7 SYN-blackhole outage on the `10.100.0.35` s7_1500 fixture).
|
||
|
||
### Deferred behavior-change follow-up (deliberately not shipped)
|
||
**R2-11 Phase C** — flip the runtime driver parsers to strict (typo'd enum ⇒ `BadNodeIdUnknown` instead of a
|
||
wrong-width `Good`). Land only after fleets have run `Deployment:TagConfigValidationMode=Error` clean.
|
||
|
||
## docker-dev live-`/run` pass (2026-07-13) — CONCLUDED; 5 gates LIVE-confirmed on the built-from-source `6aeb53a9` rig
|
||
|
||
**Bottom line:** R2-02, R2-11, R2-07 (both surgical phases), R2-04 (achievable failover), and R2-10 (resilience pipeline in
|
||
dispatch) verified live on the docker-dev central cluster. R2-05 partial-by-design; R2-03 unit-verified (rig has no driver
|
||
data). Everything still open needs infra this box lacks: the `*.IntegrationTests` sweep needs the `10.100.0.35` driver
|
||
fixtures + ephemeral LDAP up; R2-06 needs VPN to a 0.2.0 gateway; R2-08 a live GLAuth outage cycle; R2-01 the S7 blackhole
|
||
fixture. Those are the only Round-2 items not yet exercised.
|
||
|
||
Brought up the 2-node central cluster (`central-1`+`central-2`, SQL, traefik) from source at merged master `6aeb53a9`
|
||
(single-tenant rig; drove the `:9200` AdminUI via browser + Client.CLI on `:4840`/`:4841`).
|
||
|
||
- **R2-02 (S-6 High) ✅ LIVE** — on the Modbus driver's *Resilience overrides*, entering `timeoutSeconds:0` renders the
|
||
exact range-validation banner: *"Out-of-range values will be clamped to a safe value on deploy: Read timeout must be
|
||
between 1 and 86400 seconds (got 0)."* **S-8 ✅** — the Write and AlarmAcknowledge RETRIES cells are disabled ("never").
|
||
- **R2-11 (T22) ✅ LIVE** — the typed Modbus tag editor shows the **Writable** toggle; saving persists `writable: yes`.
|
||
- **R2-07 (P1 — F10b MUST) ✅ LIVE** — central-1 boot apply logged `AddressSpaceApplier: applied plan
|
||
(kind=PureAdd, added=91, removed=0, changed=0, rebuild=False)`; deleting a UNS virtual-tag + redeploy logged
|
||
`(kind=PureRemove, added=0, removed=1, changed=0, rebuild=False)` and the node vanished from the address space
|
||
(`capper-03` lost its `production-count` child). **Surgical add AND remove both execute in prod with no full rebuild —
|
||
the remove path's 3 new sink members are wired, not inert.**
|
||
- **R2-04 (S4 MUST) ✅ LIVE (achievable guarantee)** — **younger-crash** (kill `central-2`): `SBR took decision
|
||
DownUnreachable and is downing [central-2]`, `Member removed`, `central-1` stays Up, AdminUI serving, ServiceLevel
|
||
**250** (sole primary, no split-brain). **Oldest-crash** (kill `central-1`) reproduced the **documented keep-oldest
|
||
asymmetry** live: the survivor took `SBR … DownReachable … including myself` and CoordinatedShutdown (ExitCode 3) —
|
||
i.e. crashing the *oldest* node does **not** fail over. This is the pre-registered SBR topology gap (a deferred USER
|
||
strategy decision), **not** an R2 regression; R2-04's `PrimaryGatePolicy` write-gate change is orthogonal + unit-verified.
|
||
- **R2-05 (T15) ✅ (partial, as expected)** — every config/read page renders under the all-roles `multi-role-test` session
|
||
(named-policy gating didn't over-gate a legitimate admin); deny is unobservable under docker-dev `DisableLogin=true`, as documented.
|
||
- **R2-10 (T10) ✅ pipeline LIVE (breaker-OPEN state not forced).** The `/hosts` page shows the driver-health flow live
|
||
(MAIN-galaxy-eq HEALTHY; MAIN-modbus-eq FAULTED "Connection refused"; MAIN-opcua-eq FAULTED "failed to connect to any
|
||
endpoints"). The failing drivers' subscribe/discover exceptions propagate through the **full Polly resilience pipeline** —
|
||
central-1 stack traces show `Polly.CircuitBreaker` → `Polly.Retry` → `Polly.Timeout` → `CapabilityInvoker.ExecuteAsync
|
||
(CapabilityInvoker.cs:84)`, i.e. **the #10 dispatch-gap fix + R2-10's tracker data-source are live in prod.** The explicit
|
||
`breaker OPENED` log line + the active resilience chip were NOT triggered: R2-09's ConnectionBackoff throttles retries, so
|
||
consecutive failures don't accumulate fast enough to open the Polly breaker within its window (the documented FOLLOWUP-10
|
||
difficulty — the reliable trigger is a rapid idempotent write-to-dead-endpoint). Reader/store/publisher/bridge are
|
||
unit-verified (Core resilience 128, publisher+guard 6, store 8).
|
||
- **R2-03 (T12) — unit-verified; live-blocked on this data-less rig.** Every materialised node reads
|
||
`0x80320000 Bad_WaitingForInitialData` because no live driver data flows (Modbus `10.100.0.35:5020` unreachable in-rig;
|
||
VTs depend on driver data), so a VT can't be staged Good→Bad; and the Client.CLI `read` surfaces any Bad StatusCode as a
|
||
`FetchValueAsync` exception rather than printing it. Good→Bad-with-last-value transition is covered by unit suites
|
||
(Runtime 372, RoslynVTEvaluator 20). Would be live-observable with a reachable driver fixture feeding the VT.
|
||
|
||
**Rig note:** the pass mutated the persisted ConfigDb (added a `drv-modbus-line3-01` Modbus driver on MAIN; removed
|
||
`capper-03`'s `production-count` virtual tag for the PureRemove test). Reset with
|
||
`docker compose -f docker-dev/docker-compose.yml down -v`. The remaining integration-suite sweep + genuinely-external
|
||
gates (R2-06 VPN, R2-08 live LDAP, R2-01 S7 blackhole) still need their infra.
|