From c21e21423f744bbb26c5f7268732fc3ac1986f2a Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Wed, 15 Jul 2026 07:13:46 -0400 Subject: [PATCH] =?UTF-8?q?docs(archreview):=20R2-01=20S7=20live=20gate=20?= =?UTF-8?q?CLOSED=20=E2=80=94=20read-leg=20wall-clock=20fix=20(PR=20#453)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Running S7_1500ConnectTimeoutOutageTests (docker-pause blackhole, self-serviceable over SSH) uncovered + fixed the read-leg wall-clock gap STAB-14 left. Updates the R2-01 STATUS row, the needs-user/still-open lists, and the STAB-14 remediation note. --- archreview/05-protocol-drivers.md | 8 +++++++- archreview/plans/STATUS.md | 15 ++++++++------- 2 files changed, 15 insertions(+), 8 deletions(-) diff --git a/archreview/05-protocol-drivers.md b/archreview/05-protocol-drivers.md index 455ea9a7..de67dfa8 100644 --- a/archreview/05-protocol-drivers.md +++ b/archreview/05-protocol-drivers.md @@ -172,7 +172,13 @@ are the new STAB-14..17 below. > `when (token.IsCancellationRequested)` filters on the read/write ensure-wrappers (`:488`/`:1000`) and all > three poll-loop OCE catches — so a non-teardown OCE degrades/backs-off instead of killing the loop. Unit > guards T1–T4 (`S7DriverReconnectTests`, incl. the poll-loop-survival regression). Live gate -> `S7_1500ConnectTimeoutOutageTests` (env-gated SYN-blackhole, `deferred-live`). +> `S7_1500ConnectTimeoutOutageTests` **CLOSED 2026-07-15** — running it (blackhole via `docker pause`) +> exposed that the fix above covered only the *connect* leg: S7.Net's socket `ReadTimeout`/`WriteTimeout` +> are ignored by its async read/write paths, so a read on an established-but-frozen peer blocked for +> minutes and wedged the poll loop. Fixed (PR #453, master `4dc937fc`) by a new `S7OperationDeadline` +> wall-clock ceiling on every data-plane wire op (`S7PlcAdapter` → `TimeoutException`) + `IsS7ConnectionFatal` +> now treating `TimeoutException` as fatal → mark-dead → reopen (the READ-leg sibling of STAB-14). Live gate +> now GREEN; Driver.S7.Tests 246→260. **STAB-14 — High — NEW — S7's reconnect wrapper rethrows connect-timeout cancellation, and the poll loop's bare OCE catch then permanently kills the diff --git a/archreview/plans/STATUS.md b/archreview/plans/STATUS.md index 1dcfbb73..66adf4e6 100644 --- a/archreview/plans/STATUS.md +++ b/archreview/plans/STATUS.md @@ -171,7 +171,7 @@ what the unit leg cannot; (c) migrate the 3 xunit v2 holdouts once Akka ships an |---|---|---|---|---| | **R2-06** | 06/S-11 (rank 7), 06/U-7 (rank 8), 06/C-7 | `r2/06-serverhistorian-failfast` (off `1676c8f4`) | `3a049a13`…`4640ecb1` (10 task commits) | **Code-complete, offline-verified.** S-11: fail-fast `ServerHistorianOptionsValidator` (consumer-gated on `ServerHistorian:Enabled` OR `AlarmHistorian:Enabled`, fails empty/non-http(s) Endpoint at startup) + `AddValidatedOptions` wiring + adapter defense-in-depth (`TryCreate`→named `InvalidOperationException`, no more raw `UriFormatException` crash-loop) + `Validate()` warning parity + alarm-only-mode startup warning. U-7: pinned Boolean→Int1 explicit-presence (`HasDataType`) + `HISTGW_BOOL_SANDBOX_TAG` live Boolean EnsureTags leg + retype probe (skip-clean offline) + upsert/retype migration note (`docs/Historian.md`). C-7: keep-populating decision + dead-`Quality` doc comments (0.2.0 C-002). **Live leg operator-gated (T12) — pending a VPN run on a 0.2.0 gateway** (records the observed in-place retype policy back into `docs/Historian.md`; also discharges 06/U-2's full documented run). | | **R2-05** | 04/C-1 (High) | `r2/05-adminui-authz` (off `1676c8f4`) | `183b72b7`…`6c1f05cb` (5 commits) | **Code-complete, offline-verified.** All 38 routable AdminUI pages carry an explicit named-policy `@attribute` via new `AdminUiPolicies` constants (`Security/Auth/`). New `ConfigEditor` (Administrator+Designer) gates the 20-page config-authoring surface incl. the 8 driver pages that author live `ResilienceConfig` + `/api/script-analysis/*`; new `AuthenticatedRead` on the 16 read pages; `FleetAdmin`/`DriverOperator` unchanged (literals→constants). Anti-drift guard `PageAuthorizationGuardTests` (bans bare `[Authorize]`/`Roles=`/unknown-policy/unclassified) + `AdminUiPoliciesTests` semantics matrix. Corrections folded in: Reservations + ClusterRedundancy read-only (→`AuthenticatedRead`); Home had no attribute (now `AuthenticatedRead`). AdminUI.Tests 517/517, Security.Tests 80/80. **Live `/run` positive pass deferred (T15)** — docker-dev `DisableLogin=true` auto-admin can't observe deny; negative proof is CI-side (both guard tests). | -| **R2-01** | 05/STAB-14 (High regression, rank 1), 05/STAB-15 (rank 3), 05/STAB-8 seam | `r2/01-s7-fault-hardening` (off `1676c8f4`) | `fcdcf0e2`…`bdb7ed41` (13 task commits) | **Code-complete, offline-verified.** All prod changes in `S7Driver.cs`. STAB-14: connect-timeout surfaces as `TimeoutException` (not an escaping OCE) at the `EnsureConnectedAsync`/`InitializeAsync` source + `when(token.IsCancellationRequested)` filters on the read/write ensure-wrappers and all three poll-loop OCE catches — subscription poll loops survive an unreachable-host outage and recover (T2 regression RED→GREEN). STAB-15: `IsS7ConnectionFatal` broadened to the ISO-on-TCP framing surface (TPKT/TPDU/WrongNumberOfBytes/`PlcException{WrongNumberReceivedBytes}`, NOT `InvalidDataException`); mid-PDU cancel marks handle dead; probe classifies+marks-dead under the gate. STAB-8 (S7 part): `EnsureConnectedAsync` documented as the R2-09 connect-throttle seam (note-only). Driver.S7.Tests 246/246. **Live outage gate deferred (T11)** — `S7_1500ConnectTimeoutOutageTests` (env `S7_TIMEOUT_OUTAGE_START_CMD`/`STOP_CMD`, SYN-blackhole) skips clean offline; the connect-*timeout* class the `docker restart` bounce can't produce. (Pre-existing unrelated CLI docs-scan test fails at base — out of scope.) | +| **R2-01** | 05/STAB-14 (High regression, rank 1), 05/STAB-15 (rank 3), 05/STAB-8 seam | `r2/01-s7-fault-hardening` (off `1676c8f4`) | `fcdcf0e2`…`bdb7ed41` (13 task commits) | **Code-complete, offline-verified.** All prod changes in `S7Driver.cs`. STAB-14: connect-timeout surfaces as `TimeoutException` (not an escaping OCE) at the `EnsureConnectedAsync`/`InitializeAsync` source + `when(token.IsCancellationRequested)` filters on the read/write ensure-wrappers and all three poll-loop OCE catches — subscription poll loops survive an unreachable-host outage and recover (T2 regression RED→GREEN). STAB-15: `IsS7ConnectionFatal` broadened to the ISO-on-TCP framing surface (TPKT/TPDU/WrongNumberOfBytes/`PlcException{WrongNumberReceivedBytes}`, NOT `InvalidDataException`); mid-PDU cancel marks handle dead; probe classifies+marks-dead under the gate. STAB-8 (S7 part): `EnsureConnectedAsync` documented as the R2-09 connect-throttle seam (note-only). Driver.S7.Tests 246/246. **Live outage gate CLOSED 2026-07-15 (T11) — and it uncovered + fixed a real prod gap (PR #453, master `4dc937fc`).** Running `S7_1500ConnectTimeoutOutageTests` against the live s7_1500 sim (blackholed via `docker pause`, the ACK-less-established shape) exposed that STAB-14 fixed only the *connect* leg: S7.Net's `ReadTimeout`/`WriteTimeout` map to the TcpClient's *synchronous* receive/send timeout, which async reads ignore, so a read on an established-but-frozen peer blocked for minutes — the poll loop wedged (no Bad tick, no reconnect), which offline fakes (open-hang only) couldn't surface. Fix = new `S7OperationDeadline` wall-clock ceiling on every data-plane wire op in `S7PlcAdapter` (→ `TimeoutException`) + `IsS7ConnectionFatal` now classifies `TimeoutException` fatal → mark-dead → reopen (the read-leg sibling of the connect-timeout fix). Driver.S7.Tests 246→260 (6 new: 5 deadline + 1 driver-reaction); **live gate now GREEN** (8s: baseline Good → pause blackhole → Bad tick → unpause → recovered Good). (Pre-existing unrelated CLI docs-scan test fails at base — out of scope.) | | **R2-03** | 02/S13 (rank 5), 02/S12, 02/P7 | `r2/03-vt-failure-quality` (off `1676c8f4`) | `b9775485`…`6bb91e8d` (12 task commits) | **Code-complete, offline-verified.** S13: VT script failure/timeout degrades the node to `OpcUaQuality.Bad` (carrying last-known value) instead of freezing on last-Good forever — additive `EvaluationResult.Quality`, bridge historizes `0x80020000u` BadInternalError, published once per Good→Bad transition gated on all deps arrived (inputs-ready, no cold-start flash), recovery force-publishes next Good. S12: apply-boundary `ClearCompiledScripts` racing an in-flight eval into ObjectDisposedException retried once in `RoslynVirtualTagEvaluator` (re-fetch→recompile into current generation); stress guard reproduces pre-fix, deterministically green post-fix. P7: apply-boundary clear gated on a changed deployed expression set (`HashSet.SetEquals`, ordinal) — identical redeploy / rename-with-same-expressions no longer forces full VT recompile. Runtime.Tests 372/372, RoslynVirtualTagEvaluatorTests 20/20. **Deferred (T11/T12):** full cross-suite sweep + docker-dev live-`/run` (fail→Bad→recover + P7 no-recompile spot-check). | | **R2-02** | 01/S-6 (High, rank 2), 01/U-6, 01/S-8=03/S12 (rank 4), 04/C-7, 01/S-7=03/S13 | `r2/02-resilience-config-hardening` (off `1676c8f4`) | `4be13429`…`dd82e740` (17 task commits) | **Code-complete, offline-verified.** **S-6:** parser clamps timeout/retry/breaker overrides (timeout≤0→tier default, breaker==1→2, caps) with diagnostics + belt-and-braces `Build` clamp so pipeline construction can never throw Polly `ValidationException` (new `ResiliencePolicyRanges` in Core.Abstractions); brick-repro + `Build_NeverThrows_ForHostileDirectOptions` + factory-seam wiring proof; AdminUI `Validate()` range warnings. **S-8:** parser forces Write/AlarmAck retry→0 (spec invariant) **and** dispatch flips to `isIdempotent:false` (RED-first on the Runtime wiring test); non-idempotent arm caches the no-retry snapshot once/invoker (01/P-4) + tracker in-flight accounting; `WriteIdempotentAttribute` false claim corrected; retry cells disabled. **U-6:** bulkhead knob deleted end-to-end (migration-free; stored keys ignored) + `Options_properties_are_exactly_the_pipeline_wired_set` inertness guard. **C-7:** `ResilienceFormModel` JsonObject-bag round-trip preserves unknown keys; malformed JSON → `ParseFailed` lock + verbatim `ToJson` + discard button. **S-7:** per-`Create` `Interlocked` options-generation in the pipeline-cache key kills the respawn stale-pipeline race. Core resilience 121/121, AdminUI form 10/10, Analyzers 32/32, Runtime 364/364; 0 production OTOPCUA0001. **Deferred (T15/T18):** docker-dev `:9200` `/run` (warning banner, input lockout, unknown-key survival) + whole-solution sweep. | | **R2-10** | 03/U10=04/U-5 (Medium/Low, action #10) | `r2/10-resilience-observability` (off `46fedda3`) | `205d3980`…`` (11 task commits) | **Code-complete, offline-verified.** The resilience status tracker now has a production reader, mirroring the driver-health flow. Core truthfulness: `RecordBreakerClose` + derived `IsBreakerOpen` (breaker-open state derivable), `OnClosed` records the close, and `RecordSuccess` wired onto the invoker success paths (`ConsecutiveFailures` is a true consecutive counter, not stuck-forever). Transport: `DriverResilienceStatusChanged` Commons contract → periodic 5 s full-snapshot publisher `DriverResilienceStatusPublisherService` (driver nodes, registered in `AddOtOpcUaDriverFactories` with wiring guard `ResilienceStatusReaderRegistrationTests` — closes the built-but-never-wired class) → `driver-resilience-status` DPS topic → per-admin-node `DriverResilienceStatusBridge` → `IDriverResilienceStatusStore` → resilience chips (breaker-OPEN / consecutive-failures / in-flight / staleness) on `DriverStatusPanel.razor`, read in-process (self-HubConnection ban honored). Core.Tests resilience 31/31 (incl. deterministic breaker-close via a controllable TimeProvider), Host.IntegrationTests publisher+guard 6/6, AdminUI.Tests store 5/5; full-solution build clean. **Deferred (T10):** docker-dev live-`/run` — S7 dead-endpoint breaker-open observed on BOTH centrals + recovery-clear + staleness-dim (also the first live sighting of the #10 `circuit-breaker OPENED` log line, closing FOLLOWUP-10's behavioral gate). | @@ -224,7 +224,7 @@ shared docker-dev rig / VPN infra isn't contended. To run one-at-a-time: - **Full integration-suite sweep** (10 `*.IntegrationTests` projects, one at a time): discharges R2-02 T18, R2-03 T11, R2-07 T5/T6, R2-08 T14, R2-11 T24. **RAN 2026-07-14 — see [`INTEGRATION-SWEEP-STATUS.md`](INTEGRATION-SWEEP-STATUS.md).** Outcome: OpcUaClient(+Browser)/S7/Modbus-standard green, Host 105/117, TwinCAT clean-skip; the rest fail on **pre-existing test/fixture bit-rot** (AbCip static-init bug, Modbus pymodbus-4.x sim drift, FOCAS 127.0.0.1-mock topology, AbLegacy no fixture, OpcUaServer cert-store config, Host openldap:2.6 image gone). **No OtOpcUa regression.** 7 follow-ups in that file. - **docker-dev `:9200` live-`/run`** (rebuild BOTH centrals): R2-02 T15 (ResilienceConfig warning banner / input lockout / unknown-key survival), R2-03 T12 (VT fail→Bad→recover + P7 no-recompile), R2-10 T10 (breaker-open on both centrals + recovery + FOLLOWUP-10 log line), R2-11 T22 (typed-editor Writable checkbox), **R2-07 T12 (surgical add/remove/mixed — F10b inertness risk, MUST run)**, R2-05 T15 (over-gating regression only — deny unobservable under `DisableLogin`). - **2-node redundancy rig:** ~~R2-04 T13/T15~~ **BOTH ✅ PASSED 2026-07-14** — T15 live docker-dev gate + T13 in-process delivery-path test (see below). -- **Genuinely external / needs-user:** R2-06 T12 (VPN to a 0.2.0 HistorianGateway for the live EnsureTags retype probe), R2-08 live legs (real GLAuth outage/recovery cycle; `Category=LiveIntegration` gateway soak), R2-01 T11 (S7 SYN-blackhole outage on the `10.100.0.35` s7_1500 fixture). +- **Genuinely external / needs-user:** R2-06 T12 (VPN to a 0.2.0 HistorianGateway for the live EnsureTags retype probe), R2-08 live legs (`Category=LiveIntegration` gateway soak — needs VPN to a real HistorianGateway). ~~R2-01 T11~~ **CLOSED 2026-07-15** (S7 blackhole run via `docker pause` on the `10.100.0.35` s7_1500 fixture — self-serviceable over SSH, no sudo/VPN; it found + fixed the read-leg wall-clock gap, PR #453). ~~R2-08 GLAuth outage cycle~~ is *also* self-serviceable on `10.100.0.35` (deploy the Host harness compose + `docker pause` GLAuth) — not truly external, just not yet run. ### Deferred behavior-change follow-up (deliberately not shipped) **R2-11 Phase C** — flip the runtime driver parsers to strict (typo'd enum ⇒ `BadNodeIdUnknown` instead of a @@ -234,9 +234,10 @@ wrong-width `Good`). Land only after fleets have run `Deployment:TagConfigValida **Bottom line:** R2-02, R2-11, R2-07 (both surgical phases), R2-04 (achievable failover), and R2-10 (resilience pipeline in dispatch) verified live on the docker-dev central cluster. R2-05 partial-by-design; R2-03 unit-verified (rig has no driver -data). Everything still open needs infra this box lacks: the `*.IntegrationTests` sweep needs the `10.100.0.35` driver -fixtures + ephemeral LDAP up; R2-06 needs VPN to a 0.2.0 gateway; R2-08 a live GLAuth outage cycle; R2-01 the S7 blackhole -fixture. Those are the only Round-2 items not yet exercised. +data). R2-01's S7 blackhole gate is **CLOSED (2026-07-15)** — run via `docker pause` on the `10.100.0.35` s7_1500 fixture; +it found + fixed the read-leg wall-clock gap (PR #453). Still open: R2-06 needs VPN to a 0.2.0 gateway; R2-08's +`Category=LiveIntegration` gateway soak needs VPN to a real HistorianGateway (its GLAuth outage-cycle leg is self-serviceable +on `10.100.0.35` and just not yet run). Those are the only Round-2 items not yet exercised. Brought up the 2-node central cluster (`central-1`+`central-2`, SQL, traefik) from source at merged master `6aeb53a9` (single-tenant rig; drove the `:9200` AdminUI via browser + Client.CLI on `:4840`/`:4841`). @@ -275,8 +276,8 @@ Brought up the 2-node central cluster (`central-1`+`central-2`, SQL, traefik) fr **Rig note:** the pass mutated the persisted ConfigDb (added a `drv-modbus-line3-01` Modbus driver on MAIN; removed `capper-03`'s `production-count` virtual tag for the PureRemove test). Reset with -`docker compose -f docker-dev/docker-compose.yml down -v`. The remaining integration-suite sweep + genuinely-external -gates (R2-06 VPN, R2-08 live LDAP, R2-01 S7 blackhole) still need their infra. +`docker compose -f docker-dev/docker-compose.yml down -v`. R2-01's S7 blackhole gate is now CLOSED (2026-07-15, PR #453); +the remaining genuinely-external gates (R2-06 VPN, R2-08 `LiveIntegration` gateway soak) still need their infra. ---