Merge R2-03 Virtual-Tag failure quality (arch-review round 2) [PR #430]
Findings 02/S13 (VT failure/timeout degrades node to Bad quality, no stale-Good forever), 02/S12 (evaluator retry-once on disposed-race), 02/P7 (compile-cache clear gated on changed expression set). T11/T12 deferred. STATUS.md conflict resolved additively. Build clean.
This commit is contained in:
Joseph Doherty
@@ -673,3 +673,30 @@ T4, T5, T8, T10 ──► T11 (sweep + docs) ──► T12 (live /run gate)
|
||||
estimate for action item #3. **Overall risk:** Low-Medium, concentrated in S13's deliberate behavior
|
||||
change (nodes that used to freeze Good now go Bad — the point of the fix) and the one inverted legacy
|
||||
test expectation, both explicitly task-tracked.
|
||||
|
||||
## Execution deviations (R2-03)
|
||||
|
||||
Executed on branch `r2/03-vt-failure-quality` off master `1676c8f4` (not `f6eaa267` — the plan's verified
|
||||
tree; `1676c8f4` is the current master and includes the round-2 plan/docs merge, so the code anchors are
|
||||
unchanged). Deviations, all reversible:
|
||||
|
||||
- **T7 stress-test config retuned for a reliable RED on fast hardware.** The plan's prescribed shape
|
||||
(single loop × 300 sequential evaluations racing a back-to-back `Thread.Yield` clear loop) did **not**
|
||||
reproduce the ObjectDisposedException on this dev machine (arm64 macOS, many idle cores → the
|
||||
fetch→Task.Run-hop→disposed-guard window is sub-microsecond and the Task.Run body dispatches instantly).
|
||||
Escalating to a back-to-back clear loop instead compile-swamped the run into a 60 s `WaitAsync`
|
||||
timeout (every evaluation recompiles under constant clearing) — and a back-to-back clear also risks
|
||||
double-disposal within one evaluation, which a single retry cannot heal (would make the fix's GREEN
|
||||
flaky). Final config: **12 concurrent evaluation loops × 400 iterations** (thread-pool pressure widens
|
||||
the fetch→run window) racing a clear loop that clears then `Thread.SpinWait(4000)` (~tens of µs —
|
||||
frequent but not back-to-back, so at most one clear lands inside a single evaluation and the warm cache
|
||||
keeps compile cost bounded). This reproduces reliably (133 disposed-reason failures observed on the
|
||||
unfixed tree) and is deterministically green post-fix (verified stable across 3 repeat runs). The
|
||||
assertion was **not** weakened — it still requires every evaluation to succeed with no "disposed" reason.
|
||||
- **Test-run scope narrowed per controller memory constraint.** A mid-execution controller directive
|
||||
flagged a ~16 GB/run memory leak in the `*.IntegrationTests` suites and forbade whole-project/whole-
|
||||
solution `dotnet test`. Verified locally instead: `dotnet build ZB.MOM.WW.OtOpcUa.slnx` clean; the whole
|
||||
`Runtime.Tests` unit project (372/372); and `RoslynVirtualTagEvaluatorTests` filtered (20/20). **T11**
|
||||
(full cross-suite sweep) and **T12** (docker-dev live-`/run` gate) are marked `deferred-live` in
|
||||
`.tasks.json` for the controller's serialized heavy pass — the build + STATUS/tasks bookkeeping portion
|
||||
of T11 was completed and committed.
|
||||
|
||||
@@ -4,75 +4,99 @@
|
||||
{
|
||||
"id": "R2-03-T1",
|
||||
"subject": "S13 stale-Good repro test at host level (RED — must fail on f6eaa267): fail-after-success evaluator, publish probe sees Good then expects Bad AttributeValueUpdate",
|
||||
"status": "pending",
|
||||
"status": "completed",
|
||||
"blockedBy": []
|
||||
},
|
||||
{
|
||||
"id": "R2-03-T2",
|
||||
"subject": "Add OpcUaQuality Quality (default Good) to VirtualTagActor.EvaluationResult; bridge passes result.Quality into AttributeValueUpdate; historize maps Bad to 0x80020000u BadInternalError (RED bridge tests then GREEN)",
|
||||
"status": "pending",
|
||||
"blockedBy": ["R2-03-T1"]
|
||||
"status": "completed",
|
||||
"blockedBy": [
|
||||
"R2-03-T1"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "R2-03-T3",
|
||||
"subject": "VirtualTagActor failure/throw publishes Bad EvaluationResult once per transition, gated on all dependencyRefs arrived; recovery bypasses value-dedup; update Evaluator_failure_publishes_ScriptLogEntry_warning expectation (turns T1 green)",
|
||||
"status": "pending",
|
||||
"blockedBy": ["R2-03-T2"]
|
||||
"status": "completed",
|
||||
"blockedBy": [
|
||||
"R2-03-T2"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "R2-03-T4",
|
||||
"subject": "S13 transition-semantics unit guards: second consecutive failure publishes nothing; same-value recovery republishes Good; evaluator-threw path also degrades",
|
||||
"status": "pending",
|
||||
"blockedBy": ["R2-03-T3"]
|
||||
"status": "completed",
|
||||
"blockedBy": [
|
||||
"R2-03-T3"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "R2-03-T5",
|
||||
"subject": "S13 cold-start suppression guards: missing-dep failure publishes no Bad; failure after all deps arrived publishes Bad (inputs-ready gate)",
|
||||
"status": "pending",
|
||||
"blockedBy": ["R2-03-T3"]
|
||||
"status": "completed",
|
||||
"blockedBy": [
|
||||
"R2-03-T3"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "R2-03-T6",
|
||||
"subject": "S12 sequential pin: Evaluate_after_ClearCompiledScripts_recompiles_and_succeeds (green baseline the T8 retry-loop restructure must not regress)",
|
||||
"status": "pending",
|
||||
"status": "completed",
|
||||
"blockedBy": []
|
||||
},
|
||||
{
|
||||
"id": "R2-03-T7",
|
||||
"subject": "S12 concurrent stress repro (RED): 300 compiled evaluations racing ClearCompiledScripts, WaitAsync-bounded, assert zero disposed-reason failures",
|
||||
"status": "pending",
|
||||
"status": "completed",
|
||||
"blockedBy": []
|
||||
},
|
||||
{
|
||||
"id": "R2-03-T8",
|
||||
"subject": "S12 fix: retry-once on ObjectDisposedException in RoslynVirtualTagEvaluator.Evaluate (re-fetch via GetOrCompile; no retry when _disposed or second hit); update xmldoc (turns T7 green)",
|
||||
"status": "pending",
|
||||
"blockedBy": ["R2-03-T6", "R2-03-T7"]
|
||||
"status": "completed",
|
||||
"blockedBy": [
|
||||
"R2-03-T6",
|
||||
"R2-03-T7"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "R2-03-T9",
|
||||
"subject": "P7 wiring-guard rewrite (RED): ApplyVirtualTags clears the compile cache only when the expression set changes — identical redeploy must NOT clear; expression edit must; shared-expression removal must not",
|
||||
"status": "pending",
|
||||
"blockedBy": ["R2-03-T3"]
|
||||
"status": "completed",
|
||||
"blockedBy": [
|
||||
"R2-03-T3"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "R2-03-T10",
|
||||
"subject": "P7 fix: gate the OnApply ClearCompiledScripts on HashSet.SetEquals of desired vs _planByVtag expression sets (ordinal), with explanatory comment (turns T9 green)",
|
||||
"status": "pending",
|
||||
"blockedBy": ["R2-03-T9"]
|
||||
"status": "completed",
|
||||
"blockedBy": [
|
||||
"R2-03-T9"
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "R2-03-T11",
|
||||
"subject": "Full-suite sweep (Runtime.Tests whole project + RoslynVirtualTagEvaluatorTests + solution build) and STATUS.md / tasks.json bookkeeping",
|
||||
"status": "pending",
|
||||
"blockedBy": ["R2-03-T4", "R2-03-T5", "R2-03-T8", "R2-03-T10"]
|
||||
"status": "deferred-live",
|
||||
"blockedBy": [
|
||||
"R2-03-T4",
|
||||
"R2-03-T5",
|
||||
"R2-03-T8",
|
||||
"R2-03-T10"
|
||||
],
|
||||
"note": "integration/full sweep; serialized heavy pass (build clean, Runtime.Tests 372/372, RoslynVirtualTagEvaluatorTests 20/20 verified locally; whole-solution/cross-suite sweep deferred per controller memory constraint)"
|
||||
},
|
||||
{
|
||||
"id": "R2-03-T12",
|
||||
"subject": "Live-/run gate on docker-dev (rebuild BOTH centrals): author working VT, break it, Client.CLI read shows StatusCode Bad with last value, fix and read Good; P7 no-recompile spot-check on identical redeploy",
|
||||
"status": "pending",
|
||||
"blockedBy": ["R2-03-T11"]
|
||||
"status": "deferred-live",
|
||||
"blockedBy": [
|
||||
"R2-03-T11"
|
||||
],
|
||||
"note": "docker-dev live-/run; serial live pass"
|
||||
}
|
||||
],
|
||||
"lastUpdated": "2026-07-12"
|
||||
"lastUpdated": "2026-07-13"
|
||||
}
|
||||
|
||||
@@ -172,3 +172,4 @@ what the unit leg cannot; (c) migrate the 3 xunit v2 holdouts once Akka ships an
|
||||
| **R2-06** | 06/S-11 (rank 7), 06/U-7 (rank 8), 06/C-7 | `r2/06-serverhistorian-failfast` (off `1676c8f4`) | `3a049a13`…`4640ecb1` (10 task commits) | **Code-complete, offline-verified.** S-11: fail-fast `ServerHistorianOptionsValidator` (consumer-gated on `ServerHistorian:Enabled` OR `AlarmHistorian:Enabled`, fails empty/non-http(s) Endpoint at startup) + `AddValidatedOptions` wiring + adapter defense-in-depth (`TryCreate`→named `InvalidOperationException`, no more raw `UriFormatException` crash-loop) + `Validate()` warning parity + alarm-only-mode startup warning. U-7: pinned Boolean→Int1 explicit-presence (`HasDataType`) + `HISTGW_BOOL_SANDBOX_TAG` live Boolean EnsureTags leg + retype probe (skip-clean offline) + upsert/retype migration note (`docs/Historian.md`). C-7: keep-populating decision + dead-`Quality` doc comments (0.2.0 C-002). **Live leg operator-gated (T12) — pending a VPN run on a 0.2.0 gateway** (records the observed in-place retype policy back into `docs/Historian.md`; also discharges 06/U-2's full documented run). |
|
||||
| **R2-05** | 04/C-1 (High) | `r2/05-adminui-authz` (off `1676c8f4`) | `183b72b7`…`6c1f05cb` (5 commits) | **Code-complete, offline-verified.** All 38 routable AdminUI pages carry an explicit named-policy `@attribute` via new `AdminUiPolicies` constants (`Security/Auth/`). New `ConfigEditor` (Administrator+Designer) gates the 20-page config-authoring surface incl. the 8 driver pages that author live `ResilienceConfig` + `/api/script-analysis/*`; new `AuthenticatedRead` on the 16 read pages; `FleetAdmin`/`DriverOperator` unchanged (literals→constants). Anti-drift guard `PageAuthorizationGuardTests` (bans bare `[Authorize]`/`Roles=`/unknown-policy/unclassified) + `AdminUiPoliciesTests` semantics matrix. Corrections folded in: Reservations + ClusterRedundancy read-only (→`AuthenticatedRead`); Home had no attribute (now `AuthenticatedRead`). AdminUI.Tests 517/517, Security.Tests 80/80. **Live `/run` positive pass deferred (T15)** — docker-dev `DisableLogin=true` auto-admin can't observe deny; negative proof is CI-side (both guard tests). |
|
||||
| **R2-01** | 05/STAB-14 (High regression, rank 1), 05/STAB-15 (rank 3), 05/STAB-8 seam | `r2/01-s7-fault-hardening` (off `1676c8f4`) | `fcdcf0e2`…`bdb7ed41` (13 task commits) | **Code-complete, offline-verified.** All prod changes in `S7Driver.cs`. STAB-14: connect-timeout surfaces as `TimeoutException` (not an escaping OCE) at the `EnsureConnectedAsync`/`InitializeAsync` source + `when(token.IsCancellationRequested)` filters on the read/write ensure-wrappers and all three poll-loop OCE catches — subscription poll loops survive an unreachable-host outage and recover (T2 regression RED→GREEN). STAB-15: `IsS7ConnectionFatal` broadened to the ISO-on-TCP framing surface (TPKT/TPDU/WrongNumberOfBytes/`PlcException{WrongNumberReceivedBytes}`, NOT `InvalidDataException`); mid-PDU cancel marks handle dead; probe classifies+marks-dead under the gate. STAB-8 (S7 part): `EnsureConnectedAsync` documented as the R2-09 connect-throttle seam (note-only). Driver.S7.Tests 246/246. **Live outage gate deferred (T11)** — `S7_1500ConnectTimeoutOutageTests` (env `S7_TIMEOUT_OUTAGE_START_CMD`/`STOP_CMD`, SYN-blackhole) skips clean offline; the connect-*timeout* class the `docker restart` bounce can't produce. (Pre-existing unrelated CLI docs-scan test fails at base — out of scope.) |
|
||||
| **R2-03** | 02/S13 (rank 5), 02/S12, 02/P7 | `r2/03-vt-failure-quality` (off `1676c8f4`) | `b9775485`…`6bb91e8d` (12 task commits) | **Code-complete, offline-verified.** S13: VT script failure/timeout degrades the node to `OpcUaQuality.Bad` (carrying last-known value) instead of freezing on last-Good forever — additive `EvaluationResult.Quality`, bridge historizes `0x80020000u` BadInternalError, published once per Good→Bad transition gated on all deps arrived (inputs-ready, no cold-start flash), recovery force-publishes next Good. S12: apply-boundary `ClearCompiledScripts` racing an in-flight eval into ObjectDisposedException retried once in `RoslynVirtualTagEvaluator` (re-fetch→recompile into current generation); stress guard reproduces pre-fix, deterministically green post-fix. P7: apply-boundary clear gated on a changed deployed expression set (`HashSet.SetEquals`, ordinal) — identical redeploy / rename-with-same-expressions no longer forces full VT recompile. Runtime.Tests 372/372, RoslynVirtualTagEvaluatorTests 20/20. **Deferred (T11/T12):** full cross-suite sweep + docker-dev live-`/run` (fail→Bad→recover + P7 no-recompile spot-check). |
|
||||
|
||||
Reference in New Issue
Block a user