From 9fadead6a67bb76711f89c6c2cc8922766a52c37 Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Wed, 8 Jul 2026 16:14:37 -0400 Subject: [PATCH] docs(archreview): remediation plans + fix flagged doc drift Add the 7 per-domain design+implementation plans (archreview/plans/) with an index, produced from the 2026-07-08 architecture review. Fix two confirmed doc drifts the review flagged (theme #5): - CLAUDE.md KNOWN LIMITATION 2: the continuous-historization historized-ref feed IS wired (AddressSpaceApplier.FeedHistorizedRefs -> UpdateHistorizedRefs -> recorder); rewrite to reflect that value-capture is code-complete and only the live end-to-end + restart-convergence verification remains. - CLAUDE.md ScriptAnalysis gating: endpoints use Roles=Administrator,Designer via RequireAuthorization, not the FleetAdmin policy. --- CLAUDE.md | 24 +- archreview/00-OVERALL.md | 121 +++ archreview/01-core-composition.md | 372 ++++++++ archreview/02-scripting-alarms.md | 196 +++++ archreview/03-server-runtime.md | 163 ++++ archreview/04-adminui.md | 219 +++++ archreview/05-protocol-drivers.md | 616 +++++++++++++ archreview/06-gateway-integrations.md | 196 +++++ archreview/07-client-tooling-engineering.md | 411 +++++++++ archreview/plans/00-INDEX.md | 69 ++ archreview/plans/01-core-composition-plan.md | 533 ++++++++++++ archreview/plans/02-scripting-alarms-plan.md | 362 ++++++++ archreview/plans/03-server-runtime-plan.md | 599 +++++++++++++ archreview/plans/04-adminui-plan.md | 292 +++++++ archreview/plans/05-protocol-drivers-plan.md | 808 ++++++++++++++++++ .../plans/06-gateway-integrations-plan.md | 353 ++++++++ .../07-client-tooling-engineering-plan.md | 521 +++++++++++ 17 files changed, 5846 insertions(+), 9 deletions(-) create mode 100644 archreview/00-OVERALL.md create mode 100644 archreview/01-core-composition.md create mode 100644 archreview/02-scripting-alarms.md create mode 100644 archreview/03-server-runtime.md create mode 100644 archreview/04-adminui.md create mode 100644 archreview/05-protocol-drivers.md create mode 100644 archreview/06-gateway-integrations.md create mode 100644 archreview/07-client-tooling-engineering.md create mode 100644 archreview/plans/00-INDEX.md create mode 100644 archreview/plans/01-core-composition-plan.md create mode 100644 archreview/plans/02-scripting-alarms-plan.md create mode 100644 archreview/plans/03-server-runtime-plan.md create mode 100644 archreview/plans/04-adminui-plan.md create mode 100644 archreview/plans/05-protocol-drivers-plan.md create mode 100644 archreview/plans/06-gateway-integrations-plan.md create mode 100644 archreview/plans/07-client-tooling-engineering-plan.md diff --git a/CLAUDE.md b/CLAUDE.md index 2ecfa7cd..e181aa94 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -220,7 +220,8 @@ help, document formatting, and tag-path completions inside `ctx.GetTag("…")` / runtime publish gate uses, so what the editor accepts/rejects matches publish exactly. The backend lives in `src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/ScriptAnalysis/` (six minimal-API -endpoints under `/api/script-analysis/*`, gated by the `FleetAdmin` policy). +endpoints under `/api/script-analysis/*`, gated to the `Administrator,Designer` +roles via `RequireAuthorization` in `ScriptAnalysisEndpoints.MapScriptAnalysis`). See `docs/ScriptEditor.md` for the full guide. Scripts may use the `{{equip}}` token for equipment-relative tag paths that resolve per-equipment at deploy time — see the "Equipment-relative tag paths" section in `docs/ScriptEditor.md`. ## Scripted Alarm Ack/Shelve @@ -355,13 +356,18 @@ dotnet test --filter "Category=LiveIntegration" The live suite **skips cleanly** when these env vars are absent (safe to run offline on macOS). It is the gate the operator runs on the VPN before trusting the cutover. -### KNOWN LIMITATION 2 — continuous-historization value-capture is not yet live +### KNOWN LIMITATION 2 — continuous-historization value-capture wired; live end-to-end verification still pending The `ContinuousHistorizationRecorder` is fully wired (actor + FasterLog outbox + gateway value-writer + -meters) but is currently spawned with an **EMPTY historized-ref set** (`Array.Empty()` in -`WithOtOpcUaRuntimeActors`): the deployed address space — and thus the set of historized tag refs — is built -later at deploy time, not at actor-spawn time, so there is no clean ref set to resolve at wiring time. With -an empty set the recorder **registers interest in nothing and historizes nothing**. **Reads and alarm-writes -work today**; the recorder's value-capture is the remaining gap, blocked on a `SetHistorizedRefs`-style feed -driven off the deployed composition (a tracked follow-on). Until that feed lands, continuous historization -records no values. +meters). It is spawned with an **empty *initial* ref set** (`Array.Empty()` in +`WithOtOpcUaRuntimeActors`) because the deployed address space — and thus the historized-ref set — is built +later at deploy time, not at actor-spawn time. **The ref-feed gap is now closed:** on every deploy the +`AddressSpaceApplier.FeedHistorizedRefs` hook computes the added/removed historized-ref delta and posts it +through `ActorHistorizedTagSubscriptionSink` → the recorder's `UpdateHistorizedRefs` message, and the +recorder keeps the full set and re-registers its dependency-mux interest. So the recorder converges to the +deployed set of historized value tags from the first deploy onward (the feed is a non-blocking, deploy-safe +`Tell`; a faulting feed can never break a deploy). **Reads and alarm-writes work today, and the value-capture +path is code-complete.** What remains is the **live gate**: an end-to-end verification against a real gateway +(deploy → dependency-mux value delta → outbox append → `WriteLiveValues`) plus a restart-convergence test +proving the recorder re-registers the deployed set after a process restart. Until that live verification runs, +treat continuous value-capture as unproven-in-production rather than absent. diff --git a/archreview/00-OVERALL.md b/archreview/00-OVERALL.md new file mode 100644 index 00000000..0115dbcc --- /dev/null +++ b/archreview/00-OVERALL.md @@ -0,0 +1,121 @@ +# OtOpcUa — Deep Architecture Review: Overall Report + +**Date:** 2026-07-08 +**Commit:** `9cad9ed0` (master, clean tree) +**Method:** Seven parallel domain reviews, each scoring the same four dimensions (stability, performance, conventions, underdeveloped areas) with severity-rated, file:line-referenced findings. This document is the synthesis; the detail lives in the per-domain reports. + +## Report index + +| # | Report | Scope | +|---|---|---| +| 01 | [Core & composition pipeline](01-core-composition.md) | Core, Core.Abstractions, Commons, Configuration, Cluster; AddressSpace Composer/Planner/Applier | +| 02 | [Scripting, virtual tags & alarm engines](02-scripting-alarms.md) | Core.Scripting(+Abstractions), Core.VirtualTags, Core.ScriptedAlarms, Core.AlarmHistorian | +| 03 | [Server host, runtime & OPC UA](03-server-runtime.md) | OpcUaServer, Host, Runtime, ControlPlane, Security; redundancy, LDAP, node manager | +| 04 | [AdminUI (Blazor)](04-adminui.md) | AdminUI pages, ScriptAnalysis, tag editors, authorization | +| 05 | [Protocol drivers](05-protocol-drivers.md) | Modbus, S7, AbCip, AbLegacy, TwinCAT, FOCAS, OpcUaClient (+Contracts, +Cli) | +| 06 | [Gateway integrations](06-gateway-integrations.md) | Driver.Galaxy(+Browser,+Contracts), Driver.Historian.Gateway | +| 07 | [Client, tooling & engineering system](07-client-tooling-engineering.md) | Client.CLI/Shared/UI, Analyzers, build/CPM, CI, test architecture, repo hygiene | + +## Consolidated maturity matrix (1–5) + +| Subsystem | Stability | Performance | Conventions | Underdeveloped | +|---|---|---|---|---| +| 01 Core & composition | 4 | 3 | 4 | 3 | +| 02 Scripting & alarms | 3.5 | 3 | 4 | **2** | +| 03 Server & runtime | 3 | 3 | 4 | 3 | +| 04 AdminUI | 4 | 3 | 3 | 3 | +| 05 Protocol drivers | 3 | **2** | 3 | 3 | +| 06 Gateway integrations | 4 | 4 | 4 | 3 | +| 07 Client & engineering | **2.5** | 3.5 | 3 | 3 | + +**Read of the matrix:** the system's *design* is strong — conventions score 3–4 everywhere, layering is disciplined (Akka-free OpcUaServer, SDK-free Runtime, clean gateway seams), and there is essentially zero TODO/stub debt. The weaknesses cluster in *operational* stability (failover, reconnect, timeout paths that only fire under fault) and in the *verification system* (CI proves little; distributed and fault-injection behavior is outside the test net). This is the profile of a codebase built by careful review + live-verification rather than by automated gates — it works, but its correctness under failure is largely unproven. + +--- + +## Critical findings (fix first) + +These four can each take down or silently corrupt production data flow, and none is caught by any existing test. + +1. **Split-brain resolver is configured but never activated** — the SBR block exists in `akka.conf:40-46`, but nothing sets `ClusterOptions.SplitBrainResolver` (`src/Core/ZB.MOM.WW.OtOpcUa.Cluster/ServiceCollectionExtensions.cs:80-84`), so Akka runs with default NoDowning. A hard-crashed node is never downed: **cluster singletons and the driver role-leader never fail over on a hard crash**, and a partition leaves both redundancy sides advertising ServiceLevel 240 indefinitely. Invisible to the 2-node harness because it only exercises graceful stops. *(Report 03, S1)* + +2. **Production virtual-tag script timeout is ineffective** — `Host/Engines/RoslynVirtualTagEvaluator.cs:102-106` calls `ScriptEvaluator.RunAsync` directly, bypassing `TimedScriptEvaluator` (which exists and is tested). Script execution is synchronous and only checks the token at entry, so one user-authored infinite loop hangs the owning `VirtualTagActor` forever. *(Report 02, U2)* + +3. **S7 driver has no reconnect path** — the PLC connection is opened only in `InitializeAsync` (`Driver.S7/S7Driver.cs:165-167`); the probe detects the outage but nothing repairs it. A PLC reboot permanently kills the driver instance until redeploy. Zero S7 reconnect tests. *(Report 05)* + +4. **TwinCAT native ADS subscriptions are orphaned after reconnect** — the default subscription mode is never re-registered on the replacement client after reconnect (`TwinCATDriver.cs:615-619`, `AdsTwinCATClient.cs:422-436`); data silently stops with no Bad status. *(Report 05)* + +## High-severity findings (system-wide risk ranking) + +**Data-integrity / failure-visibility group** — the system tends to report success it hasn't earned: + +- **Deploy applier swallows sink failures**: a deploy whose `RebuildAddressSpace()` threw still reports success to the deploy coordinator; `AddressSpaceApplyOutcome` has no failure field (`AddressSpaceApplier.cs:150-153, 373-383`). *(01, S-1)* +- **Galaxy write success is optimistic**: empty status arrays translate to Good, and a failed supervisory-advise still lets the write "proceed" though it can never reach the Galaxy — write-outcome self-correction structurally cannot fire for Galaxy (`GatewayGalaxyDataWriter.cs:163-169, 234-243, 281-302`). *(06, S-1)* +- **Primary gate defaults to allow while role is unknown** (`DriverHostActor.cs:1018-1026`): a booting secondary services field-device writes for up to a 10 s window; compounds with at-most-once DPS delivery. *(03, S4)* +- **Modbus transport desyncs permanently after a response timeout** — timeout/TxId-mismatch aren't classified socket-fatal, leaving the single-flight stream misaligned (`ModbusTcpTransport.cs:155-165, 233-235`). *(05)* +- **AbCip races concurrent ops on shared libplctag handles with no lock** (`AbCipDriver.cs:544-575`) — torn values with Good status; AbLegacy guards this exact hazard. FOCAS has the sibling bug on plain-Dictionary caches. *(05)* + +**Availability / blocking group:** + +- **LDAP authentication block-bridges OPC UA session activation** with an unconfigurable 10 s connect timeout and no pooling (`OpcUaApplicationHost.cs:271-273`) — an LDAP outage serially stalls SDK threads. *(03, S2)* +- **HistoryRead block-bridges per node sequentially** on SDK request threads bounded only by the 30 s gateway timeout — large batches × slow historian can exhaust the request pool (`OtOpcUaNodeManager.cs:1902-2203`). *(03, S3)* +- **Any structural deploy change (one added tag) full-rebuilds the address space and kills every client subscription server-wide** — the surgical path covers only attribute edits/renames. Surgical pure-adds are the highest-leverage performance fix in the repo. *(03, P1)* +- **Evaluator-cache ALC accretion**: the live path caches script evaluators in a raw `ConcurrentDictionary` never cleared on republish, leaking collectible AssemblyLoadContexts across script edits; the purpose-built `CompiledScriptCache` is unused. *(02, U3)* + +**Security / authorization group:** + +- **AdminUI's largest mutating surface is effectively ungated** — GlobalUns, EquipmentPage, ClusterEdit, NodeEdit, all 8 driver pages, AclEdit, and Reservations carry only bare `[Authorize]`: any authenticated user, including read-only, can mutate UNS/driver/ACL config. Authorization is a three-idiom mix (bare / role-string / policy). *(04, #1)* + +**Verification-system group:** + +- **CI gates only 5 of 47 test projects** (`.github/workflows/v2-ci.yml:47-52`); Client (388 tests), Tooling, all driver and most Core suites never run in CI. The integration job runs fixture-less, so probes `Assert.Skip` and the badge is green — "skipped" is indistinguishable from "passed". *(07, S-1/S-2)* +- **The sole custom analyzer (OTOPCUA0001) is referenced by zero consuming projects** — the CapabilityInvoker-wrapping rule it enforces is inert in every build. One `ProjectReference` in `Directory.Build.props` fixes it. *(07, C-1)* + +--- + +## Cross-cutting themes + +These emerged independently from multiple reviewers and are the real architectural signal — individual bugs are symptoms of these. + +### 1. "Built-but-never-wired" is the house failure mode +At least six independent instances: the dormant `GatewayTagProvisioner` (PR #423, since fixed), the F10b `DeferredAddressSpaceSink` inertness bug (fixed), the analyzer wired to zero projects, `GatewayHistorianDataSource.RefreshConnectionStateAsync` with no production caller (health flags permanently false), the SBR HOCON block with no activating registration, and the entire ~2.4k-line `Core.VirtualTags` engine stack that production bypasses in favor of `VirtualTagHostActor` + `RoslynVirtualTagEvaluator` — which is precisely why the timeout and cache defenses (built into the dormant engine's path) are missing from the live one. **Recommendation:** every hook/capability/config block needs a *production-caller* assertion, not just unit tests of the component — e.g. a reflection-exhaustive forwarding test for `DeferredAddressSpaceSink` (only ~5 of 10 members are asserted today), a startup log line proving SBR is active, and wiring the analyzer repo-wide since it was built to catch exactly this class. + +### 2. Fixes don't flow back to shared templates +Every High in the driver fleet is a fix present in one sibling and absent in another: S7 forked the poll loop (gaining backoff the shared `PollGroupEngine` lacks, keeping a CTS race the engine fixed); AbLegacy guards the handle race AbCip has; OpcUaClient's reconnect machine is exemplary while S7 has none; the `ReadEnum` helper is copy-pasted 6×; no driver passes `onError` to the poll engine so its error sink is dead fleet-wide. The same pattern exists in Core as **duplication-by-convention**: TagConfig intent parsing is byte-parity-replicated in four projects, synced only by "MUST parse identically" comments. **Recommendation:** consolidate into `TagConfigIntent.Parse` in Commons (the `EquipmentScriptPaths` precedent) and treat `PollGroupEngine` + a shared reconnect/backoff primitive as the single home for fleet fixes. + +### 3. Optimistic success — failure signals don't reach operators +The "Safe*" swallow-and-log wrapper pattern (never fail the caller) recurs across the applier, Galaxy writes, and the primary gate's default-allow. Individually defensible; together they mean a failed deploy reports `rebuilt=true`, a lost Galaxy write reports Good, and a booting secondary accepts writes. **Recommendation:** audit each swallow site for an operator-visible signal (health flag, meter, alert row) — the pattern is fine only when the failure surfaces *somewhere*. + +### 4. The verification system trusts live-verification that CI can't reproduce +Repo-wide: no bUnit (razor `@code` logic is only live-verified), CI runs ~10% of suites, integration tier skip-gates to green, DPS delivery/actor supervision/hard-kill failover have zero tests, `OpcUaServer.IntegrationTests` has exactly one test, and driver test debt concentrates on reconnect/concurrency — exactly where the four Criticals live. The project's own history (F10b, redundancy-state delivery, enum serialization) shows live-`/run` catching what 250+ green unit tests missed. **Recommendation:** expand `v2-ci.yml` to all unit suites (cheap, immediate), make skips visible (report skipped counts as a failure condition for the integration job), and add a small fault-injection tier (hard-kill a node, drop a PLC connection) even if only run nightly against the docker host. + +### 5. Documentation drift on load-bearing claims +CLAUDE.md's KNOWN LIMITATION 2 (empty historized-ref set) is **stale — closed in code** (applier `FeedHistorizedRefs` → recorder `UpdateHistorizedRefs`), confirmed independently by two reviewers; ScriptAnalysis endpoints are documented as FleetAdmin-gated but use `Roles="Administrator,Designer"`; `docs/Client.CLI.md` documents 8 of 14 commands. **Recommendation:** update CLAUDE.md now (and add the missing restart-convergence test + live value-capture verification that the limitation's closure deserves). + +### 6. Equipment-tag (TagConfig-JSON) paths are second-class citizens +Across drivers, the equipment-tag parse/resolve path lags the authored-tag path: silent enum defaulting on typos, forced `Writable:true` (FOCAS is read-only yet defaults writable), `ResolveHost` mis-keying refs to the first device (breaking per-host breaker isolation in 4 drivers), and capability gates bypassed. Since Galaxy-standard-driver made TagConfig the primary authoring surface, this path *is* the product. **Recommendation:** a dedicated hardening pass on the `EquipmentTagRefResolver` seam across all drivers, plus the parse-strictness unification from theme 2. + +### 7. Repo hygiene debt +Retired `Historian.Wonderware*` directories still on disk (absent from slnx); 7 committed proprietary AVEVA DLLs in `lib/` referenced by nothing (license/redistribution risk); `pending.md`/`current.md` tracked despite pending.md's own "never stage" rule; five stale planning files at root. (`sql_login.txt` was verified NOT committed — gitignored, local-only.) + +--- + +## Prioritized action list + +| # | Action | Source | Effort | Impact | +|---|---|---|---|---| +| 1 | Activate the split-brain resolver (`ClusterOptions.SplitBrainResolver`) + a hard-kill failover test | 03 S1 | Small | Critical — failover currently doesn't work | +| 2 | Route production VT evaluation through `TimedScriptEvaluator` + `CompiledScriptCache` | 02 U2/U3 | Small | Critical — user script can hang an actor; ALC leak | +| 3 | Add S7 reconnect + TwinCAT ADS re-registration after reconnect (use OpcUaClient's machine as the template) | 05 | Medium | Critical — PLC power-cycle currently defeats both | +| 4 | Classify Modbus timeout/TxId-mismatch as socket-fatal; lock AbCip handles (copy AbLegacy's guard) | 05 | Small | High — silent data corruption/desync | +| 5 | Gate AdminUI mutating pages with real policies (one authz idiom, constants not literals) | 04 | Medium | High — any authenticated user can mutate config | +| 6 | Wire OTOPCUA0001 analyzer repo-wide; expand CI to all unit suites; fail-on-skip for integration | 07 | Small | High — restores meaning to green CI | +| 7 | Add a failure field to `AddressSpaceApplyOutcome`; surface Galaxy write-loss and gate default-deny on unknown role | 01/06/03 | Medium | High — failure visibility | +| 8 | Surgical pure-adds in the address-space applier (stop full rebuilds killing all subscriptions per tag add) | 03 P1 | Large | High — biggest operational perf win | +| 9 | Async LDAP + configurable timeout; channel-ize HistoryRead bridging | 03 S2/S3 | Medium | Medium-High — outage resilience | +| 10 | Consolidate TagConfig parsing (`TagConfigIntent.Parse` in Commons) + driver `ReadEnum`/parse-strictness unification + `ResolveHost` fix | 01 C-1 / 05 | Medium | Medium — kills the byte-parity duplication class | +| 11 | Update CLAUDE.md (Known Limitation 2 closed; ScriptAnalysis policy) + Client.CLI docs; delete Wonderware dirs, `lib/` AVEVA DLLs, untrack planning files | 03/04/06/07 | Small | Medium — doc trust + license risk | +| 12 | Retire or converge the dormant `Core.VirtualTags` engine; delete `MemoryRecycle`/`IDriverSupervisor` dead machinery | 02 U1 / 01 U-2 | Medium | Medium — removes the built-vs-runs divergence | + +## Overall assessment + +OtOpcUa is a well-architected system with unusually disciplined conventions, clean layering, strong seams, and near-zero TODO debt — the design work is mature. Its exposure is concentrated in three places: **fault paths** (failover, reconnect, timeout — where all four Criticals live and where test coverage is thinnest), **failure visibility** (a systematic bias toward optimistic success), and **the verification gap** (CI proves ~10% of what the test suite could; the "built-but-never-wired" pattern has now recurred six times because nothing asserts production wiring). The prioritized list above front-loads small, high-leverage fixes: items 1–6 are roughly a week of work and eliminate every Critical plus the two systemic guards (analyzer + CI) that prevent the pattern from recurring. diff --git a/archreview/01-core-composition.md b/archreview/01-core-composition.md new file mode 100644 index 00000000..274c9936 --- /dev/null +++ b/archreview/01-core-composition.md @@ -0,0 +1,372 @@ +# Architecture Review — Core Composition Pipeline and Core Libraries + +**Date:** 2026-07-08 +**Commit:** `9cad9ed0` (master) +**Reviewer scope:** + +- `src/Core/ZB.MOM.WW.OtOpcUa.Core` (driver hosting, resilience, stability, authorization trie) +- `src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions` (driver capability interfaces, `PollGroupEngine`, historian seams) +- `src/Core/ZB.MOM.WW.OtOpcUa.Commons` (cluster message contracts, deferred sinks, NodeId scheme, telemetry) +- `src/Core/ZB.MOM.WW.OtOpcUa.Configuration` (EF config persistence, generation-sealed local cache, draft validation) +- `src/Core/ZB.MOM.WW.OtOpcUa.Cluster` (Akka bootstrap, role info, ServiceLevel) +- Plus the address-space composition pipeline itself — `AddressSpaceComposer` / `AddressSpacePlanner` / `AddressSpaceApplier` + +**Scope note (location drift).** The review brief places the composition pipeline in +`ZB.MOM.WW.OtOpcUa.Core`, but since the Phase7→AddressSpace rename it lives in +`src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/` (`AddressSpaceComposer.cs`, `AddressSpacePlan.cs`, +`AddressSpaceApplier.cs`), with its tests under `tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests/`. +The `Core` project's own `OpcUa/` folder now holds only the retired `EquipmentNodeWalker` and an +unused `GenericDriverNodeManager` (see U-1). The pipeline was reviewed in full regardless. + +--- + +## Architecture Overview + +### Composition pipeline (deploy → address space) + +The pipeline is a clean three-stage compose/diff/apply design: + +1. **`AddressSpaceComposer.Compose`** (`OpcUaServer/AddressSpaceComposer.cs:290`) — a pure static + projection from EF config entities (`UnsArea`, `UnsLine`, `Equipment`, `DriverInstance`, `Tag`, + `VirtualTag`, `Script`, `ScriptedAlarm`, `Device`, `Namespace`) into an + `AddressSpaceComposition`: sorted lists of `UnsAreaProjection`/`UnsLineProjection`/ + `EquipmentNode`/`DriverInstancePlan`/`EquipmentTagPlan`/`EquipmentVirtualTagPlan`/ + `EquipmentScriptedAlarmPlan`. Tag intent (driver `FullName`, native-alarm object, historize + flags, array shape) is parsed out of the schemaless `Tag.TagConfig` JSON here. VirtualTag + scripts get `{{equip}}` token substitution and dependency-ref extraction via the shared + `Commons.Types.EquipmentScriptPaths` helper. Everything is ordinally sorted so the composition + is deterministic — the foundation of the "byte-parity" contract with the artifact-decode mirror + in `Runtime/Drivers/DeploymentArtifact.cs`. + +2. **`AddressSpacePlanner.Compute`** (`OpcUaServer/AddressSpacePlan.cs:96`) — a pure diff of two + compositions keyed on stable logical ids, using record value-equality as the changed check + (with hand-written `Equals`/`GetHashCode` on the plans carrying `IReadOnlyList` members so a + no-op redeploy diffs empty). Emits Added/Removed/Changed sets per entity class plus UNS + folder renames. `AddressSpacePlan.IsEmpty` is the short-circuit gate `OpcUaPublishActor` uses. + +3. **`AddressSpaceApplier.Apply`** (`OpcUaServer/AddressSpaceApplier.cs:73`) — the only + side-effecting stage. Decides between a **full structural rebuild** (any topology change) and a + **surgical in-place update** (`ISurgicalAddressSpaceSink.UpdateTagAttributes` / + `UpdateFolderDisplayName` for whitelisted node-irrelevant deltas — the F10b optimization), + falling back to rebuild when the sink lacks the capability or any surgical call fails. After + the address-space work it dispatches two non-blocking hooks: fire-and-forget historian tag + provisioning (`IHistorianProvisioning.EnsureTagsAsync`) and the historized-ref delta feed to + the continuous-historization recorder (`IHistorizedTagSubscriptionSink.UpdateHistorizedRefs`). + Separate `Materialise*` passes (Hierarchy / EquipmentTags / VirtualTags / ScriptedAlarms / + DiscoveredNodes) re-derive nodes from the composition after a rebuild; NodeIds are + folder-scoped (`Commons.OpcUa.EquipmentNodeIds`: `{equipment}/{folder}/{name}`), never the + driver `FullName`, to avoid collisions across identical machines. + +The sink boundary (`Commons.OpcUa.IOpcUaAddressSpaceSink`) keeps the whole pipeline SDK-free: +production binds `SdkAddressSpaceSink` via the late-swap `DeferredAddressSpaceSink` (actors resolve +the wrapper at DI time, the OPC UA hosted service swaps the real sink in after `StandardServer` +starts); dev/tests bind the `NullOpcUaAddressSpaceSink` no-op. + +### Supporting Core libraries + +- **`Core/Hosting`** — `DriverFactoryRegistry` (driver-type → factory), `DriverFactoryRegistryAdapter` + (v1 registry → v2 `IDriverFactory`), `DriverHost` (id → `IDriver` lifecycle registry). +- **`Core/Resilience`** — Polly pipelines cached per `(DriverInstanceId, HostName, Capability)` + (`DriverResiliencePipelineBuilder`), executed via `CapabilityInvoker` (with the + non-idempotent-write no-retry override), fanned out per host by `AlarmSurfaceInvoker`, observed + by `DriverResilienceStatusTracker` for Admin `/hosts`. +- **`Core/Stability`** — `MemoryTracking` (median-baseline soft/hard breach), `MemoryRecycle` and + `ScheduledRecycleScheduler` (Tier C recycle via `IDriverSupervisor`), `WedgeDetector` + (demand-aware stall detection). +- **`Core/Authorization`** — generation-sealed `PermissionTrie` per `(ClusterId, GenerationId)` + cached in `PermissionTrieCache` (CAS-pruned), walked by `TriePermissionEvaluator` against + per-session `UserAuthorizationState` (freshness window + fail-closed staleness ceiling). +- **`Commons`** — immutable record message contracts for the Akka DPS topics (deploy dispatch/ack, + alerts `AlarmTransitionEvent`, redundancy state, admin ops, script log), strongly-typed ids + (`DeploymentId`, `RevisionHash`, `CorrelationId`, `NodeId`), the deferred sink/publisher pair, + `EquipmentScriptPaths` (shared script-path/dependency extraction), and `OtOpcUaTelemetry` + (central Meter/ActivitySource). +- **`Configuration`** — `OtOpcUaConfigDbContext` (26 entities, logical-id unique indexes, JSON + check constraints, RowVersion concurrency), the generation-sealed LiteDB fallback cache + (`GenerationSealedCache` + atomic `CURRENT` pointer, fail-closed on corruption), + `ResilientConfigReader` (timeout→retry→cache-fallback with secret-scrubbed logging), + `DraftValidator` (all-errors-in-one-pass pre-publish rules). +- **`Cluster`** — `AddOtOpcUaCluster` / `WithOtOpcUaClusterBootstrap` (Akka.Hosting bootstrap with + embedded HOCON + Serilog logger wiring), `ClusterRoleInfo` (lock-guarded role topology snapshot + fed by a subscriber actor), `ServiceLevelCalculator` (pure 0–255 tiering), `RoleParser`. + +Dependency direction: `Core.Abstractions` (leaf) ← `Configuration` ← `Core`; `Commons` (leaf, +Akka+Audit packages) ← `Cluster`. The composer/applier in `OpcUaServer` reference both +`Configuration` (entities) and `Commons` (sink contracts). + +--- + +## Findings + +### 1. Stability + +#### S-1 — Applier swallows every sink failure; a deploy is reported applied even when the address space is broken — **High** + +`AddressSpaceApplier` wraps every sink call in catch-all "Safe" helpers that log and continue: +`SafeEnsureFolder`/`SafeEnsureVariable` (`AddressSpaceApplier.cs:612–622`), `SafeRebuild` +(`AddressSpaceApplier.cs:373–383`), `SafeWriteAlarmCondition`/`SafeMaterialiseAlarmCondition` +(`AddressSpaceApplier.cs:677–687`). `AddressSpaceApplyOutcome` (`AddressSpaceApplier.cs:691`) +carries only Added/Removed/Changed counts and `RebuildCalled` — **no failure count**. If +`RebuildAddressSpace()` throws, `rebuilt` is still reported `true` (`AddressSpaceApplier.cs:150–153`) +and the outcome flows back to the deploy coordinator as a success; the deployment seals while the +running server holds a partially-materialised (or entirely stale) address space. Individual +`EnsureVariable` failures during a materialise pass likewise vanish into per-node Warnings. + +The never-fail-a-deploy posture is defensible for the *detached* hooks (provisioning, historized-ref +feed — both correctly isolated), but structural materialisation failures are the deploy's core +contract. + +**Recommendation:** add `FailedNodes`/`RebuildFailed` to `AddressSpaceApplyOutcome`, propagate into +the `ApplyAck`/`DeploymentFailed` decision and the audit log, and escalate `SafeRebuild` failure to +at least a degraded ack. Tests should assert the failure surface (the fixture sink can throw). + +#### S-2 — `PollGroupEngine.Unsubscribe` blocks the calling thread up to 5 s per subscription — **Medium** + +`StopState` does a synchronous `task.Wait(TimeSpan.FromSeconds(5))` +(`Core.Abstractions/PollGroupEngine.cs:99–112`). `Unsubscribe` is the teardown path drivers call +from OPC UA subscription-management callbacks and actor message handlers; a reader stuck in a slow +network call turns every unsubscribe into a 5-second stall of the calling thread (and N stalls when +tearing down N subscriptions serially). `DisposeAsync` gets this right (parallel cancel, awaited +`WhenAll` with one shared timeout, `PollGroupEngine.cs:213–236`); the single-subscription path does +not. + +**Recommendation:** add an `UnsubscribeAsync` (or make `Unsubscribe` cancel + hand the await to a +background drain), keeping the "no callback after teardown" guarantee via the existing +LoopTask/CTS-dispose ordering. + +#### S-3 — `PollGroupEngine.Subscribe` has no disposed guard — **Low** + +`Subscribe` racing `DisposeAsync` can insert a fresh `SubscriptionState` after the dispose loop has +snapshotted `_subscriptions.Values` (`PollGroupEngine.cs:73–84` vs `213–236`), leaking a live poll +loop with no owner. Same pattern in `DriverHost.RegisterAsync` vs `DisposeAsync` +(`Core/Hosting/DriverHost.cs:53–71` vs `92–106`): a registration that wins the race after the +snapshot+clear is initialized but never shut down. Both are shutdown-window races, unlikely in +practice but cheap to close with a `_disposed` flag checked inside the lock / before insert. + +#### S-4 — `ScheduledRecycleScheduler` catch-up recycle storm after a stall — **Low** + +`TickAsync` advances `_nextRecycleUtc` by exactly one interval per fire +(`Core/Stability/ScheduledRecycleScheduler.cs:72–84`). If the host was suspended (VM pause, long +outage) across K intervals, the next K ticks each trigger a full Tier C process recycle +back-to-back. **Recommendation:** fast-forward `_nextRecycleUtc` past `utcNow` after a fire. + +#### S-5 — `GenericDriverNodeManager` re-walk teardown is not synchronized — **Low** + +`BuildAddressSpaceAsync` tears down the previous alarm forwarder and clears `_alarmSinks` +non-atomically (`Core/OpcUa/GenericDriverNodeManager.cs:58–71`); a concurrent driver alarm event +during the window between unsubscribe and re-subscribe is dropped, and two concurrent Build calls +would interleave badly. Currently moot because the class has no production caller (see U-1) — fold +this into whatever decision is made there. + +**Positive observations.** The failure-mode engineering in `Configuration/LocalCache` is exemplary: +`GenerationSealedCache` writes temp-file + atomic `File.Replace` pointer swaps, fails closed on any +corruption rather than silently serving an older generation (`GenerationSealedCache.cs:111–155`); +`LiteDbConfigCache` documents and defuses two real LiteDB concurrency traps (global `BsonMapper` +races, find-then-insert upsert races) (`LiteDbConfigCache.cs:16–36`). `ResilientConfigReader` +correctly distinguishes SQL-command `TaskCanceledException` from genuine caller cancellation +(`ResilientConfigReader.cs:117–138`) and scrubs connection-string secrets from log messages. +`PermissionTrieCache.Prune` is a correct reference-equality CAS loop (`PermissionTrieCache.cs:73–102`). +The applier's two historian hooks are properly detached (continuation observes the task, never +`.Result` on a fault — `AddressSpaceApplier.cs:259–286`). `PollGroupEngine` classifies fatal +exceptions and hardens the `onError` callback (`PollGroupEngine.cs:153–169`). + +### 2. Performance + +#### P-1 — `Compose` parses each tag's `TagConfig` JSON four times — **Medium** + +`ExtractTagFullName`, `ExtractTagAlarm`, `ExtractTagHistorize` and `ExtractTagArray` each do their +own `JsonDocument.Parse` of the same blob per tag (`AddressSpaceComposer.cs:418–433` calling +`:536–686`). A compose over a 10k-tag fleet performs 40k JSON parses (plus the same again on the +artifact-decode side). Compose runs on every deploy and every diff baseline, not on the data hot +path, so this is a scalability tax rather than a latency bug today. + +**Recommendation:** parse once per tag into a small `TagConfigIntent` record (which also collapses +the byte-parity duplication — see C-1). + +#### P-2 — Per-operation allocations on the authorization hot path — **Medium** + +`PermissionTrie.CollectMatches` allocates a fresh `HashSet` from the session's LDAP groups +plus a `List` on every call (`Core/Authorization/PermissionTrie.cs:42–45`), and +`TriePermissionEvaluator.Authorize` runs per node-operation. A recursive browse or a large +CreateMonitoredItems batch turns this into per-node garbage. `UserAuthorizationState.LdapGroups` is +immutable per membership refresh — the case-insensitive set can be computed once per session +version and reused. + +#### P-3 — `PollGroupEngine`: one background task + independent timer per subscription, no cross-subscription batching — **Medium** + +Each `Subscribe` spawns a dedicated `Task.Run` loop (`PollGroupEngine.cs:73–84`). Drivers that map +each OPC UA monitored-item group (or each poll group) to a subscription get linear task/timer +growth, and two subscriptions polling overlapping tag sets at the same interval issue duplicate +protocol reads — the engine has no interval-bucketed scheduler or read coalescing. Fine at dozens +of subscriptions; a scaling wall at hundreds per driver instance. Worth a note in the driver-facing +docs at minimum; an interval-bucketed shared loop is the structural fix if tag counts grow. + +#### P-4 — Per-write allocation in the non-idempotent write path — **Low** + +`CapabilityInvoker.ExecuteWriteAsync` builds a new options record + `Dictionary` per non-idempotent +write (`Core/Resilience/CapabilityInvoker.cs:135–152`) even though `GetOrCreate` keys only on +`(id, host, capability)` and ignores the options after the first build — the comment even cites the +"1% pipeline budget". Cache the no-retry snapshot per options generation, or key the fast path on a +precomputed pipeline. + +**Positive observations.** Pipeline resolution is lock-free `ConcurrentDictionary` reads +(`DriverResiliencePipelineBuilder.cs:58–71`). The planner's diff is O(N) dictionary passes with +deterministic sorted outputs. The surgical-apply path exists precisely to avoid the expensive full +rebuild + subscription teardown for attribute-only edits, and its whitelist-via-`with`-expression +technique (`AddressSpaceApplier.cs:630–663`) is future-field-safe (unknown new fields force the +safe rebuild). Telemetry instruments are no-op until a listener attaches (`OtOpcUaTelemetry.cs`). + +### 3. Conventions + +#### C-1 — "Byte-parity by convention": TagConfig parsing replicated in four projects with comment-enforced sync — **High** + +`ExtractTagFullName` (and friends) exist as deliberate copies in: + +- `OpcUaServer/AddressSpaceComposer.cs:536–551` (+ `ExtractTagAlarm`/`ExtractTagHistorize`/`ExtractTagArray`/`TryExtractDeviceHost`) +- `Runtime/Drivers/DeploymentArtifact.cs` (the artifact-decode mirror, per its own comments) +- `Core/OpcUa/EquipmentNodeWalker.cs:193–208` +- `Configuration/Validation/DraftValidator.cs:60–71` (a fourth "small local copy, consistent with + this codebase where the composer keeps its own") + +Each copy carries a "MUST parse identically (byte-parity)" comment. The equality of the deployed +address space across the live-compose and artifact-decode seams — and therefore the correctness of +the diff/no-op-redeploy behavior — rests on humans keeping four JSON parsers in sync. The codebase +already demonstrated the fix once: `EquipmentScriptPaths` (`Commons/Types/EquipmentScriptPaths.cs`) +was created exactly to de-duplicate the `ctx.GetTag` extraction "those two seams used to duplicate". + +**Recommendation:** move the TagConfig/DeviceConfig intent parsing into `Commons` (it has no EF or +SDK dependency; every current copy's host project already references Commons or could) as a single +`TagConfigIntent.Parse(string)` — also resolving P-1 — and keep one cross-seam parity test instead +of N replicated implementations. + +#### C-2 — Core depends on Configuration: EF entities and enums are the domain model — **Medium** + +`ZB.MOM.WW.OtOpcUa.Core.csproj` references `Configuration`; `PermissionTrie` consumes +`Configuration.Enums.NodePermissions`/`NodeAclScopeKind` (`Core/Authorization/PermissionTrie.cs:1`), +`EquipmentNodeWalker` consumes `Configuration.Entities` directly, and the composer's plan records +are projections of EF entities. There is no separate domain layer — persistence types *are* the +model. This is a deliberate, consistent choice (and the plan records do decouple the applier), but +it means: (a) any EF-driven entity change ripples straight into authorization and composition +logic; (b) `Configuration` (with EF Core, SqlServer, LiteDB, DataProtection packages) is pulled +into anything referencing Core. Flagging as accepted-risk to document rather than refactor — +new code should keep preferring the plan-record boundary the composer established. + +#### C-3 — `DraftValidator` re-derives the NodeId scheme and hard-codes a driver-type string — **Medium** + +`ValidateNoEquipmentSignalNameCollision` re-implements the folder-scoped NodeId key by hand +(`Configuration/Validation/DraftValidator.cs:75–97`) because `Configuration` cannot reference +`Commons.OpcUa.EquipmentNodeIds` (dependency direction) — a third copy of the +`{equipment}/{folder}/{name}` scheme whose drift would silently break the collision check. +`ValidateGalaxyTagFullName` hard-codes `dtype != "GalaxyMxGateway"` +(`DraftValidator.cs:41–55`) rather than a shared driver-type constant. Both are symptoms of the +same missing shared-contract home as C-1; a Commons-level scheme/constants type (or moving +`EquipmentNodeIds` down to a project Configuration can see) fixes both. + +#### C-4 — Composer violates its own purity contract with `Trace.TraceWarning` — **Low** + +The class doc says "Same inputs → same outputs, no logging" (`AddressSpaceComposer.cs:280–283`), +but the dangling-predicate-script skip emits `Trace.TraceWarning` (`AddressSpaceComposer.cs:493–496`) +— the only `System.Diagnostics.Trace` usage in a Serilog/`ILogger` codebase, invisible in the +production sinks. Either return skipped-alarm ids in the composition for the caller to log, or +accept an injected logger and fix the doc. + +#### C-5 — Inconsistent duplicate-key defensiveness inside `Compose` — **Low** + +`deviceHostById` is built with a deliberate last-wins `foreach` and a comment explaining why +`ToDictionary` (throw-on-dupe) would diverge from the decode side (`AddressSpaceComposer.cs:352–364`), +yet three lines later `driversById`/`namespacesById`/`scriptsById` use plain `ToDictionary` +(`AddressSpaceComposer.cs:401–402, 453`) — a duplicate logical id (only possible on DB-constraint +bypass) crashes the whole compose. Pick one posture; the defensive one is already argued for. + +#### C-6 — Vestigial `Akka` package reference in Commons — **Low** + +`Commons.csproj` references the `Akka` package but no Commons source uses an Akka type (only doc +comments mention actors). Harmless today because every consumer is Akka-hosted anyway, but it +contradicts Commons' role as the dependency-light contracts layer. Remove it. + +#### C-7 — Stale scaffold-era XML docs on load-bearing classes — **Low** + +`AddressSpaceApplier`'s class doc still describes the F14 scaffold ("For now we record the work", +"the SDK adapter that lands in F10b will decide…" — `AddressSpaceApplier.cs:7–26`) although both +features shipped. `DriverHost`'s doc promises "per-process isolation for Tier C … implemented in +Phase 2 via named-pipe RPC" (`Core/Hosting/DriverHost.cs:5–10`), an architecture superseded by the +out-of-repo mxaccessgw gateway. In a codebase whose XML documentation is otherwise a genuine asset +(and evidently relied on), stale top-of-class narratives are actively misleading. + +**Positive observations.** Naming and structure are highly consistent (logical-id + RowVersion +entity pattern, `Null*`/`Deferred*` sink pattern, options records with `Validate()`, message +records per DPS topic). Zero `TODO`/`HACK`/`FIXME`/`Obsolete` markers across all five projects. The +DbContext matches a documented schema with an introspection compliance test +(`SchemaComplianceTests`). The `DeferredAddressSpaceSink` explicitly forwards the surgical +capability with a comment memorializing the F10b prod-inertness bug (`DeferredAddressSpaceSink.cs:52–70`) +— institutional memory encoded where the next person will trip. + +### 4. Underdeveloped Areas + +#### U-1 — Dead/dormant code retained in Core: `GenericDriverNodeManager`, `EquipmentNodeWalker`, `TryParseRelayBody` — **Medium** + +- `GenericDriverNodeManager` (`Core/OpcUa/GenericDriverNodeManager.cs`) has **no production + references** — only its own test file. The production path is the composer→applier→sink chain. +- `EquipmentNodeWalker` (`Core/OpcUa/EquipmentNodeWalker.cs:160–165`) self-declares "retained for + unit-test support only; real server deployments never invoke Walk" — ~280 lines plus + `IdentificationFolderBuilder` and a full test suite exercising a code path that cannot run in + production. Its `ExtractFullName` copy is also one of the four C-1 duplicates. +- `EquipmentScriptPaths.TryParseRelayBody` (`Commons/Types/EquipmentScriptPaths.cs:164–172`) + serviced the relay→alias converter that was deleted when Galaxy became a standard driver + (2026-06-12); only tests call it now. + +Dormant code with green tests is worse than deleted code: it passes every gate while silently +diverging from the live path. **Recommendation:** delete the walker + GDNM + relay parser (git +preserves them), or, if the walker must survive as a test fixture, move it into the test project. + +#### U-2 — The entire Tier C recycle machinery has no `IDriverSupervisor` implementation — **Medium** + +`MemoryRecycle`, `ScheduledRecycleScheduler`, and `DriverResilienceStatusTracker.RecordRecycle` +all drive `IDriverSupervisor.RecycleAsync`, but a repo-wide search finds **zero concrete +implementations** of `IDriverSupervisor` (`Core.Abstractions/IDriverSupervisor.cs` + the two +Stability consumers are the only references). Since the out-of-process driver story moved to the +external mxaccessgw/HistorianGateway sidecars, no in-repo driver is Tier C, so the recycle path is +compiled, tested, and unreachable. Either document it as speculative infrastructure for a future +tier-migration workflow (the `MemoryRecycle` doc hints at one) or prune it alongside U-1. + +#### U-3 — Continuous-historization interest set: delta feed landed, initial set still empty — **Medium (known-limitation debt)** + +The applier now pushes add/remove historized-ref deltas per deploy +(`AddressSpaceApplier.cs:310–353` → `IHistorizedTagSubscriptionSink`), which is the convergent +design for *subsequent* deploys — but per the documented KNOWN LIMITATION 2 (CLAUDE.md), the +recorder is still spawned with an empty ref set, so after a process restart the recorder historizes +nothing until the next deploy produces a delta. The seam contract +(`Core.Abstractions/Historian/IHistorizedTagSubscriptionSink.cs`) supports the fix; the missing +piece is a full-set replay at spawn/rebuild (e.g. have the applier feed the *complete* current set +on `RebuildCalled`, not just the diff). This is the highest-leverage half-finished feature in the +subsystem. + +#### U-4 — Additive-only ACL model (no Deny) is a documented v2.0 gap — **Low** + +`PermissionTrie` is explicitly pure-union ("no explicit Deny in v2.0", +`Core/Authorization/PermissionTrie.cs:13–17`): a broad cluster-root grant cannot be carved back at +a sub-scope. Acceptable for the current flat-role deployments (AdminUI roles are global by design), +but it bounds how fine-grained OT authorization can get without a trie-walk semantics change — +worth keeping visible in the security docs. + +#### U-5 — Test coverage is strong but unevenly distributed — **Low** + +Test method counts: Core.Tests 208, Configuration.Tests 94, Core.Abstractions.Tests 65, +Commons.Tests 45, Cluster.Tests 21. The composition pipeline itself is thoroughly covered (10+ +dedicated files in `tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests/AddressSpace*`). Gaps: +`ClusterRoleInfo` — the only concurrency-bearing class in the Cluster project (lock + subscriber +actor + event fan-out) — has no test (Cluster.Tests covers only the three pure classes); +`DriverHost` dispose/register races (S-3) and the applier failure surface (S-1) are untested +because the surface doesn't exist. An Akka TestKit harness for `ClusterRoleInfo` leader-change +sequencing would close the riskiest untested seam. + +--- + +## Maturity Ratings + +| Dimension | Rating (1–5) | Justification | +|---|---|---| +| Stability | **4** | Deliberate, well-commented failure-mode engineering throughout (fail-closed caches, atomic pointer swaps, detached hooks, fatal-exception classification); docked for the applier's swallow-everything success reporting (S-1) and the blocking unsubscribe (S-2). | +| Performance | **3** | Deploy-path costs are deterministic and diffed to near-zero for no-op redeploys, and the surgical-apply path avoids rebuild churn; but hot paths carry avoidable per-operation allocations (P-2), redundant JSON parsing scales 4× with tag count (P-1), and the poll engine is task-per-subscription with no coalescing (P-3). | +| Conventions | **4** | Exceptionally consistent naming, patterns, and XML docs with zero TODO debt; docked for the systemic replicate-and-comment "byte-parity" duplication (C-1/C-3) and the Core→Configuration persistence coupling (C-2). | +| Underdeveloped areas | **3** | The live pipeline is feature-complete and well-tested, but Core retains three dead/dormant code paths with green tests (U-1), a fully-built-but-unreachable Tier C recycle subsystem (U-2), and the known continuous-historization initial-set gap (U-3). | diff --git a/archreview/02-scripting-alarms.md b/archreview/02-scripting-alarms.md new file mode 100644 index 00000000..e1918095 --- /dev/null +++ b/archreview/02-scripting-alarms.md @@ -0,0 +1,196 @@ +# Architecture Review 02 — Scripting, Virtual Tags, Scripted Alarms, Alarm Historian + +- **Date:** 2026-07-08 +- **Commit:** `9cad9ed0` +- **Scope:** + - `src/Core/ZB.MOM.WW.OtOpcUa.Core.Scripting` + `Core.Scripting.Abstractions` (Roslyn compile/eval engine) + - `src/Core/ZB.MOM.WW.OtOpcUa.Core.VirtualTags` (virtual-tag runtime) + - `src/Core/ZB.MOM.WW.OtOpcUa.Core.ScriptedAlarms` (scripted-alarm engine + Part 9 state machine) + - `src/Core/ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian` (SQLite store-and-forward sink) + - Test projects under `tests/Core/` for coverage assessment +- **Method:** full read of the load-bearing files (ScriptEvaluator, ScriptSandbox, ForbiddenTypeAnalyzer, CompiledScriptCache, TimedScriptEvaluator, VirtualTagEngine, ScriptedAlarmEngine, Part9StateMachine, SqliteStoreAndForwardSink) plus their support types; production-consumer tracing into `Runtime`/`Host` to distinguish live vs dormant code; TODO/stub grep; test-breadth vs source-surface comparison. + +--- + +## Architecture Overview + +### Script compile/eval pipeline (Core.Scripting) + +User scripts are C# statement bodies ending in `return …;`. `ScriptEvaluator.Compile` (`Core.Scripting/ScriptEvaluator.cs`) synthesizes a wrapper class (`CompiledScript.Run(globals)`), then runs a **five-step gate**: + +1. Parse the wrapper source with `#line 1` remapping so diagnostics point at operator source. +2. **Wrapper-injection guard** (`EnforceSingleRunMember`, ScriptEvaluator.cs:222) — rejects brace-balanced injections that declare sibling members/types (diagnostics LMX001/LMX002). +3. Roslyn compile against the `ScriptSandbox` reference set (pinned OtOpcUa assemblies + `System.*`/netstandard TPA subset) — the allow-list is explicitly *not* the security boundary (type forwarding makes it porous). +4. **`ForbiddenTypeAnalyzer`** — the real gate. Two semantic passes: (1) member/call surface via `GetSymbolInfo` on creations/invocations/member-access/identifiers; (2) every `TypeSyntax` via `GetTypeInfo`, recursing generic args + array element types. Deny-list = namespace prefixes (`System.IO`, `System.Net`, `System.Diagnostics`, `System.Reflection`, `System.Threading.Tasks`, `System.Runtime.InteropServices`, `System.Runtime.Loader`, `Microsoft.Win32`) plus type-granular names for dangerous residents of allowed namespaces (`Environment`, `AppDomain`, `GC`, `Activator`, `Thread`, `ThreadPool`, `Timer`, `Unsafe`). +5. Emit to an in-memory PE, load into a **per-script collectible `AssemblyLoadContext`**, bind a static `Func,TResult>` delegate. `Dispose()` unloads the ALC. + +Around the core evaluator sit three wrappers: `CompiledScriptCache` (SHA-256-source-keyed, `Lazy` single-compile, disposes ALCs on `Clear()` at publish-replace), `TimedScriptEvaluator` (`Task.Run` + `WaitAsync` wall-clock budget, default 250 ms, wraps timeout as `ScriptTimeoutException`; documents the known orphan-thread leak for CPU-bound scripts), and the script-log pipeline (`ScriptRootLogger` → rolling `scripts-*.log` + `ScriptLogCompanionSink` error mirror to the main log + `ScriptLogTopicSink` → DPS `script-logs` topic via `IScriptLogPublisher`). + +`Core.Scripting.Abstractions` holds the Roslyn-free closure the sandbox pins: `ScriptContext` (the `ctx` API — `GetTag`/`SetVirtualTag`/`Now`/`Logger`/`Deadband`), `ScriptGlobals`, `PassthroughScript` (regex fast-path classifier for the `return ctx.GetTag("X").Value;` mirror shape), and — notably — the concrete `VirtualTagContext` and `AlarmPredicateContext`, which live in this assembly but declare `Core.VirtualTags` / `Core.ScriptedAlarms` namespaces (see C2). + +`DependencyExtractor` statically harvests `ctx.GetTag("literal")` / `ctx.SetVirtualTag("literal", …)` call sites from the AST, rejecting dynamic paths at publish so the change-trigger subscription graph is knowable up front. + +### Virtual-tag runtime (Core.VirtualTags) — two implementations, one live + +The in-scope `VirtualTagEngine` is a self-contained multi-tag engine: `Load()` compiles all scripts through a `CompiledScriptCache`, validates data types and `SetVirtualTag` targets, builds a `DependencyGraph` (iterative Tarjan cycle detection + Kahn topo sort + cached rank), subscribes to upstream driver tags via `ITagUpstreamSource`, and serializes all evaluations under a single `SemaphoreSlim` gate. Change events cascade dependents in topological order; `TimerTriggerScheduler` adds interval-grouped timer triggers with in-flight tick skipping; `VirtualTagSource` adapts the engine to the driver-agnostic `IReadable`/`ISubscribable` surface. + +**However, production does not run this engine.** The live path is `VirtualTagHostActor` → per-tag `VirtualTagActor` → `IVirtualTagEvaluator` bound to `Host/Engines/RoslynVirtualTagEvaluator` (single-tag adapter; fan-out owned by `DependencyMuxActor`). Grep confirms `VirtualTagEngine`, `TimerTriggerScheduler`, `VirtualTagSource`, and `DependencyGraph` have **no production instantiation** — they are exercised only by their unit tests (see U1). Only `VirtualTagContext`, `IHistoryWriter`/`NullHistoryWriter`, and `PassthroughScript` cross into the live path. + +### Scripted-alarm flow (Core.ScriptedAlarms) — live end to end + +`ScriptedAlarmHostActor` (Runtime) owns a long-lived `ScriptedAlarmEngine` and calls `LoadAsync` per config apply (generation-tagged to discard stale completions). Inside the engine: + +1. `LoadAsync` (under `_evalGate`) tears down the prior generation (shelving timer, subscriptions, alarms, scratch, compile-cache ALCs), compiles every predicate via `CompiledScriptCache`, builds an inverse index tag-path → alarm ids (alarms are DAG leaves; no topo sort), seeds the value cache, restores persisted `AlarmConditionState` from `IAlarmStateStore` (EF-backed in prod, in-memory for dev/tests), re-derives `ActiveState` from the live predicate (startup recovery), then subscribes upstream and starts the 5 s shelving-check timer. +2. Upstream changes (`OnUpstreamChange`, fed by `DependencyMuxTagUpstreamSource`) update the value cache and spawn tracked fire-and-forget `ReevaluateAsync` tasks that serialize on `_evalGate`, run the predicate through `TimedScriptEvaluator` against a **reused per-alarm `AlarmScratch`** (read-cache dictionary + context, refilled in place — the hot path allocates nothing), and apply the result through `Part9StateMachine.ApplyPredicate`. +3. `Part9StateMachine` is a pure-function transition table over the immutable `AlarmConditionState` record (Enabled/Active/Acked/Confirmed/Shelving + audit `Comments` as `ImmutableList`). Operator verbs (Acknowledge/Confirm/OneShotShelve/TimedShelve/Unshelve/Enable/Disable/AddComment) come in via the engine's `ApplyAsync` (persist-before-update-in-memory; emissions built under the gate, fired after release to avoid re-entrancy deadlock). +4. Emissions (`ScriptedAlarmEvent`) fan out through `OnEvent` → `ScriptedAlarmSource` (the `IAlarmSource` adapter with equipment-path-prefix subscription filters) into the Part 9 condition nodes / `/alerts` / historian adapter, carrying the per-alarm `HistorizeToAveva` opt-out. `MessageTemplate` resolves `{TagPath}` tokens at emission with a stricter-than-predicate quality bar (Good only; else `{?}`). + +### Alarm-history write path (Core.AlarmHistorian) + +`IAlarmHistorianSink` is the fire-and-forget ingestion contract; production is `SqliteStoreAndForwardSink`: WAL-mode SQLite `Queue` table, `EnqueueAsync` commits a row per event (capacity fast path consults an `Interlocked` cached counter, falling back to `COUNT(*)` + oldest-row eviction at the wall, with periodic 10k-enqueue resync). A self-rescheduling one-shot timer drains batches (default 100) through `IAlarmHistorianWriter` (the HistorianGateway `SendEvent` gRPC client), applying **per-event outcomes**: `Ack` → delete, `PermanentFail` → dead-letter, `RetryPlease` → attempt-bump with a max-attempts (10) poison cap; corrupt payloads dead-letter immediately so they can't stall the queue head. Backoff ladder 1→2→5→15→60 s applied to the timer due-time; dead letters retained 30 days with operator `RetryDeadLettered()`; `GetStatus()` surfaces depth/dead-letter/last-error/evictions to the Admin UI and health checks. + +--- + +## Findings + +Severity scale: **Critical** (production correctness/availability today) / **High** (real risk under realistic conditions) / **Medium** (defect or debt worth scheduling) / **Low** (note). + +### 1. Stability + +#### S1 — HIGH — `VirtualTagEngine.Load` mutates shared state with no gate against in-flight evaluations +`VirtualTagEngine.Load` (`Core.VirtualTags/VirtualTagEngine.cs:78-173`) clears and rebuilds the plain `Dictionary` `_tags` (line 24), the `DependencyGraph`, and disposes the compile cache **without acquiring `_evalGate`**, while three concurrent paths read that state from arbitrary threads: `EvaluateInternalAsync` reads `_tags` *before* taking the gate (line 293), `CascadeAsync` walks `_graph` (line 278) whose internals are plain dictionaries plus a nullable `_cachedRank`, and timer ticks call `EvaluateOneAsync`. A `Load` during an in-flight cascade is a torn-read on non-thread-safe collections. The sibling `ScriptedAlarmEngine` solved exactly this (gate-held `LoadAsync`, `ConcurrentDictionary` with an explanatory comment at `ScriptedAlarmEngine.cs:42-50`); the VT engine never received the same treatment. Mitigating factor: the engine is dormant in production (U1) — but it is fully unit-tested and presented as production-ready, so anyone wiring it up inherits the race. +**Recommendation:** either retire the engine (preferred — see U1) or port the alarm engine's discipline: gate-held load, `ConcurrentDictionary` for `_tags`, and re-check-after-gate in the eval path. + +#### S2 — HIGH — No coalescing/backpressure on upstream-change fan-out (both engines) +Every upstream change spawns a fire-and-forget task that queues on the single eval gate: `ScriptedAlarmEngine.OnUpstreamChange` (`ScriptedAlarmEngine.cs:442-450`) spawns one `ReevaluateAsync` per change event, and `VirtualTagEngine.OnUpstreamChange` (`VirtualTagEngine.cs:263-272`) spawns one `CascadeAsync` per change (untracked). When a referenced tag changes faster than the serialized evaluations drain (a 100 Hz analog referenced by one alarm is enough), tasks accumulate without bound — memory growth plus an ever-staler processing lag, and each queued task re-evaluates against the *current* cache so the extra work is pure waste. The alarm engine tracks these tasks for dispose-drain (`_inFlight`) but does not bound them. +**Recommendation:** replace per-event task spawning with a dirty-set + single pump: mark the alarm/tag dirty, and have one loop drain the dirty set under the gate. This makes cost proportional to distinct dirty alarms, not event rate. + +#### S3 — MEDIUM — Failed reload is fail-stop, not fail-back (both engines) +Both engines tear down the previous generation *before* compiling the new one. On a compile failure, `VirtualTagEngine.Load` has already unsubscribed, cleared `_tags`/`_graph`, and cleared the compile cache (`VirtualTagEngine.cs:84-88`) before throwing (line 148) — leaving `_loaded == true` from the prior load, an empty tag set, and a stale `_valueCache` that `Read`/`VirtualTagSource` keep serving as Good values that will never update. `ScriptedAlarmEngine.LoadAsync` is the same shape (`ScriptedAlarmEngine.cs:196-209`, throw at 250-255): after a failed apply, **zero alarms are loaded** — no predicates evaluate until the next successful publish. The host actor explicitly accepts this ("log it and stay inert", `ScriptedAlarmHostActor.cs:188`, 245), and publish-time validation upstream makes a bad definition reaching the engine unlikely — but a transient `IAlarmStateStore.LoadAsync/SaveAsync` failure during the state-restore loop (`ScriptedAlarmEngine.cs:273-281`) also lands here, and that is *not* operator-preventable. +**Recommendation:** build the new generation into local structures and swap under the gate only on success (compile is already side-effect-free; the subscriptions and timer are the only external effects). At minimum, distinguish store-failures from compile-failures and retry the former. + +#### S4 — MEDIUM — `_alarmsReferencing` read without synchronization +`ScriptedAlarmEngine._alarmsReferencing` is a plain `Dictionary` (`ScriptedAlarmEngine.cs:120-121`) mutated inside gate-held `LoadAsync` (lines 201, 237-242) but read by `OnUpstreamChange` (line 446) from upstream callback threads with no gate. `LoadAsync` disposes the old subscriptions first, but `IDisposable.Dispose` on a subscription does not guarantee an already-executing callback has finished — a callback in flight during reload can race `Clear()`/`Add()` on a non-thread-safe dictionary. The class comment (lines 42-50) carefully justifies `ConcurrentDictionary` for `_alarms` for exactly this class of reader; `_alarmsReferencing` was missed. +**Recommendation:** make it a `ConcurrentDictionary` or swap-in an immutable snapshot (`Dictionary` rebuilt per load, published via `Volatile.Write` to a field). + +#### S5 — MEDIUM — Timed-shelve expiry via the predicate path swallows the `Unshelved` emission +`Part9StateMachine.ApplyPredicate` silently expires timed shelving (`Part9StateMachine.cs:47-49` via `MaybeExpireShelving`, lines 318-322) and returns `EmissionKind.None` (or `Activated`/`Cleared` for the value transition) — the `Unshelved` emission only ever comes from `ApplyShelvingCheck` on the 5 s timer. If a predicate re-evaluation lands between expiry time and the next timer tick, the state record is persisted as unshelved but **no `Unshelved` event is ever published** — OPC UA clients and the `/alerts` page tracking `ShelvingState` from events never learn the shelve ended (the next Activated/Cleared arrives un-suppressed, which is at least confusing for an operator who believes the alarm is still shelved). +**Recommendation:** have `ApplyPredicate` return a compound result (or the engine detect `Shelving` kind change between seed and result) so the expiry emits `Unshelved` before the value-transition emission. Add the missing state-machine test (`Part9StateMachineTests.cs` does not cover this path). + +#### S6 — MEDIUM — `SqliteStoreAndForwardSink.Dispose` races an in-flight drain +`Dispose` (`SqliteStoreAndForwardSink.cs:679-686`) sets `_disposed`, disposes the timer, then immediately disposes `_drainGate` and the writer. `Timer.Dispose()` (no `WaitHandle` overload used) does not wait for a running callback, so an in-flight `DrainOnceAsync` can still be holding the gate and mid-`WriteBatchAsync`: the `finally { _drainGate.Release(); }` (line 467) then throws `ObjectDisposedException` (caught by the timer callback's catch and logged as a drain fault), and the writer can be disposed under an active call. Not a crash — but shutdown noise and an undefined writer contract. +**Recommendation:** mirror the alarm engine's dispose-drain: `_drainGate.Wait()` (or an in-flight task handle) before disposing the gate/writer, or use `Timer.Dispose(WaitHandle)`. + +#### S7 — MEDIUM — Accepted sandbox limits: CPU/memory unbounded, timed-out scripts leak a thread +Documented and deliberate: the sandbox cannot bound allocation or CPU (`ScriptSandbox.cs:30-35`), and a timed-out CPU-bound script leaves its thread-pool thread spinning forever (`TimedScriptEvaluator.cs:17-26`). One orphan per bad-script evaluation *attempt* — under change-triggered evaluation a hot looping script orphans a thread per upstream change until the pool starves, since the engine keeps re-evaluating (alarm engine holds prior state on timeout, `ScriptedAlarmEngine.cs:535-538`, but does not quarantine the predicate). +**Recommendation:** add a per-script circuit breaker (N consecutive timeouts → suspend evaluation + surface to Admin UI) as a cheap interim before any out-of-process runner. The three-layer compile-time sandbox itself (allow-list refs → wrapper-injection guard → semantic deny-list with the full `TypeSyntax` pass) is **notably strong** — including the type-forwarding rationale, `var`-inference coverage via `GetTypeInfo`, and the `Unsafe`/`Activator`/`ThreadPool` type-granular entries. + +#### S8 — LOW — `ScriptedAlarmEngine.Dispose` blocks sync-over-async +`Task.WhenAll(toAwait).GetAwaiter().GetResult()` (`ScriptedAlarmEngine.cs:761`) — fine under the actor host (no sync context) but a deadlock trap if ever disposed from a context-bound thread. Consider `IAsyncDisposable`. + +#### S9 — LOW — At-least-once delivery duplicates on crash window +A crash between a successful `WriteBatchAsync` and the outcome transaction commit (`SqliteStoreAndForwardSink.cs:372-441`) re-delivers the batch. Correct choice for alarm history (dupes over loss) — worth one line in `docs/Historian.md` so the gateway side knows to tolerate replays. + +#### S10 — LOW — `VirtualTagSource.SubscribeAsync` initial-value window can miss one update +Emitting the seed read *before* registering the observer (`VirtualTagSource.cs:63-75`) guarantees ordering but means a change landing between the `Read` and the `Subscribe` is missed until the next change — the inverse trade-off of the race the comment describes. Register first, then emit the seed, and let the (idempotent, newer-wins) consumer handle a possible out-of-order pair — or document that seed-then-subscribe can serve a stale value on slow-changing tags. + +#### S11 — LOW — Capacity eviction drops the *oldest* accepted alarm events +Documented behavior with an `EvictedCount` operator counter (`SqliteStoreAndForwardSink.cs:602-636`). Note only: for a compliance-oriented alarm trail, dropping newest (or refusing enqueue) is sometimes preferred over silently losing the oldest; the current choice is defensible but should be a deliberate, documented policy in `docs/AlarmTracking.md`. + +### 2. Performance + +#### P1 — HIGH — `ScriptSandbox.Build` rebuilds the full BCL reference set on every compile +Every `ScriptEvaluator.Compile` calls `ScriptSandbox.Build(typeof(TContext))` (`ScriptEvaluator.cs:74`), which calls `MetadataReference.CreateFromFile` for **every** `System.*`/netstandard DLL in the trusted-platform-assemblies list (`ScriptSandbox.cs:83-87, 100-130`) — on the order of 100+ files, each a fresh `PortableExecutableReference` whose `AssemblyMetadata` Roslyn must load and parse per compilation. `CompiledScriptCache` de-dupes *per source*, but a publish with 200 distinct scripts pays this ~200 times, and the Admin UI `ScriptAnalysis` endpoints (which reuse this compiler context per keystroke-ish cadence) pay it continuously. The references are immutable per `contextType` for the process lifetime. +**Recommendation:** cache `SandboxConfig` (or at least the `MetadataReference` list) in a static `ConcurrentDictionary` keyed by `contextType.Assembly`. This is likely a 10-100× compile-latency win and directly reduces publish time and script-editor diagnostic latency. + +#### P2 — MEDIUM — `Task.Run` + `WaitAsync` per evaluation on the hot path +`TimedScriptEvaluator.RunAsync` (`TimedScriptEvaluator.cs:78-81`) pushes every predicate/tag evaluation to the pool and registers a timeout. For the alarm engine this happens once per referencing alarm per upstream change. The hop is required for the timeout to work at all (see U2 for what happens without it), but the common case (sub-millisecond script) pays scheduling + `WaitAsync` machinery every time. +**Recommendation:** acceptable at current scale; if profiling ever shows it, an inline-run-with-watchdog design (run synchronously, watchdog flags overruns for quarantine rather than aborting) trades hard timeout for zero hop. + +#### P3 — MEDIUM — VT engine allocates per evaluation what the alarm engine pools +`VirtualTagEngine.EvaluateInternalAsync` builds a fresh read-cache `Dictionary` and a fresh `VirtualTagContext` per evaluation (`VirtualTagEngine.cs:298, 313-317`); the alarm engine got the `AlarmScratch` reuse optimization (`ScriptedAlarmEngine.cs:52-63, 798-835`) for exactly this pattern. Inconsistent — moot while the VT engine is dormant (U1), but whichever way U1 resolves, the two should match. + +#### P4 — LOW — Single global eval gate per engine +All evaluations serialize on one `SemaphoreSlim` (`VirtualTagEngine.cs:44`; `ScriptedAlarmEngine.cs:124`). Correct and simple; it caps throughput at single-threaded script execution per engine. Fine for operator-authored low-thousands scale (the compile cache doc states the same bound); a per-alarm or sharded gate is the escape hatch if scale demands it. Not worth changing now. + +#### P5 — LOW — SQLite per-call overheads +`EnqueueAsync` opens a (pooled) connection and re-runs two PRAGMAs per event (`SqliteStoreAndForwardSink.cs:244-246`); `GetStatus` runs a synchronous `COUNT(*)` per call (line 479-484). Both fine at alarm-event rates; the capacity fast path (cached counter, probe only near the wall — lines 271-290) already removed the real hot-path cost, and the drain-order index (`IX_Queue_Drain`) is right. + +#### P6 — LOW — `DependencyGraph` is well optimized +Cached topological rank invalidated on mutation, shared empty set for misses, iterative Tarjan/Kahn (`DependencyGraph.cs:34-44, 141-149`). The per-cascade `List`/`HashSet`/`Stack` allocations in `TransitiveDependentsInOrder` are minor. No action. + +### 3. Conventions + +#### C1 — MEDIUM — `ITagUpstreamSource` is defined twice +`Core.VirtualTags/ITagUpstreamSource.cs` and a second, byte-for-byte-shaped interface at the bottom of `ScriptedAlarmEngine.cs:860-872` ("intentionally identical shape so Stream G can compose them behind one driver bridge"). Two distinct .NET types means `DependencyMuxTagUpstreamSource` and any composing bridge must implement/adapt each separately, and the "identical shape" invariant is enforced only by comment. +**Recommendation:** hoist a single `ITagUpstreamSource` into `Core.Scripting.Abstractions` (both projects already reference it) and retire the duplicate. + +#### C2 — MEDIUM — `Core.Scripting.Abstractions` declares types across three namespaces +`ScriptContext`/`ScriptGlobals`/`PassthroughScript` claim namespace `Core.Scripting`, `VirtualTagContext` claims `Core.VirtualTags`, `AlarmPredicateContext` claims `Core.ScriptedAlarms` — all physically in the Abstractions assembly. The reason is sound (the sandbox pins `contextType.Assembly`, so concrete contexts must live in the Roslyn-free assembly to keep Roslyn out of the script compilation closure — `ScriptSandbox.cs:60-70`), but it is documented only in a `PassthroughScript` remark and violates the repo's namespace-follows-assembly norm; find-by-namespace navigation actively misleads here. +**Recommendation:** add a project-level README or `AssemblyInfo` doc comment stating the rule ("this assembly is the sandbox-pinned closure; types keep their consumer namespaces deliberately"), or bite the bullet and align namespaces with a `TypeForwardedTo` migration. + +#### C3 — LOW — Trailing type definitions in engine files +`ScriptedAlarmEvent` and the duplicate `ITagUpstreamSource` live at the bottom of `ScriptedAlarmEngine.cs` (843-872); `CompilationErrorException` and `ScriptAssemblyLoadContext` at the bottom of `ScriptEvaluator.cs` (391-432); `DependencyCycleException` in `DependencyGraph.cs`. Mostly reasonable co-location, but `ScriptedAlarmEvent` is a public cross-project contract and deserves its own file. + +#### C4 — LOW — Repeated raw OPC UA status-code literals +`0x80340000u /* BadNodeIdUnknown */`, `0x80020000u /* BadInternalError */`, `0x80320000u /* BadWaitingForInitialData */`, `0x80000000u` severity mask are hand-rolled with comments in at least five files (`VirtualTagEngine.cs:215-218, 307, 328`; `VirtualTagContext.cs`; `AlarmPredicateContext.cs`; `ScriptedAlarmEngine.cs:584-587`). A small `KnownStatusCodes` static in `Core.Abstractions` would end the copy-paste drift risk (these layers deliberately avoid the OPC Foundation package, so the constants can't come from there). + +#### C5 — LOW — `Core.AlarmHistorian` mixes contract and implementation +`IAlarmHistorianSink`/`IAlarmHistorianWriter`/status types share the project with `SqliteStoreAndForwardSink`, while scripting got a dedicated Abstractions assembly. Tolerable at three files, but the Runtime adapter and the gateway driver both reference the whole project (dragging the `Microsoft.Data.Sqlite` dependency) to get the interfaces. + +#### C6 — LOW — Plan-era naming residue +XML docs throughout reference "Phase 7 plan Stream A.4/B/C/E/F/G" (`TimedScriptEvaluator.cs:5, 39`; `ScriptSandbox.cs`; `IAlarmStateStore.cs:13`; many more). The repo already purged the meaningless `Phase7` *type* prefixes (master 40e8a23e); the doc references now point at plans that have been superseded by `docs/ScriptedAlarms.md`/`docs/ScriptEditor.md`. Sweep them to current doc anchors during the next doc pass. + +#### C7 — LOW — Doc-drift: `ApplyPredicate` claims branch handling that doesn't exist +`Part9StateMachine.ApplyPredicate`'s summary promises "branch-stack increment when a new active arrives while prior active is still un-acked" (`Part9StateMachine.cs:31-34`), but no branch/previous-instance logic exists anywhere in the file — re-activation simply resets Acked/Confirmed. See U5; at minimum fix the comment. + +### 4. Underdeveloped Areas + +#### U1 — HIGH — The entire Core.VirtualTags engine stack is dormant in production +`VirtualTagEngine` (568 lines), `TimerTriggerScheduler` (162), `VirtualTagSource` (106), and `DependencyGraph` (324) have **no production instantiation** — repo-wide grep finds `new VirtualTagEngine` only in `tests/Core/ZB.MOM.WW.OtOpcUa.Core.VirtualTags.Tests`. The live virtual-tag runtime is `Runtime/VirtualTags/VirtualTagHostActor` + `DependencyMuxActor` + `Host/Engines/RoslynVirtualTagEvaluator` (single-tag adapter, `IVirtualTagEvaluator` seam), which re-implements read-cache building, cross-tag-write handling (drops them — the actor mux owns fan-out), timeout, and compile caching independently. ~1,160 lines of engine + ~1,260 lines of tests are being maintained, reviewed, and (per S1/S3) carrying latent bugs for a component nothing runs. The `RoslynVirtualTagEvaluator` doc comment even acknowledges the split ("Cycle detection + cascade ordering live in VirtualTagEngine; this adapter stays single-tag scoped"). +**Recommendation:** decide explicitly. Either (a) retire `VirtualTagEngine`/`TimerTriggerScheduler`/`VirtualTagSource` (keep `DependencyGraph` only if the publish-time cycle check uses it — verify) and let the actor pipeline be the one implementation, or (b) document the engine as the sanctioned embedded/non-Akka path and fix S1-S3. The current state — two divergent implementations where the tested one is dormant and the live one bypasses the safety wrappers (U2/U3) — is the worst of both. + +#### U2 — CRITICAL (cross-cutting; defect sits in Host, root cause is a dormant in-scope component) — Production script timeout is ineffective: `TimedScriptEvaluator` is bypassed on the live path +`RoslynVirtualTagEvaluator.Evaluate` runs `evaluator.RunAsync(context, cts.Token).GetAwaiter().GetResult()` with a 2 s `CancellationTokenSource` (`Host/Engines/RoslynVirtualTagEvaluator.cs:102-106`). But `ScriptEvaluator.RunAsync` executes the compiled delegate **synchronously on the calling thread** and only checks the token *before* invoking it (`ScriptEvaluator.cs:178-190` — "TimedScriptEvaluator wraps this in Task.Run so a long-running script still honours its wall-clock budget"). The token can therefore never interrupt a running script: the `catch (OperationCanceledException)` timeout branch in the Host adapter is dead code, and a CPU-bound or infinite-loop virtual-tag script **hangs the owning `VirtualTagActor`'s message loop indefinitely** — no timeout, no log, no recovery. The in-scope `TimedScriptEvaluator` exists precisely to prevent this and is used by the (live) alarm engine and the (dormant) VT engine — but not by the one component that evaluates operator-authored virtual-tag scripts in production. +**Recommendation:** wrap the adapter's evaluator in `TimedScriptEvaluator` (accepting the documented orphan-thread trade-off), or at minimum `Task.Run` + `WaitAsync` inside `Evaluate`. Add a regression test with a `while(true)` script. This should be fixed before any other finding in this report. + +#### U3 — HIGH (same file) — Live path also bypasses `CompiledScriptCache`: unbounded ALC accretion across republishes +`RoslynVirtualTagEvaluator` caches compiled evaluators in a raw `ConcurrentDictionary>` keyed by source (`RoslynVirtualTagEvaluator.cs:25-26, 69`), never evicted until process dispose. Every edited-script republish adds a new entry while the old source's evaluator — and its collectible `AssemblyLoadContext` — stays rooted forever. This is *exactly* the "per-publish recompile accretion" leak that `CompiledScriptCache.Clear()` was built to fix (its own doc, `CompiledScriptCache.cs:34-40`, and the engine comments at `ScriptedAlarmEngine.cs:66-77` describe the production leak from skipping it). Additionally, `GetOrAdd` with a value factory can compile twice under a race and silently leak the losing ALC (the cache solved this with `Lazy` + `ExecutionAndPublication`). +**Recommendation:** switch the adapter to `CompiledScriptCache` and clear it on config apply (the host actor already sees apply boundaries). + +#### U4 — MEDIUM — `IHistoryWriter` is a permanently-Null surface +`VirtualTagDefinition.Historize` routes to `IHistoryWriter.Record`, but DI binds only `NullHistoryWriter` (`Runtime/ServiceCollectionExtensions.cs:57-58, 236-237` — "durable AVEVA sink is infra-gated; a deployment binding a real IHistoryWriter overrides"). No real implementation exists in the tree; the operator-visible Historize checkbox on virtual tags is a silent no-op (the separate `ContinuousHistorization` path taps the mux directly and is itself gated + ref-set-empty per CLAUDE.md Known Limitation 2). Either wire `IHistoryWriter` to the gateway value-writer or hide/annotate the checkbox until it does something. + +#### U5 — MEDIUM — Part 9 surface gaps are real but undeclared in one place +No condition branches / previous-instances (re-activation while un-acked just resets Acked — and the xmldoc overpromises, C7); `Severity` is static per definition ("not currently computed by the predicate", `ScriptedAlarmDefinition.cs`); `AlarmKind` affects only node typing; `MessageTemplate` has no brace escaping (documented as "reach for a feature request", `MessageTemplate.cs:16-19`); `Retain` is carried on the definition but not consulted anywhere in this scope. Each is a defensible v1 cut; collect them in `docs/ScriptedAlarms.md` as an explicit conformance statement so OPC UA client integrators know what subset to expect. + +#### U6 — LOW — No TODO/HACK/stub markers +Grep found zero `TODO|HACK|FIXME|NotImplemented` across all five projects — unfinished edges are instead documented in XML remarks (a healthier pattern, though it makes gaps greppable only by reading). + +#### U7 — LOW — Test coverage: broad and behavior-focused, with specific holes +~6,690 test lines against ~5,760 source lines, all four impl projects have dedicated suites: `SqliteStoreAndForwardSinkTests` (888 — outcomes, poison cap, corrupt payloads, capacity fast path), `ScriptedAlarmEngineTests` (1,270), `ScriptSandboxTests` (614 — the escape catalog), `CompiledScriptCacheTests` (409 — including ALC-reclaim via `WeakReference`), `VirtualTagEngineTests` (800), state machine, extractor, graph, loggers, sources. Holes worth filling: (a) S5's expiry-during-predicate emission swallow — no `Part9StateMachineTests` case; (b) no concurrency test for load-vs-cascade (would expose S1/S4); (c) no sink `Dispose`-during-drain test (S6); (d) `ForbiddenTypeAnalyzer` has no dedicated unit suite (exercised only through end-to-end `Compile` in sandbox tests — fine functionally, but analyzer-pass regressions produce opaque failures); (e) nothing tests the *production* `RoslynVirtualTagEvaluator` timeout (which would have caught U2 — it lives in Host, whose test project should own it). + +--- + +## Maturity Ratings + +Scale: 1 (prototype) → 5 (production-hardened). Rating reflects the scoped code as it stands, including how it is (or isn't) consumed. + +| Dimension | Rating | Justification | +|---|---|---| +| Stability | **3.5** | Alarm engine + sink show genuinely careful engineering (gate discipline, emit-outside-gate, dispose drains, persist-before-memory, poison-row caps); dragged down by fail-stop reloads, unbounded change fan-out, the VT engine's un-gated `Load`, and the shelve-expiry emission swallow. | +| Performance | **3** | Right structures where it counts (compile cache, alarm scratch reuse, graph rank cache, SQLite capacity fast path), but every compile rebuilds the full BCL reference set (P1), every evaluation pays a pool hop, and one global gate serializes each engine. | +| Conventions | **4** | Strongly consistent with repo norms (Serilog, Null-object defaults, xUnit/Shouldly, exemplary "why"-comments); duplicated `ITagUpstreamSource`, the three-namespace Abstractions assembly, and literal status codes are the debt. | +| Underdeveloped areas | **2** | A complete, tested virtual-tag engine nobody runs, while the live path bypasses the scope's two key defenses (timeout → hung actor risk, ALC cache → leak); Historize is a Null no-op; Part 9 gaps undeclared. The split between "what is built and tested" and "what actually runs" is the defining problem of this subsystem. | + +--- + +## Priority Recommendations (ordered) + +1. **U2** — Wrap the production virtual-tag evaluation in `TimedScriptEvaluator` (or equivalent). A single bad script can currently hang a `VirtualTagActor` forever with no timeout. +2. **U3** — Use `CompiledScriptCache` (with apply-boundary `Clear`) in `RoslynVirtualTagEvaluator` to stop cross-publish ALC accretion. +3. **P1** — Statically cache the `ScriptSandbox` reference set; large publish-time and script-editor latency win for one small change. +4. **U1** — Decide the fate of the dormant `VirtualTagEngine` stack (retire preferred); until then S1/S3 fixes are latent-bug maintenance on dead code. +5. **S2** — Add dirty-set coalescing to `ScriptedAlarmEngine` upstream fan-out before a fast tag meets a popular alarm in production. +6. **S5 + C7** — Fix the timed-shelve expiry emission and the state-machine doc drift together, with a state-machine test. +7. **S3/S4/S6** — Schedule the reload-swap, `_alarmsReferencing` synchronization, and sink dispose-drain as one hardening pass. diff --git a/archreview/03-server-runtime.md b/archreview/03-server-runtime.md new file mode 100644 index 00000000..91855927 --- /dev/null +++ b/archreview/03-server-runtime.md @@ -0,0 +1,163 @@ +# Architecture Review — Server & Runtime Subsystem + +- **Date:** 2026-07-08 +- **Commit:** `9cad9ed0` +- **Scope:** `src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer`, `.Host`, `.Runtime`, `.ControlPlane`, `.Security`, plus the sink/publisher wrappers in `src/Core/ZB.MOM.WW.OtOpcUa.Commons/OpcUa` (they are the load-bearing seam between Host and Runtime) and `tests/Server` for coverage assessment. +- **Dimensions:** Stability · Performance · Conventions · Underdeveloped areas + +--- + +## Architecture overview + +### Host composition + +`Program.cs` (`src/Server/ZB.MOM.WW.OtOpcUa.Host/Program.cs`) is a single fused binary whose wiring is driven by the `OTOPCUA_ROLES` env var (`admin`, `driver`, or both). Notable mechanics: per-role appsettings overlays with env-vars/args re-appended so deployment overrides keep precedence (`Program.cs:61-73`); CWD anchored to `AppContext.BaseDirectory` for Windows-service path sanity (`:83`); Serilog via the shared `AddZbSerilog` with the static `Log.Logger` assigned post-build so Akka's `SerilogLogger` routes actor logs (`:305`). + +On `driver` nodes the host registers the historian stack (three config-gated layers: `AddServerHistorian` read client, `AddAlarmHistorian` SQLite store-and-forward, `AddHistorianProvisioning` EnsureTags, plus the ContinuousHistorization outbox/writer pair), the driver-factory registry (8 drivers, `DriverFactoryBootstrap.cs`), the LDAP data-plane authenticator, and — critically — the **late-binding seams**: `DeferredAddressSpaceSink` and `DeferredServiceLevelPublisher` are DI singletons that actors capture at construction; `OtOpcUaServerHostedService` swaps the real SDK-backed implementations in after `StandardServer` start and swaps Nulls back in on stop (`OtOpcUaServerHostedService.cs:131, 226-231, 241-253`). + +### OPC UA server & node manager + +`OtOpcUaSdkServer` (a `StandardServer`) creates the single `OtOpcUaNodeManager` (`CustomNodeManager2`, 2 424 lines) that owns the entire writable address space: UNS Area/Line/Equipment folders, typed equipment-tag variables, Part 9 `AlarmConditionState` nodes, historized-node registration, and four HistoryRead override arms (Raw with synthesized continuation-point paging + tie-cluster over-fetch, Processed, AtTime, Events). Reverse paths out of the node manager are deliberately Akka-free delegates/interfaces set by the Host at start: `AlarmCommandRouter` (→ DPS `alarm-commands` topic), `NativeAlarmAckRouter` (→ `DriverHostActor.RouteNativeAlarmAck`), `NodeWriteGateway` (→ `ActorNodeWriteGateway` Ask), `HistorianDataSource` (→ gateway read client). Inbound writes are role-gated (`WriteOperate`), dispatched fire-and-forget under the SDK `Lock`, optimistically applied, and self-corrected (Bad-quality blip + revert + Part 8 audit event) on a failed device outcome. + +### Runtime actor topology (per driver node) + +`WithOtOpcUaRuntimeActors` (`Runtime/ServiceCollectionExtensions.cs:194-347`) spawns, in order: `DbHealthProbeActor` → `DependencyMuxActor` → (gated) `ContinuousHistorizationRecorder` → `OpcUaPublishActor` (pinned single-thread dispatcher `opcua-synchronized-dispatcher`, owns the `AddressSpaceApplier`) → `PeerProbeSupervisor` → `DriverHostActor` → `HistorianAdapterActor`. `DriverHostActor` spawns one `DriverInstanceActor` per deployed driver plus the `VirtualTagHostActor` and `ScriptedAlarmHostActor` sub-hosts, maintains the `(driver, FullName) ⇄ NodeId` routing maps, and fans every `AttributeValuePublished` to the mux + the publish actor (`ForwardToMux`, `DriverHostActor.cs:561-589`). **No actor in Runtime overrides `SupervisorStrategy`** — everything runs under Akka's default one-for-one unlimited restart decider. + +### Redundancy model + +Non-transparent warm redundancy. `RedundancyStateActor` (admin-role cluster singleton, `ControlPlane/Redundancy/RedundancyStateActor.cs`) derives Primary/Secondary purely from Akka's `driver` role-leader, debounces 250 ms, publishes `RedundancyStateChanged` on the DPS `redundancy-state` topic, and re-publishes on a 10 s heartbeat (the late-subscriber fix). Consumers: `OpcUaPublishActor` computes ServiceLevel (`ServiceLevelCalculator` + DB-health probe + peer OPC UA probes; 240/200/100/0 bands, first-publish-always so the SDK's 255 default never stands — `OpcUaPublishActor.cs:418-437, 508-542`); `DriverHostActor`, `ScriptedAlarmHostActor`, and `HistorianAdapterActor` use the same snapshot to Primary-gate device writes, alarm emission, and historization. There is no lease or fencing token — correctness rests entirely on Akka's single-role-leader guarantee within a converged cluster. + +--- + +## Findings + +Severity scale: **Critical** (correctness/availability broken in a realistic scenario) · **High** (significant risk or debt, plan a fix) · **Medium** (real but bounded) · **Low** (hygiene). + +### 1. Stability + +**S1 — CRITICAL: The split-brain resolver is configured but never activated; unreachable members are never downed.** +`akka.conf` carries a full `split-brain-resolver { active-strategy = "keep-oldest" … }` block (`src/Core/ZB.MOM.WW.OtOpcUa.Cluster/Resources/akka.conf:40-46`), but there is **no `akka.cluster.downing-provider-class` anywhere in the repo**, and `WithClustering` sets only `SeedNodes`/`Roles` — not `ClusterOptions.SplitBrainResolver` (`src/Core/ZB.MOM.WW.OtOpcUa.Cluster/ServiceCollectionExtensions.cs:80-84`; repo-wide grep for `downing-provider|SplitBrainResolver` returns nothing). In Akka.NET the SBR HOCON block is inert without the provider, so the cluster runs the default **NoDowning** provider. Consequences: (a) a *hard-crashed* node stays "unreachable" forever, so **cluster singletons (`ConfigPublishCoordinator`, `AdminOperationsActor`, `RedundancyStateActor`, `AuditWriterActor`) never fail over** and the `driver` role-leader never moves — i.e. redundancy failover works only for graceful shutdowns; (b) a network partition never resolves, and with `min-nr-of-members = 1` each side elects its own role-leader, so **both sides advertise ServiceLevel 240 indefinitely**. The two-node integration harness only exercises graceful `StopAsync`, which is why this is invisible in tests. +*Recommendation:* set `ClusterOptions.SplitBrainResolver = SplitBrainResolverOption` (or `downing-provider-class = "Akka.Cluster.SBR.SplitBrainResolverProvider"` in `akka.conf`), then add a two-node kill-9 failover test asserting singleton handover + ServiceLevel demotion. + +**S2 — HIGH: LDAP authentication blocks the OPC UA session-activation thread with an effectively non-cancellable, non-configurable timeout.** +The SDK invokes `ImpersonateUser` synchronously; `OpcUaApplicationHost.HandleImpersonation` block-bridges the authenticator (`OpcUaApplicationHost.cs:271-273`) with `CancellationToken.None`. The doc comment shifts timeout responsibility to the authenticator ("callers must enforce their own timeouts"), but `LdapOpcUaUserAuthenticator` adds none (`Host/OpcUa/LdapOpcUaUserAuthenticator.cs:30-48`), and the shared library's `AuthenticateAsync` is a **synchronous method wrapped in `Task.FromResult`** that observes the token only at entry; its 10 s connect timeout is a library default that OtOpcUa's `LdapOptions.ToLibraryOptions()` does not project or expose (`Security/Ldap/LdapOptions.cs:94-107`). Every authentication also opens a fresh TCP connection + service-account bind + search + user bind (no pooling). Under a directory outage or slow DC, session activations stall serially for ~10-20 s each on SDK threads. +*Recommendation:* enforce a hard timeout in `LdapOpcUaUserAuthenticator` (`Task.WhenAny` + deny on timeout), surface a `Security:Ldap:TimeoutMs` option, and consider a short negative-cache to shed load during an outage. Fail-closed semantics are otherwise correct end-to-end (deny on error, opaque messages, zero-role fallback on mapper fault). + +**S3 — HIGH: HistoryRead block-bridges the gateway per node, sequentially, on SDK request threads.** +All four HistoryRead arms call `.GetAwaiter().GetResult()` per node handle with `CancellationToken.None` (`OtOpcUaNodeManager.cs:1902-1911, 2059, 2169-2171, 2201-2203`). This is safe w.r.t. the node-manager `Lock` (the overrides run outside it — correctly documented), but bounded only by the gateway client's per-call `CallTimeout` (30 s default). A single request naming N historized nodes against a slow historian holds one SDK request thread up to N × 30 s; the server is configured `MaxRequestThreadCount = 100`, `MaxQueuedRequestCount = 200` (`OpcUaApplicationHost.cs:329-331`), so a handful of misbehaving history clients can exhaust the request pool and degrade *all* OPC UA services. +*Recommendation:* parallelize per-node reads within a batch (bounded), pass a per-request deadline token, and consider a server-side concurrent-HistoryRead limiter. + +**S4 — HIGH: Primary-gate "default-allow while role unknown" opens a dual-primary window on the data plane.** +`DriverHostActor.HandleRouteNodeWrite` / `HandleRouteNativeAlarmAck` / native-alarm emission all gate on `_localRole is Secondary or Detached` and deliberately default to *allow* until the first redundancy snapshot arrives (`DriverHostActor.cs:1018-1026, 1074-1083, 956-960; OnRedundancyStateChanged :1109-1114`). Combined with DPS delivery being at-most-once and heartbeat-repaired every 10 s, a freshly-booted or snapshot-missed **secondary services device writes and emits alerts as if primary** for up to the heartbeat interval — and indefinitely if the NodeId identity mismatch (S5) bites. The boot-window rationale (single-node deploys) is sound, but the same signal gates *shared-field-device writes*. +*Recommendation:* for the write/ack gates specifically, prefer default-*deny* once the cluster has >1 driver member (discoverable from cluster state), or require at least one received snapshot before servicing writes on multi-node clusters. + +**S5 — MEDIUM: Redundancy NodeId identity is string-matched across three independently-derived sources with no diagnostic on mismatch.** +`OpcUaPublishActor` matches `n.NodeId == _localNode.Value` (`OpcUaPublishActor.cs:512`); the snapshot side derives `host:port` from the gossiped `Member.Address` (`RedundancyStateActor.cs:147-148`) while the local side derives it from configured `PublicHostname:Port` (`ClusterRoleInfo`). If `PublicHostname` ever diverges from the gossiped address (DNS vs IP, container advertised name), the node silently computes ServiceLevel 0 / keeps a stale role — the exact shape of the historical "silently inert delivery" bug, with no log distinguishing it from legitimate Detached. +*Recommendation:* log-once (Warning) when the local node is absent from a received snapshot; assert the identity equality in a startup self-check. + +**S6 — MEDIUM: A crashed-and-restarted `DriverInstanceActor` loses its message-delivered state (desired subscriptions) until the next deploy.** +No actor in Runtime overrides `SupervisorStrategy` (grep confirms), so an exception in a driver child triggers Akka's default one-for-one **unlimited, non-backoff restart**. Restart re-runs `PreStart` (driver re-init from the Props spec) but the desired-subscription set arrives post-spawn via `SetDesiredSubscriptions` messages stored in actor state (`DriverInstanceActor.cs:84, 181, 861`); a restart wipes it and `DriverHostActor` has no restart-detection to re-send (restarts don't fire `Terminated`). A persistently-throwing driver also hot-loops restarts with no backoff. +*Recommendation:* wrap driver children in `BackoffSupervisor` (restart with exponential backoff) and have the child request its subscription set from the parent in `PreStart`, or have the parent push it on a `PostRestart` signal. Add a supervision test (none exists — see U6). + +**S7 — MEDIUM: `DriverInstanceActor.PostStop` blocks the actor shutdown path on driver shutdown.** +`_driver.ShutdownAsync(CancellationToken.None).GetAwaiter().GetResult()` (`DriverInstanceActor.cs:950`) is the only synchronous block in the Runtime actors; with `CancellationToken.None` a hung protocol stack can stall stop/re-deploy of the whole child set. +*Recommendation:* bound it (`WaitAsync(timeout)`) and log-and-abandon on expiry. + +**S8 — MEDIUM: Inbound Part 9 commands and native acks are at-most-once; the client sees `Good` even if the command is lost.** +The alarm-command router is a fire-and-forget DPS `Publish` and the native-ack router a fire-and-forget `Tell`, both wrapped in catch-log-drop (`OtOpcUaServerHostedService.cs:139-194`); the node manager has already returned `Good` so the SDK applies local condition state regardless (`OtOpcUaNodeManager.cs:769-772`). A mediator hiccup or missing `DriverHostActor` silently strands the engine/upstream state. This is a documented, deliberate trade-off (non-blocking under `Lock`), but it is invisible in operation. +*Recommendation:* add a counter/metric for dropped routes (both routers currently only log at Warning) and consider an engine-side reconciliation sweep. + +**S9 — MEDIUM: Certificate lifecycle has no renewal or expiry monitoring.** +The application certificate is auto-created self-signed with SDK defaults (2048-bit, **12-month lifetime**) and only checked at boot (`OpcUaApplicationHost.cs:305-317`). Nothing monitors expiry; ~12 months after first deploy, Sign/SignAndEncrypt endpoints and UserName-token encryption fail on live servers with no advance warning. +*Recommendation:* startup + periodic expiry check with a metric/health-check and a documented rotation runbook (the AdminUI cert-actions work covers client certs; the *server* cert lifecycle is the gap). + +**S10 — LOW: SDK start failure is swallowed and the node keeps running with Null sinks.** +`StartAsync` catches, logs, and returns (`OtOpcUaServerHostedService.cs:110-129`) — a deliberate availability choice (AdminUI stays up), but the node then silently no-ops all OPC UA work and nothing surfaces the condition through `/health` or ServiceLevel (which is itself served by the dead server). +*Recommendation:* set a health-check flag consumed by `MapOtOpcUaHealth` so fleet status shows the degraded state. + +**S11 — LOW: `AuditWriterActor` buffer is unbounded between flushes and drops whole batches on DB outage** (`ControlPlane/Audit/AuditWriterActor.cs:35, 83, 113-116`). Explicitly best-effort by contract; flag only if compliance-grade audit is ever required. (Currently moot — see U3: the pipeline has no producers.) + +**Positive stability notes (worth preserving):** the node-manager locking discipline is exemplary — every mutation under `Lock`, every `Server.ReportEvent` deliberately outside it (`OtOpcUaNodeManager.cs:1101-1136, 1503-1504, 1595-1614`), the write-dispatch continuation forced off the SDK thread via `RunContinuationsAsynchronously` (`:884-897`), and the revert guarded by "still holds the optimistic value" (`ShouldRevert`, `:1048`). The Deferred-sink swap-to-Null on stop prevents post-stop writes hitting a disposed manager. `ConfigPublishCoordinator` has real crash-recovery (rebuilds expected-acks from DB on `PreStart`). `HandleServiceLevelChanged`'s first-publish-always correctly defeats the SDK's 255 default. `ActorNodeWriteGateway` bounds its Ask at 10 s with the inner driver Ask at 8 s and the backend call at 5 s — a correctly nested timeout ladder. + +### 2. Performance + +**P1 — HIGH: Any structural deploy change triggers a full address-space teardown/rebuild that severs every client's monitored items server-wide.** +`AddressSpaceApplier.Apply` forces `RebuildAddressSpace()` for *any* added/removed equipment, alarm, tag, or virtual tag (`AddressSpaceApplier.cs:134-154`); the rebuild removes every variable/folder/condition under one `Lock` hold and re-materialises from scratch (`OtOpcUaNodeManager.cs:1690-1736`). Recreated nodes are new `NodeState` instances, so existing client subscriptions on the old nodes go dead — the F10b surgical path (`UpdateTagAttributes`/`UpdateFolderDisplayName`) exists precisely because of this, but covers only attribute edits and renames. Adding one tag to one equipment still drops every subscription on a production SCADA server. +*Recommendation:* extend the surgical path to pure-add deltas first (`EnsureFolder`/`EnsureVariable` are already idempotent — adds need no teardown; `RaiseNodesAddedModelChange` already exists for announcement), then scope removals per-equipment. This is the highest-leverage performance/stability item in the subsystem. + +**P2 — MEDIUM: The value hot path is one global lock acquisition + one actor message per value, with no batching.** +Per published value: driver child `Tell` → `DriverHostActor.ForwardToMux` (dictionary lookup + up to two `Tell`s, `DriverHostActor.cs:561-589`) → `OpcUaPublishActor` (pinned thread) → `sink.WriteValue`, which takes the **global node-manager `Lock`** per value (`OtOpcUaNodeManager.cs:266-281`), contending with SDK read/subscription/publish threads. The mux itself is lean (`DependencyMuxActor.cs:95-108` — early-drop for uninterested refs, set-based fan-out). At high tag counts × fast poll intervals this serialises everything through one lock and allocates one record per hop. +*Recommendation:* add a batched `WriteValues(IReadOnlyList<…>)` sink call (one lock hold per driver publish cycle) — the seam (`IOpcUaAddressSpaceSink`) makes this a contained change, but remember the Deferred-wrapper forwarding trap (U2). + +**P3 — MEDIUM: HistoryRead-Events has no paging and translates `NumValuesPerNode == 0` to `int.MaxValue`.** +`EventMaxEvents` deliberately maps "no limit" to unbounded (`OtOpcUaNodeManager.cs:1944-1954`), and the Events arm never issues continuation points ("full window in one shot", `:1914-1921`). A wide time window over a busy alarm source materialises the entire result in memory on both gateway and server. +*Recommendation:* impose a server-side max (mirroring `MaxTieClusterOverfetch`'s philosophy) and fail loudly or page beyond it. + +**P4 — LOW: Every deploy re-runs the four `Materialise*` passes over the full composition** (`OpcUaPublishActor.cs:338-354`) — acceptable because `EnsureFolder`/`EnsureVariable` early-return on existing ids, but each pass still takes/releases the `Lock` per node; fold into P2's batching if it shows up in profiles. On the positive side, the Raw-paging tie-cluster over-fetch is explicitly bounded (`MaxTieClusterOverfetch`, default 65 536) with a loud-fail backstop, and `HandleDiscoveredNodes` has an unchanged-plan short-circuit preventing ~15× re-subscribe blips during the discovery window (`DriverHostActor.cs:658-673`) — both good designs. + +### 3. Conventions + +**C1 — MEDIUM: The `ServerHistorian` section is bound imperatively in five places with no `IOptions` registration.** +`Program.cs:120-122` (own bind + `Validate()` warnings), `AddServerHistorian`, `AddAlarmHistorian`, `AddHistorianProvisioning` (each re-`Get<>`s the section, `Runtime/ServiceCollectionExtensions.cs:86, 132, 168`), and `OtOpcUaServerHostedService`'s constructor (`.cs:88-90`, self-justified as "self-contained"). Warnings can log twice; five bind sites will drift. +*Recommendation:* one `AddValidatedOptions` and inject `IOptions<>` everywhere, matching the `OpcUa`/`Ldap` pattern that already exists (`Program.cs:102, 254-255`). + +**C2 — MEDIUM: Options validation is two-tier and the tiers disagree.** +`OpcUa` and `Security:Ldap` get real fail-fast `ValidateOnStart` validators (`OpcUaApplicationHostOptionsValidator` requires ≥1 security profile; `LdapOptionsValidator` refuses plaintext-without-opt-in) — good. `ServerHistorian` / `ContinuousHistorization` / `AlarmHistorian` only get advisory `Validate()` warning lists, and `DevStubMode=true` (accept-any-credentials Administrator grant) is merely log-warned, never blocked, in production (`OtOpcUaLdapAuthService.cs:79-88`). +*Recommendation:* promote the historian sections to the validator pattern; add an environment-gated guard (or at minimum a startup validator failure) for `DevStubMode` outside Development. + +**C3 — LOW: Library code logs via the static Serilog logger** (`Runtime/ServiceCollectionExtensions.cs:90, 100, 136`) while everything else uses MEL `ILogger` — works only because `Program.cs:305` assigns `Log.Logger`, an ordering-sensitive hidden coupling. The node manager similarly logs through the obsolete `Utils.LogError` static trace with six `#pragma warning disable CS0618` sites (`OtOpcUaNodeManager.cs:449-451` et al.) — the acknowledged follow-up (wire an `ITelemetryContext`) should actually land. + +**C4 — LOW: `WithOtOpcUaRuntimeActors` is service-locator style with silent Null fallbacks** (`Runtime/ServiceCollectionExtensions.cs:211-237`) — the accepted Akka.Hosting idiom, and the missing-registration warnings (evaluator, recorder deps) are good; but `resolver.GetService()` (`:298`) has *no* fallback warning parallel to the others, despite being the exact seam that shipped dormant once (PR #423). The `dispatched=N, requested=0` tally in the applier (`AddressSpaceApplier.cs:274-282`) partially compensates. + +**Positive conventions notes:** the layering is clean and deliberate — `OpcUaServer` is Akka-free (delegates/gateway seams), `Runtime` is SDK-free (sink seams), Core holds the pure calculators (`ServiceLevelCalculator`, planners); the Null-object + Deferred late-binding pattern is applied consistently (sink, ServiceLevel, write gateway, historian source, provisioning, history writer); actor `Props` factories + registry marker keys are uniform; scope-correctness around the scoped `IGroupRoleMapper` inside the singleton authenticator is explicitly handled (`LdapOpcUaUserAuthenticator.cs:63-67`); NodeId is canonically `host:port` everywhere except one dormant spot (U4). + +### 4. Underdeveloped areas + +**U1 — HIGH (documentation drift on a load-bearing claim): CLAUDE.md "KNOWN LIMITATION 2" is stale — the ContinuousHistorizationRecorder ref-feed gap is CLOSED in code.** +The recorder is still *spawned* with `historizedRefs: Array.Empty()` (`Runtime/ServiceCollectionExtensions.cs:272`), but the applier now feeds it the add/remove delta on every deploy via `ActorHistorizedTagSubscriptionSink` → `UpdateHistorizedRefs` (`Runtime/ServiceCollectionExtensions.cs:287-292`; `AddressSpaceApplier.cs:210-213, 310-353`; `ContinuousHistorizationRecorder.cs:67-78`). Restart convergence works because `OpcUaPublishActor._lastApplied` is in-memory, so the first post-boot rebuild diffs against empty and emits the full set as Added (`OpcUaPublishActor.cs:326-336`). CLAUDE.md still says "the recorder registers interest in nothing and historizes nothing… Until that feed lands, continuous historization records no values." +*Recommendation:* update CLAUDE.md; add an explicit restart-convergence test (boot → apply full plan → assert mux interest registered), since the chain (empty seed + delta feed + in-memory `_lastApplied`) is load-bearing and only implicitly covered today. + +**U2 — MEDIUM: The `DeferredAddressSpaceSink` capability-forwarding trap is currently closed but only half test-guarded.** +Audit result: **all** current members forward correctly, including both `ISurgicalAddressSpaceSink` methods with capability-sniffing fallback (`src/Core/…/DeferredAddressSpaceSink.cs:58-70`) — no inert capability today. But tests assert forwarding for only ~5 of the 10 members; `WriteAlarmCondition`, `MaterialiseAlarmCondition`, `EnsureFolder`, `RebuildAddressSpace`, `RaiseNodesAddedModelChange` have no per-member forwarding assertion (`tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests` / `DeferredAddressSpaceSinkTests.cs`). The F10b incident (surgical path dead in prod for exactly this reason) proves the failure mode is real. +*Recommendation:* add a reflection-driven exhaustive test: every method of `IOpcUaAddressSpaceSink` + every optional capability interface the wrapper claims must reach a recording inner sink. Same treatment for `DeferredServiceLevelPublisher`. + +**U3 — MEDIUM: The structured audit pipeline is fully built, tested, and has zero producers.** `AuditWriterActor` + `AuditOutcomeMapper` state "no live structured AuditEvent emit sites; all production audit flows through the bespoke stored-procedure path" (`ControlPlane/Audit/AuditOutcomeMapper.cs:12-18`; `Security/Audit/AuditActor.cs:18-24`). Either wire producers or delete — dormant tested code decays. + +**U4 — MEDIUM: `FleetStatusBroadcaster.DriverHostStatusHeartbeat` is dead code with a latent NodeId-mismatch repeat.** No producer anywhere sends the heartbeat (`FleetStatusBroadcaster.cs:38, 110-121`), so `CurrentRevision` is always null and the freshness branch never fires; and the broadcaster keys nodes by **host only** (`:152-159`) while every other subsystem uses `host:port` — if the heartbeat is ever wired with a `host:port` NodeId it will create phantom node records, re-enacting the historical NodeId bug. +*Recommendation:* wire the producer (DriverHostActor already knows its applied revision) or remove; either way canonicalise the key to `host:port`. + +**U5 — MEDIUM: Native Part 9 conditions support Acknowledge-to-driver only.** Confirm/AddComment/Shelve on a *native* condition still route to the scripted engine, which does not own them (documented H6c scope, `OtOpcUaNodeManager.cs:647-658`); Enable/Disable short-circuits to `BadNotSupported` (`:698-703`). Operators using a Part 9 client will find shelving a Galaxy alarm silently ineffective upstream. + +**U6 — MEDIUM: Test-coverage gaps concentrate exactly where the architecture is most fragile** (from the tests/Server sweep, ~910 tests total across 6 projects): +- **DPS delivery of redundancy/ServiceLevel state is untested** — `RedundancyStateActorTests` stub the broadcast, `ServiceLevelEndToEndTests` inject `NodeRedundancyState` directly, and the `TwoNodeClusterHarness` runs in-process with EF InMemory. This is the known "unit tests can't catch it" blind spot, and it isn't even documented in a test comment. +- **No actor supervision/restart test** anywhere (relates to S6) — only "init failure keeps Reconnecting state" is covered. +- **Outbox durability across process restart** and sustained-outage backpressure untested (a single failure→retry→ack cycle is). +- `OpcUaServer.IntegrationTests` contains **one test** (dual-endpoint ServerArray) — no over-the-wire security-mode matrix, subscription, or HistoryRead round-trip. +- Docker-fixture-dependent driver-protocol tests self-skip when `10.100.0.35` is unreachable — green local/macOS runs can hide protocol regressions. +Strengths worth noting: node-manager HistoryRead/paging (~60 tests), the write-gate/revert path, LDAP fail-closed edges, and Composer/Applier (~120 tests) are genuinely well covered. + +**U7 — LOW: `IHistoryWriter` is permanently Null-wired** (vtag historization, infra-gated on a nonexistent live-write RPC — `Runtime/ServiceCollectionExtensions.cs:56-58`); **H2 HistoryUpdate** remains unimplemented (HistoryRead-only server; `IsReadModified` rejected, `OtOpcUaNodeManager.cs:1797-1801`) — both tracked backlog, listed for completeness. `LdapAuthFailure` lacks a `DirectoryUnavailable` value (directory-down is overloaded onto `ServiceAccountBindFailed`). + +**U8 — LOW: `BuildSecurityPolicies` doc-comment claims the empty-profile fallback is "logged and very visible" but the method logs nothing** (`OpcUaApplicationHost.cs:376-418`). Defused in production by the options validator (`MinCount ≥ 1`, `OpcUaApplicationHostOptionsValidator.cs`), but the silent fail-open-to-None remains for direct embedders/tests; fix the comment or add the log. + +**U9 — LOW: `EnsureVariable` silently ignores changed historize-intent on an existing node** (documented, `OtOpcUaNodeManager.cs:1345-1349`) — correct today because the planner routes such deltas to `UpdateTagAttributes` or a rebuild, but the invariant lives in two files' comments rather than an assert. + +--- + +## Maturity ratings + +| Dimension | Rating (1-5) | Justification | +|---|---|---| +| Stability | **3** | Excellent node-manager locking/self-correction discipline and Null/Deferred boot safety, but the inert SBR (S1) breaks hard-crash failover cluster-wide, and LDAP/HistoryRead thread-blocking (S2/S3) plus the primary-gate boot window (S4) are real production exposures. | +| Performance | **3** | Hot paths are lean per-message and history paging is carefully bounded, but the rebuild-on-any-structural-change subscription wipe (P1) and per-value global-lock writes (P2) will not survive large tag counts or frequent deploys gracefully. | +| Conventions | **4** | Deliberate, consistent layering (Akka-free OpcUaServer, SDK-free Runtime, Null/Deferred seams, uniform actor patterns); loses a point for the five-way `ServerHistorian` binding, two-tier options validation, and static-logger islands. | +| Underdeveloped | **3** | Core capabilities are complete and the once-dormant provisioning/ref-feed seams are now wired, but stale load-bearing docs (U1), a half-guarded forwarding trap (U2), dormant audit/heartbeat code (U3/U4), and test blind spots exactly on the fragile seams (U6) leave meaningful debt. | + +--- + +## Cross-cutting themes + +1. **Config-in-HOCON is not config-in-effect** — the SBR block shows Akka behavior declared in `akka.conf` can be silently inert without the activating registration; audit other HOCON-declared behaviors (dispatcher blocks, singleton settings) the same way. +2. **The Deferred/Null seam pattern is powerful but trap-prone**: every new optional capability must be forwarded through the wrappers *and* exhaustively test-guarded, or it ships inert (proven twice: F10b surgical sink, PR #423 provisioning). A reflection-based forwarding test would retire the trap class. +3. **"Default-allow until told otherwise" gates** (primary write gate, peer-probe benefit-of-the-doubt, ServiceLevel legacy seam) consistently trade split-brain safety for boot-window availability; each is individually reasoned but they compound with at-most-once DPS delivery. +4. **Unit-test greenness systematically overstates distributed correctness** here: DPS delivery, supervision restarts, hard-kill failover, and Docker-gated protocol paths all sit outside the test net — and the project's own history (redundancy-delivery bug, F10b inertness) shows these are exactly where the bugs live. diff --git a/archreview/04-adminui.md b/archreview/04-adminui.md new file mode 100644 index 00000000..fb4a6493 --- /dev/null +++ b/archreview/04-adminui.md @@ -0,0 +1,219 @@ +# Architecture Review 04 — AdminUI (Blazor Server) + +- **Date:** 2026-07-08 +- **Commit:** `9cad9ed0` +- **Scope:** `src/Server/ZB.MOM.WW.OtOpcUa.AdminUI` (77 razor components, ScriptAnalysis backend, minimal APIs, SignalR bridges, UNS service layer), plus `tests/Server/ZB.MOM.WW.OtOpcUa.AdminUI.Tests` for coverage assessment. Auth *policy definitions* live in `ZB.MOM.WW.OtOpcUa.Security/ServiceCollectionExtensions.cs` and are referenced where the AdminUI consumes them. + +--- + +## Architecture Overview + +### Packaging and mounting + +The AdminUI is a **Razor Class Library**, not an app: the fused Host calls `AddAdminUI()` / `MapAdminUI()` (`EndpointRouteBuilderExtensions.cs:23–63`). All pages render `InteractiveServer`. `Routes.razor` uses `AuthorizeRouteView` (fixing the historical Admin-001 inert-attribute bug) with login redirect + a permission-denied slot. `MainLayout` delegates to the shared `ZB.MOM.WW.Theme` `ThemeShell` chassis. + +### Page map + +| Area | Pages | +|---|---| +| Dashboards | `Home`, `Fleet` (per-node deploy state, 10 s poll), `Hosts` (Akka topology + driver health, 5 s poll + event push), `AlarmsHistorian` (sink status, 5 s poll) | +| Live tails | `Alerts` (alarm transitions + DriverOperator Ack/Shelve), `ScriptLog` | +| Config: clusters | `ClustersList`, `ClusterEdit/Overview/Namespaces/Acls/Audit/Redundancy/Drivers`, `NodeEdit`, `NamespaceEdit`, `AclEdit`, `NewCluster`, 8 driver pages behind `DriverEditRouter` + `DriverTypePicker` | +| Config: UNS | `GlobalUns` (tree + Area/Line modals + CSV import), `EquipmentPage` (tabbed Details/Tags/VirtualTags/Alarms with `TagModal`/`VirtualTagModal`/`ScriptedAlarmModal`) | +| Scripting | `Scripts`, `ScriptEdit` (Monaco), `ScriptLog` | +| Ops/admin | `Deployments`, `Certificates`, `Reservations`, `RoleGrants`, `Account`, `Login` | + +### API surface + +- `POST /api/deployments` (`Api/DeployApiEndpoints.cs`) — headless deploy trigger; `AllowAnonymous` but self-gated on `Security:DeployApiKey` via `X-Api-Key` with fixed-time compare, 503 when unconfigured. Clean, well tested. +- `POST /api/script-analysis/{diagnostics,completions,hover,signature-help,format,inlay-hints}` (`ScriptAnalysis/ScriptAnalysisEndpoints.cs`) — Roslyn-backed Monaco language services, gated `Roles = "Administrator,Designer"`. +- SignalR hubs `/hubs/{alerts,driverstatus,fleetstatus,scriptlog}` — all `[Authorize]`; retained for out-of-process clients only. + +### State / broadcast patterns + +The load-bearing pattern is **DPS topic → per-node bridge actor → in-process singleton → component event subscription**: + +- `AlertSignalRBridge` / `ScriptLogSignalRBridge` publish to `IInProcessBroadcaster` (stream feeds, with `IsConnected` health for the "live" pill). +- `DriverStatusSignalRBridge` feeds `IDriverStatusSnapshotStore` (last-value feed). +- Components subscribe to the singleton's .NET events and marshal via `InvokeAsync`; **no component anywhere opens a server-side `HubConnection` to its own hub** — the known forbidden pattern is documented in `IInProcessBroadcaster.cs:9–15` and honored consistently across all four live surfaces (`Alerts`, `ScriptLog`, `DriverStatusPanel`, `Hosts`). + +Data access is `IDbContextFactory` per call. UNS goes through a real service seam (`Uns/UnsTreeService.cs`, 1531 lines, scoped, RowVersion-aware). Driver browse uses a singleton `BrowseSessionRegistry` + `BrowseSessionReaper` (2 min idle TTL, 30 s tick, dispose-outside-dictionary) + per-call 20 s timeouts — a tidy lifecycle design. + +--- + +## 1. STABILITY + +### S-1 (Medium) — Timer disposal is inconsistent; three components don't drain in-flight callbacks + +The codebase's own best practice (stated in comments) is `System.Threading.Timer` + `await DisposeAsync()` so an in-flight tick can't call `StateHasChanged` on a disposed component. `Hosts.razor:348–355`, `Alerts.razor:291–299`, and `DriverStatusPanel.razor:301–310` do this correctly. Three components don't: + +- `Components/Pages/Fleet.razor:171` — `public void Dispose() => _timer?.Dispose();` (sync, no drain). +- `Components/Pages/AlarmsHistorian.razor:90` — same. +- `Components/Shared/Drivers/DriverTestConnectButton.razor:68–81` — uses `System.Timers.Timer` with an `async (_, _)` `Elapsed` handler (line 69; effectively async-void) and a sync `Dispose`, directly contradicting the comment convention in `Alerts.razor:243–244`. + +**Recommendation:** converge on the `IAsyncDisposable` + `Threading.Timer` idiom in all three; it is already the documented house pattern. + +### S-2 (Medium) — `Fleet.razor` auto-refresh has no error handling; DB faults become unobserved task exceptions + +`Fleet.razor:112` schedules `_ = InvokeAsync(LoadAsync)`; `LoadAsync` (`Fleet.razor:119–159`) has `try/finally` but **no catch** — a transient SQL failure faults the discarded task silently and the page shows stale data with no error surface. Contrast `Hosts.razor:269–287`, which catches, logs, and degrades gracefully. `AlarmsHistorian.razor:73–78` has the opposite problem: a bare `catch { }` that leaves the page on "Loading…" forever (the comment admits the role-gating message is a TODO). + +**Recommendation:** copy the `Hosts.LoadConfigAsync` catch-log-degrade shape into `Fleet.LoadAsync`; give `AlarmsHistorian` an explicit "no historian on this node role" state. + +### S-3 (Medium) — No `` anywhere + +`Routes.razor` / `MainLayout.razor` wrap nothing in `ErrorBoundary`; a single unhandled exception in any event handler or render tears down the entire circuit to the framework error UI. For a 77-component operations console this is a coarse failure mode. + +**Recommendation:** wrap `@Body` in `MainLayout` in an `ErrorBoundary` with a recover button; consider per-panel boundaries around the live-tail surfaces. + +### S-4 (Medium) — Delete flows re-fetch RowVersion at click time, defeating the optimistic-concurrency guard + +`UnsTreeService` implements RowVersion concurrency rigorously (22 `OriginalValue` sites, 10 `DbUpdateConcurrencyException` handlers), and the *edit* paths carry the RowVersion loaded when the modal opened — correct. But the *delete* paths load a **fresh** DTO immediately before deleting and pass its current RowVersion: + +- `GlobalUns.razor:279–345` (`ConfirmDeleteAsync` — Area/Line/Equipment branches, e.g. line 293). +- `EquipmentPage.razor:370–380, 415–425, 460–470` (`DeleteTag`/`DeleteVirtualTag`/`DeleteAlarm`). + +The guard therefore always passes: a concurrent edit between rendering the row and clicking Delete is silently destroyed. The comments ("Load the tag fresh to capture its current RowVersion") show this is deliberate but it makes delete a *last-writer-wins* operation while updates are *first-writer-wins* — an inconsistent contract. + +**Recommendation:** carry the RowVersion from the rendered row/DTO into the delete call (the list DTOs would need to carry it), or document delete as intentionally unconditional. + +### S-5 (Medium) — Destructive actions on `EquipmentPage` and `ScriptEdit` have no confirmation + +`GlobalUns` has a proper delete-confirm modal (`GlobalUns.razor:71–98`) and `DriverStatusPanel` has an inline Restart confirm (`DriverStatusPanel.razor:119–131`). But `EquipmentPage.razor:172/217/263` delete a tag/virtual-tag/alarm on a single click, and `ScriptEdit.razor:140` deletes a script the same way. + +**Recommendation:** reuse the `GlobalUns` confirm-modal pattern (or a shared `ConfirmButton` component) for all deletes. + +### S-6 (Low) — `DriverStatusPanel` result-chip timer lacks the stale-fire token guard `Alerts` added + +`Alerts.razor:244–257` guards against a disposed-but-already-queued timer callback clearing a *newer* result via a token compare. `DriverStatusPanel.razor:287–299` has the identical construct without the guard. Harmless-looking but the race the Alerts comment describes applies equally here. + +### S-7 (Low) — `AdminOperationsClient.AskAsync` bypasses the 10 s ask timeout + +`Clients/AdminOperationsClient.cs:58–59` — the generic `AskAsync` (used by `DriverStatusPanel` Reconnect/Restart) forwards only the caller's token; the three typed methods double-guard with `AskTimeout`. Callers do pass a 15 s CTS today, so this is latent, but the seam's contract is inconsistent. + +### Positive stability notes + +- The forbidden self-`HubConnection` pattern is absent; every live surface reads the in-process singleton, and the rationale is documented at the seam. +- Zero `async void` in the project (the one async event-handler lambda is S-1's `System.Timers` case). +- Event-handler hygiene is excellent: subscribe-before-read TOCTOU ordering (`Alerts.razor:154–159`), unsubscribe-before-timer-drain dispose ordering, capture-then-invoke in `InProcessBroadcaster.SetConnected` (`IInProcessBroadcaster.cs:86–103`). +- `AdminOperationsActor` access is exclusively via `Ask` through the singleton proxy — no shared mutable state in the AdminUI layer itself; the snapshot store and broadcaster are the only singletons and both are lock/`ConcurrentDictionary`-guarded. +- `MonacoEditor.razor` disposes cleanly and distinguishes `JSDisconnectedException` (silent) from real `JSException` (logged). + +--- + +## 2. PERFORMANCE + +### P-1 (Medium) — Monaco pushes the full document to .NET on every keystroke + +`wwwroot/js/monaco-init.js:205–208`: `onDidChangeModelContent` → `invokeMethodAsync("OnValueChanged", editor.getValue())` with **no debounce** — the entire script text crosses the SignalR circuit per keystroke, and `ValueChanged.InvokeAsync` re-renders the parent (`ScriptEdit`, virtual-tag modal). Diagnostics are correctly debounced at 500 ms, but the value sync is not. + +**Recommendation:** debounce the value push (~200–300 ms) or send it only on blur/save/diagnostic-tick; the .NET side only needs the value for Save and the inline problems panel. + +### P-2 (Medium) — ScriptAnalysis recompiles from scratch on every request, six endpoints deep + +`ScriptAnalysis/ScriptAnalysisService.cs:60–67`: each call to any endpoint re-parses and creates a fresh `CSharpCompilation`; `Diagnose` additionally runs full `GetDiagnostics()`. A typing user triggers diagnostics (500 ms debounce) + completions + hover + signature-help against the *same text*, each paying a full compile. Static caching of the sandbox references and preamble is already done (good); there is no memoization of `(text → tree/compilation)`. + +**Recommendation:** add a small LRU (even size-1 keyed on the normalized text) shared by the endpoints; this collapses the common completions-after-diagnostics case. Fine at single-admin scale — flag before multi-user rollout. + +### P-3 (Low) — Per-circuit DB polling on the dashboard pages + +`Hosts.razor:229–245` reloads `ClusterNodes` + `DriverInstances` from SQL every 5 s *per open circuit*; `Fleet.razor` runs a `GroupBy` over `NodeDeploymentStates` every 10 s. Driver health itself is event-pushed (good) — only the config enrichment polls. Acceptable today; a shared `IMemoryCache` (or reacting to the deploy event) would decouple cost from viewer count. + +### P-4 (Low) — `Deployments.razor:98` runs `ConfigComposer.SnapshotAndFlattenAsync` (full config snapshot + hash) on every page load and after every deploy, purely to compute the drift badge. On a large fleet this is the page's dominant cost. + +### P-5 (Low) — No `` anywhere; lists are instead bounded by design (`Alerts` cap 200, `Deployments` `Take(50)`, cluster-scoped tables). Adequate. Two nits: `Alerts.razor:63` renders rows without `@key` while prepending (`Insert(0, …)`), forcing whole-table diffs per event ( `EquipmentPage` tables do use `@key`); the `UnsTree` filter re-runs the recursive `VisibleUnder` walk per render (self-documented as bounded/cheap — agreed). + +### P-6 (Low) — `inlay-hints` round-trips for a documented no-op + +`monaco-init.js:99–117` registers an inlay-hints provider that POSTs on every model change; `ScriptAnalysisService.InlayHints` (`ScriptAnalysisService.cs:415`) always returns empty. Skip registering the provider until the feature exists. + +--- + +## 3. CONVENTIONS + +### C-1 (High) — Three authorization idioms coexist, and the largest mutating surface is effectively ungated + +Observed gates: + +| Idiom | Where | +|---|---| +| `Roles = "Administrator,Designer"` | `Deployments.razor:12`, `Scripts.razor`, `ScriptEdit.razor:5`, `ScriptAnalysisEndpoints.cs:18` | +| `Policy = "FleetAdmin"` | `RoleGrants.razor:2` (page), `Certificates.razor:90,186` (per-action) | +| `Policy "DriverOperator"` (imperative) | `Alerts.razor:165`, `DriverStatusPanel.razor:166` | +| **bare `[Authorize]`** | **everything else** — `GlobalUns`, `EquipmentPage`, `ClusterEdit`, `NodeEdit`, `NamespaceEdit`, `AclEdit`, all 8 driver pages, `Reservations` | + +Consequence: any authenticated user — including a read-only viewer — can create/edit/delete areas, lines, equipment, tags, virtual tags, scripted alarms, driver instances, ACLs, and namespaces; only the *deploy* button is role-gated. Project memory says global-roles/simplest-authz is a deliberate posture, but the current state isn't "simplest" — it's three idioms plus an unguarded majority, which is exactly the drift that produces surprises later. + +**Recommendation:** centralize policy names as constants (they exist only as string literals today), pick policies over role strings (`FleetAdmin` already means `RequireRole("Administrator")`), and apply one write-gate (`Administrator,Designer` equivalent) to every config-mutating page. This is a one-attribute-per-page change. + +### C-2 (Medium) — CLAUDE.md / code drift on the ScriptAnalysis gate + +CLAUDE.md states the `/api/script-analysis/*` endpoints are "gated by the `FleetAdmin` policy"; the code uses `Roles = "Administrator,Designer"` (`ScriptAnalysisEndpoints.cs:17–18`, comment says it deliberately matches the Scripts page). Fix the doc (or converge per C-1). + +### C-3 (Good) — Driver-typed tag-editor pattern is complete and consistent + +`Uns/TagEditors/TagConfigEditorMap.cs:10–20` and `TagConfigValidator.cs:12–22` cover **all seven** non-Galaxy drivers (Modbus, S7, AbCip, AbLegacy, TwinCat, Focas, OpcUaClient) with parallel keys; Galaxy intentionally uses the live-browse picker + raw `{"FullName": …}` JSON (`TagModal.razor:96–125`). Every editor follows the same shell-over-pure-model shape (`FromJson`/`ToJson`/`Validate`, preserve-unknown-keys), with the parse-only-on-real-change guard against parent re-renders (`ModbusTagConfigEditor.razor:52–57`). The composable root-key seams (`TagHistorizeConfig`, `TagArrayConfig`, `NativeAlarmModel`) merge onto the same JSON without clobbering — a genuinely good design. + +### C-4 (Good) — The numeric-enum serialization bug class is closed + +All 8 driver pages carry `JsonStringEnumConverter` in their serializer options; tag editors bind enums as strings via `Enum.GetValues`/`TryParse`; and the fix is pinned by per-driver `*FormSerializationTests` plus `DriverPageJsonConverterTests` in the test project — regression-proofed, not just patched. + +### C-5 (Medium) — Two data-access idioms: service seam vs. EF-in-page + +UNS pages go through `IUnsTreeService` (testable, RowVersion-aware, 12 test files). But `ScriptEdit.razor:87–156`, `Scripts`, `Deployments.razor:83–100`, `Fleet`, and `Hosts` inject `IDbContextFactory` and run EF (including entity mutation and `SaveChangesAsync`) directly in `@code`. The service-seam pages are the ones with meaningful test coverage; the EF-in-page logic is untestable without a host. New pages copy whichever file is nearest. + +**Recommendation:** declare the service-seam the convention; migrate `ScriptEdit`'s save/delete into the existing `UnsTreeService` script methods (`UpdateScriptSourceAsync` already exists at `IUnsTreeService.cs:502`). + +### C-6 (Low) — Naming/marker nits + +`"GalaxyMxGateway"` is a bare string literal in `TagModal.razor:311`; the `DataTypes` array (`TagModal.razor:244`) duplicates type lists that exist elsewhere; casing drifts across `TwinCat` (map key) / `TwinCATTagConfigEditor` / `FOCASAddressPickerBody` / `FocasTagConfigEditor`. CSS is disciplined (one `site.css` + shared theme kit; chip/panel classes reused consistently, `chip-warn` vs `chip-caution` both exist — verify both are defined in the theme). + +--- + +## 4. UNDERDEVELOPED AREAS + +### U-1 — Test coverage: strong C# seams, zero component rendering coverage + +443 `[Fact]`/`[Theory]` cases covering: all tag-config models + validator, `UnsTreeService` (structure/tags/vtags/import/equip-token/script CRUD against a test DB), browse session service/registry/reaper, `CertificateStoreManager`, audit writers, `DeployApiEndpoints` auth matrix, driver-page form serialization (the enum bug class), 8 ScriptAnalysis suites (diagnose/completions/hover/format/tag-path/equip-token), `InProcessBroadcaster`, `HostsDriverView`, address builders. That is a deliberately shaped portfolio around the pure seams. + +There is **no bUnit** — 77 razor components have no render/binding coverage, which is the documented repo posture ("no bUnit — live-verify"; the `@`-binding bug class passes all unit tests). The consequence is that everything in `@code` blocks that isn't extracted (e.g. all of `EquipmentPage`'s tab/delete handlers, `GlobalUns`'s modal orchestration, `Alerts`' row-action state machine) is verified only by manual `/run` sessions. + +**Recommendation:** either adopt bUnit for the modal/table state machines, or keep extracting page logic into plain classes (the `HostsDriverView`/`VirtualTagModalHelpers` precedent) so the untested residue in `@code` stays trivial. + +### U-2 — Stub/known-gap behaviors + +- `InlayHints` is a wired no-op endpoint + active client provider (P-6); documented in `ScriptAnalysisService.cs:22–25`. +- `AlarmsHistorian.razor:73–78` — admits the missing role-gated "no historian on this node" message; page silently shows "Loading…" forever on admin-only nodes. +- `MonacoEditor.razor:33–37` — `MarkersChanged` is typed `object[]` with a comment that the DTO "is not modelled yet". +- `GlobalUns.ConfirmDeleteAsync` default branch returns "Delete for this node kind is not yet available." for Tag/VirtualTag nodes (`GlobalUns.razor:~320`) — consistent with equipment-page ownership, but the message leaks a not-implemented posture into the UI. + +### U-3 — Raw-JSON fallback surfaces + +`TagModal.razor:130–136` falls back to a schemaless textarea for unmapped drivers ("Validated server-side at deploy" — i.e., no client validation), and the Galaxy path allows direct JSON editing beside the picker. With all current drivers mapped (C-3), the fallback today only serves Galaxy + future drivers, but it remains the escape hatch through which invalid configs reach deploy time. `_form.TagConfig` is `[Required]` yet never checked for *well-formed JSON* in the fallback path. + +**Recommendation:** at minimum, `JsonDocument.Parse` the fallback textarea in `SaveAsync` before submitting. + +### U-4 — Accessibility + +~31 `aria-*`/`role=` attributes across 77 components, concentrated in the modals (`role="dialog"`, `btn-close` aria-labels). Gaps: text-glyph expander buttons (`▼`/`▶` in `UnsTree.razor:49`) with no `aria-expanded`/label; modals have no focus trap or Escape handling; live tails (`Alerts`) have no `aria-live` region; tables lack captions/scope. Form labeling is decent (most inputs have `for`/`id` pairs). Acceptable for an internal ops tool; worth a targeted pass on the tree and the modals if operator diversity matters. + +--- + +## Maturity Ratings + +| Dimension | Rating (1–5) | Justification | +|---|---|---| +| Stability | **4** | Disciplined disposal/marshaling with documented race reasoning and the self-HubConnection ban honored everywhere; docked for three non-draining timers, `Fleet`'s uncaught refresh path, no `ErrorBoundary`, and the RowVersion-refetch delete pattern. | +| Performance | **3** | Event-push where it matters and bounded lists, but keystroke-grain Monaco interop, per-request Roslyn recompiles ×6 endpoints, per-circuit DB polling, and a full config snapshot per Deployments render. | +| Conventions | **3** | The tag-editor/validator pattern is exemplary and test-pinned, but authorization is a three-idiom mix with the main mutating surface ungated, plus service-seam vs EF-in-page duality and a CLAUDE.md gate-description drift. | +| Underdeveloped | **3** | 443 well-aimed seam tests, but zero rendering coverage for 77 components (by policy), several admitted stubs, an unvalidated raw-JSON escape hatch, and thin accessibility. | + +--- + +## Top Recommendations (ordered) + +1. **C-1:** Centralize policy constants and put a write-role gate on every config-mutating page (`/uns`, equipment, cluster/driver/ACL/namespace editors, Reservations). +2. **S-4 + S-5:** Make deletes carry the rendered RowVersion (or document last-writer-wins) and add confirm dialogs on `EquipmentPage`/`ScriptEdit` deletes. +3. **S-3:** Add an `ErrorBoundary` around `@Body`. +4. **P-1:** Debounce Monaco's value push to .NET. +5. **S-1/S-2:** Converge the three straggler timers on the async-dispose idiom and give `Fleet.LoadAsync` the `Hosts`-style catch. +6. **C-5/U-1:** Declare the service-seam pattern canonical and keep extracting `@code` logic into testable classes. diff --git a/archreview/05-protocol-drivers.md b/archreview/05-protocol-drivers.md new file mode 100644 index 00000000..2f355571 --- /dev/null +++ b/archreview/05-protocol-drivers.md @@ -0,0 +1,616 @@ +# Architecture Review 05 — Protocol Drivers + +**Date:** 2026-07-08 +**Commit:** `9cad9ed0` +**Scope:** the seven cross-platform protocol drivers and their satellites — +`Driver.Modbus` (+ `.Addressing`, `.Contracts`), `Driver.S7` (+ `.Contracts`), +`Driver.AbCip` (+ `.Contracts`), `Driver.AbLegacy` (+ `.Contracts`), +`Driver.TwinCAT` (+ `.Contracts`), `Driver.FOCAS` (+ `.Contracts`), +`Driver.OpcUaClient` (+ `.Contracts`, `.Browser`), and `src/Drivers/Cli/*` +(six per-driver CLI harnesses + `Cli.Common`). Test projects under +`tests/Drivers/` and `tests/Drivers/Cli/` were assessed for coverage only. +Galaxy and the Historian drivers are out of scope. + +**Method:** Modbus was read exhaustively as the reference driver (it is the +documented origin of the shared `PollGroupEngine` and the richest read +planner); each other driver's main client/poller/factory/probe/contracts +files were then read in full and diffed structurally against the Modbus +shape. Cross-cutting greps covered TODO/HACK/NotImplemented, JSON +serializer options, `PollGroupEngine` construction, and interface +declarations. File references are repo-relative; drivers are abbreviated to +their suffix (e.g. `Driver.S7/S7Driver.cs`). + +--- + +## Architecture Overview + +### The common driver shape + +Every protocol driver is expected to follow the same template: + +1. **Factory extension** (`DriverFactoryExtensions.Register`) registers a + `DriverTypeName → CreateInstance(id, configJson)` closure with the + `DriverFactoryRegistry` (wired centrally in + `src/Server/ZB.MOM.WW.OtOpcUa.Host/Drivers/DriverFactoryBootstrap.cs`). + Config parses through a string-typed DTO with fail-loud `ParseEnum` + helpers and `PropertyNameCaseInsensitive` + comment/trailing-comma + tolerance. +2. **Driver class** implements `IDriver, ITagDiscovery, IReadable, + IWritable, ISubscribable, IHostConnectivityProbe, IPerCallHostResolver, + IDisposable, IAsyncDisposable` (Modbus, + `Driver.Modbus/ModbusDriver.cs:21-22`), with optional capability + interfaces (`IAlarmSource`, `IRediscoverable`, `IHistoryProvider`) per + protocol. +3. **Reference resolution** goes through the shared + `EquipmentTagRefResolver` + (`src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/EquipmentTagRefResolver.cs`), + bridging authored tag-table names and raw equipment-tag `TagConfig` JSON + blobs (parsed once by a per-driver `EquipmentTagParser` in the + Contracts project, cached, cleared on teardown). +4. **Subscriptions** overlay a polling loop via the shared `PollGroupEngine` + (`Core.Abstractions/PollGroupEngine.cs`) for protocols without a push + model; the driver supplies the batch reader + on-change bridge. +5. **Health** is an immutable `DriverHealth` snapshot published via + `Volatile.Read/Write`; failures set `Degraded` while retaining + `LastSuccessfulRead`; `Faulted` is reserved for permanent config faults. +6. **Writes** return per-item `WriteResult` with protocol-error → OPC UA + StatusCode mapping (a per-driver `StatusMapper` or switch). Accurate + per-item statuses are load-bearing: the server's write-outcome + self-correction (`OtOpcUaNodeManager` reverting the node value on a + failed device write) only works if the driver reports the failure. +7. **Probe loop** — a background task issuing a cheap protocol-level + handshake, raising `OnHostStatusChanged` on Running↔Stopped transitions + under a probe lock; plus a stateless `DriverProbe : IDriverProbe` + backing the AdminUI Test Connect button. +8. **Teardown** — one shared `TeardownAsync` used by both `ShutdownAsync` + and `DisposeAsync`: cancel loops, await them, dispose CTSs, clear + caches, dispose the transport. + +Modbus adds the most advanced read planner in the fleet: block-read +coalescing (`MaxReadGap`), auto-prohibition of failing ranges with +bisection re-probing (`ModbusDriver.cs:687-795, 553-677`), chunking over +per-device caps, per-register bit-RMW locks, deadband publish filtering, +and `WriteOnChangeOnly` suppression with read-side invalidation. + +### Per-driver conformance matrix + +| Aspect | Modbus (ref) | S7 | AbCip | AbLegacy | TwinCAT | FOCAS | OpcUaClient | +|---|---|---|---|---|---|---|---| +| LOC (driver proj) | 2,374 | 1,979 | 3,762 | 1,830 | 2,261 | 5,007 | 2,347 | +| `IPerCallHostResolver` | yes | **no** | yes | yes | yes | yes | no (single endpoint, defensible) | +| Extra capabilities | — | — | `IAlarmSource` | — | `IRediscoverable` | `IAlarmSource` | `IAlarmSource`, `IHistoryProvider` | +| Subscription model | PollGroupEngine | **bespoke forked loop** | PollGroupEngine | PollGroupEngine | native ADS push (default) + PollGroupEngine fallback | PollGroupEngine | **real OPC UA MonitoredItems** | +| `onError` sink passed to engine | no | n/a | no | no | no | no | n/a | +| Reconnect story | transport auto-reconnect + geometric backoff | **none** | evict + recreate handle, no backoff | **no eviction on data path** | lazy per-call reconnect, **native subs orphaned** | lazy reconnect per tick, no backoff | `SessionReconnectHandler` + re-arm (best) | +| Read batching | block coalescing + bisection | per-tag | per-tag; UDT planner **half-built** | per-tag | per-tag (no ADS sum-read) | per-address, single-flight socket | batched `ReadValueIdCollection` | +| Deadband / WriteOnChangeOnly | yes / yes | no / no | no / no | no / no | no / no | no / no | n/a (passthrough) | +| Bit-level writes | RMW + per-register lock | native S7 bit write | RMW + per-parent lock (DINT-only assumption) | RMW + per-parent lock, width-aware | RMW + per-parent lock (**4-byte width assumption unverified**) | n/a (read-only backend) | n/a | +| Health via `Volatile`/`volatile` | yes | **plain field** | **plain field** | `volatile` | **plain field** | yes | **plain field** | +| `InitializeAsync` honours configJson | **no** | yes | yes | **no** | yes (empty-config quirk) | yes | yes | +| Factory `Register` takes `ILoggerFactory` | yes | **no** | **no** | yes | **no** | **no** | yes | +| Options `Validate()` | no (none in fleet) | no | no | no | no | no | no | +| Live browse / discovery | declared tags only | declared tags only | declared + opt-in controller browse + UDT fan-out | declared only (correct for PCCC) | declared + opt-in symbol browse + expander | declared + capability-gated fixed tree | full recursive remote browse | +| CLI harness | yes | yes | yes | yes | yes (+`browse`) | yes | **none** | + +--- + +## Findings + +Severity scale: **Critical** = live data-loss/unavailability in a realistic +field scenario; **High** = correctness or stability defect likely to bite; +**Medium** = architectural debt / divergence with concrete failure modes; +**Low** = hygiene. + +### 1. Stability + +**STAB-1 — Critical — S7 has no reconnect path at all.** +`Plc.OpenAsync` is called exactly once, in `InitializeAsync` +(`Driver.S7/S7Driver.cs:165-167`). Read/write failures set `Degraded` and +keep using the dead `Plc` (`S7Driver.cs:479-483, 995-1001`); the probe loop +detects the outage, transitions `Stopped` (`S7Driver.cs:1351-1383`) — and +keeps probing the same dead connection forever. A transient PLC reboot or +network blip permanently kills the driver instance until an external +redeploy/restart. Every sibling driver has *some* recovery path (Modbus: +transport-level reconnect+backoff; TwinCAT/FOCAS: lazy recreate; AbCip: +handle eviction). *Recommendation:* on socket-level `PlcException`, dispose +and lazily reopen the `Plc` under `_gate` (mirror TwinCAT's +`EnsureConnectedAsync`), or let the probe loop own recycle-on-Stopped. +Note the test suite has **zero reconnect tests for S7** (see TEST-1), which +is how this shipped. + +**STAB-2 — Critical — TwinCAT native subscriptions are silently orphaned +after reconnect.** Native ADS notifications are the *default* mode +(`Driver.TwinCAT.Contracts/TwinCATDriverOptions.cs:31`). When +`EnsureConnectedAsync` discards a stale client +(`Driver.TwinCAT/TwinCATDriver.cs:615-619`), `AdsTwinCATClient.Dispose` +clears its notification table (`AdsTwinCATClient.cs:422-436`) and nothing +re-runs `AddNotificationAsync` on the replacement client — all subscribed +data flow stops with no Bad status and no error. Compounding: the fast path +keys on `IsConnected` (`TwinCATDriver.cs:606`), which reflects the local +AMS port, not wire liveness, so the replacement path may never even +trigger; and a probe failure never recycles the client +(`TwinCATDriver.cs:542-544`). *Recommendation:* record registration intent +(symbol, interval, handler) per subscription and replay it whenever the +per-device client instance changes; make probe-detected outage force a +client recycle. + +**STAB-3 — High — Modbus transport desynchronizes after a response +timeout.** `SendAsync` reconnect-retries only on `IsSocketLevelFailure` +(`Driver.Modbus/ModbusTcpTransport.cs:155-165, 257-261`). A per-op timeout +surfaces as `OperationCanceledException` — not socket-level — so the socket +is kept with a half-read (or still-in-flight) response. The next +transaction then reads the stale response, hits the TxId mismatch check +(`ModbusTcpTransport.cs:233-235`), and throws `InvalidDataException` — +which is *also* not socket-level, so the stream is still not torn down. +The single-flight connection stays permanently desynchronized until a real +socket error or the (optional, default-off) idle-disconnect fires. +*Recommendation:* treat per-op timeout and any framing violation +(TxId mismatch, truncated header) as connection-fatal: tear down the socket +before propagating. + +**STAB-4 — High — AbCip runs concurrent operations on shared libplctag +handles with no serialization.** `ReadSingleAsync` / `ReadGroupAsync` / +`WriteAsync` execute Read→GetStatus→Decode and Encode→Write→GetStatus on +cached `IAbCipTagRuntime` instances with zero locking +(`Driver.AbCip/AbCipDriver.cs:544-575, 614-637, 708-726`) while the server +read path, every poll-group loop, and the alarm-projection loop +(`AbCipAlarmProjection.cs:228`) can hit the same handle concurrently. +AbLegacy documents this exact hazard and guards it with a per-runtime +operation lock (“A libplctag Tag handle is not safe for concurrent +Read/GetStatus/DecodeValue”, `Driver.AbLegacy/AbLegacyDriver.cs:775-787`); +the fix never propagated to AbCip. Consequences: torn/wrong values decoded +with Good status, cross-attributed `GetStatus` results. +*Recommendation:* port AbLegacy's `GetRuntimeLock` pattern to AbCip. + +**STAB-5 — High — FOCAS fixed-tree caches are plain `Dictionary`s mutated +across threads.** `LastFixedSnapshots` / `LastServoLoads` / +`LastSpindleLoads` / `LastTimers` (`Driver.FOCAS/FocasDriver.cs:1184-1194`) +are written by `FixedTreeLoopAsync` (`FocasDriver.cs:752, 793, 836-839`) +while `TryReadFixedTree` reads them on `ReadAsync` callers' threads +(`FocasDriver.cs:912, 920, 945`). Concurrent write-during-read on +`Dictionary` is undefined behaviour (corruption or throw on resize). +The neighbouring `_parsedAddressesByTagName` already got the +`ConcurrentDictionary` treatment (`FocasDriver.cs:40`); these did not. +*Recommendation:* convert to `ConcurrentDictionary` or swap immutable +snapshots. + +**STAB-6 — High — AbLegacy probe-loop shutdown race; S7 unsubscribe +race.** AbLegacy discards the probe task at spawn (`_ = Task.Run(...)`, +`Driver.AbLegacy/AbLegacyDriver.cs:118`) and `ShutdownAsync` cancels then +immediately disposes the CTS and runtimes without awaiting loop exit +(`AbLegacyDriver.cs:156-170`) — a winding-down loop can raise +`OnHostStatusChanged` after shutdown and touch a disposed CTS. AbCip fixed +this identical bug (retained `ProbeTask` + phased shutdown, +`Driver.AbCip/AbCipDriver.cs:295-298, 332-365`); the fix never propagated +back. S7 has the mirror-image bug in `UnsubscribeAsync`: cancel + dispose +the CTS without draining `PollTask` (`Driver.S7/S7Driver.cs:1185-1193`), +so `Task.Delay` on the disposed source can throw `ObjectDisposedException` +that the loop only catches as OCE (`S7Driver.cs:1233-1234`) — +`PollGroupEngine.StopState` exists precisely to prevent this +(`PollGroupEngine.cs:99-112, 136`). *Recommendation:* await the loop task +(bounded) before disposing the CTS in both places. + +**STAB-7 — High — `InitializeAsync` ignores `driverConfigJson` in Modbus +and AbLegacy.** AbLegacy's `_options` is readonly ctor state and +`Initialize/Reinitialize` never parse the parameter +(`Driver.AbLegacy/AbLegacyDriver.cs:16, 78-153`); Modbus likewise uses only +ctor options (`Driver.Modbus/ModbusDriver.cs:167-197`). S7, AbCip and +TwinCAT deliberately re-parse (`Driver.AbCip/AbCipDriver.cs:229-236` +documents “rather than being silently discarded”). Any config change +delivered through `ReinitializeAsync` on a live instance is silently +discarded for these two drivers. TwinCAT has a related quirk: a parsed +config with zero devices AND zero tags is silently discarded in favour of +ctor options (`Driver.TwinCAT/TwinCATDriver.cs:89-94`), so an intentional +“empty the driver” redeploy is ignored. *Recommendation:* make config +re-parse on initialize a template requirement; add a consuming test per +driver. + +**STAB-8 — Medium — No reconnect backoff anywhere except the Modbus +transport and S7's poll loop.** A dead device pays: AbCip — a full `Tag` +create + Forward Open per tag per poll tick (evict-recreate, +`AbCipDriver.cs:891-897`); FOCAS — a full two-socket reconnect attempt per +tick of *each* of three loops (`Driver.FOCAS/FocasDriver.cs:1089-1120`); +TwinCAT — a connect attempt per call (`TwinCATDriver.cs:602-651`). +`PollGroupEngine` itself polls at a fixed cadence regardless of failures — +S7's bespoke loop is the only subscriber-side backoff in the fleet +(`S7Driver.cs:1284-1293`, capped 30 s). *Recommendation:* add optional +failure backoff to `PollGroupEngine` (adopt S7's `ComputeBackoffDelay`) +and a small connect-attempt throttle to the lazy-reconnect drivers. + +**STAB-9 — Medium — `PollGroupEngine`'s `onError` sink is dead code: no +driver passes it.** All five consumers construct the engine without the +error callback (`ModbusDriver.cs:115`, `AbCipDriver.cs:133`, +`AbLegacyDriver.cs:65`, `TwinCATDriver.cs:69`, `FocasDriver.cs:72`), so a +reader exception in a poll tick (e.g. `RequireTransport` after teardown, +reader contract violation) is silently swallowed — the precise +“silently-broken subscription” failure mode the engine's own doc comment +warns about (`PollGroupEngine.cs:21-25, 164-169`). *Recommendation:* pass +`onError` routing to the driver's health surface + logger in all five; +consider making the parameter required. + +**STAB-10 — Medium — AbLegacy never evicts failed tag handles on the data +path.** Non-zero status and transport exceptions leave the cached runtime +in place (`AbLegacyDriver.cs:246-259, 269-274`) with no recreate-on-failure +(contrast `AbCipDriver.cs:891-897` and AbLegacy's own probe loop at +`AbLegacyDriver.cs:456-461`). A handle invalidated by a PLC state change +pins the tag Bad until full driver reinit. + +**STAB-11 — Medium — RMW locks don't cover direct parent writes (AbCip, +AbLegacy, TwinCAT).** The per-parent semaphore serializes bit-RMW callers +against each other, but a direct write to the parent word/DINT itself goes +through the normal write path without taking the same lock +(`AbCipDriver.cs:708-710` vs `787-818`; same shape in AbLegacy and +TwinCAT), so it can interleave with an in-flight read-modify-write → lost +update. Additionally AbCip's RMW hard-codes DINT while `AbCipTagPath` +accepts `.N` on any parent (`Driver.AbCip/AbCipTagPath.cs:17-19`), and +TwinCAT's RMW always accesses the parent as 4-byte `UDInt` — explicitly +flagged as an unverified assumption that will likely fail +`DeviceInvalidSize` on WORD/BYTE parents on real hardware +(`Driver.TwinCAT/AdsTwinCATClient.cs:96-107`). + +**STAB-12 — Medium — Blocking/uncancellable teardown paths.** FOCAS's sync +`Dispose()` performs a close-PDU exchange (`SendPdu` + `ReadPdu`) on a +`NetworkStream` with no ReadTimeout and no token +(`Driver.FOCAS/Wire/FocasWireClient.cs:164-175`) — a stalled CNC can wedge +`ShutdownAsync` indefinitely. TwinCAT's `ConnectAsync` calls the blocking +SDK `Connect` and ignores both its token and timeout parameter for the +connect itself (`AdsTwinCATClient.cs:73-80`). + +**STAB-13 — Positive.** Write-outcome surfacing — the seam the server's +value-revert self-correction depends on — is honoured by every protocol +driver: per-item `WriteResult` with typed exception→status mapping (Modbus +`ModbusDriver.cs:897-942`; S7 `S7Driver.cs:921-1006` including the +PUT/GET-denied → `Faulted` escalation; AbCip `AbCipDriver.cs:732-775`; +AbLegacy `AbLegacyDriver.cs:348-366`; TwinCAT `TwinCATDriver.cs:332-349` +with a numerically-verified ADS error table; OpcUaClient passes upstream +codes verbatim and distinguishes post-dispatch cancellation as +`BadTimeout` "outcome unknown" — `OpcUaClientDriver.cs:753-779`, the most +careful non-idempotency handling in the fleet). Nothing is +fire-and-forget except the AbCip alarm *acknowledge* path (UNDER-5). +OpcUaClient's reconnect state machine (double-arm guard, re-arm on +exhaustion, session swap under `_probeLock` — +`OpcUaClientDriver.cs:1908-2058`) is the strongest reconnect +implementation in the subsystem. + +### 2. Performance + +**PERF-1 — High — Five of seven drivers read one tag per wire round-trip.** +Only Modbus (block coalescing + auto-prohibition/bisection, +`ModbusDriver.cs:687-795`) and OpcUaClient (single +`ReadValueIdCollection` per batch, `OpcUaClientDriver.cs:640-666`) batch. +The others are O(N) round trips per poll tick on serialized links: +- S7: one PDU per tag under one `_gate`d connection + (`S7Driver.cs:443-484`); S7comm multi-var reads (S7.Net + `ReadMultipleVarsAsync`) unused. +- TwinCAT: one ADS call per symbol; ADS **sum-read** — Beckhoff's + documented batching mechanism — unused (`TwinCATDriver.cs:203-248`). +- FOCAS: one request/response per address on a strictly single-flight + socket (`SynchronizedFocasClient.cs:34`); `pmc_rdpmcrng` supports ranges + but each read fetches exactly one value's width + (`Wire/WireFocasClient.cs:296-334`). Throughput ceiling ≈ tags × RTT. +- AbLegacy: one PCCC transaction per scalar; no span coalescing of + adjacent file elements (N7:0..N7:2), the natural Modbus analog. +*Recommendation:* per-protocol batching is the single biggest scalability +lever; priority order = TwinCAT sum-read, S7 multi-var, FOCAS PMC range +reads, AbLegacy spans. + +**PERF-2 — High — AbCip's whole-UDT batched read exists in two halves that +were never joined.** The read planner +(`Driver.AbCip/AbCipUdtReadPlanner.cs:32-100`) can collapse N members into +one parent read, but only via declaration-order layout — off by default +because it produces “silently-plausible wrong numbers” when member order +diverges (`AbCipDriverOptions.cs:60-72`). Meanwhile the driver already +fetches and caches *true* member offsets via the CIP Template Object +(`AbCipDriver.cs:151-182`, `AbCipTemplateCache.cs`) — used only for +discovery, never consulted by the planner +(`AbCipUdtMemberLayout.cs:22-24` even points at the richer path). Net: in +every safe configuration, N UDT members = N CIP round trips per tick. +*Recommendation:* feed `AbCipUdtShape` offsets into the planner; retire +the unsafe declaration-order mode. + +**PERF-3 — High — S7's forked poll loop diffs arrays by reference.** +`PollOnceAsync` compares `Equals(lastSeen?.Value, current.Value)` +(`S7Driver.cs:1304`); array reads return a fresh CLR array every poll, so +every subscribed array tag fires `OnDataChange` **every tick**, flooding +the dependency mux/DPS downstream. `PollGroupEngine` fixed exactly this +with `StructuralComparisons` (`PollGroupEngine.cs:203-209`); the bespoke +S7 copy never picked it up. (Also a Conventions finding — see CONV-1.) + +**PERF-4 — Medium — OpcUaClient serializes all traffic on one global +`SemaphoreSlim(1,1)`.** Reads, writes, discovery, subscription creates, +acks and HistoryReads all queue on `_gate` +(`OpcUaClientDriver.cs:78, 626, 713, 836, 1186, 1377, 1451, 1653, 1825`). +OPC UA sessions multiplex service calls natively; a multi-second +HistoryRead or full re-discovery blocks every live read/write behind it. +The gate is only needed for session-swap consistency, which `_probeLock` + +re-read-inside-critical-section already provide. + +**PERF-5 — Medium — OpcUaClient discovery enrichment issues one giant +un-chunked Read and silently degrades on rejection.** +`EnrichAndRegisterVariablesAsync` sends `pending.Count * 4` ReadValueIds in +a single call (`OpcUaClientDriver.cs:1029-1046`) — 40k operations for a +10k-node server. A server enforcing `MaxNodesPerRead` returns +`BadTooManyOperations`; the blanket catch (`:1049-1057`) then registers +*every* variable as Int32/ViewOnly/non-historized with no log or health +signal. *Recommendation:* chunk client-side; log the fallback. + +**PERF-6 — Low — Assorted per-tick allocation churn.** TwinCAT re-parses +`TwinCATSymbolPath` on every read/write/subscribe call — lists + +StringBuilder per tag per tick (`TwinCATDriver.cs:220, 284, 468`); S7 +caches parses (`_parsedByName`) and is the model. Modbus publishes a fresh +`DriverHealth` record per successful tag read (`ModbusDriver.cs:285`). +AbCip's `BuildArray` boxes per element +(`LibplctagTagRuntime.cs:107-118`). None are hot enough to be urgent. + +**PERF-7 — Low — Deadband exists only in Modbus.** The publish-suppression +deadband (`ModbusDriver.cs:136-159`) and `WriteOnChangeOnly` never +generalized; noisy analog CNC/PLC signals on the other five poll-based +drivers publish every jitter. The mechanism is driver-agnostic and belongs +in or beside `PollGroupEngine`. + +### 3. Conventions + +**CONV-1 — High — S7 forked the poll loop instead of using +`PollGroupEngine`, and the fork has diverged in both directions.** +`PollGroupEngine`'s own doc claims it serves “Modbus, AB CIP, S7, FOCAS” +(`PollGroupEngine.cs:8-9`) but S7 re-ships the entire loop +(`S7Driver.cs:1165-1307`). The fork is *better* in two ways the engine +never absorbed (capped exponential failure backoff `:1284-1293`; poll +failures logged + degrade health `:1258-1274`) and *worse* in two ways the +engine already fixed (reference-equality array diff — PERF-3; CTS disposed +without draining the loop — STAB-6). This is the textbook cost of template +divergence: each side owns fixes the other needs. *Recommendation:* +extend `PollGroupEngine` with backoff + onError-driven health, then move +S7 onto it and delete the fork. + +**CONV-2 — High — Three different strictness tiers parse the same config, +and the most-used one is the most permissive.** For every driver the same +enum field is parsed three ways: +- **Factory**: string-typed DTO + fail-loud `ParseEnum` listing valid + values (`ModbusDriverFactoryExtensions.cs:191-201` and equivalents) — + deliberate, good. +- **Probe**: deserializes with `JsonStringEnumConverter`; Modbus's probe + binds the *runtime options type* (`ModbusDriverOptions`, TimeSpan + fields) rather than the factory DTO shape + (`ModbusDriverProbe.cs:19-24, 36`), so a DB config with `timeoutMs` + silently loses its timeout in the probe and tag-shape mismatches can + make the probe reject/misread a config the factory accepts. OpcUaClient + documents probe/factory parse-parity as the rule + (`OpcUaClientDriverProbe.cs:22`); Modbus violates it. +- **Equipment-tag parser**: silent fallback — an unknown or typo'd + `dataType` string quietly becomes the default type (`Int16`/`DInt`/ + `Int`/`Int32` per driver) instead of a rejection + (`Driver.Modbus.Contracts/ModbusEquipmentTagParser.cs:65-67`, + `S7EquipmentTagParser.cs:59-61`, `AbCipEquipmentTagParser.cs:86-88`, + `AbLegacyEquipmentTagParser.cs:60-62`, + `TwinCATEquipmentTagParser.cs:47-49`, + `FocasEquipmentTagParser.cs:43-45`). A mis-typed equipment tag reads and + writes the wrong width with Good status. The identical private + `ReadEnum` helper is copy-pasted six times instead of living beside + `EquipmentTagRefResolver` in Core.Abstractions. +*Recommendation:* hoist a shared strict `ReadEnum` (present-but-invalid ⇒ +parse failure ⇒ `BadNodeIdUnknown`); make probes parse the factory DTO. + +**CONV-3 — Medium — Factory registration asymmetry leaves four drivers +logging to `NullLogger` in production.** S7, TwinCAT, AbCip and FOCAS +`Register(registry)` take no `ILoggerFactory` +(`S7DriverFactoryExtensions.cs:19`, `TwinCATDriverFactoryExtensions.cs:18`, +`AbCipDriverFactoryExtensions.cs:21`, `FocasDriverFactoryExtensions.cs:36`) +even though all four driver classes accept a logger; the bootstrap +consequently can't pass one +(`Host/Drivers/DriverFactoryBootstrap.cs:100-107`). All the warning paths +those drivers carefully log (poll failures, prohibitions, live-risk +assumptions) are invisible in production. Modbus/AbLegacy/OpcUaClient/ +Galaxy have the logger-aware overload. *Recommendation:* one-line fix per +factory; align all eight `Register` signatures. + +**CONV-4 — Medium — `ResolveHost` mis-keys equipment-tag references in +every multi-device driver.** The implementations consult only +`_tagsByName` (authored names); an equipment-tag TagConfig-JSON reference +never matches and falls back to the driver-level/first-device host +(`AbCipDriver.cs:483-488`, `AbLegacyDriver.cs:496-501`, +`TwinCATDriver.cs:585-592`, `FocasDriver.cs:1082-1087`; Modbus +`ModbusDriver.cs:125-131` has the same shape but per-tag `UnitId` isn't in +the equipment parser anyway). Per-host Polly bulkhead/circuit-breaker keys +are therefore wrong for equipment tags on multi-device instances — a +broken device B can trip device A's breaker and vice versa. Since +equipment tags are the *primary* authoring model post-Galaxy-standardization, +the per-call host resolver is effectively inert for the flagship path. +*Recommendation:* route `ResolveHost` through +`EquipmentTagRefResolver` and derive the host from the parsed def +(`deviceHostAddress` / `unitId`). + +**CONV-5 — Medium — Health-field publication conventions drift.** The +template documents `Volatile.Read/Write` (`ModbusDriver.cs:217-228`); +FOCAS conforms (`FocasDriver.cs:43-46`), AbLegacy uses a `volatile` field +with rationale (`AbLegacyDriver.cs:29-34`), while S7 +(`S7Driver.cs:124, 1272`), AbCip (`AbCipDriver.cs:95`), TwinCAT and +OpcUaClient (`OpcUaClientDriver.cs:96`) use plain fields. Immutable record +swap means no torn reads — this is a visibility/staleness nit — but four +different idioms for the same one-line concern is exactly the divergence +this review is hunting. Related one-offs: TwinCAT declares `Healthy` at +init with zero wire contact (`TwinCATDriver.cs:115`) where every other +driver proves the connection first; TwinCAT reads host state without its +probe lock (`TwinCATDriver.cs:524-525`); AbLegacy never refreshes health +on successful writes (`AbLegacyDriver.cs:344-346` vs +`AbCipDriver.cs:725`); AbLegacy silently clobbers duplicate tag names +(`AbLegacyDriver.cs:91`) where AbCip fails fast +(`AbCipDriver.cs:249-256`). + +**CONV-6 — Medium — Contracts layering is uniform except OpcUaClient, +which drags the whole OPC UA SDK into its Contracts.** Six Contracts +projects are POCO options + equipment-tag parser (FOCAS's csproj even +enforces “NO PackageReference. NO ProjectReference.”), but +`Driver.OpcUaClient.Contracts` carries a full OPC UA stack reference +because `NamespaceMap` lives there +(`OpcUaClient.Contracts.csproj:9`) — every consumer wanting just the +options DTO (probe hosts, AdminUI editors, Browser) transitively loads the +SDK. This is the previously-deferred OpcUaClient.Contracts-002 finding; +it remains the right call to move `NamespaceMap` out. Minor: all Contracts +projects use the parent namespace without a `.Contracts` suffix — +consistent, just surprising. + +**CONV-7 — Low/Medium — CLI harness divergences.** The six CLIs share +`DriverCommandBase` + `SnapshotFormatter` and a uniform +probe/read/write/subscribe verb set, but: `ParseValue`/`ParseBool` is +copy-pasted six times (`*/Commands/WriteCommand.cs`), option validation is +a different shape in every base and **absent entirely in AbLegacy** +(`AbLegacyCommandBase.cs` — `--timeout-ms 0` reaches the driver), the +abstract `Timeout` init-setter is a silent no-op in five bases but throws +`NotSupportedException` in AbCip (`AbCipCommandBase.cs:43`), and only +TwinCAT exposes a `browse` command despite AbCip also supporting +controller browse (hardcoded `EnableControllerBrowse=false`, +`AbCipCommandBase.cs:63`). OpcUaClient has no CLI at all (the server-side +Client.CLI tests the *server*, not this driver against a third-party +endpoint). *Recommendation:* hoist value parsing + a standard `Validate()` +into Cli.Common; add an OpcUaClient CLI or document the gap. + +**CONV-8 — Low — Dead/no-op config knobs and stale scaffold docs.** +`DisableFC23` is a documented no-op (`ModbusDriverOptions.cs:83-89`); +AbCip `ConnectionSize` is plumbed but never applied +(`AbCipDriverOptions.cs:97-103`); AbLegacy's class doc still says +read/write “ship in PRs 2 and 3” (`AbLegacyDriver.cs:9-11`); OpcUaClient's +header still says “PR 66 ships the scaffold: IDriver only” +(`OpcUaClientDriver.cs:12-14`). No options class in the fleet has a +`Validate()` method — validation is scattered across factory throws and +init guards (S7's init guards are the best of breed, +`S7Driver.cs:314-385`). + +### 4. Underdeveloped areas + +**UNDER-1 — High — FOCAS advertises writes it cannot perform.** The only +real backend returns `BadNotWritable` for every address +(`Wire/WireFocasClient.cs:71-73`), yet `FocasTagDefinition.Writable` +defaults `true` with a doc-comment citing write tests +(`FOCAS.Contracts/FocasDriverOptions.cs:152-158`) and +`FocasEquipmentTagParser` hard-codes `Writable: true` +(`FocasEquipmentTagParser.cs:35`) — so equipment tags can surface as +Operate-writable OPC UA nodes whose writes always fail. The driver-side +write status mapping (`FocasDriver.cs:353-381`) is currently dead code. +*Recommendation:* either implement PMC writes in the wire client or force +`Writable:false` at the parser/discovery seams until it exists. + +**UNDER-2 — High — OpcUaClient's `UnsMappingTable` is validated as +mandatory but consumed nowhere.** `ValidateNamespaceKind` hard-fails +Equipment-kind configs lacking a mapping table +(`OpcUaClientDriver.cs:330-355`), yet no code reads the table — +`DiscoverAsync` builds the local tree purely from remote browse structure +(`:824-1105`). Operators are forced to author config with zero runtime +effect; the promised remote→UNS remapping feature does not exist. +*Recommendation:* delete the gate or build the feature. + +**UNDER-3 — High — OpcUaClient alarm source-node filtering compares +mismatched NodeId encodings.** `OnEventNotification` filters +session-relative `ns=N;…` renderings against caller-supplied refs by +ordinal string compare (`OpcUaClientDriver.cs:1337, 1538-1539`), while the +driver's own persisted references are stable `nsu=…` strings +(`:808-809`). Any non-empty filter built from stored refs silently drops +every event; only empty-filter subscriptions work today. + +**UNDER-4 — Medium — S7 feature gaps make authored array tags +unreachable.** `S7TagDto` has no `ArrayCount` field and `BuildTag` never +sets it (`S7DriverFactoryExtensions.cs:80-90, 137-151`) — driver-config +array tags are silently scalar; arrays only work as equipment tags +(`S7EquipmentTagParser.cs:73-87`). Array *writes* are unsupported +(`S7Driver.cs:1014-1017`), wide-type arrays are a stated follow-up +(`:861-862`), and `UInt32` surfaces lossily as `Int32` +(`S7Driver.cs:1149-1152`). + +**UNDER-5 — Medium — AbCip alarm acknowledge outcomes are discarded.** +`AcknowledgeAsync` fire-and-forgets the write results +(`AbCipAlarmProjection.cs:143-149`); a failed ack (undeclared `.Acked` +member, non-writable, comms error) is invisible to the operator, health, +and the Part 9 caller — conditions stick un-acked with no visible cause. +Related discovery weak spots: with `EnableControllerBrowse` on, one +unreachable PLC faults the entire multi-device `DiscoverAsync` +(uncaught `await foreach`, `AbCipDriver.cs:970-1037`), and browsed tags +default to writable/Operate (`CipSymbolObjectDecoder.cs:91`). + +**UNDER-6 — Medium — Equipment-tag parsers lag the authored-tag feature +set.** Modbus's parser ignores `unitId`, `deadband`, `stringByteOrder`, +`writable`, `coalesceProhibited` and the address-grammar string, and +forces `Writable: true` (`ModbusEquipmentTagParser.cs:54-57`); AbLegacy +and FOCAS also hard-code writable-true +(`AbLegacyEquipmentTagParser.cs:52`, `FocasEquipmentTagParser.cs:35`) — +read-only equipment tags are unauthorable on three drivers (AbCip honours +a `writable` key, `AbCipEquipmentTagParser.cs:53-54`). FOCAS equipment +tags additionally bypass the `FocasCapabilityMatrix` pre-flight gate that +authored tags get (`FocasDriver.cs:243-247` vs `:115-119`). As equipment +tags are the primary authoring model, the parsers deserve parity plus a +shared conformance test. + +**UNDER-7 — Medium — Live-risk assumptions documented but ungated.** +TwinCAT carries two explicit "unverified against real hardware" blocks on +core paths — array read shape (`AdsTwinCATClient.cs:109-116`) and +Flat-mode `SubSymbols` population (`:263-277`, struct members could +silently vanish from discovery) — plus the bit-RMW width assumption +(STAB-11). AbLegacy's array decode rests on five explicitly unproven +layout assumptions with no PCCC fixture +(`LibplctagLegacyTagRuntime.cs:64-88`). FOCAS's `cnc_getfigure` command id +and servo-load scaling are sim-validated only +(`FocasWireClient.cs:391-395, 943-947`). These are honest, well-documented +debts — but nothing aggregates them into a "needs bench time" checklist. + +**UNDER-8 — Medium — Test coverage is codec-heavy, failure-path-light, and +uneven.** Approximate unit-test counts: Modbus 191 (+71 addressing), +AbCip 234, FOCAS 143, S7 130, TwinCAT 119, AbLegacy 114, OpcUaClient 93 +(+10 browser). Integration suites are all skip-gated on fixture +reachability; Modbus's is the richest (30 tests incl. exception injection +and device-quirk profiles), AbLegacy's the thinnest (2), and TwinCAT's 13 +require a real TC3 XAR runtime — no docker fixture exists, so it never +runs automated. Systemic gaps: **zero reconnect tests for S7 and +AbLegacy** (the STAB-1/STAB-10 findings live exactly there), zero +concurrency tests for S7 and AbCip (STAB-4 lives there; only AbLegacy has +a dedicated runtime-concurrency suite), and OpcUaClient has the widest +feature surface on the lowest test density (though its failure-path +*spread* — reconnect, failover, stale-session race, continuation points — +is the best). `Cli.Common`'s `DriverCommandBase` has no direct tests. +There is **zero TODO/HACK/FIXME/NotImplementedException debt** in the +subsystem — all `NotSupportedException` sites are deliberate fail-fast +gates with handlers. + +**UNDER-9 — Low — OpcUaClient is a 2,154-line monolith.** Six separable +responsibilities share one file (config/PKI/identity, failover sweep, +recursive discovery, live subscriptions, alarms/acks, the full +IHistoryProvider, plus the reconnect state machine). The reconnect state +machine is the trickiest concurrency in the subsystem and deserves +extraction for isolated testability. Also: subscription transfer rests on +the *unset* SDK default `TransferSubscriptionsOnReconnect` +(`OpcUaClientDriver.cs:1946-1949`), and `BuildCertificateIdentity` only +loads PKCS#12 while the options doc promises PEM support +(`:479-484` vs `OpcUaClientDriverOptions.cs:68-75`). + +--- + +## Maturity ratings + +| Dimension | Rating (1-5) | Justification | +|---|---|---| +| Stability | **3** | The template (Modbus transport + OpcUaClient reconnect machine) is genuinely robust and write-outcome surfacing is universal, but two Critical connection-loss failures (S7 no-reconnect, TwinCAT orphaned native subs), an AbCip handle-concurrency hazard, and fleet-wide missing backoff mean a routine PLC power-cycle defeats half the fleet. | +| Performance | **2** | Only 2 of 7 drivers batch reads; the rest are O(N) serialized round trips per poll tick, AbCip's flagship UDT batching is half-built, and deadband exists only in Modbus — the subsystem scales by tag count, not by request count. | +| Conventions | **3** | The shared seams (PollGroupEngine, EquipmentTagRefResolver, factory/probe pattern, StatusMapper) are real and mostly followed, but the S7 poll-loop fork, three-tier config-parse strictness, six-way `ReadEnum` copy-paste, logger-registration asymmetry, and drifting health idioms show fixes not flowing back to the template. | +| Underdeveloped areas | **3** | Zero stub/TODO debt and honest live-risk documentation, offset by advertised-but-dead features (FOCAS writes, `UnsMappingTable`, `DisableFC23`, `ConnectionSize`), equipment-tag parsers lagging the primary authoring model, and reconnect/concurrency test gaps precisely where the Critical bugs live. | + +--- + +## Recommended remediation order + +1. **S7 reconnect** (STAB-1) + a reconnect test template applied to S7 and + AbLegacy (UNDER-8) — one field scenario, two drivers, currently fatal. +2. **TwinCAT native-subscription replay on client recycle** (STAB-2). +3. **AbCip per-runtime operation lock** (STAB-4, port from AbLegacy) and + **FOCAS ConcurrentDictionary caches** (STAB-5) — silent-corruption + class. +4. **Modbus transport: timeout/framing-violation ⇒ connection teardown** + (STAB-3). +5. **PollGroupEngine v2**: absorb S7's backoff + failure-health, wire + `onError` in all five consumers, migrate S7 off its fork + (CONV-1, STAB-9, PERF-3, STAB-8). +6. **Shared strict equipment-tag enum parsing + writable/capability + parity** (CONV-2, UNDER-6) — one Core.Abstractions helper, six deletions. +7. **Per-protocol read batching** (PERF-1/PERF-2), starting with the AbCip + planner⇄template-cache join (design already exists in-tree). +8. Sweep the one-liners: factory `ILoggerFactory` (CONV-3), `ResolveHost` + via resolver (CONV-4), FOCAS writable-false (UNDER-1), delete + `UnsMappingTable` gate (UNDER-2), OpcUaClient alarm-filter encoding + (UNDER-3). diff --git a/archreview/06-gateway-integrations.md b/archreview/06-gateway-integrations.md new file mode 100644 index 00000000..5898bb1e --- /dev/null +++ b/archreview/06-gateway-integrations.md @@ -0,0 +1,196 @@ +# Architecture Review 06 — Gateway Integrations (Galaxy Driver + Historian Gateway Driver) + +- **Date:** 2026-07-08 +- **Commit:** `9cad9ed0` (master) +- **Scope:** + - `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Galaxy` (+ `.Contracts`, `.Browser`) + - `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway` + - Test coverage: `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Tests`, `.Galaxy.Browser.Tests`, `.Historian.Gateway.Tests` + - Peripheral (read for context, not reviewed in depth): `Runtime/Historian/*` (recorder, options), `AddressSpaceApplier` provisioning/subscription hooks, `OtOpcUaNodeManager` tie-cluster paging. +- **Dimensions:** Stability, Performance, Conventions, Underdeveloped Areas. + +--- + +## Architecture Overview + +### Galaxy driver data flow (mxaccessgw) + +`GalaxyDriver` (`GalaxyDriver.cs`) is a standard Tier-A in-process Equipment-kind driver registered under type name `GalaxyMxGateway`. All Galaxy access flows over gRPC to the external **mxaccessgw** gateway (sibling repo), consumed via the Gitea-feed packages `ZB.MOM.WW.MxGateway.Client` / `.Contracts` (`MxCommand` / `MxEvent` protos). The driver holds **three logical gRPC clients**: + +1. **Worker session** — `GalaxyMxSession` wraps `MxGatewayClient.OpenSessionAsync` + MXAccess `Register`. All data-plane traffic rides it: + - **Subscribe** — `GatewayGalaxySubscriber.SubscribeBulkAsync` (session-level `SetBufferedUpdateInterval` applied first, cached last-applied value), events consumed by the shared `EventPump` off the bidirectional `StreamEvents` RPC, fanned out through `SubscriptionRegistry`'s handle→subscription reverse map onto `OnDataChange`. + - **Read** — MxAccess has no one-shot read; `ReadViaSubscribeOnceAsync` synthesises Read as SubscribeBulk → first-event wait per handle → UnsubscribeBulk. + - **Write** — `GatewayGalaxyDataWriter`: lazy `AddItem` (or borrow a live handle from the subscription registry), `AdviseSupervisory` once per handle (required for commit under WriteUserId=0), then raw `Write` or `WriteSecured` routed on the discovery-captured per-tag `SecurityClassification`. +2. **Session-less client** (`_ownedMxClient`) — the always-on central alarm monitor: `GatewayGalaxyAlarmFeed` (StreamAlarms: active-alarm snapshot → `snapshot_complete` → live transitions, self-reconnecting) and `GatewayGalaxyAlarmAcknowledger` (unary `AcknowledgeAlarm`). +3. **Repository client** (`GalaxyRepositoryClient`) — hierarchy browse for `DiscoverAsync` (via `IGalaxyHierarchySource` → `GalaxyDiscoverer`) and the `DeployWatcher` deploy-event stream that raises `IRediscoverable.OnRediscoveryNeeded`. + +Recovery is owned by `ReconnectSupervisor` (Healthy → TransportLost → Reopening → Replaying → Healthy, capped exponential backoff, never gives up): the `EventPump` stream fault feeds `ReportTransportFailure`; reopen routes through `GalaxyMxSession.RecreateAsync` (dispose stale session + client, rebuild) and invalidates the writer's handle/advise caches; replay recreates the EventPump then re-issues SubscribeBulk per tracked subscription and `Rebind`s the registry with the fresh handles. Host connectivity is surfaced through `HostStatusAggregator` (transport state from the supervisor + per-platform `ScanState` probes via `PerPlatformProbeWatcher`). + +### Historian gateway data flow (HistorianGateway sidecar) + +`Driver.Historian.Gateway` is the **sole historian backend**, consuming the `ZB.MOM.WW.HistorianGateway.Client` package (`historian_gateway.v1`) behind the proto-typed `IHistorianGatewayClient` seam (`HistorianGatewayClientAdapter` is a pure pass-through; channels are lazy — no I/O at construction). Four independent paths, each with **its own gRPC channel** to the same sidecar (documented deliberate trade-off, `GatewayHistorianServiceCollectionExtensions.cs:57-64`): + +1. **Read** — `GatewayHistorianDataSource` (`IHistorianDataSource`): OPC UA HistoryRead Raw/Processed/AtTime/Events → `ReadRaw` / `ReadAggregate` / `ReadAtTime` / `ReadEvents`; at-time replies re-aligned one-snapshot-per-requested-timestamp; health counters under a single lock. Tie-cluster paging (`MaxTieClusterOverfetch`) lives server-side in `OtOpcUaNodeManager` (~line 2200), bound from `ServerHistorianOptions`. +2. **Alarm history write** — `GatewayAlarmHistorianWriter` (`SendEvent` per event) behind the durable `SqliteStoreAndForwardSink`; maps every outcome (ack, typed client exceptions, raw `RpcException`) to exactly one `Ack` / `RetryPlease` / `PermanentFail` per event and never throws. +3. **Continuous historization** — `ContinuousHistorizationRecorder` (Runtime actor) taps the dependency-mux value fan-out, appends to the crash-safe `FasterLogHistorizationOutbox` (PerEntry fsync or Periodic commit; bounded drop-oldest capacity; startup recovery scan), drains through `GatewayHistorianValueWriter` (`WriteLiveValues`, non-throwing bool). +4. **Tag provisioning** — `GatewayTagProvisioner` (`EnsureTags`), dispatched fire-and-forget from `AddressSpaceApplier.Apply` with a tally log; non-historizable data types skipped; can never block or fail a deploy. + +--- + +## Findings + +### 1. Stability + +#### S-1 (High) — Galaxy write success is optimistic; a committed-write failure can never surface + +`GatewayGalaxyDataWriter.TranslateReply` (`GatewayGalaxyDataWriter.cs:281-302`) honours the protocol status and the first MXAccess status row, **defaulting to `Good` when the statuses array is empty** — and the gateway's write execution is effectively fire-and-forget past dispatch (the reply reflects command acceptance, not the eventual COM-side commit). Two concrete consequences: + +- The supervisory-advise failure path (`GatewayGalaxyDataWriter.cs:234-243`) logs a warning, forgets the handle, and **lets the write proceed anyway** — but the file's own comment (`:163-165`) states a raw Write without supervisory advise "doesn't throw (reply looks OK) but the value never reaches the galaxy". That is a silent write loss returning `Good` to the OPC UA client. +- The server's write-outcome self-correction (#5, reverts the node on a failed device write) can structurally never trigger for Galaxy, so a lost write leaves a phantom-Good node value indefinitely. + +**Blast radius:** operator writes (WriteOperate-gated) to Galaxy attributes that silently don't commit, with no Bad status, no health degradation, no metric. Recommendation: (a) return `Uncertain` (not proceed-to-Good) when supervisory advise fails; (b) pursue a gateway-side `WriteComplete` correlation (the gw backlog's `OnWriteComplete` event family already exists in the proto — `EventPump.cs:200-206` filters it out) so the reply/statuses row carries the real commit outcome; (c) add a `galaxy.writes.unconfirmed` counter in the interim. + +#### S-2 (Medium) — EventPump saturation drops the *newest* events (inverted staleness bias) + +`EventPump.RunAsync` (`EventPump.cs:128-140`) uses `TryWrite` against a bounded channel: when the fan-out consumer stalls, the **just-arrived** event is dropped and 50 000 stale queued events are preserved. For last-value-wins OPC UA telemetry this is the wrong bias — after a stall clears, subscribers replay old values and the freshest one may be the dropped one. The drop is at least metered (`galaxy.events.dropped`). Recommendation: per-item-handle conflation (keep newest per handle) or drop-oldest ring semantics; either preserves the no-backpressure requirement while keeping recent data. + +#### S-3 (Medium) — `ReadViaSubscribeOnceAsync` can wait forever on a non-cancellable token + +The read synthesis (`GalaxyDriver.cs:738-755`) fills pending snapshots with `BadTimeout` **only via `cancellationToken.Register`**. If a caller passes `CancellationToken.None` (or a token that never fires) and a successfully subscribed tag never publishes an initial event (gateway hiccup between SubscribeBulk and first push), the awaits at `:753` never complete and the read hangs, holding the subscription open. Recommendation: race each pending TCS against an internal deadline derived from `DefaultCallTimeoutSeconds` / `PublishingIntervalMs` regardless of the caller's token. + +#### S-4 (Medium) — Transport-failure detection is EventPump-only; a subscription-less driver never degrades + +`ReconnectSupervisor.ReportTransportFailure` has exactly one production caller: the EventPump stream-fault callback (`GalaxyDriver.cs:929-949`). The pump only starts on the first subscribe/read. A driver doing only writes (or idle after discovery) whose gateway restarts will keep `GetHealth() == Healthy`; each write then fails per-request with `BadCommunicationError`, but no reopen/replay/`Degraded` transition ever runs, and the stale session persists until something subscribes. Failed unary RPCs (SubscribeBulk, Write, AcknowledgeAlarm) do not feed the supervisor either. Recommendation: report classified transport exceptions from the write/subscribe paths into the supervisor (idempotent by design, so this is cheap). + +#### S-5 (Medium) — FasterLog outbox `RemoveAsync` truncates the FIFO prefix; out-of-order acks silently drop unacked entries + +`FasterLogHistorizationOutbox.RemoveAsync` (`FasterLogHistorizationOutbox.cs:158-182`) removes the target **plus every older live entry** and truncates the log to the target's successor. The contract "recorder acks in FIFO order" is enforced only by comment. If the drain ever acks a mid-batch id first (e.g. per-tag partial write success in a future recorder change), all older unacked values are durably discarded without incrementing `DroppedCount`. Recommendation: count and warn when the prefix removal drops non-target entries, or make the API batch-oriented (`RemoveThroughAsync`) so the semantics are explicit at the call site. + +#### S-6 (Medium) — Historian health snapshot's connection flags are dormant in production + +`GatewayHistorianDataSource.RefreshConnectionStateAsync` (`GatewayHistorianDataSource.cs:219-244`) is documented as "intended to be driven by a periodic health hosted-service", but **no production caller exists** — only `GatewayHealthSnapshotTests` invokes it. `ProcessConnectionOpen` / `EventConnectionOpen` in `GetHealthSnapshot` are therefore permanently `false` in a deployed host, which any dashboard consuming the snapshot will read as "historian down" (or, if ignored, the flags are dead weight). Same shape as the memory's "register-AND-pass-into-consumer" trap that bit `GatewayTagProvisioner` in PR #423. Recommendation: add the periodic refresh hosted-service (or fold a refresh into the existing health probe cadence), or remove the flags from the snapshot. + +#### S-7 (Low) — Alarm feed reconnect has no backoff + +`GatewayGalaxyAlarmFeed.RunAsync` (`GatewayGalaxyAlarmFeed.cs:102-148`) re-opens on a fixed 5 s delay forever. `DeployWatcher` (capped exponential + jitter, `DeployWatcher.cs:212-233`) and `ReconnectSupervisor` both do this properly. A dead gateway gets hammered every 5 s per driver instance. Cosmetic at this scale, but inconsistent; align on capped exponential. + +#### S-8 (Low) — `GalaxyMxSession` state is unsynchronized across reopen + +`_session` / `_connected` (`GalaxyMxSession.cs:28-32`) are plain fields; `RecreateAsync` tears down while concurrent writers/subscribers may hold or fetch `Session`. In practice the supervisor's single-flight recovery plus the per-call try/catch (mapped to `BadCommunicationError`) contain it, and the caches are invalidated post-reopen — but the safety is emergent, not designed. Acceptable; document the invariant on `Session`. + +#### S-9 (Positive) — Store-and-forward + outbox crash-safety discipline is strong + +- `GatewayAlarmHistorianWriter` (`GatewayAlarmHistorianWriter.cs:66-118`) short-circuits remaining batch entries to `RetryPlease` on shutdown, classifies auth failures as retryable (an auth blip never dead-letters), and defaults unknowns to `PermanentFail` so poison events cannot loop. +- `FasterLogHistorizationOutbox` PerEntry mode fsyncs before append returns; recovery rebuilds the index from `BeginAddress`; the capacity-overflow-after-crash convergence case is analysed in-source (`FasterLogHistorizationOutbox.cs:200-204`). +- Partial-open recovery in `GalaxyMxSession.ConnectAsync` (`GalaxyMxSession.cs:95-101`) tears down half-open state so retry rebuilds cleanly. + +#### S-10 (Positive) — TLS / API-key posture + +Both integrations support TLS with CA pinning (`CaCertificatePath`), the historian's `AllowUntrustedServerCertificate` inversion is explicitly documented at the mapping site (`HistorianGatewayClientAdapter.cs:49-52`), and keys are never logged. See C-1 for the resolver. + +### 2. Performance + +#### P-1 (Medium) — Galaxy Read costs three gateway round-trips plus a publish wait per OPC UA Read + +`ReadViaSubscribeOnceAsync` = SubscribeBulk + first-event wait (up to `PublishingIntervalMs`) + UnsubscribeBulk, with server-side AddItem/RemoveItem churn per read (`GalaxyDriver.cs:644-780`). This is forced by MxAccess (no one-shot read RPC) and is correctly batched per request, but a client polling via Read instead of subscribing multiplies gateway/COM load ~3×. Recommendation: document "subscribe, don't poll" as the supported pattern; consider a short-lived read-through cache keyed on full reference if polling clients appear. + +#### P-2 (Medium) — Alarm-history drain is one unary `SendEvent` per event + +`GatewayAlarmHistorianWriter.WriteBatchAsync` (`GatewayAlarmHistorianWriter.cs:66-80`) serializes the batch — deliberate poison-event isolation, but an alarm storm of N events costs N sequential RPC round-trips out of the SQLite drain worker. If the gateway ever grows a batched `SendEvents` with per-event status rows, adopt it; until then this is an accepted, well-documented ceiling. + +#### P-3 (Low) — Four channels to the historian sidecar; two-plus to mxaccessgw + +Each historian path owns its channel (read / alarm-write / provisioner / value-writer) and the Galaxy driver holds a session client, a session-less client, and a repository client. All are lazy, HTTP/2-multiplexed, and the trade-off (clean independent dispose ownership over channel sharing) is argued in-source (`GatewayHistorianServiceCollectionExtensions.cs:57-64`). No action; noted so nobody "optimizes" it into a shared-singleton dispose bug. + +#### P-4 (Positive) — Fan-in/fan-out paths are indexed, bounded, and metered + +- `SubscriptionRegistry` maintains a handle→subscriptions reverse map with a per-entry handle→fullRef index, so `ResolveSubscribers` is O(subscribers), not O(bindings) (`SubscriptionRegistry.cs:97-111`), and `TryResolveItemHandle` lets the writer skip AddItem for subscribed tags with a stale-entry cross-check (`:128-142`). +- Subscribe/replay correlation uses a one-pass result index (`GalaxyDriver.BuildResultIndex`, fixing a former O(n²) on the 50k-tag path). +- EventPump decouples network read from dispatch via a bounded channel (default 50 000, configurable) with received/dispatched/dropped counters. + +#### P-5 (Low) — HistoryRead buffers whole replies; `maxEvents <= 0` is unbounded + +`ReadRawAsync` / `ReadProcessedAsync` / `ReadEventsAsync` (`GatewayHistorianDataSource.cs:59-182`) drain the stream into a list before mapping. Raw reads are capped by `maxValuesPerNode`, but an events read with `maxEvents <= 0` collects without limit (the gateway's `RuntimeDb:EventReadMaxRows` is the only cap, and it's a remote deployment knob). The tie-cluster paging bound (`MaxTieClusterOverfetch`, validated > 0 in `ServerHistorianOptions.Validate`, enforced at `OtOpcUaNodeManager.cs:2200-2214`) is well-designed. Recommendation: clamp events reads to a server-side default cap when the caller passes 0. + +#### P-6 (Low) — Outbox peek holds the state lock across disk I/O + +`PeekBatchAsync` scans FasterLog from the logical head under `_state` (`FasterLogHistorizationOutbox.cs:140-155`), blocking a concurrent `AppendAsync`'s index update for the duration of a (mostly page-cached) disk scan. At the recorder's 64-entry default batch this is negligible; revisit only if drain batches grow. + +### 3. Conventions + +#### C-1 (Positive) — Secret handling is a model for the other drivers + +`GalaxySecretRef` (`Galaxy.Contracts/GalaxySecretRef.cs`) resolves `env:` / `file:` / `dev:` / literal with fail-fast on unset env vars and a startup warning on unprefixed cleartext; it lives in Contracts so the runtime driver and the AdminUI browser share one implementation (`GalaxyDriverBrowser.cs:125`). The historian side takes `ServerHistorian__ApiKey` via env with a "never commit" doc contract and an empty-key `Validate()` warning. No key value is ever logged on either path. + +#### C-2 (Positive) — Seam quality and driver-family consistency + +- `IHistorianGatewayClient` is a clean single seam: every consumer (`GatewayHistorianDataSource`, `GatewayAlarmHistorianWriter`, `GatewayTagProvisioner`, `GatewayHistorianValueWriter`) depends only on it; `FakeHistorianGatewayClient` backs the offline suite; the adapter is a zero-translation pass-through. +- The Galaxy capability seams (`IGalaxyHierarchySource`, `IGalaxyDataReader/Writer`, `IGalaxySubscriber`, `IGalaxyAlarmFeed/Acknowledger`) plus the internal test ctor mirror the protocol-driver pattern; tracing decorators (`Traced*`) wrap the production seams without an OpenTelemetry dependency. +- `GalaxyDriverFactoryExtensions.CreateInstance` throws precise, instance-named errors on missing required fields (`GalaxyDriverFactoryExtensions.cs:56-75`), and `GalaxyDriverProbe` uses `JsonStringEnumConverter` (the FB-9/FB-10 enum-serialization lesson applied). + +#### C-3 (Medium) — The historian seam leaks wire types by design; keep it contained + +`IHistorianGatewayClient` signatures use `HistorianGateway.Contracts.Grpc` types (`HistorianSample`, `RetrievalMode`, `WriteAck`, ...) — documented as deliberate (`IHistorianGatewayClient.cs:5-11`). This is fine while the driver is the only consumer, but the pure `Mapping/` layer is the real anti-corruption boundary: any future consumer must sit **above** `GatewayHistorianDataSource`, never on the seam. Worth a one-line guard note in the interface doc. + +#### C-4 (Low) — Options-validation styles diverge + +Galaxy: throw-on-construct for required fields; DataAnnotations attributes on the records are decorative (nothing runs `Validator`). Historian: `Validate()` returns operator warnings and never throws (registration logs them). Both are defensible, but the repo now has two idioms for "driver-adjacent options validation". Pick one for new sections (the warnings-list style is the friendlier operational contract). + +#### C-5 (Low) — Retired Wonderware project directories still on disk + +`src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Wonderware{,.Client,.Client.Contracts}` (and their test dirs) remain in the tree but are absent from `ZB.MOM.WW.OtOpcUa.slnx` (0 references). Dead directories invite grep noise and accidental resurrection; delete them (git history preserves the code). + +#### C-6 (Low) — EventPump silently drops unknown event families with no metric + +`Dispatch` (`EventPump.cs:192-207`) returns without counting on any non-`OnDataChange` family. The rationale comment is good, but a gateway version emitting `OnBufferedDataChange` (or a new family) would silently discard data with zero observability — the alarm feed counts its decode drops (`AlarmTransitionsDecodingFailures`); the pump should count filtered/unknown families the same way. + +### 4. Underdeveloped Areas + +#### U-1 (High, documentation-drift) — CLAUDE.md KNOWN LIMITATION 2 is stale: the historized-ref feed is now wired + +CLAUDE.md states the recorder is "spawned with an EMPTY historized-ref set … registers interest in nothing and historizes nothing" pending a `SetHistorizedRefs`-style feed. The code has closed that gap: `WithOtOpcUaRuntimeActors` still spawns with `historizedRefs: Array.Empty()` (`Runtime/ServiceCollectionExtensions.cs:272`) **but** wraps the recorder in `ActorHistorizedTagSubscriptionSink` and hands it to the applier (`:287-307`), `AddressSpaceApplier` pushes the per-deploy add/remove delta (`AddressSpaceApplier.cs:343`), and the recorder handles `UpdateHistorizedRefs` (`ContinuousHistorizationRecorder.cs:186`) — the in-source comment explicitly says this "clos[es] the T18 ref-feed gap". Actions: (1) update CLAUDE.md; (2) the end-to-end value-capture path (deploy → delta → mux tap → outbox → `WriteLiveValues`) has, as far as the tree shows, **never been live-verified** — given this repo's repeated "wired-but-inert in prod" history (F10b `DeferredAddressSpaceSink`, PR #423 provisioner), a live `/run` against a real gateway is the required close-out. + +#### U-2 (Medium) — KNOWN LIMITATION 1: the live-validation gate is built but only partially run + +The `Category=LiveIntegration` suite is complete and skip-clean: `GatewayLiveFixture` env-gates on `HISTGW_GATEWAY_ENDPOINT`/`APIKEY` (+ per-test `HISTGW_TEST_TAG`, `HISTGW_WRITE_SANDBOX_TAG`, `HISTGW_ALARM_SOURCE`) with a bounded 3 s TCP probe so a down VPN skips instead of hanging (`Live/GatewayLiveFixture.cs`). Four live tests cover read, EnsureTags+write, alarm SendEvent→ReadEvents round-trip, and a SendEvent contract check. Recent history (merge `245316d8`, "assert FU-1 alarm SendEvent→ReadEvents round-trip (gateway C4 fixed)") shows the alarm leg has been run live at least once, but CLAUDE.md still flags the whole cutover as unvalidated. A full documented run of the suite (and a CLAUDE.md status update) is the remaining work — the infrastructure is not the gap. + +#### U-3 (Medium) — WriteSecured / VerifiedWrite user identity is stubbed at zero + +`InvokeWriteSecuredAsync` hardcodes `CurrentUserId = 0, VerifierUserId = 0` (`GatewayGalaxyDataWriter.cs:263-264`). ArchestrA secured/verified writes exist precisely to attribute and dual-authorize the operation; user 0 will be rejected or unattributed by any galaxy that actually classifies tags SecuredWrite/VerifiedWrite. There is no mapping from the OPC UA session identity (the LDAP-authenticated principal is available server-side) to an ArchestrA user id. Until that lands, writes to secured-classification tags are effectively unsupported — document it, or fail fast with a clear status instead of sending user 0. + +#### U-4 (Medium) — Dormant paths inventory + +- `GatewayHistorianDataSource.RefreshConnectionStateAsync` — no production caller (see S-6). +- Replay still fans out per-subscription `SubscribeBulk`; the gateway's batched `ReplaySubscriptionsCommand` remains a "PR 6.x can swap this" note (`GalaxyDriver.cs:314-316`). +- `GalaxyDriver.FlushOptionalCachesAsync` is a no-op and `GetMemoryFootprint` is a constants-based estimate (`GalaxyDriver.cs:561-575`) — fine, but the server's cache-flush heuristic gets synthetic data from this driver. +- `HistorianGatewayClientAdapter.ReadEventsAsync` never forwards `maxEvents` on the wire (client-side cap only, documented at `IHistorianGatewayClient.cs:58-63`) and the source filter is re-applied client-side defensively (`GatewayHistorianDataSource.cs:152-157`) because the gateway-side filter "may not be present" — both are polite workarounds for gateway gaps that should be tracked against the sidecar repo. + +#### U-5 (Low) — Outbox serializer has no version/format byte + +`HistorizationOutboxEntrySerializer` writes a fixed positional layout with no version prefix (`HistorizationOutboxEntrySerializer.cs:14-16`). Any future field addition breaks recovery of pre-existing on-disk entries with an undiagnosable deserialization failure mid-`RecoverState`. One reserved byte now is free; a migration later is not. + +#### U-6 — Test coverage assessment + +- **Galaxy** (~250 test methods across 34 files): strong unit coverage of the hard parts — reconnect orchestration (`GalaxyMxSessionReconnectTests`, `ReconnectSupervisorTests`), pump fault/bounded-channel behaviour, registry rebind/handle-resolve, writer caches, alarm feed decode, value encode/decode, status mapping, probe, factory. Three skip-gated live smokes (`GatewayGalaxyLiveReopenAndWriteTests` — proven 2/2 against `10.100.0.48:5120` per project memory — and `GatewayGalaxyAlarmFeedLiveTests`) gated on `MXGW_ENDPOINT` + `GALAXY_MXGW_API_KEY`. +- **Historian.Gateway** (~15 files): mappers fully covered, writer/provisioner/data-source outcome classification covered via `FakeHistorianGatewayClient`, outbox recovery/capacity covered, plus the 4-test live suite. +- **Thin spots:** no test drives `GalaxyDriver` write-through the supervisory-advise *failure* branch asserting the returned status (S-1's blast radius is untested); no test exercises out-of-order `RemoveAsync` on the outbox (S-5); nothing covers the read-hang case in S-3; `EventPump` unknown-family filtering has no assertion (C-6). + +--- + +## Maturity Ratings + +| Dimension | Rating (1-5) | Justification | +|---|---|---| +| Stability | **4** | Layered recovery (supervisor, self-reconnecting feeds, backoff, partial-open teardown) and disciplined never-throw write sinks; docked for the Galaxy silent-write-loss blast radius (S-1), EventPump-only fault detection (S-4), and the dormant health refresh (S-6). | +| Performance | **4** | Bounded channels with drop metering, O(1) fan-out indexes, handle borrowing, and a bounded tie-cluster over-fetch; docked for the 3-round-trip read synthesis (P-1) and serial alarm sends (P-2), both documented and gateway-constrained. | +| Conventions | **4** | Exemplary seams, shared secret resolver, consistent driver-family shape and tracing decorators; docked for the dual options-validation idiom, on-disk retired Wonderware dirs, and the unmetered pump filter. | +| Underdeveloped areas | **3** | Live-validation infrastructure is complete but the gate is only partially run and CLAUDE.md's limitation 2 is stale against code that has closed the gap; WriteSecured identity is stubbed; several polite client-side workarounds paper over gateway gaps. | + +--- + +## Cross-Cutting Themes + +1. **"Wired but never invoked" is this codebase's recurring failure mode** — the provisioner (PR #423), the F10b sink forwarding, and now `RefreshConnectionStateAsync` (S-6). Any new capability interface or hook needs an explicit production-caller check plus a live `/run`, not just unit tests. +2. **Optimistic-Good on fire-and-forget writes** — Galaxy structurally cannot report a failed commit (S-1); reviewers of the node-write router / write-outcome self-correction should not assume driver parity here. +3. **CLAUDE.md drift** — KNOWN LIMITATION 2 is closed in code (U-1) and LIMITATION 1 is partially exercised (U-2); the doc should be reconciled before it misleads the next session. +4. **Gateway-gap workarounds accrue client-side** — defensive source filters, client-side event caps, missing batched replay/SendEvents; these belong on the sister repos' backlogs with links from the in-source comments. diff --git a/archreview/07-client-tooling-engineering.md b/archreview/07-client-tooling-engineering.md new file mode 100644 index 00000000..c550ad54 --- /dev/null +++ b/archreview/07-client-tooling-engineering.md @@ -0,0 +1,411 @@ +# Architecture Review 07 — Client Tooling, Analyzers, and the Cross-Cutting Engineering System + +| | | +|---|---| +| **Date** | 2026-07-08 | +| **Commit** | `9cad9ed0` (master) | +| **Reviewer** | Architecture review agent (deep-review sweep, slice 07) | +| **Scope** | `src/Client/*` (Client.CLI, Client.Shared, Client.UI) + `tests/Client/*`; `src/Tooling/ZB.MOM.WW.OtOpcUa.Analyzers` + `tests/Tooling/*`; cross-cutting: `Directory.Build.props`, `Directory.Packages.props`, `NuGet.config`, `ZB.MOM.WW.OtOpcUa.slnx`, `.github/workflows/`, `ci/`, `scripts/`, `docker-dev/`, overall test architecture, docs freshness, repo-root hygiene | + +--- + +## Architecture Overview + +### Client stack + +Three projects form a clean layered client stack, all .NET 10: + +- **`Client.Shared`** — the OPC UA client library. `OpcUaClientService` + (`src/Client/ZB.MOM.WW.OtOpcUa.Client.Shared/OpcUaClientService.cs`, 975 lines) implements + `IOpcUaClientService` behind five adapter seams (`IApplicationConfigurationFactory`, + `IEndpointDiscovery`, `ISessionFactory`, `ISessionAdapter`, `ISubscriptionAdapter` under + `Adapters/`) so the OPC Foundation SDK is fully fakeable — the test project supplies + `Fakes/Fake*` for every seam. The service owns connect/disconnect, read/write with + string→typed value coercion, browse with continuation points, data + alarm subscriptions + with **failover replay** (multi-endpoint round-robin on keep-alive failure), Part 9 alarm + method calls (Acknowledge/Confirm/Shelve/Enable/Disable), HistoryRead (raw + aggregate), + and redundancy-info reads. +- **`Client.CLI`** — CliFx-based terminal client (`otopcua-cli`). 14 commands + (`Commands/`: connect, read, write, browse, subscribe, historyread, alarms, redundancy, + acknowledge, confirm, shelve, enable, disable + base). `CommandBase` centralises + connection options; `Program.cs` is a 14-line CliFx bootstrap with a type-activator that + injects a shared `IOpcUaClientServiceFactory`. Commands are session-per-invocation + (create → connect → work → disconnect/dispose in `finally`). +- **`Client.UI`** — Avalonia 11.2 desktop app (CommunityToolkit.Mvvm). **Real, not + vestigial**: ~2,905 lines of C# + 551 lines of AXAML across 10 viewmodels, 9 views, a + custom `DateTimeRangePicker` control, JSON settings persistence, and a UI-dispatcher + seam (`IUiDispatcher` with a synchronous test double). Covers browse tree, read/write, + subscriptions, alarms (ack/confirm/shelve dialogs), and history. 126 headless + (Avalonia.Headless) tests. + +### Tooling + +`src/Tooling/ZB.MOM.WW.OtOpcUa.Analyzers` ships exactly one Roslyn analyzer: +**OTOPCUA0001 `UnwrappedCapabilityCallAnalyzer`** — flags async calls to the seven guarded +driver-capability interfaces (`IReadable`, `IWritable`, `ITagDiscovery`, `ISubscribable`, +`IHostConnectivityProbe`, `IAlarmSource`, `IHistoryProvider`) that are not wrapped in +`CapabilityInvoker.ExecuteAsync/ExecuteWriteAsync` (the Polly breaker/retry/bulkhead +pipeline). Semantically sophisticated (symbol-identity matching, DIM handling, +wrapper-lambda containment), netstandard2.0, `EnforceExtendedAnalyzerRules`, tracked +`AnalyzerReleases.*.md`, 31 tests. **But it is wired into zero consuming projects** — see +finding C-1. + +### Test architecture map + +47 test projects mirror `src/` under `tests//` with three tiers: + +1. **`*.Tests`** — pure unit suites (fakes/in-memory), 40 projects. xunit.v3 + Shouldly + everywhere except three legacy v2 holdouts still on xunit 2.9.2 (`AdminUI.Tests`, + `ControlPlane.Tests`, `Runtime.Tests`). +2. **`*.IntegrationTests`** — 10 projects needing a live fixture. Pattern: a collection + fixture does a **one-shot TCP reachability probe** against an env-var endpoint whose + default is the shared Docker host `10.100.0.35` (e.g. + `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Modbus.IntegrationTests/ModbusSimulatorFixture.cs:37` + — `MODBUS_SIM_ENDPOINT` default `10.100.0.35:5020`), and every test `Assert.Skip`s when + unreachable. `dotnet test` therefore passes cleanly offline — by *skipping*. +3. **`Category=LiveIntegration`** — env-gated live suites against real infrastructure + (`tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway.Tests/Live/GatewayLiveIntegrationTests.cs`, + gated on `HISTGW_GATEWAY_ENDPOINT` etc.). Skip-clean when env vars absent. +4. **`Category=E2E`** — reserved; **no project or test carries it yet** (the nightly E2E + workflow is a documented no-op). + +### Build / CI pipeline shape + +- `Directory.Build.props` — global net10.0 / nullable / implicit usings / latest lang, plus + the CVE-2025-6965 `NuGetAuditSuppress` carve-out. `TreatWarningsAsErrors` is deliberately + **opt-in per project** (legacy xUnit1051 debt), with a written promotion plan. +- `Directory.Packages.props` — central package management, all 100+ versions pinned, no + floating versions, three inline security-pin rationales (Roslyn 5.0.0 CS9057 pin, + OpenTelemetry 1.15.3, Tmds.DBus.Protocol 0.21.3). +- `NuGet.config` — three sources with **strict packageSourceMapping** (nuget.org wildcard, + repo-local `nuget-packages/` for MxGateway, Gitea feed for the ZB.MOM.WW.* shared libs). +- CI: two workflow files in `.github/workflows/` (remote is Gitea; header claims + Gitea-Actions compatibility). `v2-ci.yml`: build → 5-project unit matrix → 2-project + integration matrix. `v2-e2e.yml`: nightly docker-dev fleet + `Category=E2E` filter that + matches zero tests. +- `ci/` contains only `ab-server.lock.json` (pinned libplctag release for the AB fixture). +- `scripts/` — PowerShell operational tooling: `compliance/` (6 phase-gate scripts), `e2e/` + (per-driver E2E harnesses + sample config), `install/` (Windows service + Traefik), + `migration/`, `smoke/` (SQL seeds), `focas/` (protocol capture), and + `check-code-reviews-readme.ps1` (review-index consistency check — not run by CI). +- `docker-dev/` — the local 8-service dev fleet (SQL, migrator, cluster-seed, central-1/2, + site-a/b pairs, Traefik). + +--- + +## Findings + +Severity scale: **Critical** (broken now, corrupts trust) / **High** (material gap, fix soon) +/ **Medium** (real debt) / **Low** (polish). + +### 1. STABILITY + +#### S-1 (High) — CI gates ~15% of the test matrix; client, tooling, driver, and most core suites are never run by CI +`.github/workflows/v2-ci.yml:47-52` enumerates **5** unit-test projects (Cluster, +ControlPlane, Runtime, Security, OpcUaServer) out of **47** in the solution. Everything in +this review's scope — `Client.CLI.Tests` (104 tests), `Client.Shared.Tests` (158), +`Client.UI.Tests` (126), `Analyzers.Tests` (31) — plus all driver unit suites, driver CLI +suites, and most Core suites (Core, Core.Scripting, VirtualTags, ScriptedAlarms, +AlarmHistorian, Commons, Configuration, Abstractions) run only when a developer remembers +to run them. The matrix predates most of these projects and was never widened. +**Recommendation:** replace the hand-maintained matrix with a single +`dotnet test ZB.MOM.WW.OtOpcUa.slnx --filter "Category!=E2E&Category!=LiveIntegration"` +leg (the skip-gated integration fixtures already tolerate missing endpoints), or generate +the matrix from the slnx. Any new-project drift then becomes impossible. + +#### S-2 (High) — "green CI" for integration tests means "skipped", and nothing distinguishes skip from pass +The integration job (`v2-ci.yml:61-76`) runs `Host.IntegrationTests` and +`OpcUaServer.IntegrationTests` on `ubuntu-latest` with **no service containers**, while +the fixtures default to `10.100.0.35` (unreachable from any hosted runner — +`ModbusSimulatorFixture.cs:37`, `AbServerFixture.cs`, `Snap7ServerFixture.cs`, +`OpcPlcFixture.cs`, and `Host.IntegrationTests/DriverTestConnectE2eTests.cs` all +hard-default to it). Probe-fails → `Assert.Skip` → job green. The design is intentional +and well-documented for dev boxes, but in CI it silently converts the entire integration +tier into a no-op with a passing badge. **Recommendation:** in CI, either start the +fixtures as workflow `services:` (modbus/opc-plc images exist) and set the `*_ENDPOINT` +env vars to `localhost`, or add a post-test step that fails the job when skipped-count > +threshold (`--logger trx` + parse), so a silent fixture outage cannot masquerade as green. + +#### S-3 (Medium) — the nightly E2E workflow is a permanent green no-op +`v2-e2e.yml:6-10` says it plainly: the E2E test project "does not yet exist… this workflow +is a green no-op". No test in the tree carries `Category=E2E` (verified by grep). A +nightly job that always passes trains people to ignore it; it also boots the full +docker-dev fleet for nothing. **Recommendation:** either land the minimal E2E round-trip +project (the `scripts/e2e/test-*.ps1` harnesses show exactly what it should assert) or +disable the schedule until it exists. + +#### S-4 (Medium) — fixed-sleep timing tests are the dominant wait style +`grep Task.Delay|Thread.Sleep` across `tests/` (excluding obj) shows heavy fixed-delay +usage: Driver.Galaxy.Tests (30), Driver.Modbus.Tests (22), Driver.AbCip.Tests (20), +Host.IntegrationTests (14), and in this slice **Client.CLI.Tests** — +`SubscribeCommandTests.cs:30,55,78,103`, `AlarmsCommandTests.cs:26,51,76,104,127,149` +(`await Task.Delay(100)` to let a background command loop start before cancelling), +`EventHandlerLifecycleTests.cs:54` (150 ms). These pass locally and flake under CI load — +which is currently masked because CI never runs them (S-1). **Recommendation:** replace +start-up sleeps with a readiness signal from the fake service (e.g. a +`TaskCompletionSource` completed on first `SubscribeAsync`), which `FakeOpcUaClientService` +can expose cheaply. + +#### S-5 (Medium) — Client.Shared subscription bookkeeping has lock-discipline gaps +`OpcUaClientService.cs` documents that `_subscriptionLock` guards the subscription state +(lines 18-22), but: +- `SubscribeAsync` checks/creates the shared `_dataSubscription` **outside** the lock + (line 262) — two concurrent first-subscribers can both see null and create two SDK + subscriptions, leaking one. +- `RunFailoverAsync` nulls `_dataSubscription` / `_alarmSubscription` **without** the lock + (lines 695-696), while `DisconnectAsync`'s comment (lines 138-140) assumes the failover + path nulls them under the lock. +- `UnsubscribeAlarmsAsync` (line 335) does not take `_alarmSubscribeSemaphore`, so an + unsubscribe racing a `SubscribeAlarmsAsync` can delete the adapter the subscriber just + created and leave `_alarmSubscription` non-null-but-deleted. +- Keep-alive failover is fire-and-forget (`_ = HandleKeepAliveFailureAsync()`, lines + 115, 718) — correct re-entrancy guard via `Interlocked.CompareExchange` (line 659), but + an exception escaping `TransitionState`'s event invocation is unobserved. + +For the CLI's session-per-command usage these races are near-unhittable; for Client.UI +(long-lived service, UI-thread + keep-alive-thread concurrency) they are real. +**Recommendation:** move the `_dataSubscription` null-check/create inside a small async +gate (mirror `_alarmSubscribeSemaphore`), take the lock in `RunFailoverAsync`, and route +`UnsubscribeAlarmsAsync` through the semaphore. + +#### S-6 (Medium) — global `Log.Logger` swap per CLI command +`CommandBase.ConfigureLogging()` (`CommandBase.cs:120-134`) does `Log.CloseAndFlush()` then +replaces the static `Log.Logger` on every command execution. In-process this is a race for +any parallel test collections that execute commands concurrently (xunit.v3 parallelises +collections by default), and it makes the CLI hostile to embedding. The +`LoggerLifecycleTests` suite exists precisely to police this. **Recommendation:** give +each command an instance `ILogger` (Serilog `LoggerConfiguration.CreateLogger()` held per +execution) instead of mutating the global. + +#### S-7 (Low) — no `global.json`, but the build depends on an exact SDK band +`v2-ci.yml:10-12` claims "The .NET 10 SDK is pinned via global.json at the repo root" — +**no `global.json` exists**. Meanwhile `Directory.Packages.props:44-49` pins +`Microsoft.CodeAnalysis.CSharp` to 5.0.0 explicitly because "SDK 10.0.105 ships compiler +5.0.0.0… until the SDK rolls to 10.0.110+". An SDK roll on a runner or dev box silently +changes the compiler this pin was matched to. **Recommendation:** add `global.json` with +`rollForward: latestFeature` and revisit the Roslyn pin note when it lands. + +#### S-8 (Low) — first-caller interval wins for the shared data subscription +`SubscribeAsync` creates one `_dataSubscription` with the first caller's `intervalMs` +(`OpcUaClientService.cs:262`); later subscriptions with different intervals join the same +publish interval silently (the monitored-item sampling interval is honoured, the publish +cadence is not). Fine for the CLI; surprising for UI users mixing 100 ms and 10 s items. +Document or create per-interval subscriptions. + +### 2. PERFORMANCE + +#### P-1 (Medium) — CI does 7 independent restore+build passes per push with no caching +Every matrix leg in `v2-ci.yml` checks out and implicitly restores/builds from scratch +(`dotnet test` without `--no-build`), and the dedicated `build` job's outputs are thrown +away. No `actions/cache` for the NuGet package folder. For a solution this size (70+ +projects) that is the bulk of CI wall-clock. **Recommendation:** cache +`~/.nuget/packages` keyed on `Directory.Packages.props`, and either share the build via +artifacts or accept rebuild but add `--no-restore` after an explicit cached restore. + +#### P-2 (Low) — recursive browse is N+1 over `HasChildrenAsync` +`OpcUaClientService.BrowseAsync` issues an extra `HasChildrenAsync` round-trip per Object +child (`OpcUaClientService.cs:230-231`), and `SubscribeCommand.CollectVariablesAsync` +walks the tree serially (`SubscribeCommand.cs:256-281`). On the fleet address space +(thousands of equipment folders) `subscribe -r` start-up is dominated by this. Acceptable +for a diagnostic tool; batch the browse (`BrowseNext` on multiple nodes / read +`References` in bulk) if it becomes a soak-test bottleneck. + +#### P-3 (Low, positive) — integration-fixture cost is well-engineered +The probe-once collection-fixture pattern (`ModbusSimulatorFixture` remarks: "checking +every test would waste several seconds against a firewalled endpoint") is consistently +applied across all 10 IntegrationTests projects, and fixtures do not hold sockets open. +No per-test container spin-up anywhere. This is the right shape; the problem is what CI +does with it (S-2), not the fixtures themselves. + +#### P-4 (Low) — CLI subscription output path is allocation-sane +`SubscribeCommand` serialises SDK callbacks through an unbounded single-reader `Channel` +(`SubscribeCommand.cs:118-119`) rather than locking the console writer — correct and +cheap. The unbounded channel could balloon if the console blocks while thousands of +monitored items update; a bounded channel with `DropOldest` would cap it. Cosmetic at +current scale. + +### 3. CONVENTIONS + +#### C-1 (High) — the custom analyzer is wired into **zero** projects; OTOPCUA0001 enforces nothing +A repo-wide grep of every `.csproj`, `.props`, and `.targets` finds only two references to +`ZB.MOM.WW.OtOpcUa.Analyzers`: its own csproj and its test project's `ProjectReference`. +No consuming project references it with `OutputItemType="Analyzer"`, and +`Directory.Build.props` does not inject it. The carefully built rule — "every +IReadable/IWritable/… call must route through CapabilityInvoker" — is enforced **only by +its 31 unit tests asserting the analyzer itself works**, not against the actual Core/ +Server/Driver code it was written to police. This is the same failure mode the project +has already been bitten by twice at runtime (the dormant `GatewayTagProvisioner`, the +non-forwarding `DeferredAddressSpaceSink`): a capability built and tested but never +plugged in. **Recommendation:** add to `Directory.Build.props` (conditioned on +`'$(MSBuildProjectName)' != 'ZB.MOM.WW.OtOpcUa.Analyzers'`): +```xml + + + +``` +then triage the first wave of OTOPCUA0001 warnings (add the documented test-project +`NoWarn` where intended). + +#### C-2 (Medium) — `TreatWarningsAsErrors` opt-in never reached the Client/Tooling slice (or most tests) +`Directory.Build.props:2-11` explains TWE is per-project pending legacy cleanup, and every +Core, Server, Driver, and Driver-CLI **src** project has opted in — but +`Client.CLI.csproj`, `Client.Shared.csproj`, `Client.UI.csproj`, and +`Analyzers.csproj` have not, and only 9 of 47 test projects have. The comment's stated +blocker ("pre-v2 test projects… xUnit1051") does not apply to the client stack, which is +v2-era code. **Recommendation:** add TWE to the four remaining src projects now (they +build warning-clean or nearly so), and burn down the test-project debt per module. + +#### C-3 (Low) — central package management discipline is exemplary; the audit carve-out is still valid +Verified current state: CPM enabled, every version pinned, `packageSourceMapping` prevents +dependency-confusion for the private `ZB.MOM.WW.*` namespaces (`NuGet.config:9-29`), and +the `NuGetAuditSuppress` for GHSA-2m69-gcr7-jv3q (`Directory.Build.props:19-32`) is still +present, still scoped to a single advisory, and still accurate as written (transitive +SQLitePCLRaw native bundle, no patched release; documented removal condition). The two +transitive CVE pins (OpenTelemetry 1.15.3 at `Directory.Packages.props:80-87`, +Tmds.DBus.Protocol 0.21.3 at lines 104-107 with the matching direct reference in +`Client.UI.csproj`) follow the memory-documented "surgical direct reference" strategy. +No action; keep the removal reminders alive. + +#### C-4 (Low) — test naming/layout is consistent; one categorisation deviation +`*.Tests` vs `*.IntegrationTests` naming is uniform and mirrors `src/` exactly. The single +deviation: live-gated tests live *inside* a unit-suite project +(`Driver.Historian.Gateway.Tests/Live/GatewayLiveIntegrationTests.cs`, +`Category=LiveIntegration`) rather than a `*.IntegrationTests` sibling — harmless because +they skip-gate on env vars, but a `dotnet test tests/Drivers/...Gateway.Tests` run now has +a hidden live-test dependency surface. Framework split: 44 projects on xunit.v3, 3 legacy +on xunit 2.x (`AdminUI.Tests`, `ControlPlane.Tests`, `Runtime.Tests`) — finish the +migration and drop the `xunit` 2.9.2 pin from `Directory.Packages.props:108`. + +#### C-5 (Low) — no mechanical code-style enforcement exists; StyleGuide.md is a *docs* style guide +There is **no `.editorconfig`** in the repo, no `dotnet format` CI step, and +`StyleGuide.md` is exclusively a documentation-writing guide — whose opening line still +says "for all **ScadaBridge** documentation" (`StyleGuide.md:3`), a copy-paste from the +sister repo. Code style is therefore enforced only by review culture. **Recommendation:** +add a root `.editorconfig` (the implicit conventions are already consistent — file-scoped +namespaces, 4-space, `_camelCase` fields) and fix the StyleGuide title/product name. + +#### C-6 (Low) — csproj boilerplate duplicates Directory.Build.props +Nearly every csproj restates `TargetFramework`/`Nullable`/`ImplicitUsings` that +`Directory.Build.props:12-17` already supplies (e.g. `Client.CLI.csproj:5-8`). Harmless +but it means an SDK bump requires touching 70+ files instead of one. Strip on next sweep. + +### 4. UNDERDEVELOPED AREAS + +#### U-1 — Client.UI maturity: real product, adequately tested, current docs +Verdict: **not vestigial**. ~2.9 k LOC C#, 551 lines AXAML, MVVM with dispatcher and +settings seams, alarm ack/confirm/shelve dialogs, history view with custom range picker, +126 headless tests, and `docs/Client.UI.md` matches the code (verified stack table and +window-layout claims). It is the least exercised layer in anger (no CI — S-1 — and no +E2E), and headless tests can't catch AXAML binding regressions (the AdminUI "no bUnit — +live-verify" lesson applies equally here). Rating it "maturing", not "underdeveloped". + +#### U-2 (Medium) — source → test coverage matrix: three structural gaps +Matrix over the 41 solution src projects (test projects verified against the slnx): + +| Source project | Unit tests | Integration | Gap | +|---|---|---|---| +| Client.Shared | ✅ Client.Shared.Tests (158) | — | none | +| Client.CLI | ✅ Client.CLI.Tests (104) | — | none | +| Client.UI | ✅ Client.UI.Tests (126, headless) | — | none | +| Tooling/Analyzers | ✅ Analyzers.Tests (31) | — | **not wired into builds (C-1)** | +| Core, Commons, Cluster, Configuration, Core.Scripting, VirtualTags, ScriptedAlarms, AlarmHistorian, Core.Abstractions | ✅ each | — | none | +| **Core.Scripting.Abstractions** | ❌ | — | no dedicated tests (thin interfaces; covered incidentally by Core.Scripting.Tests) | +| AdminUI, ControlPlane, Runtime, Security, OpcUaServer | ✅ each | OpcUaServer.IntegrationTests | none | +| **Host** | ❌ **no Host.Tests** | Host.IntegrationTests only | Host's DI/wiring logic (the layer where both "dormant wiring" bugs lived) has no unit tier | +| All 8 protocol/gateway drivers + Modbus.Addressing + 2 Browsers + Historian.Gateway | ✅ each | 7 × IntegrationTests + LiveIntegration | none | +| **8 × `*.Contracts` projects** (Galaxy, Modbus, S7, AbCip, AbLegacy, TwinCAT, FOCAS, OpcUaClient) | ❌ | — | DTO-only; consuming-suite coverage; OpcUaClient.Contracts NamespaceMap gap already known-deferred | +| 7 × Driver CLIs + Cli.Common | ✅ each | — | none | + +The one gap worth acting on is **Host**: it is where registration/forwarding mistakes +land, and its only automated coverage requires the 2-node harness. A `Host.Tests` project +asserting DI composition (e.g. "when `ServerHistorian:Enabled`, `IHistorianProvisioning` +resolves to `GatewayTagProvisioner` and the applier receives it") would have caught the +PR #423 dormancy at unit speed. + +#### U-3 (Medium) — docs drift: `docs/Client.CLI.md` documents 8 of 14 commands +Spot-checks performed: +1. `docs/Client.CLI.md` command sections: connect, read, write, browse, subscribe, + historyread, alarms, redundancy — **`ack`, `confirm`, `shelve`, `enable`, `disable` + have zero mentions** (grep), yet the commands ship + (`Commands/AcknowledgeCommand.cs`, `ConfirmCommand.cs`, `ShelveCommand.cs`, + `EnableCommand.cs`, `DisableCommand.cs`) and CLAUDE.md explicitly says "Client.CLI + supports `ack`, `confirm`, `shelve` commands. See `docs/Client.CLI.md` for full + documentation." Five commands undocumented. +2. The doc's "104 unit tests" claim (`docs/Client.CLI.md`, Testing section) — **accurate**: + exactly 104 `[Fact]`/`[Theory]` in Client.CLI.Tests today. +3. `docs/Client.UI.md` stack/layout claims — **accurate** vs csproj and Views/. +4. `v2-ci.yml:10-12` global.json claim — **false** (S-7). +Recommendation: add the five missing command sections; they are the operator-facing alarm +workflow. + +#### U-4 (Medium) — repo-root planning-file sprawl contradicts its own rules +Five point-in-time planning/state files are committed at the root: `looseends.md` (state +as of 2026-05-18), `pending.md` (2026-06-16), `stillpending.md`, `current.md`, +`HISTORIAN-GATEWAY-INTEGRATION-ISSUES.md`. `pending.md` itself declares "HARD RULE: never +`git add .`; never stage `pending.md` / `current.md` / …" — **yet `pending.md` and +`current.md` are tracked** (they appear in `git ls-files`; last commits `cd20c3c0`, +`384dbd7d`). The memory index says the stillpending backlog is ~95% shipped, so most of +this content is stale snapshots that now contradict `docs/` and CLAUDE.md. +**Recommendation:** either move them under `docs/plans/` with dated names (the existing +convention) or delete + gitignore them; make the file's own rule true. + +#### U-5 (Medium) — orphaned proprietary AVEVA DLLs tracked in `lib/` +`lib/` contains 7 **committed** vendor binaries (`aahClient.dll`, `aahClientCommon.dll`, +`aahClientManaged.dll`, `ArchestrA.MXAccess.dll`, `ArchestrA.CloudHistorian.Contract.dll`, +`Historian.CBE.dll`, `Historian.DPAPI.dll`). No csproj anywhere references them (zero +`HintPath` hits repo-wide) — they are leftovers from the retired Wonderware sidecar / +in-process MXAccess era. Committed proprietary SDK DLLs are a redistribution/licence risk +in any clone of this repo and dead weight in history. **Recommendation:** `git rm -r lib/` +(the bitness/COM story now lives entirely in the mxaccessgw repo, per CLAUDE.md). + +#### U-6 (Low) — secrets hygiene: `sql_login.txt` is NOT committed (gitignored), but is still a plaintext credential at the root +Verified: `.gitignore:47` lists it, `git check-ignore` confirms, and +`git log -- sql_login.txt` is empty — the file has never been committed. The remaining +risk is purely local (plaintext `wwadmin` password for `wonder-sql-vd03` sitting in a +Desktop directory that agents and sync tools read). **Recommendation:** move to +`dotnet user-secrets` / an env file outside the repo; at minimum keep it out of any +export bundle (`export-clean-copy.bat` at the root should be checked to exclude it). +Also note the retired `Driver.Historian.Wonderware*` project dirs still exist on disk as +ignored `bin/obj` husks — local-only debris, safe to delete. + +#### U-7 (Low) — missing CI stages (inventory) +Not gated anywhere today: analyzer tests + analyzer enforcement (C-1), all client/driver/ +core unit suites (S-1), `dotnet format`/style (C-5), NuGet vulnerability audit as an +explicit failing step (currently only implicit in restore warnings, which are not TWE'd +in most projects), the `scripts/check-code-reviews-readme.ps1` consistency check, and any +docs link/freshness check. The `ci/ab-server.lock.json` fixture pin references a +"GitHub Actions step" in `docs/v2/test-data-sources.md` that does not exist in either +workflow — the AB fixture download step was planned but never landed. + +#### U-8 (Low) — CLI security posture is dev-tool-grade by construction +`CommandBase.CreateConnectionSettings()` hardcodes `AutoAcceptCertificates = true` +(`CommandBase.cs:91`), so `--security signandencrypt` encrypts but never authenticates the +server (any cert accepted → MITM-able), and `-P` takes the password as a process-visible +argv. Acceptable for a diagnostic tool, but the doc should say so, and a +`--strict-certs` opt-in would be cheap. Also `Client.Shared`'s alarm fallback path bakes +Galaxy-specific attribute names (`.InAlarm`, `.Acked`, `.TimeAlarmOn`, `.DescAttrName`) +into the generic client library (`OpcUaClientService.cs:851-871`) — a documented but +layering-violating convenience. + +--- + +## Maturity Ratings (1 = ad hoc, 5 = exemplary) + +| Dimension | Rating | Justification | +|---|---|---| +| Stability | **2.5** | The client code and fixture patterns are careful, but CI runs ~15% of the suite, the integration tier silently skips to green on hosted runners, and the nightly E2E is a documented no-op — the safety net exists mostly on developer machines. | +| Performance | **3.5** | Build/test structure is lean (CPM, probe-once fixtures, channel-serialised CLI output); loses points for 7× uncached restore/builds in CI and the N+1 recursive browse. | +| Conventions | **3** | Package management and test layout are exemplary and self-documenting, but the flagship convention-enforcement tool (the analyzer) is wired into nothing, TWE never reached the client slice, and there is no .editorconfig — conventions hold by culture, not mechanism. | +| Underdeveloped areas | **3** | Client.UI is genuinely mature and the coverage matrix is nearly complete (Host is the one real hole); dragged down by 5-command docs drift, committed stale planning files that violate their own rules, and orphaned proprietary DLLs in `lib/`. | + +--- + +## Top recommendations (ordered) + +1. Wire `ZB.MOM.WW.OtOpcUa.Analyzers` into every project via `Directory.Build.props` (C-1). +2. Widen `v2-ci.yml` to the whole solution and make skipped integration tests visible/failing in CI (S-1, S-2). +3. Delete tracked `lib/` vendor DLLs (U-5) and resolve the root planning-file contradiction (U-4). +4. Add the five missing command sections to `docs/Client.CLI.md` (U-3). +5. Add `global.json` (S-7), `Host.Tests` (U-2), and TWE to the four client/tooling csprojs (C-2). +6. Fix the three `OpcUaClientService` lock-discipline gaps before Client.UI grows long-lived multi-thread usage (S-5). diff --git a/archreview/plans/00-INDEX.md b/archreview/plans/00-INDEX.md new file mode 100644 index 00000000..360b9532 --- /dev/null +++ b/archreview/plans/00-INDEX.md @@ -0,0 +1,69 @@ +# Arch-Review Remediation Plans — Index + +Design + implementation plans for every finding in the 2026-07-08 architecture review +(`archreview/*.md`, review commit `9cad9ed0`). Each plan verifies every finding against +the current tree, then gives root cause → design (with alternatives) → concrete file-level +steps → tests → effort/risk. + +## Plan documents + +| # | Plan | Findings | Verification | +|---|---|---|---| +| 01 | [Core & composition](01-core-composition-plan.md) | 21 | all confirmed, 0 stale | +| 02 | [Scripting & alarms](02-scripting-alarms-plan.md) | 30 | all confirmed, 0 stale | +| 03 | [Server & runtime](03-server-runtime-plan.md) | 28 | all confirmed (U1 = stale *doc* only) | +| 04 | [AdminUI](04-adminui-plan.md) | 20 | all confirmed (C-3/C-4 good, no action) | +| 05 | [Protocol drivers](05-protocol-drivers-plan.md) | 37 + 1 positive | all confirmed, 0 stale | +| 06 | [Gateway integrations](06-gateway-integrations-plan.md) | 28 (23 actionable) | all confirmed (U-1 = stale *doc*) | +| 07 | [Client, tooling & engineering](07-client-tooling-engineering-plan.md) | 26 | all confirmed, 0 stale | + +**Total: ~190 findings planned. Zero were stale-because-already-fixed; the only "stale" items are +two documentation drifts (CLAUDE.md Known Limitation 2) both flagged for correction.** + +## The 4 Criticals (fix first) + +| # | Critical | Plan | Fix shape | +|---|---|---|---| +| 1 | Split-brain resolver never activated → no hard-crash failover | 03 S1 | `ClusterOptions.SplitBrainResolver = KeepOldestOption` + hard-kill failover test | +| 2 | Production VT script timeout is dead code → one loop wedges the actor | 02 U2 (+U3) | Route through `TimedScriptEvaluator` + `CompiledScriptCache` | +| 3 | S7 has no reconnect path → PLC reboot kills the driver | 05 STAB-1 | `IS7PlcFactory` seam + lazy `EnsureConnectedAsync` | +| 4 | TwinCAT ADS subs orphaned after reconnect → silent data stop | 05 STAB-2 | Store replayable registration intent, replay on client swap | + +## Cross-cutting guardrails (from OVERALL themes, span multiple plans) + +- **Theme #1 built-but-never-wired** — the recurring failure mode. Guardrails proposed across plans: + reflection-exhaustive `DeferredAddressSpaceSink` forwarding test (03 U2, sequenced *before* the + surgical-add work), wiring the OTOPCUA0001 analyzer repo-wide (07 C-1), a startup log line proving + SBR is active (03 S1), and wiring/removing the dormant historian health surface (06 S-6). +- **Theme #2/#6 fixes don't flow to shared templates** — Part B of plan 05: `PollGroupEngine` v2 + (backoff + onError + delete the S7 fork), shared strict `ReadEnum`/`ReadBool`, `ConnectionBackoff` + primitive, `ResolveHost`-via-resolver; plus `TagConfigIntent.Parse` in Commons (01 C-1) killing the + 4-project byte-parity duplication. +- **Theme #3 optimistic success** — failure-visibility fixes: `AddressSpaceApplyOutcome` failure field + (01 S-1), Galaxy write fail-closed (06 S-1), primary-gate default-deny on unknown role (03 S4). +- **Theme #4 verification gap** — CI to all unit suites + fail-on-skip (07 S-1/S-2), fault-injection tier + (hard-kill / drop-PLC), the missing bUnit substitute = reflection/policy guards (04 C-1). +- **Theme #5 doc drift** — fix CLAUDE.md Known Limitation 2 + scadaproj index (03 U1 / 06 U-1), + ScriptAnalysis policy doc, Client.CLI docs (07 U-3). +- **Theme #7 repo hygiene** — `git rm -r lib/` AVEVA DLLs, untrack planning files, delete Wonderware + husks (07 U-4/U-5/U-6). + +## Suggested execution order + +Front-loaded per the OVERALL prioritized action list — items 1–6 eliminate every Critical plus the two +systemic guards (analyzer + CI) that stop the built-but-never-wired pattern recurring: + +1. **Criticals** — 03 S1, 02 U2+U3, 05 STAB-1, 05 STAB-2 (each with its missing test) +2. **Systemic guards** — 07 C-1 (analyzer) + 07 S-1/S-2 (CI to all suites, fail-on-skip); 03 U2 (Deferred-sink forwarding test) +3. **Silent-corruption Highs** — 05 STAB-4/5 (AbCip lock, FOCAS caches), 05 Modbus timeout-fatal classification +4. **Failure-visibility Highs** — 01 S-1, 06 S-1, 03 S4 +5. **AdminUI authz** — 04 C-1 (ConfigEditor policy + reflection guard) +6. **Perf** — 03 P1 (surgical pure-adds, after 03 U2 guard) +7. **Consolidation + hygiene + docs** — 01 C-1, 05 Part B, 07 U-4/U-5, doc fixes + +## Open maintainer decision + +- **Prune vs. keep the dormant machinery** (01 U-2 / 02 U-1): the callerless Tier-C recycle machinery + + `IDriverSupervisor`, and the ~2.4k-line `Core.VirtualTags` engine that production bypasses. Plans + recommend retiring both *after* the live paths are hardened (02 U2/U3 first), but the delete-vs-keep + call is the maintainer's. diff --git a/archreview/plans/01-core-composition-plan.md b/archreview/plans/01-core-composition-plan.md new file mode 100644 index 00000000..bdab392d --- /dev/null +++ b/archreview/plans/01-core-composition-plan.md @@ -0,0 +1,533 @@ +# Design + Implementation Plan — Core & Composition Pipeline + +**Domain report:** `archreview/01-core-composition.md` +**Baseline commit reviewed:** `9cad9ed0` (master) +**Plan author verification date:** 2026-07-08 +**Scope:** Core, Core.Abstractions, Commons, Configuration, Cluster; AddressSpace Composer/Planner/Applier. + +## Verification summary + +All 21 findings were opened at their cited file:line against the current tree. **Every finding is CONFIRMED +accurate** — none are stale or already-fixed. (One nuance: U-3's *delta feed* half is already implemented and +shipped, matching the report's own "delta feed landed, initial set still empty" framing — so U-3 is confirmed as +partially-done debt, not stale.) + +Key structural fact that shapes the C-1/C-3 fix: **`Commons` is a pure leaf project** (zero `ProjectReference`s, +verified from `Commons.csproj`), and **`Configuration` does not currently reference it** but *can* without creating a +cycle (Commons references neither Configuration nor Core). This makes "move shared parsing/scheme into Commons" the +correct, low-risk home for the byte-parity duplication class. + +## Priority ordering (this plan) + +Ordered by severity, then by the OVERALL prioritized action list (items #7 and #10 touch this domain directly). + +| Order | ID | Sev | Title | Effort | OVERALL ref | +|---|---|---|---|---|---| +| 1 | S-1 | High | Applier swallows sink failures; deploy reports success on broken address space | M | Action #7 | +| 2 | C-1 | High | TagConfig parsing byte-parity-replicated in 4 projects | M | Action #10 | +| 3 | C-3 | Med | DraftValidator re-derives NodeId scheme + hard-codes driver-type string | S | Action #10 | +| 4 | P-1 | Med | Compose parses each TagConfig JSON 4× | S (folds into C-1) | — | +| 5 | S-2 | Med | `PollGroupEngine.Unsubscribe` blocks caller up to 5 s | M | — | +| 6 | P-2 | Med | Per-operation allocations on authorization hot path | S | — | +| 7 | P-3 | Med | PollGroupEngine task-per-subscription, no coalescing | L (doc-only now) | — | +| 8 | C-2 | Med | Core → Configuration coupling (accepted-risk, document) | S | — | +| 9 | U-1 | Med | Dead/dormant code: GDNM, EquipmentNodeWalker, TryParseRelayBody | S | Action #12 | +| 10 | U-2 | Med | Tier C recycle machinery has no `IDriverSupervisor` impl | S | Action #12 | +| 11 | U-3 | Med | Continuous-historization initial ref set still empty at spawn | M | — | +| 12 | P-4 | Low | Per-write allocation in non-idempotent write path | S | — | +| 13 | S-3 | Low | `Subscribe`/`RegisterAsync` have no disposed guard | S | — | +| 14 | S-4 | Low | ScheduledRecycleScheduler catch-up storm after stall | S | — | +| 15 | S-5 | Low | GDNM re-walk teardown not synchronized (moot — dead code) | — (folds into U-1) | — | +| 16 | C-4 | Low | Composer violates purity with `Trace.TraceWarning` | S | — | +| 17 | C-5 | Low | Inconsistent duplicate-key defensiveness in Compose | S | — | +| 18 | C-6 | Low | Vestigial Akka package reference in Commons | S | — | +| 19 | C-7 | Low | Stale scaffold-era XML docs on load-bearing classes | S | — | +| 20 | U-4 | Low | Additive-only ACL model (no Deny) — documented v2.0 gap | S (doc) | — | +| 21 | U-5 | Low | Uneven test coverage (ClusterRoleInfo untested) | M | — | + +--- + +## 1. S-1 — Applier swallows every sink failure; deploy reports applied even when address space is broken (High) + +**Verification: CONFIRMED.** +- `AddressSpaceApplyOutcome` (`AddressSpaceApplier.cs:691`) carries only `RemovedNodes/AddedNodes/ChangedNodes/RebuildCalled` — **no failure count**. +- `SafeRebuild` (`:373-383`) catches every exception from `_sink.RebuildAddressSpace()` and only logs; the caller still sets `rebuilt = true` (`:150-153`). +- `SafeEnsureFolder`/`SafeEnsureVariable` (`:612-622`), `SafeWriteAlarmCondition`/`SafeMaterialiseAlarmCondition` (`:677-687`) swallow per-node failures into Warnings. +- Consumer confirmed: `OpcUaPublishActor.cs:335` calls `_applier.Apply(plan)` and **only logs** the outcome (`:357`). The whole rebuild path is itself inside a try/catch that logs (`:360-363`). There is no `ApplyAck`/`DeploymentFailed` reply carrying success/failure back to a deploy coordinator — the deploy seals independently of apply health. + +**Root cause.** The "never fail a deploy" posture (correct for the *detached* provisioning/historized-ref hooks) was applied uniformly, including to the structural materialisation that is the deploy's core contract. The outcome record has no channel to express partial/total failure, so the publish actor has nothing to act on even if it wanted to. + +**Proposed design.** Give the applier a truthful failure surface and route it to an operator-visible signal. This is the domain-01 slice of OVERALL cross-cutting theme #3 ("optimistic success"). + +Two-part approach: +1. **Count failures in the outcome.** Change the `Safe*` helpers to return a `bool` (or increment a private counter) and add `RebuildFailed` + `FailedNodes` to `AddressSpaceApplyOutcome`. `SafeRebuild` returning false must flip `rebuilt`'s meaning: keep `RebuildCalled` (it *was* attempted) but set `RebuildFailed = true` so callers can distinguish "rebuilt OK" from "rebuild threw". +2. **Surface it.** In `OpcUaPublishActor`, when `outcome.RebuildFailed || outcome.FailedNodes > 0`, log at `Error` (not Info) **and** emit an operator-visible signal. Reuse the existing telemetry meter `OtOpcUaTelemetry.OpcUaSinkWrite` with a `kind:"apply-failed"` tag (pattern already used at `OpcUaPublishActor.cs:303,356`), and — if a deploy-status/alert channel is reachable from the actor — post a degraded status. Do **not** convert this into a hard deploy abort in v1: a partially-materialised address space is still better than none, and the publish actor is already past the seal decision. The goal is *visibility*, matching the report's "escalate `SafeRebuild` failure to at least a degraded ack." + +**Alternatives considered.** +- *Throw from Apply on rebuild failure* — rejected: the actor's catch would swallow it anyway and it removes the counts other callers want; also risks aborting the subsequent `Materialise*` passes that could still succeed. +- *Full ApplyAck→DeploymentFailed wiring* — larger blast radius touching ControlPlane deploy coordination; deferred. The OVERALL list scopes item #7 as "add a failure field ... surface"; the deploy-abort decision belongs with report 03's deploy-coordination work. + +**Implementation steps.** +- `AddressSpaceApplier.cs`: + - Extend record: `AddressSpaceApplyOutcome(int RemovedNodes, int AddedNodes, int ChangedNodes, bool RebuildCalled, bool RebuildFailed, int FailedNodes)`. Keep a 4-arg → 6-arg default or update the two construction sites (`:80` empty-plan, `:215` main return). + - `SafeRebuild()` → `private bool SafeRebuild()` returning `false` on catch; caller records `rebuildFailed`. + - `SafeEnsureFolder`/`SafeEnsureVariable`/`SafeWriteAlarmCondition`/`SafeMaterialiseAlarmCondition` → return `bool`; the `Materialise*` passes tally a `_failedNodes` counter. Because the `Materialise*` passes run from `OpcUaPublishActor` *after* `Apply` returns (they are separate public methods, `:338-354`), expose the tally via either (a) an out-param/return on each `Materialise*` method, or (b) an instance counter reset at the start of the rebuild sequence and read after. Prefer (a) — each `Materialise*` returns an `int failedNodes`; the actor sums them. +- `OpcUaPublishActor.cs` (`:335-358`): capture the failure tally across `Apply` + the four `Materialise*` calls; branch to `_log.Error` + `OtOpcUaTelemetry.OpcUaSinkWrite.Add(1, kind:"apply-failed")` when non-zero. + +**Tests (unit).** `tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests/AddressSpace*`: +- A throwing fixture sink (the `NullOpcUaAddressSpaceSink` pattern extended to throw on `RebuildAddressSpace` / `EnsureVariable`): assert `outcome.RebuildFailed == true` when rebuild throws, and `FailedNodes == N` when N `EnsureVariable` calls throw during a materialise pass. +- Assert the happy path still reports `RebuildFailed == false, FailedNodes == 0`. + +**Effort: M. Risk: low-medium** — record shape change ripples to every `Apply` caller + test asserting the record; contained to OpcUaServer + Runtime. + +--- + +## 2. C-1 — TagConfig parsing byte-parity-replicated in four projects (High) + +**Verification: CONFIRMED.** Four independent copies of the top-level `FullName` (and sibling `alarm`/`isHistorized`/`isArray`/`HostAddress`) parsing: +- `AddressSpaceComposer.cs:536-551` (`ExtractTagFullName`) + `:601-686` (`ExtractTagAlarm`/`ExtractTagHistorize`/`ExtractTagArray`) + `:567` (`TryExtractDeviceHost`). +- `Runtime/Drivers/DeploymentArtifact.cs` (artifact-decode mirror; comment at `:666` cites `EquipmentNodeWalker.ExtractFullName`). +- `Core/OpcUa/EquipmentNodeWalker.cs` `ExtractFullName` (dormant — see U-1). +- `Configuration/Validation/DraftValidator.cs:60-71` (`ExtractTagConfigFullName`, "a small local copy"). + +Each copy carries a "MUST parse identically (byte-parity)" comment. `EquipmentScriptPaths` (`Commons/Types/`) is the in-repo precedent for de-duplicating exactly this kind of cross-seam parser. + +**Root cause.** OpcUaServer does not reference the Core driver assembly and Configuration does not reference Commons, so no shared home existed — each seam grew its own parser, synced by human-enforced comments. This is the OVERALL cross-cutting theme #2 ("fixes don't flow back to shared templates") in the Core layer. + +**Proposed design.** Create a single `TagConfigIntent` parser in **Commons** (leaf, no EF/SDK deps — verified) that parses the blob **once** and returns all intents, then retrofit every seam onto it. This simultaneously resolves **P-1** (one parse instead of four). + +```csharp +// src/Core/ZB.MOM.WW.OtOpcUa.Commons/Types/TagConfigIntent.cs +public sealed record TagConfigIntent( + string FullName, // top-level "FullName" or raw blob fallback + TagAlarmIntent? Alarm, // "alarm" object or null + bool IsHistorized, string? HistorianTagname, + bool IsArray, uint? ArrayLength) +{ + public static TagConfigIntent Parse(string? tagConfig); // single JsonDocument.Parse, never throws +} +public sealed record TagAlarmIntent(string AlarmType, int Severity, bool? HistorizeToAveva); +``` + +Add a companion `DeviceConfigIntent.TryExtractHost(string?)` (or fold `TryExtractDeviceHost` + `NormalizeDeviceHost` into Commons) so the device-host normalization single-source-of-truth also lives there. + +**Why Commons, and why one record.** The report's own recommendation, and the `EquipmentScriptPaths` precedent, both point here. A single record parsed once (a) kills the 4× duplication, (b) collapses P-1's 4 parses/tag → 1, and (c) lets the byte-parity contract be enforced by **one** cross-seam parity test instead of N. + +**Alternatives considered.** +- *Put it in Core.Abstractions* — rejected: DeploymentArtifact (Runtime) and DraftValidator (Configuration) would still need it; Commons is the common ancestor all of them can see (or cheaply add). +- *Keep separate parsers, add a shared golden-file parity test* — rejected: tests the symptom, not the cause; still 4 code paths to drift. + +**Implementation steps.** +1. Add `Commons/Types/TagConfigIntent.cs` (+ `DeviceConfigIntent` helper). Port the *exact* current parse semantics verbatim from `AddressSpaceComposer` (the canonical copy) — especially the raw-blob fallback for `FullName`, the `historizeToAveva` bool? default-on rule, and the `arrayLength`-only-when-`isArray` rule — so no byte-parity behavior changes. +2. `AddressSpaceComposer.cs`: replace the per-tag `ExtractTagFullName`/`ExtractTagAlarm`/`ExtractTagHistorize`/`ExtractTagArray` calls (`:418-433`) with a single `var intent = TagConfigIntent.Parse(t.TagConfig);` then project fields off `intent`. Replace `TryExtractDeviceHost`/`NormalizeDeviceHost` usages with the Commons helper. Delete the now-private methods. +3. `Runtime/Drivers/DeploymentArtifact.cs`: replace its mirror `Extract*` methods with `TagConfigIntent.Parse`. +4. `Configuration/Validation/DraftValidator.cs`: add a `ProjectReference` to Commons (cycle-safe — verified), replace `ExtractTagConfigFullName` (`:60-71`) with `TagConfigIntent.Parse(t.TagConfig).FullName` (note: current DraftValidator returns null for the missing case; preserve that by checking `string.IsNullOrWhiteSpace` against the parsed FullName, being careful that the Commons parser returns the raw blob as fallback — the Galaxy check wants the *explicit* FullName, so either expose a `HasExplicitFullName` bool on the intent or keep DraftValidator's null-on-absent semantics by adding a `TagConfigIntent.TryGetExplicitFullName`). +5. `EquipmentNodeWalker.ExtractFullName`: deleted with the walker under U-1 (or repointed at Commons if the walker survives as a test fixture). + +**Tests.** +- `Commons.Tests`: exhaustive `TagConfigIntent.Parse` table (object/non-object/blank/malformed/each field present-absent-wrong-type) — this becomes the *single* byte-parity authority. +- Keep one cross-seam parity test asserting `AddressSpaceComposer` compose output and `DeploymentArtifact` decode output agree on a fixture set (this already exists in the OpcUaServer.Tests byte-parity suite — repoint it at the shared parser). + +**Effort: M. Risk: medium** — touches the byte-parity contract, the crown-jewel invariant. Mitigation: port semantics verbatim, land behind the existing byte-parity test suite, no behavior change intended. + +--- + +## 3. C-3 — DraftValidator re-derives NodeId scheme + hard-codes driver-type string (Medium) + +**Verification: CONFIRMED.** +- `DraftValidator.cs:75-97` (`ValidateNoEquipmentSignalNameCollision`) hand-rolls the folder-scoped key `"{eq}/{folder}/{name}"` via a local `Key(...)` because Configuration can't reference `Commons.OpcUa.EquipmentNodeIds` (`Commons/OpcUa/EquipmentNodeIds.cs` confirmed to exist). +- `DraftValidator.cs:48` hard-codes `dtype != "GalaxyMxGateway"`. + +**Root cause.** Same missing shared-contract home as C-1: dependency direction blocked Configuration from seeing the Commons scheme/constants. + +**Proposed design.** Once C-1 adds the `Configuration → Commons` ProjectReference, this becomes trivial: +- Replace the local `Key(...)` in `ValidateNoEquipmentSignalNameCollision` with the same key `EquipmentNodeIds` produces (add a public `EquipmentNodeIds.Variable(eq, folder, name)` — it already exists per `AddressSpaceApplier.cs:177` usage — and call it, so the collision check keys on the *exact* materialiser scheme, eliminating drift risk). +- Replace `"GalaxyMxGateway"` with a shared driver-type constant. Add `DriverTypes.GalaxyMxGateway` (a `const string`) in Commons (or reuse an existing driver-type-names home if one exists — grep `DriverType` constants first). Point DraftValidator at it. + +**Implementation steps.** +1. (Depends on C-1's `Configuration → Commons` reference.) +2. Ensure `EquipmentNodeIds.Variable` is public and matches DraftValidator's `null-folder ⇒ "{eq}/{name}"` behavior (verify against `AddressSpaceApplier.cs:177`). If the VirtualTag branch (which passes `folder=null`) needs the no-folder overload, add it. +3. Add `Commons` driver-type constant; replace the literal at `DraftValidator.cs:48`. +4. Grep for other `"GalaxyMxGateway"` literals (composer comment references it; `DriverFactoryBootstrap` registers it) and repoint the *logic* sites (not doc comments) at the constant. + +**Tests.** Existing `Configuration.Tests` DraftValidator collision + Galaxy-FullName tests should pass unchanged (behavior-preserving). Add one asserting the collision key equals `EquipmentNodeIds.Variable(...)` for a folder + no-folder case, so the shared-scheme binding is guarded. + +**Effort: S. Risk: low.** + +--- + +## 4. P-1 — Compose parses each TagConfig JSON four times (Medium) + +**Verification: CONFIRMED.** `AddressSpaceComposer.cs:418-433` invokes `ExtractTagHistorize`, `ExtractTagArray`, `ExtractTagFullName`, `ExtractTagAlarm` — each doing its own `JsonDocument.Parse` (`:541,606,638,670`). 4 parses/tag × 2 seams (compose + artifact-decode). + +**Root cause / fix.** This is fully resolved by C-1's `TagConfigIntent.Parse` (single parse per tag per seam). No separate work item — **fold into C-1**. Note in the C-1 PR that P-1 is closed by the same change. + +**Effort: S (subsumed). Risk: as C-1.** + +--- + +## 5. S-2 — `PollGroupEngine.Unsubscribe` blocks the calling thread up to 5 s per subscription (Medium) + +**Verification: CONFIRMED.** `StopState` (`PollGroupEngine.cs:99-112`) does synchronous `task.Wait(TimeSpan.FromSeconds(5))` (`:109`). `Unsubscribe` (`:89-97`) calls it inline. `DisposeAsync` (`:213-236`) does the correct parallel-cancel + awaited `WhenAll(...).WaitAsync(5s)`. + +**Root cause.** The single-subscription teardown path was written synchronously to preserve the "no `_onChange` after teardown" guarantee, but blocks the caller (an OPC UA subscription-management callback / actor handler) while a reader is mid-slow-call. N serial unsubscribes = N × up-to-5 s. + +**Proposed design.** Add `ValueTask UnsubscribeAsync(ISubscriptionHandle)` that cancels the CTS, then awaits `LoopTask.WaitAsync(timeout)` (mirroring `DisposeAsync`'s discipline) rather than blocking. Keep the synchronous `Unsubscribe` for compatibility but re-implement it to **cancel + hand the drain to a background continuation** that disposes the CTS after the loop actually exits — preserving the "CTS not disposed while `Task.Delay` holds the token" ordering the current comment (`:102-106`) protects. + +Because the poll loop already treats `ObjectDisposedException` from a disposed CTS as normal cancellation (`:133-136`), the background-drain variant is safe: worst case the loop observes cancellation, the continuation disposes the CTS, no callback fires post-teardown (the loop's `while (!ct.IsCancellationRequested)` + `Task.Delay(ct)` exits before any further `_onChange`). + +**Alternatives considered.** +- *Just shorten the 5 s timeout* — rejected: still blocks, just less; and a short timeout risks disposing the CTS while the loop runs. +- *Fire-and-forget cancel with no drain* — rejected: loses the "no callback after teardown" guarantee that the current design deliberately holds. + +**Implementation steps.** +- `PollGroupEngine.cs`: add `public async ValueTask UnsubscribeAsync(ISubscriptionHandle handle)` doing `TryRemove` + `state.Cts.Cancel()` + `await (state.LoopTask ?? Task.CompletedTask).WaitAsync(5s)` (swallow) + `state.Cts.Dispose()`. +- Re-implement `Unsubscribe` to remove + cancel synchronously, then schedule `_ = DrainAndDisposeAsync(state)` (a private async method that awaits the loop then disposes the CTS) instead of blocking. Return `true` immediately. +- Audit the driver callers (Modbus/S7/AbCip/FOCAS subscription-management callbacks) — where the callback is `async`, prefer `await UnsubscribeAsync`. Grep `\.Unsubscribe(` across `src/Drivers`. + +**Tests (Core.Abstractions.Tests).** +- `UnsubscribeAsync` completes promptly even when the reader delegate blocks (inject a reader that awaits a `TaskCompletionSource`); assert no `_onChange` fires after `UnsubscribeAsync` returns. +- Synchronous `Unsubscribe` returns without a multi-second stall (assert wall-clock under a small bound with a slow reader) and the CTS is eventually disposed (no leak) — the loop count drops to zero. + +**Effort: M. Risk: medium** — teardown ordering is subtle; the existing `DisposeAsync` is the proven template to mirror. Blast radius: all poll-based drivers. + +--- + +## 6. P-2 — Per-operation allocations on the authorization hot path (Medium) + +**Verification: CONFIRMED.** `PermissionTrie.CollectMatches` (`PermissionTrie.cs:42`) does `ldapGroups.ToHashSet(OrdinalIgnoreCase)` per call, plus `new List()` (`:45`). Called per node-operation by `TriePermissionEvaluator.Authorize` (verified consumer). A recursive browse / large CreateMonitoredItems batch = per-node garbage. + +**Root cause.** The group set is rebuilt from the session's LDAP groups on every authorize even though `UserAuthorizationState.LdapGroups` is immutable per membership refresh. + +**Proposed design.** Compute the case-insensitive `HashSet` **once per session-version** and pass it in. Two clean options: +1. **Overload `CollectMatches(NodeScope, IReadOnlySet groups)`** and have the evaluator hold a cached `HashSet` on `UserAuthorizationState` (or a small per-session wrapper) built when the membership/version changes. Keep the `IEnumerable` overload for callers that don't have a prebuilt set. +2. Cache the set on `UserAuthorizationState` itself (lazy, `Lazy>` keyed by the version field). + +Prefer (1): it keeps `PermissionTrie` allocation-free on the hot path and puts the cache where the immutability actually lives (the session state). The `List` is small and bounded by trie depth (≤ 6 levels × grants); leave it unless profiling flags it — or return via a pooled/stackalloc'd span if the evaluator only needs the OR-ed bitmask (it does: the caller "OR-s the flag bits"). A tighter design: add `CollectMask(scope, groups) -> NodePermissions` that OR-s inline and allocates nothing, keeping `CollectMatches` for diagnostics. + +**Implementation steps.** +- `Core/Authorization/UserAuthorizationState.cs`: add a cached `IReadOnlySet LdapGroupSet` built once per version (verify the version/refresh field name). +- `PermissionTrie.cs`: add `CollectMatches(NodeScope, IReadOnlySet)` overload; optionally add `NodePermissions CollectMask(NodeScope, IReadOnlySet)` that avoids the `List`. +- `TriePermissionEvaluator.Authorize`: pass the cached set / call `CollectMask`. + +**Tests.** Existing authorization tests must pass unchanged (behavior identical). Add a test asserting the same grants resolve via the new set overload as the enumerable overload. A micro-bench is optional (no bench harness in repo). + +**Effort: S. Risk: low** — additive overload; existing path preserved. + +--- + +## 7. P-3 — PollGroupEngine: task-per-subscription, no cross-subscription batching (Medium) + +**Verification: CONFIRMED.** `Subscribe` (`:73-84`) spawns a dedicated `Task.Run(PollLoopAsync)` per subscription with its own interval timer. No interval-bucketed scheduler or read coalescing. + +**Root cause.** The engine was extracted from ModbusDriver as a per-subscription loop; fine at dozens of subscriptions, a scaling wall at hundreds/driver-instance. + +**Proposed design.** This is a **structural fix disproportionate to current need** — the report itself says "Worth a note in the driver-facing docs at minimum." Recommend: +- **v1 (this pass): documentation only.** Add a remarks paragraph to `PollGroupEngine`'s XML doc stating the task-per-subscription cost model and the scaling ceiling, and note it in the driver-authoring doc (`docs/` driver guide) so integrators map poll groups, not per-tag subscriptions. +- **v2 (tracked follow-on, only if tag counts grow):** an interval-bucketed shared scheduler — one loop per distinct clamped interval, reading the union of that bucket's references once and fanning results to each subscription's diff state. This is the real fix and pairs naturally with OVERALL theme #2's "treat PollGroupEngine as the single home for fleet fixes." + +**Implementation steps (v1).** Doc-only edit to `PollGroupEngine.cs` remarks + driver-authoring doc. No code/test change. + +**Effort: S now (doc), L later (bucketed scheduler). Risk: none (doc); high-ish later (touches every poll driver's timing).** + +--- + +## 8. C-2 — Core depends on Configuration; EF entities are the domain model (Medium) + +**Verification: CONFIRMED.** `Core.csproj` references `Configuration` (`:16`). `PermissionTrie.cs:1` consumes `Configuration.Enums`; `EquipmentNodeWalker` consumes `Configuration.Entities`; composer plan records project EF entities. + +**Root cause / posture.** Deliberate, consistent choice — persistence types *are* the model; plan records decouple the applier. The report flags this as **accepted-risk to document, not refactor.** + +**Proposed design.** No refactor. Add an ADR / architecture-note entry documenting: (a) Configuration is the domain model by design; (b) the ripple cost (EF entity changes reach authorization + composition); (c) the mitigation guidance already in practice ("new code keeps preferring the plan-record boundary the composer established"). Cross-link from `CLAUDE.md`'s Architecture Overview. + +**Implementation steps.** Doc-only: add to `docs/` (architecture/decisions) + a one-line note in the composer/PermissionTrie XML docs pointing at it. + +**Effort: S. Risk: none.** + +--- + +## 9. U-1 — Dead/dormant code retained in Core (Medium) + +**Verification: CONFIRMED — all three are test-only:** +- `GenericDriverNodeManager` — only `tests/Core/.../GenericDriverNodeManagerTests.cs` references it (grep: zero production refs). +- `EquipmentNodeWalker` (+ `IdentificationFolderBuilder`) — only `tests/Core/.../OpcUa/EquipmentNodeWalkerTests.cs` + doc-comment mentions; self-declares "retained for unit-test support only." +- `EquipmentScriptPaths.TryParseRelayBody` (`Commons/Types/EquipmentScriptPaths.cs:164`) — only `Commons.Tests/EquipmentScriptPathsTests.cs` calls it (relay→alias converter deleted 2026-06-12). + +**Root cause.** Code retained after its production caller was removed (Phase7→AddressSpace rename retired the walker; Galaxy-standard-driver retired the relay parser; GDNM superseded by composer→applier→sink). + +**Proposed design.** **Delete all three + their test files** (git preserves history). This is OVERALL cross-cutting theme #1's remediation ("dormant code with green tests is worse than deleted code"). Order the deletion to also close **S-5** and part of **C-1**: +- Deleting `EquipmentNodeWalker` removes one of C-1's four `ExtractFullName` copies for free. +- Deleting `GenericDriverNodeManager` makes **S-5** (its non-synchronized re-walk teardown) moot — no separate fix needed. + +**Implementation steps.** +1. Delete `src/Core/.../OpcUa/GenericDriverNodeManager.cs` + `tests/.../GenericDriverNodeManagerTests.cs`. +2. Delete `src/Core/.../OpcUa/EquipmentNodeWalker.cs` (+ `IdentificationFolderBuilder` if co-located and unused) + `tests/.../OpcUa/EquipmentNodeWalkerTests.cs`. Update the doc-comment references in `DeploymentArtifact.cs:666`, `AddressSpaceComposer.cs:287,530`, `OpcUaClientTagConfigModel.cs:11`, `IScriptTagCatalog.cs:56,182`, `OtOpcUaNodeManager.cs:34` to point at `TagConfigIntent.Parse` (they currently cite `EquipmentNodeWalker.ExtractFullName` as the canonical parser — that citation moves to Commons under C-1). +3. Delete `TryParseRelayBody` from `EquipmentScriptPaths.cs:164-172` + its two test cases in `EquipmentScriptPathsTests.cs:215-236`. +4. Verify the `Core/OpcUa/` folder is empty afterward; if so, remove it. + +**Tests.** Removal only — the deleted tests go with the code. Full `dotnet build` + `dotnet test` to confirm no lingering references. + +**Effort: S. Risk: low** (pure deletion; the compiler proves no production caller). Coordinate ordering with C-1 so the doc-comment repointing is consistent. + +--- + +## 10. U-2 — Tier C recycle machinery has no `IDriverSupervisor` implementation (Medium) + +**Verification: CONFIRMED.** `IDriverSupervisor` refs are exactly: the interface (`Core.Abstractions/IDriverSupervisor.cs`), two Stability consumers (`MemoryRecycle.cs`, `ScheduledRecycleScheduler.cs`), and their tests. **Zero concrete implementations.** Out-of-process drivers moved to external mxaccessgw/HistorianGateway sidecars, so no in-repo driver is Tier C — the recycle path is compiled, tested, unreachable. + +**Root cause.** Speculative infrastructure for a tier-migration workflow that the external-gateway architecture superseded. + +**Proposed design.** Two defensible options; recommend **prune** consistent with U-1's philosophy and OVERALL Action #12 ("delete `MemoryRecycle`/`IDriverSupervisor` dead machinery"): +- **Option A (recommended): delete** `MemoryRecycle`, `ScheduledRecycleScheduler`, `IDriverSupervisor`, `DriverResilienceStatusTracker.RecordRecycle` (verify it has no other caller first), and their tests. Removes a fully-built-but-unreachable subsystem. +- **Option B (if there's a real near-term tier-migration plan): keep + document** with an explicit "speculative — no production caller today; wired when in-process Tier C returns" note on each class, plus a tracking issue. The report leans toward pruning ("prune it alongside U-1"). + +**Decision input needed:** confirm with the maintainer that no tier-migration workflow is imminent. Absent that, prune. + +**Implementation steps (Option A).** +1. Grep `RecordRecycle` — if only Stability + tests, remove it from `DriverResilienceStatusTracker`. +2. Delete `MemoryRecycle.cs`, `ScheduledRecycleScheduler.cs`, `IDriverSupervisor.cs`, `WedgeDetector`? (No — `WedgeDetector` is demand-aware stall detection, separate; keep unless it also proves callerless — grep first). +3. Delete `MemoryRecycleTests.cs`, `ScheduledRecycleSchedulerTests.cs`. +4. Note: `MemoryTracking` (median-baseline breach) may feed `MemoryRecycle` — verify `MemoryTracking` has an independent purpose (it likely feeds meters/health) before keeping; keep it if it has other consumers. + +This also closes **S-4** (the ScheduledRecycleScheduler catch-up storm) by deletion. If Option B is chosen instead, apply the S-4 fast-forward fix below. + +**Effort: S. Risk: low** (deletion, compiler-verified). **Blocked on maintainer confirmation** of no imminent tier-migration. + +--- + +## 11. U-3 — Continuous-historization initial ref set still empty at spawn (Medium — known-limitation debt) + +**Verification: CONFIRMED (partially-done, as the report frames it).** +- The **delta feed is implemented**: `AddressSpaceApplier.FeedHistorizedRefs` (`:310-353`) → `IHistorizedTagSubscriptionSink.UpdateHistorizedRefs` → `ActorHistorizedTagSubscriptionSink` → `ContinuousHistorizationRecorder.UpdateHistorizedRefs` (all present). +- The **initial set is still empty**: `Runtime/ServiceCollectionExtensions.cs:272` spawns the recorder with `historizedRefs: Array.Empty()`, with a comment acknowledging it. So after a **process restart** with no subsequent deploy, the recorder historizes nothing (CLAUDE.md KNOWN LIMITATION 2). **Note:** OVERALL theme #5 flags KNOWN LIMITATION 2's *first half* (delta feed) as stale-in-docs but the *restart-convergence* gap this finding names is real. + +**Root cause.** The deployed address space (and thus the historized-ref set) is materialised at deploy time, not at actor-spawn time, so there's no clean set to resolve at wiring time. Only *subsequent* deploys feed deltas. + +**Proposed design.** **Full-set replay on rebuild.** The convergent fix is: whenever the applier does a full `RebuildCalled` rebuild (which re-materialises the entire composition), feed the recorder the **complete current historized-ref set** (an absolute "set to exactly this" message), not just the diff. On a fresh process, the first deploy/rebuild after boot then converges the recorder to the full set even with no prior state; on steady-state edits the delta feed continues to apply. + +Design detail — add a `SetHistorizedRefs(IReadOnlyList full)` absolute-set operation to `IHistorizedTagSubscriptionSink` (the seam contract already supports the fix per the report), and have `AddressSpaceApplier.Apply` call it (instead of / in addition to the delta) **when `rebuilt == true`**, computing the full historized set from `plan`... but `plan` is a *diff*, not the full composition. So the full set must come from the **composition**, which the applier's `Materialise*` methods receive (`MaterialiseEquipmentTags(composition)`). Cleanest: after the rebuild's materialise passes in `OpcUaPublishActor` (`:349`), compute the full historized-ref set from `composition.EquipmentTags` (filter `IsHistorized && Alarm is null`, resolve override-or-FullName exactly as `HistorizedRef` does) and call `SetHistorizedRefs`. Recorder's `OnUpdateHistorizedRefs` already "holds the full set and re-registers it" — add an `OnSetHistorizedRefs` that replaces the set wholesale. + +**Alternatives considered.** +- *Feed the full set from `Apply` using the plan* — rejected: the plan is a diff; a fresh boot's first deploy against `_lastApplied = null` would produce an all-adds plan that *happens* to equal the full set, but that's fragile and breaks for a restart mid-stream where `_lastApplied` was restored. Deriving from `composition` is robust. +- *Persist the recorder's set across restarts* — larger; the FasterLog outbox is for values, not interest. Rebuild-replay is simpler and already aligned with the deploy lifecycle. + +**Implementation steps.** +1. `Core.Abstractions/Historian/IHistorizedTagSubscriptionSink.cs`: add `void SetHistorizedRefs(IReadOnlyList full)`. Update `NullHistorizedTagSubscriptionSink` no-op. +2. `Runtime/Historian/ActorHistorizedTagSubscriptionSink.cs`: forward to a new `ContinuousHistorizationRecorder.SetHistorizedRefs` message. +3. `ContinuousHistorizationRecorder.cs`: `Receive(OnSetHistorizedRefs)` — replace `_historizedRefs` wholesale and re-register dependency-mux interest. +4. `OpcUaPublishActor.cs` (after `:354`, when a rebuild ran): derive the full historized set from `composition` and call the sink's `SetHistorizedRefs`. (Or add an applier method `FeedFullHistorizedRefs(composition)` symmetric with `FeedHistorizedRefs(plan)`.) +5. Update CLAUDE.md KNOWN LIMITATION 2 to reflect the closed restart-convergence gap (coordinate with OVERALL Action #11). + +**Tests.** +- `Runtime.Tests`: recorder converges to the full set on `SetHistorizedRefs` (registers interest in exactly the historized tags), and a fresh-spawn → rebuild → `SetHistorizedRefs` path historizes the full set. +- Restart-convergence integration test (the one OVERALL theme #5 says the closure "deserves"): spawn recorder empty, apply a composition with historized tags, assert values flow without a prior delta. + +**Effort: M. Risk: medium** — touches the historization actor + the seam contract; live value-capture is still gated by the separate ContinuousHistorization live-validation (CLAUDE.md), so verify offline via unit/actor tests plus the env-gated live suite when a gateway is reachable. + +--- + +## 12. P-4 — Per-write allocation in the non-idempotent write path (Low) + +**Verification: CONFIRMED.** `CapabilityInvoker.ExecuteWriteAsync` (`:135-152`) builds `noRetryOptions = snapshot with { CapabilityPolicies = new Dictionary<...>{...} }` per non-idempotent write, then `GetOrCreate(id, "{host}::non-idempotent", Write, noRetryOptions)` — and `GetOrCreate` keys only on `(id, host, capability)`, ignoring the options after first build (comment cites the "1% pipeline budget"). + +**Root cause.** The no-retry options snapshot is rebuilt every call even though the resolved pipeline is cached and the options are only consumed on first build. + +**Proposed design.** Since the cache key already ignores the options after first build, the allocation is pure waste on the hot path. Fastest fix: **only build `noRetryOptions` on a cache miss.** Add a `GetOrCreate` overload that takes an options *factory* (`Func`) invoked lazily only when the pipeline isn't cached. Then `ExecuteWriteAsync` passes a factory closing over `snapshot`; the closure (small) is built but the record+Dictionary only allocate on miss. + +**Alternatives considered.** +- *Cache the no-retry pipeline separately keyed on options generation* — more state; the factory-on-miss approach reuses the existing cache and is minimal. +- *Precompute a per-host non-idempotent pipeline at options-change time* — most efficient but requires an options-change hook; overkill for a Low. + +**Implementation steps.** +- `DriverResiliencePipelineBuilder.cs`: add `GetOrCreate(id, host, capability, Func optionsFactory)` overload that only calls the factory on a `ConcurrentDictionary` miss (use `GetOrAdd` with the factory). +- `CapabilityInvoker.ExecuteWriteAsync`: still snapshot `_optionsAccessor()` once (needed for correctness per the existing comment), but move the `with { ... }` construction into the factory lambda so it only runs on miss. (Minor: the snapshot itself is cheap; the Dictionary is the cost.) + +**Tests.** `Core.Tests` resilience: assert a non-idempotent write on a warm cache does not rebuild options (spy the factory invocation count == 0 on the second call), and retries are still disabled. + +**Effort: S. Risk: low.** + +--- + +## 13. S-3 — `Subscribe`/`RegisterAsync` have no disposed guard (Low) + +**Verification: CONFIRMED.** +- `PollGroupEngine.Subscribe` (`:73-84`) inserts into `_subscriptions` with no `_disposed` check; `DisposeAsync` (`:213-236`) snapshots `.Values` then clears — a `Subscribe` winning the race after the snapshot leaks a live loop. +- `DriverHost.RegisterAsync` (`:53-71`) vs `DisposeAsync` (`:92-106`): a registration after the snapshot+clear is initialized but never shut down. + +**Root cause.** Shutdown-window races; no disposed flag. + +**Proposed design.** Add a `volatile bool _disposed` to each, set at the top of `DisposeAsync` (inside the lock for `DriverHost`), and check it before insert: +- `PollGroupEngine.Subscribe`: if `_disposed`, either throw `ObjectDisposedException` (strict) or return a no-op handle. Given callers may race legitimately during teardown, prefer throwing `ObjectDisposedException` (the SDK convention) so callers don't silently believe they subscribed. +- `DriverHost.RegisterAsync`: check `_disposed` inside the existing `_lock` before adding; throw `ObjectDisposedException` if set. + +Note `DisposeAsync` for `PollGroupEngine` isn't lock-guarded (it's `ConcurrentDictionary`-based); set `_disposed = true` first, then the snapshot loop, and have `Subscribe` check `_disposed` *after* `Interlocked.Increment` but *before* `_subscriptions[id] = state` — and if disposed-after-insert, remove+stop. Simplest robust form: check `_disposed` before starting the loop; if a dispose is concurrent, the double-check pattern (insert, then re-check `_disposed`, and if set, `TryRemove` + cancel) closes the window. + +**Implementation steps.** +- `PollGroupEngine.cs`: add `_disposed`; guard `Subscribe` with the insert-then-recheck pattern; set flag in `DisposeAsync`. +- `DriverHost.cs`: add `_disposed`; guard `RegisterAsync` inside `_lock`; set flag in `DisposeAsync` inside `_lock`. + +**Tests (Core.Tests / Core.Abstractions.Tests).** +- `Subscribe` after `DisposeAsync` throws `ObjectDisposedException` and leaves no live loop (assert `ActiveSubscriptionCount == 0`). +- `RegisterAsync` after `DisposeAsync` throws and doesn't leave an initialized-but-unshut driver. + +**Effort: S. Risk: low.** + +--- + +## 14. S-4 — ScheduledRecycleScheduler catch-up recycle storm after a stall (Low) + +**Verification: CONFIRMED.** `TickAsync` (`:72-84`) advances `_nextRecycleUtc += _recycleInterval` (`:82`) by exactly one interval per fire. K missed intervals → K back-to-back recycles. + +**Root cause.** Fixed-increment scheduling with no fast-forward past `utcNow`. + +**Proposed design.** **Depends on U-2's decision.** If U-2 prunes this subsystem (recommended), S-4 disappears with it — no work. If U-2 keeps it, apply the report's fix: after a fire, fast-forward `_nextRecycleUtc` past `utcNow` in whole intervals: +```csharp +// after RecycleAsync: +do { _nextRecycleUtc += _recycleInterval; } while (_nextRecycleUtc <= utcNow); +``` +(or compute the number of elapsed intervals arithmetically). Fires exactly once for any K-interval gap. + +**Tests.** `ScheduledRecycleSchedulerTests`: feed a `utcNow` K intervals past `_nextRecycleUtc`; assert `RecycleAsync` called exactly once and `NextRecycleUtc > utcNow`. + +**Effort: S. Risk: low. Gated on U-2 (likely deleted).** + +--- + +## 15. S-5 — GDNM re-walk teardown not synchronized (Low — moot) + +**Verification: CONFIRMED but moot.** `GenericDriverNodeManager.BuildAddressSpaceAsync` (`:58-71`) unsubscribes `_alarmForwarder` then `_alarmSinks.Clear()` non-atomically; a concurrent alarm event in the window is dropped. **But GDNM has no production caller (U-1).** + +**Proposed design.** **Fold into U-1 — delete GDNM.** No standalone fix. If (against the recommendation) GDNM is kept, guard the teardown+re-subscribe under a lock. + +**Effort: 0 (subsumed by U-1).** + +--- + +## 16. C-4 — Composer violates its own purity contract with `Trace.TraceWarning` (Low) + +**Verification: CONFIRMED.** Class doc says "Same inputs → same outputs, no logging" (`AddressSpaceComposer.cs:281`); the dangling-predicate-script skip emits `Trace.TraceWarning` (`:493-496`) — the only `System.Diagnostics.Trace` use in a Serilog codebase, invisible in production sinks. + +**Root cause.** A warning was needed at a skip site in a static pure method with no logger; `Trace` was the path of least resistance. + +**Proposed design.** **Return skipped-alarm ids in the composition** for the caller (which has an `ILogger`) to log. Add `IReadOnlyList SkippedAlarmIds` (or a richer `SkippedAlarm(id, equipmentId, predicateScriptId)` record list) to `AddressSpaceComposition`; the composer collects skips instead of tracing; `OpcUaPublishActor` (which already logs the apply outcome) logs them at `Warning`. This preserves purity (same inputs → same outputs, including the skip list) and makes the warning visible. + +**Alternatives considered.** +- *Inject an `ILogger`* — rejected: breaks the pure-static contract the byte-parity design leans on (and the artifact-decode mirror stays logger-free). +- *Just delete the trace* — rejected: loses a genuine operator signal (a dangling predicate reference that the draft validator *should* have caught but is the last line of defense). + +**Implementation steps.** +- `AddressSpaceComposition`: add `SkippedAlarmIds` init-only member (default empty). +- `AddressSpaceComposer.cs:486-516`: collect skips into a list, drop the `Trace.TraceWarning`, set the member on return. +- `OpcUaPublishActor`: after `ParseComposition`/compose, if `SkippedAlarmIds` non-empty, `_log.Warning(...)`. (Note: the actor consumes `DeploymentArtifact.ParseComposition`, not the composer directly — so the artifact-decode mirror must also carry/emit the skip list, or the skip must be surfaced at compose time on the publish side. Verify which seam the runtime uses and thread the skip list through it.) +- Fix the class doc if any residual logging remains (it won't). + +**Tests.** `OpcUaServer.Tests`: compose with a scripted alarm whose predicate script is absent → assert the alarm is skipped AND its id appears in `SkippedAlarmIds`. + +**Effort: S. Risk: low.** + +--- + +## 17. C-5 — Inconsistent duplicate-key defensiveness in Compose (Low) + +**Verification: CONFIRMED.** `deviceHostById` uses a deliberate last-wins `foreach` with a comment (`AddressSpaceComposer.cs:359-364`), but `driversById`/`namespacesById` (`:401-402`) and `scriptsById` (`:453`) use plain `ToDictionary` — a duplicate logical id (only on DB-constraint bypass) throws and crashes the whole compose. + +**Root cause.** Copy-paste divergence; the defensive posture wasn't applied uniformly. + +**Proposed design.** **Adopt the defensive posture uniformly** (the report: "the defensive one is already argued for"). Replace the three `ToDictionary` calls with last-wins `foreach` builds (or a shared `static Dictionary LastWins(IEnumerable, Func)` helper in the composer). Rationale: a DB-constraint bypass should degrade gracefully (last-wins, matching the decode side) rather than crash the whole compose — consistent with the byte-parity contract the `deviceHostById` comment already defends. + +**Alternatives considered.** *Make `deviceHostById` throw too (fail-fast everywhere)* — rejected: diverges from the artifact-decode side's last-wins, breaking byte-parity, which the existing comment explicitly warns against. + +**Implementation steps.** +- Add a private `LastWins` helper; replace `driversById`, `namespacesById`, `scriptsById` construction. Confirm the decode side (`DeploymentArtifact`) uses last-wins for the same maps and matches. + +**Tests.** `OpcUaServer.Tests`: compose with a duplicate `DriverInstanceId` (or `ScriptId`) → assert no throw and last-wins selection, matching decode. + +**Effort: S. Risk: low.** + +--- + +## 18. C-6 — Vestigial Akka package reference in Commons (Low) + +**Verification: CONFIRMED.** `Commons.csproj:9` references `Akka`; the only "Akka" token in Commons source is a **doc comment** in `NodeDiagnosticsSnapshot.cs:15` ("over Akka"). No Akka type is used. + +**Root cause.** Leftover from an earlier design; harmless today (every consumer is Akka-hosted) but contradicts Commons' dependency-light contracts role. + +**Proposed design.** **Remove the `` from `Commons.csproj`.** Build to confirm nothing breaks (grep already proves no type usage). + +**Implementation steps.** Delete the line; `dotnet restore`/`build`. If CPM (`Directory.Packages.props`) versions it, leave the version entry (other projects may use it) but drop the Commons reference. + +**Tests.** Build + full test run. + +**Effort: S. Risk: low.** + +--- + +## 19. C-7 — Stale scaffold-era XML docs on load-bearing classes (Low) + +**Verification: CONFIRMED.** +- `AddressSpaceApplier.cs:7-26` class doc still describes the F14 scaffold ("For now we record the work", "the SDK adapter that lands in F10b will decide…") though both shipped (the class *does* the surgical-vs-rebuild decision now). +- `DriverHost.cs:5-10` doc promises "per-process isolation for Tier C … implemented in Phase 2 via named-pipe RPC" — superseded by the out-of-repo mxaccessgw gateway. + +**Root cause.** Docs not updated when the features shipped / architecture changed. + +**Proposed design.** Rewrite both class docs to describe current reality. For `AddressSpaceApplier`: describe the actual full-rebuild-vs-surgical decision, the two detached hooks, and the SDK-free sink boundary. For `DriverHost`: describe it as the in-process id→`IDriver` lifecycle registry; drop the named-pipe/Tier C narrative (or add a one-line "out-of-process drivers now live in external gateways" pointer). Coordinate with the `fixdocs`/`checkdocs` skill conventions already used in this repo (recent commit `9cad9ed0` was an XML-doc sweep). + +**Implementation steps.** Edit the two class-doc blocks. While there, if U-2 keeps `DriverHost`'s Tier C references anywhere, align them. + +**Tests.** None (doc). Optionally run `checkdocs`/CommentChecker. + +**Effort: S. Risk: none.** + +--- + +## 20. U-4 — Additive-only ACL model (no Deny) — documented v2.0 gap (Low) + +**Verification: CONFIRMED.** `PermissionTrie.cs:13-17` explicitly states "pure union (additive grants; no explicit Deny in v2.0)." A broad cluster-root grant can't be carved back at a sub-scope. + +**Root cause.** Deliberate v2.0 simplification aligned with flat, global AdminUI roles. + +**Proposed design.** **No code change — document the boundary in the security docs.** Add a "Authorization model: additive-only, no Deny" subsection to `docs/security.md` stating the bound (fine-grained OT authz needs a trie-walk semantics change to add Deny/most-specific-wins) so the limitation is visible to deployment planners. If/when Deny is needed, it's a scoped follow-on: add a `Deny` flag dimension to `TrieGrant` and change `CollectMatches` from pure-OR to a precedence walk (most-specific-scope Deny wins) — out of scope here. + +**Implementation steps.** Doc-only edit to `docs/security.md`. + +**Effort: S. Risk: none.** + +--- + +## 21. U-5 — Uneven test coverage; ClusterRoleInfo untested (Low) + +**Verification: CONFIRMED.** `Cluster.Tests` contains only `RoleParserTests`, `HoconLoaderTests`, `ServiceLevelCalculatorTests` (the three pure classes). `ClusterRoleInfo` — the only concurrency-bearing class in Cluster (lock + subscriber actor + event fan-out) — has **no dedicated unit test** (it appears only in Server-level integration harnesses: `RedundancyStateActorTests`, `MultiClusterScopingTests`, `TwoNodeClusterHarness`, `ServiceCollectionExtensionsTests`). The applier failure surface (S-1) and `DriverHost`/`PollGroupEngine` dispose races (S-3) are untested because those surfaces don't exist yet. + +**Root cause.** Coverage concentrated on pure/deterministic classes; the concurrency-bearing seam and fault surfaces were left to live-verification / integration. + +**Proposed design.** Add an **Akka TestKit harness for `ClusterRoleInfo`** exercising leader-change sequencing — the report's "riskiest untested seam." Cover: role topology snapshot updates under the lock, event fan-out on membership/leader change, and no-lost-update under concurrent snapshot reads during a leader change. The S-1 and S-3 tests are already specified under their own findings (they land as those surfaces are built), which is the bulk of closing U-5's named gaps. + +**Implementation steps.** +- `tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests/ClusterRoleInfoTests.cs` using Akka.TestKit (verify TestKit is already a test dep — `Host.IntegrationTests` uses cluster harnesses, so it's available): drive a fake cluster-event stream through the subscriber actor and assert `ClusterRoleInfo`'s snapshot + fan-out. + +**Tests.** This finding *is* a test-addition; no product code changes. + +**Effort: M. Risk: low** (test-only; the risk is flaky async assertions — use TestKit's deterministic probes). + +--- + +## Cross-finding sequencing (recommended PR order) + +1. **PR A (C-1 + P-1 + C-3):** land `TagConfigIntent.Parse` + `DeviceConfigIntent` in Commons; add `Configuration → Commons` reference; retrofit composer, DeploymentArtifact, DraftValidator; add shared driver-type constant + `EquipmentNodeIds`-keyed collision check. Behind the byte-parity test suite. *Highest-leverage; unblocks C-3 and the U-1 doc-repointing.* +2. **PR B (U-1 + S-5):** delete GDNM, EquipmentNodeWalker, TryParseRelayBody + tests; repoint doc comments at Commons parser (depends on PR A). +3. **PR C (S-1):** applier failure surface + publish-actor signal. +4. **PR D (U-2 decision):** prune Tier C recycle (closes S-4) — *gated on maintainer confirmation*; else apply S-4 fast-forward. +5. **PR E (U-3):** full-set replay on rebuild + CLAUDE.md KNOWN LIMITATION 2 update. +6. **PR F (S-2 + S-3):** PollGroupEngine `UnsubscribeAsync` + disposed guards + DriverHost guard. +7. **PR G (P-2 + P-4):** authorization set caching + non-idempotent-write factory-on-miss. +8. **PR H (Lows/docs):** C-4, C-5, C-6, C-7, C-2 note, U-4 doc, P-3 doc, U-5 ClusterRoleInfo tests. + +--- + +## Notes for the executor + +- **Byte-parity is the crown jewel.** Any change to composer/DeploymentArtifact/DraftValidator parsing (C-1/C-3/C-5) must keep the existing cross-seam parity test green; port semantics verbatim, no behavior change intended. +- **`Commons` is a verified pure leaf** — moving shared code there is cycle-safe and the correct home per the `EquipmentScriptPaths` precedent. +- **Deletions (U-1/U-2) are compiler-verified** — no production caller exists for GDNM, EquipmentNodeWalker, TryParseRelayBody, or any `IDriverSupervisor` impl. Git preserves history. +- **U-2 and the U-2-dependent S-4 need one maintainer decision** (prune vs. keep-and-document the Tier C recycle machinery). Everything else is unblocked. +- Live-verify surfaces in this domain are thin (compose/authorize/teardown are unit-testable); the only item needing a real gateway is U-3's continuous-historization value capture, which stays behind the existing env-gated live suite. diff --git a/archreview/plans/02-scripting-alarms-plan.md b/archreview/plans/02-scripting-alarms-plan.md new file mode 100644 index 00000000..b5a5ba2b --- /dev/null +++ b/archreview/plans/02-scripting-alarms-plan.md @@ -0,0 +1,362 @@ +# Design + Implementation Plan — 02 Scripting, Virtual Tags, Scripted Alarms, Alarm Historian + +- **Source report:** `archreview/02-scripting-alarms.md` +- **Review commit:** `9cad9ed0` · **Plan verified against tree at:** `9cad9ed0` (master) +- **Scope:** `Core.Scripting`(+`Abstractions`), `Core.VirtualTags`, `Core.ScriptedAlarms`, `Core.AlarmHistorian`, plus the live consumers in `Host`/`Runtime`. + +## Verification summary + +Every finding in the report was opened at the cited file:line and checked against the current tree. **All findings CONFIRMED — none stale.** Line numbers in the report match the current source. Key confirmations: + +- **U2** — `RoslynVirtualTagEvaluator.Evaluate` (`Host/Engines/RoslynVirtualTagEvaluator.cs:101-108`) builds a `CancellationTokenSource(_runTimeout)` and calls `evaluator.RunAsync(context, cts.Token).GetAwaiter().GetResult()`. `ScriptEvaluator.RunAsync` (`Core.Scripting/ScriptEvaluator.cs:178-190`) runs the compiled delegate **synchronously** (`var result = _func(globals);` line 188) and only checks the token at entry (line 182). The CTS cannot interrupt a running script; the `catch (OperationCanceledException)` branch (line 105-108) is dead. `VirtualTagActor.OnDependencyChanged` (`Runtime/VirtualTags/VirtualTagActor.cs:113`) calls `_evaluator.Evaluate(...)` **inline in the actor's message handler** → a `while(true)` script hangs that actor's message loop forever. Registered live in `Host/Program.cs:217-221`. **Confirmed.** +- **U3** — same file: `_cache` is a raw `ConcurrentDictionary>` keyed by expression source (`RoslynVirtualTagEvaluator.cs:25-26`), populated via `GetOrAdd(expression, …Compile)` (line 67), only ever cleared in `Dispose()` (line 132-141). No apply-boundary eviction → cross-publish ALC accretion. `GetOrAdd` value-factory can double-compile under a race and leak the losing ALC. **Confirmed.** +- **P1** — `ScriptEvaluator.Compile` calls `ScriptSandbox.Build(typeof(TContext))` (line 74) on every compile; `Build` runs `MetadataReference.CreateFromFile` for every pinned assembly + every `System.*`/netstandard TPA path (`ScriptSandbox.cs:83-87`, `EnumerateBclAssemblyPaths` 100-130). Immutable per `contextType.Assembly`, rebuilt every call. **Confirmed.** +- **U1** — `new VirtualTagEngine`, `TimerTriggerScheduler`, `new VirtualTagSource` appear only under `tests/Core/…Core.VirtualTags.Tests`. No production instantiation. `DependencyGraph` (Tarjan/Kahn) has **no production consumer** — repo grep for topo/cycle in the live path returns only unrelated matches (`MemoryRecycle`, resilience). Live cascade fan-out is `DependencyMuxActor` (no topo sort; cross-tag writes dropped). **Confirmed.** +- **S1–S6, S8, S10, S11, P2–P6, C1, C3, C7, U4, U5** — all confirmed at cited lines (details in each section below). + +--- + +## Priority ordering + +Ordered by the report's Priority Recommendations and the OVERALL prioritized action list (item #2 = U2/U3): + +1. **U2** (Critical) — production VT script timeout ineffective +2. **U3** (High) — live path bypasses `CompiledScriptCache`; ALC accretion +3. **P1** (High) — `ScriptSandbox.Build` rebuilds full BCL ref set every compile +4. **U1** (High) — dormant `Core.VirtualTags` engine stack +5. **S2** (High) — no coalescing/backpressure on upstream fan-out +6. **S1** (High) — `VirtualTagEngine.Load` un-gated (resolves with U1) +7. **S5 + C7** (Medium) — timed-shelve expiry swallows `Unshelved` + doc drift +8. **S3 / S4 / S6** (Medium) — reload-swap, `_alarmsReferencing` sync, sink dispose-drain (one hardening pass) +9. **U4 / U5 / C1 / C2 / C5** (Medium) — Null Historize, Part 9 conformance doc, dup interface, namespace doc, contract split +10. **Lows** (batched) — S7, S8, S9, S10, S11, P2–P6, C3, C4, C6, U6, U7 + +--- + +## 1. U2 — CRITICAL — Production virtual-tag script timeout is ineffective + +**Restatement:** `RoslynVirtualTagEvaluator` runs scripts on the calling (actor) thread with a token that can never interrupt them; a CPU-bound/infinite-loop virtual-tag script hangs the owning `VirtualTagActor` forever — no timeout, no log, no recovery. + +**Verification:** Confirmed (see summary). The dead code is `RoslynVirtualTagEvaluator.cs:105-108`. Root cause: `ScriptEvaluator.RunAsync` is synchronous by design (line 184-189 documents "TimedScriptEvaluator wraps this in Task.Run…"), but the Host adapter never wraps it — it hand-rolls a `CancellationTokenSource` that a synchronous `_func(globals)` invocation ignores. + +**Root cause:** The one component that evaluates operator-authored VT scripts in production (`RoslynVirtualTagEvaluator`) reimplemented timeout handling incorrectly instead of reusing `TimedScriptEvaluator`, which was built for exactly this and is already used by the live alarm engine (`ScriptedAlarmEngine.cs:227`) and the dormant VT engine (`VirtualTagEngine.cs:117`). + +**Proposed design:** Route the adapter's execution through `TimedScriptEvaluator`, accepting the documented orphan-thread trade-off (S7) — this is the sanctioned mechanism (`Task.Run` + `WaitAsync`) that makes the wall-clock budget real for CPU-bound scripts. Alternatives considered: +- *Inline `Task.Run` + `WaitAsync` in `Evaluate`* — duplicates `TimedScriptEvaluator` logic (the exact anti-pattern U1/theme-#2 warns against). Rejected in favor of reuse. +- *Out-of-process runner* — the real fix for CPU budgeting but a v3 concern (per `TimedScriptEvaluator` docs); out of scope. + +Because `TimedScriptEvaluator` wraps a *single* `ScriptEvaluator`, and the adapter caches evaluators per source, the wrapper is constructed per compiled entry. Combine with U3: the cache value should become the wrapped/timed evaluator (or a small record holding both). Simplest coherent shape — cache the `ScriptEvaluator` (from `CompiledScriptCache`, U3) and construct a `TimedScriptEvaluator` per call (cheap — it's a thin struct-like wrapper over the inner evaluator; the inner compiled delegate is what's expensive and that stays cached). + +**Implementation steps:** +1. In `RoslynVirtualTagEvaluator.Evaluate` (`Host/Engines/RoslynVirtualTagEvaluator.cs`), replace the block at lines 99-113: + - Construct `var timed = new TimedScriptEvaluator(evaluator, _runTimeout);` + - Call `var raw = timed.RunAsync(context).GetAwaiter().GetResult();` (no CTS needed — the wall-clock budget lives in `TimedScriptEvaluator`). + - Change the catch to `catch (ScriptTimeoutException) { return VirtualTagEvalResult.Failure($"script timed out after {_runTimeout.TotalSeconds:F1}s"); }` (keep the generic `catch (Exception)` for user-code faults). +2. Keep `_runTimeout` default 2 s (existing). Optionally expose per-tag timeout later (out of scope). +3. Note the orphan-thread interaction with S7: a hot looping script now orphans one pool thread per evaluation attempt. Track the S7 circuit-breaker as a fast follow (below) — but U2's fix (stop hanging the *actor*) is strictly better than today regardless. + +**Tests (unit + regression):** +- Add to the existing `tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/RoslynVirtualTagEvaluatorTests.cs`: + - `Evaluate_WithInfiniteLoopScript_ReturnsFailureWithinTimeout` — expression `while(true){} return 0;` (or a spin that exceeds the budget); assert `result.Success == false`, `result.Reason` contains "timed out", and the call returns within ~timeout + slack (wrap in its own `Task.Run` + `WaitAsync(5s)` so a regression *fails the test* instead of hanging the suite). + - `Evaluate_WithFastScript_StillReturnsValue` — regression that the wrapping didn't break the happy path or the passthrough fast-path. +- Optionally an actor-level test in `Runtime.Tests` proving `VirtualTagActor` continues processing messages after a timed-out evaluation (message loop not wedged). + +**Effort:** S. **Risk/blast-radius:** Low-Medium. Touches the single live VT evaluation path; the happy path is unchanged (passthrough fast-path still short-circuits before compile). The behavioral change is that a runaway script now returns Bad instead of hanging — strictly safer. Combine the edit with U3 (same method). + +--- + +## 2. U3 — HIGH — Live path bypasses `CompiledScriptCache`: unbounded ALC accretion + +**Restatement:** `RoslynVirtualTagEvaluator` caches compiled evaluators in a raw `ConcurrentDictionary` never evicted on republish; every edited-script publish roots a new collectible `AssemblyLoadContext` forever, and the `GetOrAdd` factory can double-compile under a race and leak the loser. + +**Verification:** Confirmed (`RoslynVirtualTagEvaluator.cs:25-26, 67, 132-141`). This is the same leak `CompiledScriptCache.Clear()` was built to fix (`CompiledScriptCache.cs:34-40, 87-118`) and that both engines already route through (`ScriptedAlarmEngine.cs:66-77`, `VirtualTagEngine.cs:26-38`). The adapter is the lone hold-out. + +**Root cause:** Adapter reimplemented caching with a plain dictionary + value-factory `GetOrAdd` instead of the purpose-built `CompiledScriptCache` (`Lazy` + `ExecutionAndPublication` single-compile, dispose-on-`Clear`). No apply-boundary `Clear()` is called because the adapter doesn't see config-apply — but the host actor does. + +**Proposed design:** Swap the raw dictionary for `CompiledScriptCache` and give the adapter a `Clear()`/`OnConfigApply()` seam the host actor invokes at each deploy boundary. Two wiring options for the apply-boundary clear: +- *(a) Adapter exposes `void ClearCompiledScripts()`; `VirtualTagHostActor` calls it when it (re)loads a plan generation.* Preferred — mirrors how `ScriptedAlarmHostActor` drives `ScriptedAlarmEngine.LoadAsync` at apply, and how the alarm engine's `_compileCache.Clear()` runs per generation. +- *(b) `IVirtualTagEvaluator` gains an optional `Clear` member.* Broader surface change; the interface has a `NullVirtualTagEvaluator` and other consumers — only do this if the host actor can't reach the concrete type. Given `Host/Program.cs:217-221` registers the concrete `RoslynVirtualTagEvaluator` as a singleton AND as `IVirtualTagEvaluator`, the host actor can resolve the concrete singleton; prefer (a) but if the actor only has the interface, add a narrow `IScriptCacheOwner { void ClearCompiledScripts(); }` the adapter implements and the actor optionally casts to (theme #1: assert the wiring with a test). + +**Implementation steps:** +1. In `RoslynVirtualTagEvaluator`: replace `_cache` with `private readonly CompiledScriptCache _cache = new();` +2. Replace the `GetOrAdd` at line 67 with `_cache.GetOrCompile(expression)` inside the same try/catch (it throws the same `CompilationErrorException`/`ScriptSandboxViolationException`). +3. Add `public void ClearCompiledScripts() => _cache.Clear();` +4. `Dispose()` → `_cache.Dispose()` (drops the per-evaluator dispose loop; `CompiledScriptCache.Dispose` already disposes every materialised ALC). +5. Wire the apply-boundary clear: in `Runtime/VirtualTags/VirtualTagHostActor.cs`, at the point it receives a new plan generation / rebuilds its child `VirtualTagActor` set, call `ClearCompiledScripts()` on the injected evaluator (resolve the concrete type, or via the narrow interface from option (b)). Verify the host actor already has an apply/rebuild message it handles — if it recreates children per generation, that handler is the hook. +6. Confirm `CompiledScriptCache.GetOrCompile`'s `Lazy` single-compile removes the double-compile race automatically. + +**Tests:** +- Unit (`Host.IntegrationTests` or a new `RoslynVirtualTagEvaluator` unit test): compile source A, then call `ClearCompiledScripts()`, assert the prior evaluator's ALC is reclaimed via `WeakReference` + `GC.Collect()` (mirror `CompiledScriptCacheTests` ALC-reclaim test at `tests/Core/…Core.Scripting.Tests/CompiledScriptCacheTests.cs`). +- Wiring assertion (theme #1): a `VirtualTagHostActor` test proving a config-apply/rebuild triggers `ClearCompiledScripts` (spy evaluator or count cache entries before/after). + +**Effort:** S. **Risk:** Low. Behavior-preserving except that stale ALCs are now released. Do in the same PR as U2 (same file). Blast radius = VT evaluation + one host-actor hook. + +--- + +## 3. P1 — HIGH — `ScriptSandbox.Build` rebuilds the full BCL reference set on every compile + +**Restatement:** Each compile re-reads 100+ `System.*`/netstandard DLLs into fresh `MetadataReference`s; a 200-script publish pays this ~200×, and the AdminUI ScriptAnalysis endpoints pay it per keystroke-cadence request. + +**Verification:** Confirmed (`ScriptSandbox.cs:47, 83-87, 100-130`; called from `ScriptEvaluator.cs:74`). The reference list is a pure function of `contextType.Assembly` (the only per-call input) and the process TPA set — immutable for the process lifetime. + +**Root cause:** No memoization; `MetadataReference.CreateFromFile` (which loads + parses `AssemblyMetadata`) runs unconditionally per compile. + +**Proposed design:** Memoize the built `SandboxConfig` (references + imports) in a `static readonly ConcurrentDictionary` keyed by `contextType.Assembly`. The pinned OtOpcUa assemblies and the BCL path set are stable per key. `MetadataReference` instances are immutable and thread-safe to share across compilations (Roslyn is designed for this). Alternative: cache just the BCL `List` (process-global, key-independent) and rebuild only the 4 pinned entries per key — marginal extra win, more code. Prefer caching the whole `SandboxConfig` per `contextType.Assembly` (there are only ~3 distinct context assemblies in practice: VirtualTags, ScriptedAlarms, test contexts). + +**Implementation steps:** +1. In `ScriptSandbox`, add `private static readonly ConcurrentDictionary _cache = new();` +2. Change `Build` body to `return _cache.GetOrAdd(contextType.Assembly, static asm => BuildUncached(asm));` — but note `Build` currently takes `contextType` and validates `ScriptContext`-assignability. Keep the validation in `Build` (before the cache lookup, since it's cheap and guards the contract), then key the cache on `contextType.Assembly`. Move the reference-list construction (lines 54-97) into a private `BuildUncached(Type contextType)` / or key on assembly and pin `typeof(DataValueSnapshot).Assembly` etc. which are constant. + - Caveat: the four pinned assemblies at lines 59-70 include `contextType.Assembly` itself — keying on `contextType.Assembly` keeps them correct. +3. No API change to callers; `SandboxConfig` is already an immutable record. + +**Tests:** +- `ScriptSandboxTests` (`tests/Core/…Core.Scripting.Tests/ScriptSandboxTests.cs`): assert two `Build(sameContextType)` calls return the **same** `SandboxConfig` reference (memoized), and `Build` for two different context types returns configs whose reference sets both resolve `ctx.GetTag`. Assert the existing escape-catalog tests still pass (they exercise the analyzer through `Compile`, which will now hit the cache). +- Optional micro-benchmark note in the PR: compile-latency before/after (report predicts 10-100×). + +**Effort:** S. **Risk:** Low. `MetadataReference` sharing across compilations is a supported Roslyn pattern. The only subtlety is the `contextType.Assembly` key correctly capturing the pinned-set variation. Blast radius = all script compilation (VT, alarm, ScriptAnalysis) — but purely a perf/allocation change; output is byte-identical. + +--- + +## 4. U1 — HIGH — The entire `Core.VirtualTags` engine stack is dormant in production + +**Restatement:** `VirtualTagEngine` (568 lines), `TimerTriggerScheduler` (162), `VirtualTagSource` (106), `DependencyGraph` (324) have no production instantiation; the live path (`VirtualTagHostActor` + `DependencyMuxActor` + `RoslynVirtualTagEvaluator`) reimplements the same concerns. ~1,160 engine + ~1,260 test lines maintained for dead code that carries latent bugs (S1, S3, P3). + +**Verification:** Confirmed — `new VirtualTagEngine`/`TimerTriggerScheduler`/`new VirtualTagSource` only in `tests/Core/…Core.VirtualTags.Tests`. `DependencyGraph` has no production consumer (grep for Tarjan/topo/cycle in the live path finds only unrelated `MemoryRecycle`/resilience). `RoslynVirtualTagEvaluator`'s own xmldoc acknowledges the split (lines 20-21). + +**Root cause:** The Akka actor pipeline (F8b) superseded the embedded engine, but the engine was never retired — a "built-but-never-wired" instance (OVERALL theme #1). + +**Proposed design — RETIRE (report's preferred option (a)).** The live actor pipeline is the single sanctioned implementation. Once U2/U3 land, the actor path has the timeout + cache defenses the engine held, so nothing of value is lost. Retire: +- `VirtualTagEngine.cs`, `TimerTriggerScheduler.cs`, `VirtualTagSource.cs`, `DependencyGraph.cs` (+ `DependencyCycleException`) and their test suites (`VirtualTagEngineTests`, `TimerTriggerSchedulerTests`, `VirtualTagSourceTests`, `DependencyGraphTests`). +- **Keep** in `Core.VirtualTags`: `ITagUpstreamSource` (subject of C1 — hoist, don't delete), `VirtualTagDefinition`, `IHistoryWriter`/`NullHistoryWriter` (U4), and anything the composer/actor path references. **Verify before deleting** each type has no non-test consumer (grep each; `VirtualTagContext` lives in Abstractions and is consumed live, per C2). + +*Alternative — sanction the engine as an embedded/non-Akka path and fix S1/S3/P3.* Rejected: doubles the maintenance surface with no consumer, and there is no roadmap need for a non-Akka embedded runtime. If one ever appears, the git history preserves the engine. + +Retiring the engine **subsumes S1, S3, P3** (they are latent bugs in the deleted code) and removes half of C1/C3's duplication pressure. + +**Implementation steps:** +1. Grep each candidate type for non-test references (`VirtualTagEngine`, `TimerTriggerScheduler`, `VirtualTagSource`, `DependencyGraph`, `DependencyCycleException`). Confirm zero `src/` consumers (done for the four engine types; re-verify at delete time). +2. Delete the four source files + their four test files; remove from any `.csproj`/slnx include if explicitly listed (they use globbing — likely no edit needed). +3. If `ITagUpstreamSource` is being hoisted (C1), do that first so the delete doesn't strand the interface. +4. Update `docs/VirtualTags.md` (referenced by `VirtualTagEngine` xmldoc) to describe only the live actor pipeline; drop engine-specific sections. +5. Build + run the full `Core.VirtualTags.Tests` (now trimmed) and the `Runtime`/`Host` suites. + +**Tests:** No new tests; deletion. Confirm the remaining `Core.VirtualTags.Tests` (definition/`IHistoryWriter`/`VirtualTagSource`-adjacent) still build. Confirm `Runtime.Tests` VT-actor coverage is the surviving safety net. + +**Effort:** M (mechanical but touches ~2.4k lines + verification). **Risk:** Low if the grep confirms no `src/` consumers — but it is a large deletion, so land it **after** U2/U3 (so the live path is provably hardened first) and in its own PR for a clean revert. Blast radius = the dormant subtree only. + +--- + +## 5. S2 — HIGH — No coalescing/backpressure on upstream-change fan-out + +**Restatement:** Every upstream change spawns a fire-and-forget task queued on the single eval gate; a fast tag (e.g. 100 Hz) referenced by an alarm accumulates tasks without bound — memory growth, staleness, and wasted re-evaluation against the current cache. + +**Verification:** Confirmed. `ScriptedAlarmEngine.OnUpstreamChange` (`ScriptedAlarmEngine.cs:442-450`) spawns one `ReevaluateAsync` per change via `TrackBackgroundTask`. The `_inFlight` set tracks but does not bound them. (The VT-engine sibling `OnUpstreamChange` at `VirtualTagEngine.cs:263-272` is retired under U1 — S2 applies to the *live* alarm engine only after U1.) + +**Root cause:** Per-event task spawning instead of a dirty-set + single pump; cost scales with event rate, not distinct dirty alarms. + +**Proposed design:** Replace per-event `ReevaluateAsync` spawning with a **dirty-set + single-consumer pump** inside `ScriptedAlarmEngine`: +- `OnUpstreamChange` updates `_valueCache[path]` (unchanged), then for each referencing alarm id, `_dirtyAlarmIds.Add(id)` (a `ConcurrentDictionary` used as a set, or a `HashSet` under a lock), and signals a single long-lived pump (e.g. `_pumpSignal.Release()` on a `SemaphoreSlim(0)`, or an `AutoResetEvent`/`Channel<>`). +- One pump loop (started in `LoadAsync`, stopped in `Dispose`) waits on the signal, drains the current dirty set under `_evalGate`, and re-evaluates each distinct dirty alarm once against the current cache. New changes during a pass simply re-mark dirty for the next pass — this is the coalescing. +- This makes cost proportional to distinct dirty alarms per drain, not to event rate, and naturally bounds memory (the dirty set is bounded by the alarm count). + +Alternatives: *bounded channel with drop-oldest* (simpler but drops evaluations — acceptable since re-eval reads current cache, but a dirty-set is strictly better because it never loses the *fact* that an alarm needs re-eval); *debounce timer per alarm* (more timers, more complexity). Prefer the dirty-set + single pump — it also composes with a future VT need if one arises. + +Reuse the existing `_inFlight`/drain discipline for the pump task (the pump is one tracked task; `Dispose` already drains `_inFlight`). Keep the shelving timer as-is. + +**Implementation steps:** +1. Add `_dirtyAlarmIds` (concurrent set) + a signal primitive + a `Task _pumpTask` + a `CancellationTokenSource _pumpCts` to `ScriptedAlarmEngine`. +2. `LoadAsync`: after `_loaded = true`, start the pump loop (once per engine lifetime, not per load — or restart per load; simplest is start-once in the constructor and gate work on `_loaded`). Guard: the pump must acquire `_evalGate` and re-check `_disposed`/`_loaded` (mirror `ReevaluateAsync` lines 460-463). +3. `OnUpstreamChange`: mark dirty + signal instead of `TrackBackgroundTask(ReevaluateAsync(...))`. +4. Pump body reuses the existing `EvaluatePredicateToStateAsync` + persist-before-memory + emit-outside-gate pattern (lines 465-489). Snapshot + clear the dirty set under a short lock, then process under `_evalGate`. +5. `Dispose`: cancel `_pumpCts`, signal to unblock, await the pump (fold into the existing `_inFlight` drain or await `_pumpTask` explicitly). + +**Tests:** `tests/Core/…Core.ScriptedAlarms.Tests/ScriptedAlarmEngineTests.cs`: +- `RapidUpstreamChanges_CoalesceToBoundedEvaluations` — push N (e.g. 1000) changes to one referenced tag faster than eval drains (controllable clock / instrumented predicate counter); assert the predicate ran far fewer than N times and the final state reflects the last value. +- `FanOut_DoesNotLeakTasks` — assert the in-flight/dirty structures return to empty after quiescence. +- Regression: existing change-driven activation/clear tests still pass (a single change still promptly re-evaluates). + +**Effort:** M. **Risk:** Medium — reworks the alarm engine's hot path and its concurrency model. The `_evalGate` + persist-before-memory + emit-outside-gate invariants must be preserved exactly. Land standalone with careful review; it is the alarm engine's most-exercised path. Blast radius = scripted-alarm evaluation only. + +--- + +## 6. S1 — HIGH — `VirtualTagEngine.Load` mutates shared state with no gate + +**Restatement:** `Load` clears/rebuilds `_tags`, `_graph`, and disposes the compile cache without holding `_evalGate`, racing in-flight `CascadeAsync`/`EvaluateInternalAsync`/timer ticks reading those non-thread-safe collections. + +**Verification:** Confirmed. `Load` (`VirtualTagEngine.cs:78-173`) does `UnsubscribeFromUpstream(); _tags.Clear(); _graph.Clear(); _compileCache.Clear();` at 84-87 with no gate; `EvaluateInternalAsync` reads `_tags` at 293 before taking the gate (295); `CascadeAsync` walks `_graph` at 278. + +**Root cause:** The engine never received the alarm engine's gate-held-load + concurrent-collection discipline. + +**Proposed design:** **Resolved by U1 (retire the engine).** Since S1 is a latent bug in dormant code, deleting `VirtualTagEngine` eliminates it. **No separate fix.** If U1 is deferred or the engine is instead sanctioned (option (b)), then port the alarm engine's discipline: acquire `_evalGate` for the whole `Load`, make `_tags` a `ConcurrentDictionary`, and re-check-after-gate in `EvaluateInternalAsync` (it already reads `_tags` before the gate at 293 — move the read inside). Effort in that case: M; risk: Medium. + +**Recommendation:** Fold into U1's deletion. **Effort:** none (subsumed). **Risk:** none. + +--- + +## 7. S5 + C7 — MEDIUM — Timed-shelve expiry via predicate path swallows `Unshelved`; doc drift + +**S5 restatement:** When `ApplyPredicate` expires a timed shelve (via `MaybeExpireShelving`), it returns `Activated`/`Cleared`/`None` and **never** `Unshelved` — that emission only comes from `ApplyShelvingCheck` on the 5 s timer. A predicate re-eval landing between expiry time and the next tick persists the state as unshelved but publishes no `Unshelved` event, so OPC UA clients / `/alerts` never learn the shelve ended. + +**C7 restatement:** `ApplyPredicate`'s xmldoc (`Part9StateMachine.cs:31-34`) promises "branch-stack increment when a new active arrives while prior active is still un-acked" — no such logic exists (re-activation just resets Acked/Confirmed at lines 62-63). + +**Verification:** Both confirmed. `ApplyPredicate` (`Part9StateMachine.cs:39-90`): `MaybeExpireShelving` at 48, expiry produces `stateWithShelving` but the only emissions returned are `Suppressed`/`Activated`/`Cleared`/`None` (67, 84, 89). `ApplyShelvingCheck` (300-316) is the sole `Unshelved` source. Doc-drift at 31-34 confirmed against the whole file (no branch logic anywhere). + +**Root cause:** `TransitionResult` carries a single `EmissionKind`, so a compound "shelve expired AND value transitioned" can only report one. The engine's `EvaluatePredicateToStateAsync` (`ScriptedAlarmEngine.cs:546-552`) fires exactly one emission per predicate result. + +**Proposed design:** Make the shelve-expiry emission explicit. Two options: +- *(a) Engine-side detection (minimal, preferred).* In `EvaluatePredicateToStateAsync`, before calling `ApplyPredicate`, note `seed.Shelving.Kind`; after, if the result's `Shelving.Kind` changed from `Timed` to `Unshelved` (i.e. expiry happened inside `ApplyPredicate`), append a synthetic `Unshelved` emission to `pendingEmissions` **before** the value-transition emission. Order matters: `Unshelved` first, then `Activated`/`Cleared`. No state-machine signature change. Add an `AutoUnshelve` audit comment to match `ApplyShelvingCheck` for consistency (or centralize expiry so both paths append the same comment). +- *(b) Compound result.* Change `ApplyPredicate` to return a list of emissions (or a nullable secondary `Unshelved`). Cleaner semantically but a wider signature change touching all callers/tests. Prefer (a) — smaller blast radius, keeps `Part9StateMachine` pure-function-per-transition. + +Either way, centralize the expiry so `MaybeExpireShelving` and `ApplyShelvingCheck` don't drift. + +**C7 fix:** Correct the `ApplyPredicate` xmldoc (lines 30-34) to describe actual behavior: "activation, clearing, timed-shelve expiry, and shelving suppression; re-activation resets Acked/Confirmed. No condition-branch / previous-instance stack (see U5 conformance note)." + +**Implementation steps:** +1. `Part9StateMachine.cs`: fix the `ApplyPredicate` summary (C7). Optionally factor a private `ExpireTimedShelvingWithAudit` so both entry points append the identical `AutoUnshelve` comment. +2. `ScriptedAlarmEngine.EvaluatePredicateToStateAsync` (lines 504-553): capture `seed.Shelving.Kind`; after `ApplyPredicate`, detect Timed→Unshelved and prepend an `Unshelved` `ScriptedAlarmEvent` (via `BuildEmission(state, result.State, EmissionKind.Unshelved)`) to `pendingEmissions` ahead of the value emission. + +**Tests:** +- `Part9StateMachineTests.cs`: add `ApplyPredicate_WhenTimedShelveExpired_StatePersistsUnshelved` (state-level assertion — the machine already unshelves; assert `Shelving.Kind == Unshelved` after expiry). +- `ScriptedAlarmEngineTests.cs`: `PredicateReeval_AfterTimedShelveExpiry_EmitsUnshelvedBeforeTransition` — set a timed shelve, advance the controllable clock past `UnshelveAtUtc`, drive a predicate change via upstream push (not the shelving timer), assert two emissions in order: `Unshelved` then `Activated`/`Cleared`. +- Doc-only assertion not needed for C7. + +**Effort:** S-M. **Risk:** Low-Medium — touches emission ordering; keep it additive (never suppress the existing value emission). Blast radius = scripted-alarm shelving semantics; OPC UA clients gain a correct `Unshelved` event. + +--- + +## 8. S3 / S4 / S6 — MEDIUM — Reload-swap, `_alarmsReferencing` sync, sink dispose-drain (one hardening pass) + +### S3 — Failed reload is fail-stop, not fail-back + +**Restatement:** Both engines tear down the prior generation before compiling the new one; a compile OR transient `IAlarmStateStore` failure leaves the engine inert (alarm engine: zero alarms until next publish; VT engine: serving stale Good values). + +**Verification:** Confirmed. `ScriptedAlarmEngine.LoadAsync` clears/unsubscribes at 196-209, compiles, throws at 250-255 on any failure — but the store-restore loop (273-281) can also throw (`_store.LoadAsync`/`SaveAsync`), and that path is **not operator-preventable** (unlike compile errors, which publish-time validation catches). VT-engine equivalent is retired under U1. + +**Design:** For the alarm engine, split the two failure classes: +- **Compile failures** — already aggregated (250-255); publish-time validation makes these unlikely. Keep fail-stop but note the operator-preventable nature. Optionally build the new generation into locals and swap under the gate only on success (compile is side-effect-free; subscriptions + timer are the only external effects). This is the cleaner "build-then-swap" the report recommends, but it is a larger rewrite of `LoadAsync`. +- **Transient store failures** in the restore loop (273-281) — wrap each `_store.LoadAsync`/`SaveAsync` so a transient failure for one alarm falls back to `state.Condition` (the Fresh seed already compiled) and logs, rather than aborting the entire load. At minimum, distinguish store-failures from compile-failures and don't let a store hiccup zero out every alarm. + +**Recommended scope:** the *targeted* fix (don't abort the whole load on a single-alarm store failure) is S; the full build-then-swap is M. Do the targeted fix now; note build-then-swap as optional follow-up. + +**Implementation:** In `LoadAsync`'s restore loop (273-281), try/catch per alarm around `_store.LoadAsync`/`EvaluatePredicateToStateAsync`/`_store.SaveAsync`; on failure, keep the Fresh-seeded `AlarmState`, log a warning, and continue. The alarm still loads and evaluates live; only its persisted-state restore is skipped. + +### S4 — `_alarmsReferencing` read without synchronization + +**Restatement:** `_alarmsReferencing` is a plain `Dictionary` (`ScriptedAlarmEngine.cs:120-121`) mutated under `_evalGate` in `LoadAsync` (201, 237-242) but read by `OnUpstreamChange` (446) from upstream callback threads with no gate; a callback in flight during reload races `Clear()`/`Add()`. + +**Verification:** Confirmed. `_alarms` was deliberately made `ConcurrentDictionary` for exactly this reader class (comment 42-50); `_alarmsReferencing` was missed. Note: S2's dirty-set rework also reads `_alarmsReferencing` in `OnUpstreamChange` — fix S4 **together with S2** since they touch the same method. + +**Design:** Publish `_alarmsReferencing` as an immutable snapshot swapped atomically: keep a `private volatile IReadOnlyDictionary> _alarmsReferencing`; `LoadAsync` builds a fresh `Dictionary` under the gate and `Volatile.Write`s it; `OnUpstreamChange` reads the current reference (a coherent snapshot). Alternative: `ConcurrentDictionary` — but the value is a mutable `HashSet` also mutated during build, so the immutable-snapshot swap is cleaner and matches the "rebuilt per load" lifecycle. Prefer the volatile-snapshot swap. + +**Implementation:** Change the field type; build a local `Dictionary>` in the `LoadAsync` compile loop, freeze it (or keep `HashSet` values but never mutate after publish), and `Volatile.Write`. `OnUpstreamChange` reads via `Volatile.Read` (or just the `volatile` field). + +### S6 — `SqliteStoreAndForwardSink.Dispose` races an in-flight drain + +**Restatement:** `Dispose` (`SqliteStoreAndForwardSink.cs:679-686`) sets `_disposed`, disposes the timer, then immediately disposes `_drainGate` + the writer without waiting for an in-flight `DrainOnceAsync` — the drain's `finally { _drainGate.Release(); }` (line 467) then throws `ObjectDisposedException`, and the writer can be disposed mid-`WriteBatchAsync`. + +**Verification:** Confirmed. `DrainOnceAsync` takes `_drainGate.WaitAsync(0)` (323) and releases in `finally` (467); `Timer.Dispose()` (683) doesn't wait for a running callback (`DrainTimerCallback` is `async void`, 199-203). + +**Design:** Mirror the alarm engine's dispose-drain (`ScriptedAlarmEngine.cs:751-769`). Before disposing `_drainGate`/writer, acquire the gate to guarantee no drain is mid-flight: +- `Dispose`: set `_disposed`, dispose `_drainTimer`, then `_drainGate.Wait()` (blocking; sink is disposed from the host shutdown path, no sync-context deadlock) — this waits for any in-flight `DrainOnceAsync` to release — then dispose the writer, then dispose `_drainGate` (or don't re-release; since we hold it, just dispose after). Add a `_disposed` re-check at the top of `DrainOnceAsync` after acquiring the gate (it may already have one — verify around line 323-330) so a callback that passed the timer can bail. + +**Implementation:** Rewrite `Dispose` (679-686) to: `_disposed = true; _drainTimer?.Dispose(); _drainGate.Wait(); try { if (_writer is IDisposable d) d.Dispose(); } finally { _drainGate.Dispose(); }`. Confirm `DrainOnceAsync` returns early when `_disposed` after acquiring the gate. + +**Tests (S3/S4/S6):** +- S3: `LoadAsync_WhenStoreThrowsForOneAlarm_OtherAlarmsStillLoad` (fake `IAlarmStateStore` throwing for a specific id) — assert the engine loads and evaluates the rest. +- S4: covered indirectly by S2's concurrency test; optionally a targeted `Load_ConcurrentWithUpstreamCallback_NoTornRead` stress test. +- S6: `Dispose_DuringInFlightDrain_DoesNotThrow` — a slow fake `IAlarmHistorianWriter.WriteBatchAsync` (blocks on a gate the test controls), start a drain, call `Dispose` on another thread, release the writer, assert no `ObjectDisposedException` surfaces and the writer's dispose ran after the batch completed. Add to `SqliteStoreAndForwardSinkTests.cs`. + +**Effort:** S3 targeted S / S4 S / S6 S — bundle as one "alarm/sink hardening" PR (M total). **Risk:** Low-Medium; all three are fault-path hardening with clear tests. Sequence S4 with S2 (same method). Blast radius = alarm engine load + sink shutdown. + +--- + +## 9. Medium conventions / underdeveloped: U4, U5, C1, C2, C5 + +### U4 — MEDIUM — `IHistoryWriter` is a permanently-Null surface + +**Restatement:** `VirtualTagDefinition.Historize` routes to `IHistoryWriter.Record`, but DI binds only `NullHistoryWriter` (`Runtime/ServiceCollectionExtensions.cs:58`, `237`); the operator-visible Historize checkbox on virtual tags is a silent no-op. + +**Verification:** Confirmed. `services.TryAddSingleton(NullHistoryWriter.Instance)` (line 58); no real impl in tree. Separate `ContinuousHistorization` path taps the mux directly (CLAUDE.md — and note the OVERALL report says Known Limitation 2 is *closed*, so continuous historization is now live via `FeedHistorizedRefs`/`UpdateHistorizedRefs`). + +**Design:** Two coherent choices; pick per product intent: +- *(a) Wire `IHistoryWriter` to the gateway value-writer* — make the checkbox real by binding a gateway-backed `IHistoryWriter` (the same `WriteLiveValues` path continuous-historization uses). Larger; overlaps the continuous-historization recorder. Only do this if VT-result historization is a distinct requirement from continuous historization. +- *(b) Hide/annotate the checkbox until it does something* (preferred interim) — in the AdminUI VirtualTag modal, disable/hide the Historize toggle (or add an inline "not yet wired" note) and add an xmldoc/appsettings note that `IHistoryWriter` is Null unless a deployment overrides it. Avoids operator confusion at near-zero cost. + +**Recommendation:** (b) now; open a tracked follow-up for (a) if VT-result historization is on the roadmap. Cross-reference the OVERALL note that continuous historization is the live historization path. + +**Effort:** S (option b). **Risk:** Low. Touches AdminUI VT modal + a doc line. + +### U5 — MEDIUM — Part 9 surface gaps real but undeclared + +**Restatement:** No condition branches/previous-instances; `Severity` static per definition; `AlarmKind` affects only node typing; `MessageTemplate` has no brace escaping; `Retain` carried but not consulted. Each a defensible v1 cut, but scattered. + +**Verification:** Confirmed against `Part9StateMachine.cs` (no branch logic), `ScriptedAlarmDefinition.cs` (static Severity), `MessageTemplate.cs`, and the engine (Retain unused). + +**Design:** Documentation, not code. Add an explicit **OPC UA Part 9 conformance statement** section to `docs/ScriptedAlarms.md` listing the supported subset and the v1 cuts (no branching/previous-instances, static Severity, kind→node-typing only, no template brace escaping, Retain carried-not-enforced) so client integrators know what to expect. Fold C7's corrected `ApplyPredicate` doc into this. + +**Effort:** S. **Risk:** None. + +### C1 — MEDIUM — `ITagUpstreamSource` defined twice + +**Restatement:** Identical-shape interface in `Core.VirtualTags/ITagUpstreamSource.cs` AND at `ScriptedAlarmEngine.cs:860-872`; two distinct .NET types, invariant enforced only by comment. + +**Verification:** Confirmed — two `interface ITagUpstreamSource` definitions. + +**Design:** Hoist a single `ITagUpstreamSource` into `Core.Scripting.Abstractions` (both projects reference it — verify the reference direction) and retire both duplicates. If `Core.VirtualTags` is being retired (U1), the alarm engine's copy is the survivor to relocate. Sequence: do C1 as part of/just before U1 so the delete doesn't strand the interface. Update `DependencyMuxTagUpstreamSource` and any composing bridge to the single type (they currently must adapt each separately). + +**Implementation:** Add `ITagUpstreamSource.cs` to `Core.Scripting.Abstractions`; delete the copy from `ScriptedAlarmEngine.cs` (843-872 also holds `ScriptedAlarmEvent` — see C3) and the `Core.VirtualTags` file; fix usings. Confirm `Core.ScriptedAlarms` and `Core.VirtualTags` already reference `Core.Scripting.Abstractions`. + +**Effort:** S-M. **Risk:** Low-Medium — a shared contract move; ensure all implementors (`DependencyMuxTagUpstreamSource`) compile against the single type. Blast radius = both engines + the mux bridge. + +### C2 — MEDIUM — `Core.Scripting.Abstractions` declares types across three namespaces + +**Restatement:** `ScriptContext`/`ScriptGlobals`/`PassthroughScript` → ns `Core.Scripting`; `VirtualTagContext` → ns `Core.VirtualTags`; `AlarmPredicateContext` → ns `Core.ScriptedAlarms` — all physically in the Abstractions assembly (deliberate: the sandbox pins `contextType.Assembly`, so concrete contexts must live Roslyn-free). Documented only in a `PassthroughScript` remark; misleads find-by-namespace. + +**Verification:** Confirmed (rationale at `ScriptSandbox.cs:60-70`). + +**Design:** Documentation, not a namespace migration (a `TypeForwardedTo` migration is higher-risk churn for a convention nit). Add a project-level `README.md` (or an `AssemblyInfo.cs` doc comment / a `_Namespaces.md`) in `Core.Scripting.Abstractions` stating the rule: "this assembly is the sandbox-pinned Roslyn-free closure; concrete script-context types keep their **consumer** namespaces deliberately so `ScriptSandbox.Build(contextType)` pins this assembly without dragging Roslyn." Reference `ScriptSandbox.cs:60-70`. + +**Effort:** S. **Risk:** None. + +### C5 — LOW/MEDIUM — `Core.AlarmHistorian` mixes contract + implementation + +**Restatement:** `IAlarmHistorianSink`/`IAlarmHistorianWriter`/status types share the project with `SqliteStoreAndForwardSink`, so the Runtime adapter + gateway driver drag the `Microsoft.Data.Sqlite` dependency to get the interfaces. + +**Verification:** Confirmed by project structure. + +**Design:** Optional. Extract the interfaces + status DTOs into a `Core.AlarmHistorian.Abstractions` (mirror the scripting split) so consumers reference the contract without the SQLite impl. Tolerable at three files — schedule only if the SQLite transitive dependency becomes a problem for the gateway driver. Defer unless bundled with a broader Abstractions cleanup. + +**Effort:** M (new project + reference rewiring). **Risk:** Low. **Recommend: defer** (note only). + +--- + +## 10. Lows (batched) + +| ID | Restatement | Verification | Action | Effort | +|---|---|---|---|---| +| **S7** | Sandbox CPU/mem unbounded; timed-out CPU-bound script leaks a pool thread; hot looping script orphans one thread per upstream change | Confirmed (`ScriptSandbox.cs:30-35`, `TimedScriptEvaluator.cs:17-26`). Note U2's fix *increases* orphan exposure on the VT path | Add a **per-script circuit breaker**: N consecutive timeouts → suspend evaluation + surface to Admin UI (meter/health). Cheap interim before out-of-process runner. Applies to both `RoslynVirtualTagEvaluator` (post-U2) and `ScriptedAlarmEngine` (quarantine the predicate after repeated timeouts — currently holds prior state but keeps re-evaluating, `ScriptedAlarmEngine.cs:535-538`) | M | +| **S8** | `ScriptedAlarmEngine.Dispose` `Task.WhenAll(...).GetAwaiter().GetResult()` (`ScriptedAlarmEngine.cs:761`) — deadlock trap off the actor host | Confirmed | Implement `IAsyncDisposable` alongside `IDisposable`; actor host awaits. Low urgency (fine under current host) | S | +| **S9** | At-least-once delivery duplicates on crash between `WriteBatchAsync` and outcome commit (`SqliteStoreAndForwardSink.cs:372-441`) | Confirmed (correct choice: dupes over loss) | Document in `docs/Historian.md` that the gateway `SendEvent` side must tolerate replays | S | +| **S10** | `VirtualTagSource.SubscribeAsync` emits seed Read before registering observer (`VirtualTagSource.cs:62-73`) → a change between Read and Subscribe is missed | Confirmed | Retired with U1 (VirtualTagSource is dormant). If kept: document the seed-then-subscribe trade-off, or register-first + let idempotent newer-wins consumer dedupe | none (U1) | +| **S11** | Capacity eviction drops *oldest* accepted alarm events (`SqliteStoreAndForwardSink.cs:602-636`), `EvictedCount` counter exists | Confirmed | Document the drop-oldest policy as deliberate in `docs/AlarmTracking.md`; note drop-newest/refuse-enqueue as the compliance alternative | S | +| **P2** | `Task.Run`+`WaitAsync` per evaluation on the hot path (`TimedScriptEvaluator.cs:78-81`) | Confirmed; required for the timeout to work | No action now; note inline-run-with-watchdog as the escape hatch if profiling shows it. (U2 *adds* this hop to the VT path — accepted) | none | +| **P3** | VT engine allocates per-eval what the alarm engine pools (`VirtualTagEngine.cs:298, 313-317`) | Confirmed | Retired with U1 | none (U1) | +| **P4** | Single global eval gate per engine | Confirmed; correct + simple | No action (per-alarm/sharded gate is the escape hatch) | none | +| **P5** | SQLite per-call PRAGMA + `COUNT(*)` overheads (`SqliteStoreAndForwardSink.cs:244-246, 479-484`) | Confirmed; capacity fast path already removed the hot cost | No action | none | +| **P6** | `DependencyGraph` well-optimized | Confirmed | Retired with U1 | none (U1) | +| **C3** | Trailing type defs: `ScriptedAlarmEvent` + dup `ITagUpstreamSource` at `ScriptedAlarmEngine.cs:843-872`; `CompilationErrorException`/`ScriptAssemblyLoadContext` at `ScriptEvaluator.cs:391-432`; `DependencyCycleException` in `DependencyGraph.cs` | Confirmed | Move `ScriptedAlarmEvent` (public cross-project contract) to its own file (do with C1's interface hoist). Others are reasonable co-location — leave | S | +| **C4** | Repeated raw OPC UA status-code literals (`0x80340000u` etc.) across ≥5 files | Confirmed | Add a `KnownStatusCodes` static in `Core.Abstractions` (these layers avoid the OPC Foundation package deliberately) and replace the hand-rolled literals | S | +| **C6** | Plan-era naming residue ("Phase 7 plan Stream A.4/B/C/…") in xmldocs across many files | Confirmed | Sweep to current doc anchors (`docs/ScriptedAlarms.md`/`docs/ScriptEditor.md`) during the next doc pass | S | +| **U6** | No TODO/HACK/FIXME markers (gaps in xmldoc remarks) | Confirmed — healthy pattern | No action | none | +| **U7** | Broad behavior-focused tests with specific holes: (a) S5 expiry-emission, (b) load-vs-cascade concurrency (S1/S4), (c) sink Dispose-during-drain (S6), (d) `ForbiddenTypeAnalyzer` no dedicated suite, (e) production `RoslynVirtualTagEvaluator` timeout (would've caught U2) | Confirmed — no `*ForbiddenType*` test file exists; `RoslynVirtualTagEvaluatorTests.cs` exists in `Host.IntegrationTests` | Holes (a)/(c)/(e) covered by the tests above (S5/S6/U2). Add a dedicated `ForbiddenTypeAnalyzerTests` suite (analyzer-pass unit tests against `CSharpCompilation`s) so analyzer regressions produce clear failures | S-M | + +--- + +## Suggested PR sequencing + +1. **PR-A (Critical, small):** U2 + U3 together (same file `RoslynVirtualTagEvaluator.cs` + one `VirtualTagHostActor` hook) + their tests. Ship first — closes the OVERALL action item #2 Critical. +2. **PR-B (perf, small):** P1 sandbox-reference memoization + `ScriptSandboxTests`. +3. **PR-C (cleanup, medium):** C1 (hoist `ITagUpstreamSource`) + C3 (`ScriptedAlarmEvent` file) → then U1 retire (`VirtualTagEngine`/`TimerTriggerScheduler`/`VirtualTagSource`/`DependencyGraph` + tests). Subsumes S1, S3-VT, P3, P6, S10. Land after PR-A. +4. **PR-D (alarm hardening, medium):** S2 (dirty-set pump) + S4 (`_alarmsReferencing` snapshot, same method) + S6 (sink dispose-drain) + S3 (targeted store-failure fallback) + their tests. +5. **PR-E (semantics, small-medium):** S5 + C7 (unshelve emission + doc) with state-machine + engine tests. +6. **PR-F (docs/nits):** U4 (annotate checkbox), U5 (Part 9 conformance doc), C2 (Abstractions README), C4 (`KnownStatusCodes`), C6 (plan-era doc sweep), S9/S11 (doc policies), U7 (`ForbiddenTypeAnalyzerTests`). Optional S7 circuit-breaker + S8 `IAsyncDisposable` as tracked follow-ups. diff --git a/archreview/plans/03-server-runtime-plan.md b/archreview/plans/03-server-runtime-plan.md new file mode 100644 index 00000000..d905084b --- /dev/null +++ b/archreview/plans/03-server-runtime-plan.md @@ -0,0 +1,599 @@ +# Design & Implementation Plan — Server & Runtime Subsystem (Report 03) + +- **Source report:** `archreview/03-server-runtime.md` +- **Review commit:** `9cad9ed0` +- **Verification pass performed against tree at:** current `master` (same tree; all cited files present) +- **Scope:** `OpcUaServer`, `Host`, `Runtime`, `ControlPlane`, `Security`; redundancy, LDAP, node manager, address-space applier. + +## Verification summary + +| ID | Sev | Status | One-liner | +|----|-----|--------|-----------| +| S1 | Critical | **Confirmed** | SBR configured in HOCON but never activated → NoDowning; hard-crash failover broken | +| S2 | High | **Confirmed** | LDAP auth block-bridges session activation; no timeout, no pooling | +| S3 | High | **Confirmed** | HistoryRead block-bridges gateway per node, sequentially, on SDK threads | +| S4 | High | **Confirmed** | Primary write/ack gate defaults to allow while role unknown → dual-primary window | +| P1 | High | **Confirmed** | Any structural add full-rebuilds address space, kills all subscriptions | +| S5 | Medium | **Confirmed** | Redundancy NodeId string-matched across 3 sources, silent on mismatch | +| S6 | Medium | **Confirmed** | No `SupervisorStrategy` anywhere in Runtime; restart wipes subscription state, hot-loops | +| S7 | Medium | **Confirmed** | `DriverInstanceActor.PostStop` block-bridges `ShutdownAsync(None)` | +| S8 | Medium | **Confirmed** | Inbound Part 9 / native-ack routes are at-most-once fire-and-forget, no metric | +| S9 | Medium | **Confirmed** | Server cert 12-month lifetime, boot-only check, no expiry monitoring | +| S10 | Low | **Confirmed** | SDK start failure swallowed; no health surface | +| S11 | Low | **Confirmed** | `AuditWriterActor` unbounded buffer, drops batches (moot — no producers, U3) | +| P2 | Medium | **Confirmed** | Per-value global-lock write, one actor msg per value, no batching | +| P3 | Medium | **Confirmed** | HistoryRead-Events unbounded (`NumValuesPerNode==0 → int.MaxValue`), no paging | +| P4 | Low | **Confirmed** | Every deploy re-runs 4 Materialise passes over full composition | +| C1 | Medium | **Confirmed** | `ServerHistorian` bound imperatively in 5 places, no `IOptions` | +| C2 | Medium | **Confirmed** | Two-tier options validation; `DevStubMode` only log-warned in prod | +| C3 | Low | **Confirmed** | Library code logs via static Serilog; node manager uses obsolete `Utils.LogError` | +| C4 | Low | **Confirmed** | `IHistorianProvisioning` resolve has no missing-registration warning | +| U1 | High (doc) | **Confirmed stale doc** | CLAUDE.md "Known Limitation 2" stale — recorder ref-feed IS wired | +| U2 | Medium | **Confirmed** | Deferred-sink forwarding correct today but only ~5/10 members test-guarded | +| U3 | Medium | **Confirmed** | Audit pipeline built/tested, zero producers | +| U4 | Medium | **Confirmed** | `FleetStatusBroadcaster.DriverHostStatusHeartbeat` dead code, host-only key | +| U5 | Medium | **Confirmed** | Native Part 9 conditions: Acknowledge-to-driver only; Confirm/Shelve/Enable no-op upstream | +| U6 | Medium | **Confirmed** | Test gaps concentrate on DPS delivery / supervision / hard-kill failover | +| U7 | Low | **Confirmed** | `IHistoryWriter` Null-wired; H2 HistoryUpdate unimplemented; enum gap | +| U8 | Low | **Confirmed** | `BuildSecurityPolicies` doc claims a log it doesn't emit | +| U9 | Low | **Confirmed** | `EnsureVariable` silently ignores changed historize-intent (invariant in comments) | + +**Nothing in this report is stale-because-already-fixed.** The one "stale" item (U1) is stale *documentation* — the code it describes as broken is actually wired. Every other finding reproduces against the current tree. + +**Top 3 by priority:** (1) **S1** — activate the split-brain resolver (OVERALL action #1); (2) **P1** — surgical pure-adds in the address-space applier (OVERALL action #8, highest operational perf win); (3) **S4 + S2/S3** — the availability/blocking cluster (primary-gate default-deny, async LDAP, channelized HistoryRead; OVERALL actions #7/#9). + +--- + +## S1 — CRITICAL — Split-brain resolver configured but never activated + +**Restatement:** `akka.conf` carries a `split-brain-resolver { active-strategy = "keep-oldest" … }` block but nothing registers a downing provider, so the cluster runs Akka's default **NoDowning**. Hard-crashed nodes stay unreachable forever → singletons + `driver` role-leader never fail over; a partition leaves both sides at ServiceLevel 240. + +**Verification — Confirmed.** +- `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/Resources/akka.conf:40-46` — the SBR block is present, inside `akka.cluster { … }`. +- `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/ServiceCollectionExtensions.cs:80-84` — `WithClustering(new ClusterOptions { SeedNodes = …, Roles = … })` sets **only** those two properties; `SplitBrainResolver` is never assigned. +- Repo-wide grep for `downing-provider|SplitBrainResolver|KeepOldest|downing-provider-class` returns **zero** hits (confirmed empty). +- Packages: `Akka.Cluster` / `Akka.Cluster.Hosting` = **1.5.62** (`Directory.Packages.props:6-13`). This version ships `Akka.Cluster.SBR.SplitBrainResolverProvider` and the `Akka.Cluster.Hosting` `ClusterOptions.SplitBrainResolver` option surface. +- `min-nr-of-members = 1` (`akka.conf:38`) makes each partition side self-sufficient for leader election → the "both sides at 240" partition case is real. + +**Root cause:** In Akka.NET the `split-brain-resolver` HOCON block is *inert configuration* — it only takes effect when `akka.cluster.downing-provider-class` points at `Akka.Cluster.SBR.SplitBrainResolverProvider` (or the Hosting `ClusterOptions.SplitBrainResolver` option, which sets that provider under the hood). The HOCON block was written but the activating registration was never added. Classic "config-in-HOCON is not config-in-effect" (OVERALL cross-cutting theme #1). + +**Proposed design.** Prefer the **Akka.Hosting typed option** (`ClusterOptions.SplitBrainResolver`) over hand-writing `downing-provider-class`, because the rest of the cluster bootstrap already goes through `WithClustering(ClusterOptions)` and Akka.Hosting owns provider wiring — mixing a raw HOCON `downing-provider-class` with Hosting risks the same "Hosting doesn't honor HOCON" foot-gun already documented for `akka.loggers` (`akka.conf:9-13`). Keep the HOCON block as the tuning source (`stable-after`, `keep-oldest.down-if-alone`) — the SBR provider reads those keys once activated — OR move the strategy fully into the typed option and delete the HOCON block to avoid a two-source drift. **Recommendation: use the typed option and keep HOCON only for values the typed option can't express, documenting which wins.** + +Strategy choice: **keep-oldest with `down-if-alone = on`** matches the current HOCON intent and is correct for a 2-node warm-redundancy pair (the oldest — typically the long-running primary — survives an even split; `down-if-alone` downs a singleton that loses quorum). `keep-majority` is wrong for 2 nodes (no majority in a 1-1 split). `static-quorum` would need `quorum-size = 2` which defeats single-node deploys. Keep-oldest is the right call; document the reasoning. + +`stable-after = 15s` must be ≥ the failure detector's `acceptable-heartbeat-pause` (10s) plus margin — current 15s is fine but tie it explicitly to `down-removal-margin` (also 15s) in a comment. + +**Implementation steps.** +1. `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/ServiceCollectionExtensions.cs` — in `WithOtOpcUaClusterBootstrap`, set the SBR option on the `ClusterOptions`: + ```csharp + builder.WithClustering(new ClusterOptions + { + SeedNodes = options.SeedNodes, + Roles = options.Roles, + SplitBrainResolver = new KeepOldestOption { DownIfAlone = true }, // Akka.Cluster.Hosting + }); + ``` + (Confirm the exact type name against Akka.Cluster.Hosting 1.5.62 — it is `KeepOldestOption : SplitBrainResolverOption`, with `DownIfAlone` and optional `Role`. `StableAfter` is set on the option or left to the HOCON default.) +2. Add a **startup assertion log line** proving SBR is active (OVERALL theme #1 recommendation). In the same method after `WithClustering`, register a small `IActorRef`-less startup hook (or log in `WithOtOpcUaClusterBootstrap`) that reads `Context.System.Settings.Config.GetString("akka.cluster.downing-provider-class")` at first `MemberUp` and logs `Info` if it equals the SBR provider, `Error` if empty. A cheap version: a tiny actor subscribed to `ClusterEvent.IMemberEvent` that logs the resolved downing provider once. This converts the silent-inert failure mode into an operator-visible signal. +3. Reconcile the HOCON: either delete the `split-brain-resolver` block from `akka.conf` (typed option now owns it) OR keep it and add a comment that `WithClustering(SplitBrainResolver=…)` is what activates it. Do not leave both un-cross-referenced. +4. Update `docs/Redundancy.md` and `docs/v2/Architecture.md` — document that SBR is now active, the keep-oldest rationale, and the hard-crash failover expectation. + +**Tests.** +- **Unit:** a config test asserting `akka.cluster.downing-provider-class == "Akka.Cluster.SBR.SplitBrainResolverProvider, Akka.Cluster"` after `WithOtOpcUaClusterBootstrap` runs (proves the typed option activated the provider). This is the "assert SBR is active" guard. +- **Integration (the load-bearing one):** extend the 2-node harness (`tests/Server/.../TwoNodeClusterHarness`) with a **hard-kill failover test**: bring both nodes up, confirm role-leader on node A, `kill -9`-equivalent (abrupt `ActorSystem` terminate / drop the transport without `CoordinatedShutdown`), then assert within `stable-after + margin` that (a) node B becomes `driver` role-leader, (b) the singletons (`RedundancyStateActor`, `ConfigPublishCoordinator`, `AdminOperationsActor`, `AuditWriterActor`) hand over to B, (c) B's ServiceLevel rises to 240 and the (dead) A demotes. The current harness only exercises graceful `StopAsync` — this is exactly the U6 blind spot. If in-process abrupt-kill is hard to simulate deterministically, use `Cluster.Get(sys).Down(selfAddress)` suppression + transport fault injection, or run it as a nightly docker-host multi-process test. +- **Partition test (optional, nightly):** simulate a 1-1 partition (block the transport between the two), assert exactly one side downs the other and only one side keeps 240. + +**Effort:** S (the fix) + M (the failover test harness). **Risk/blast-radius:** Medium. The one-line option change alters cluster downing behavior globally — a mis-tuned `stable-after` could down a node during a transient GC pause. Mitigate by keeping `stable-after ≥ acceptable-heartbeat-pause + margin` and validating on the 2-node rig before merge. + +--- + +## P1 — HIGH — Structural deploy full-rebuilds the address space, severing every client subscription + +**Restatement:** `AddressSpaceApplier.Apply` forces `RebuildAddressSpace()` for any added/removed/structurally-changed equipment, alarm, tag, or virtual tag. The rebuild removes and recreates every `NodeState`, so all existing client monitored items go dead server-wide. Adding one tag to one equipment drops every subscription on the server. + +**Verification — Confirmed.** +- `AddressSpaceApplier.cs:134-140` — `structuralRebuild` is true whenever `AddedEquipmentTags.Count > 0 || RemovedEquipmentTags.Count > 0 || AddedEquipment/AddedAlarms/… > 0`. Pure adds fall into this branch. +- `AddressSpaceApplier.cs:150-154` — `if (structuralRebuild) { SafeRebuild(); rebuilt = true; }`. +- `OtOpcUaNodeManager.cs:1690-1736` — `RebuildAddressSpace()` clears `_variables`, `_alarmConditions`, `_folders`, `_notifierFolders`, `_eventNotifierSources` under one `Lock`, detaching/removing every `NodeState`. +- `OpcUaPublishActor.cs:338-354` — after `Apply`, the actor *unconditionally* re-runs the four `Materialise*` passes, which recreate all nodes from scratch. Existing subscriptions bind to the old (now-removed) `NodeState` instances → dead. +- The surgical path (`AddressSpaceApplier.cs:155-199`) covers only **changed** tags (`TagDeltaIsSurgicalEligible`) and folder renames — **not adds or removes**. +- Building blocks already exist and are idempotent: `EnsureFolder` (`:1284`), `EnsureVariable` (`:1369`, early-returns on existing id — U9 confirms), `MaterialiseAlarmCondition` (`:586`, idempotent guard `:57`), and `RaiseNodesAddedModelChange` (`:1592`). + +**Root cause:** The applier's rebuild predicate treats "add" and "remove" identically to "structural change." But a pure add needs no teardown at all — the idempotent `EnsureFolder`/`EnsureVariable`/`MaterialiseAlarmCondition` passes add exactly the new nodes and no-op the existing ones; `RaiseNodesAddedModelChange` already exists to announce them to subscribed clients. The rebuild is unnecessary for the pure-add case and catastrophic for live subscriptions. + +**Proposed design — surgical pure-adds (phase 1), scoped removals (phase 2).** This is the highest-leverage item in the subsystem (OVERALL action #8) and the report's own top recommendation. Do it in two phases so phase 1 (adds) ships value fast and low-risk. + +**Phase 1 — pure-add surgical path.** Split the rebuild predicate: a rebuild is only *required* when there is a **removal** or a **node-affecting change** (the existing non-surgical-eligible change set). Pure adds route to the idempotent Materialise passes + a model-change announcement, no teardown: +- New predicate: + - `requiresRebuild = RemovedEquipment/RemovedAlarms/RemovedEquipmentTags/RemovedEquipmentVirtualTags any > 0` **OR** `ChangedEquipment.Count > 0` **OR** `ChangedAlarms.Count > 0` **OR** `ChangedEquipmentTags.Any(!surgicalEligible)` **OR** `ChangedEquipmentVirtualTags.Any(!nodeIrrelevant)`. + - `pureAdds = !requiresRebuild && (AddedEquipment/AddedAlarms/AddedEquipmentTags/AddedEquipmentVirtualTags any > 0)`. +- When `pureAdds` (and not requiresRebuild): **do not** call `RebuildAddressSpace()`. The existing unconditional Materialise passes in `OpcUaPublishActor` already add only the new nodes (idempotent). Add a step that calls `RaiseNodesAddedModelChange` for each newly-added node/folder so subscribed clients get a `GeneralModelChangeEvent` and can re-browse. +- Because `OpcUaPublishActor` currently re-runs the Materialise passes regardless, the safest minimal change is: in `Apply`, compute `rebuilt=false` for the pure-add case and rely on the downstream passes; then emit model-change events for the plan's added ids. Confirm the Materialise passes run *after* `Apply` returns (they do — `OpcUaPublishActor.cs:335-354`), so the added nodes exist before the announce; if announce must follow materialise, move the announce into the applier's post-materialise hook or have `OpcUaPublishActor` call a new `applier.AnnounceAdds(plan)` after the passes. + +**Phase 2 — scoped per-equipment removal.** For removals, add surgical `RemoveVariable(nodeId)` / `RemoveFolder(nodeId)` / `RemoveAlarmCondition(nodeId)` methods to the node manager (mirroring the per-node teardown already inside `RebuildAddressSpace`'s loops but scoped to one id), forwarded through `ISurgicalAddressSpaceSink` + `DeferredAddressSpaceSink` (respect the U2 forwarding trap — every new capability method MUST be added to the deferred wrapper and test-guarded). A remove of equipment E only tears down E's subtree, preserving every other equipment's subscriptions. Announce via `RaiseNodesDeletedModelChange` (add if the SDK exposes it; else a `GeneralModelChangeEvent`). Only fall back to a full rebuild for the genuinely-ambiguous changed-topology cases. + +**Alternatives considered:** (a) Leave rebuild but *re-attach* subscriptions to new nodes — not feasible; the SDK binds monitored items to `NodeState` instances, and there's no supported re-home API. (b) Diff-and-patch every deploy generically — larger and riskier than the add/remove split; the plan already carries typed add/remove/change deltas, so use them. The add/remove split reuses the exact pattern F10b established for surgical changes. + +**Implementation steps.** +1. `AddressSpaceApplier.cs` — refactor the rebuild predicate into `requiresRebuild` / `pureAdds`; route pure adds away from `SafeRebuild()`. Add `AnnounceAdds(AddressSpacePlan)` calling `_sink.RaiseNodesAddedModelChange(id)` for each added folder/variable/alarm node id (compute ids via `EquipmentNodeIds` as the surgical change path already does at `:177`). +2. `OtOpcUaNodeManager.cs` — Phase 2: add `RemoveVariable`/`RemoveFolder`/`RemoveAlarmCondition` (scoped teardown under `Lock`, drop the matching `_historizedTagnames`/`_eventNotifierSources`/`_nativeAlarmNodeIds` registrations); add `RaiseNodesDeletedModelChange` if available. +3. `ISurgicalAddressSpaceSink` (+ `SdkAddressSpaceSink`, `NullAddressSpaceSink`, `DeferredAddressSpaceSink`) — Phase 2: add the remove methods with capability-sniffing forwarding, matching the existing `UpdateTagAttributes`/`UpdateFolderDisplayName` pattern (`DeferredAddressSpaceSink.cs:58-69`). +4. `OpcUaPublishActor.cs` — ensure the announce runs after the Materialise passes (call `_applier.AnnounceAdds(plan)` after line 354, or fold the announce into `Apply` if ordering allows). + +**Tests.** +- **Unit:** applier test — a pure-add plan (one added tag) does **not** call `RebuildAddressSpace` (assert via a recording sink) and **does** call `RaiseNodesAddedModelChange` for the added id. A changed-only plan still routes surgical/rebuild as today. A remove plan (phase 2) calls `RemoveVariable` for the removed id and does not rebuild if no other change. +- **Live-verify (the decisive one — F10b/OpcUaServer-001 precedent proves unit-green is insufficient):** on docker-dev (rebuild both central-1/2), subscribe a Client.CLI monitored item to an existing equipment tag (`subscribe -n ns=2;s=…`), then `POST /api/deployments` a config that **adds one new tag** to a *different* equipment. Assert the CLI subscription keeps delivering (no `Bad`/dead notifications) and the new tag becomes browsable. This is the exact regression the whole item exists to kill. Reuse the OpcUaServer-001 live-verify recipe from memory (`project_full_codebase_review_2026-06-19.md`). +- **Integration:** `OpcUaServer.IntegrationTests` — over-the-wire subscribe-then-add-tag round-trip asserting subscription survival (also addresses U6's "one test only" gap). + +**Effort:** L (phase 1 M, phase 2 M). **Risk/blast-radius:** High — this touches the address-space mutation core. Mitigate with the recording-sink unit tests + the mandatory subscription-survival live-verify + the U2 forwarding guard (a new remove capability that isn't forwarded through `DeferredAddressSpaceSink` ships inert, exactly like F10b). + +--- + +## S4 — HIGH — Primary gate defaults to allow while role unknown (dual-primary data-plane window) + +**Restatement:** `DriverHostActor`'s inbound-write / native-ack / native-alarm-emit gates gate on `_localRole is Secondary or Detached` and default to *allow* until the first redundancy snapshot arrives. A freshly-booted or snapshot-missed secondary services shared-field-device writes and emits alerts as primary for up to the 10s heartbeat interval — indefinitely if S5's NodeId mismatch bites. + +**Verification — Confirmed.** +- `DriverHostActor.cs:195` — `private RedundancyRole? _localRole;` (nullable, **null at boot**). +- `HandleRouteNodeWrite` (`:1018-1026`): `if (_localRole is RedundancyRole.Secondary or RedundancyRole.Detached) { reject }` — null falls through to allow. +- `HandleRouteNativeAlarmAck` (`:1074-1083`): same pattern, drops on Secondary/Detached, null → allow. +- Native-alarm emit (`:956-984`): `if (_localRole is Secondary or Detached) continue;` — null → emit. +- `OnRedundancyStateChanged` (`:1109-1116`): a snapshot not mentioning this node leaves `_localRole` unchanged → stays null → default-allow. + +**Root cause:** The boot-window default-allow is correct for single-node deploys (where no snapshot ever demotes) but wrong for multi-node clusters where a booting secondary has a real primary peer. The gate can't distinguish "single node, no peer" from "multi-node, snapshot not yet arrived." It uses the *role signal* to gate *shared-device writes*, and the role signal is under-determined at boot. + +**Proposed design — cluster-membership-aware default.** Discover whether this is a multi-driver cluster from Akka cluster state (already available: `OpcUaPublishActor.cs:81` does `Akka.Cluster.Cluster.Get(Context.System)`). Change the gate default per cluster size: +- **Single-driver cluster** (≤1 member with the `driver` role): keep default-allow (preserves single-node deploys + boot window). +- **Multi-driver cluster** (≥2 `driver` members): **default-deny** the write/ack gates until at least one redundancy snapshot has been received (`_localRole is not null`). Native-alarm *condition writes* stay ungated (the secondary keeps its address space warm — that's `:953-954`, correct); only the *alerts publish* + *device write* + *device ack* gate deny-on-unknown. + +Implementation: `DriverHostActor` subscribes to `ClusterEvent.IMemberEvent` (or reads `_cluster.State.Members` on demand) and tracks `driverMemberCount`. The gate becomes: +```csharp +bool servicedByThisNode = _localRole switch { + RedundancyRole.Secondary or RedundancyRole.Detached => false, + RedundancyRole.Primary => true, + null => DriverMemberCount() <= 1, // unknown: allow only on a single-node cluster +}; +``` +Add a `_hasReceivedSnapshot` flag (distinct from role, to handle the "snapshot arrived but didn't mention us" case → that's the S5 mismatch; log-once per S5 and deny on multi-node). + +**Alternatives considered:** (a) Always default-deny — breaks single-node deploys (no snapshot ever arrives to enable writes). Rejected. (b) A fixed boot grace timer that flips to deny after N seconds — fragile, and a slow snapshot on a healthy multi-node cluster would wrongly deny. The membership-count approach is deterministic and uses signal already in hand. (c) Require a lease/fencing token — larger architectural change (the report notes "no lease or fencing token"); out of scope, membership-gate is the pragmatic fix. + +**Implementation steps.** +1. `DriverHostActor.cs` — add `private readonly Akka.Cluster.Cluster _cluster = Akka.Cluster.Cluster.Get(Context.System);` (mirror `OpcUaPublishActor`); add `DriverMemberCount()` counting `Up` members carrying the `driver` role from `_cluster.State.Members`. +2. Introduce a single private `bool ShouldServiceAsPrimary()` helper encoding the switch above; call it from all three gate sites (`:960`, `:1022`, `:1078`) so the policy lives in one place. +3. On a snapshot that omits this node while `DriverMemberCount() > 1`, log-once `Warning` (this is the S5 hook — the two findings share the diagnostic). +4. Document the semantics in `docs/Redundancy.md`. + +**Tests.** +- **Unit (TestKit):** with `_localRole == null` and a stubbed single-driver membership → write is serviced; with `_localRole == null` and a two-driver membership → write is rejected ("not primary / role unknown on multi-node"); after a Primary snapshot → serviced; after a Secondary snapshot → rejected. +- **Integration:** on the 2-node harness, boot the secondary and immediately issue a write before the first snapshot; assert it is rejected (not silently applied to the field device). This pairs with the S1 failover test. + +**Effort:** S/M. **Risk/blast-radius:** Medium — changes write-admission policy on multi-node clusters; a bug could deny legitimate primary writes during a snapshot gap. Mitigate: the count check is conservative (only denies when a real peer exists) and the single-node path is untouched. + +--- + +## S2 — HIGH — LDAP authentication block-bridges session activation with a non-cancellable, non-configurable timeout + +**Restatement:** The SDK invokes `ImpersonateUser` synchronously; `OpcUaApplicationHost.HandleImpersonation` block-bridges the authenticator with `CancellationToken.None`. The authenticator adds no timeout; the shared library's connect timeout is a default that `LdapOptions.ToLibraryOptions()` neither projects nor exposes. Each auth opens a fresh TCP connect + service bind + search + user bind. A DC outage stalls SDK threads ~10-20s each. + +**Verification — Confirmed.** +- `OpcUaApplicationHost.cs:271-273` — `authenticator.AuthenticateUserNameAsync(…, CancellationToken.None).GetAwaiter().GetResult();` on the SDK impersonation callback thread. +- `LdapOpcUaUserAuthenticator.cs:30-48` — awaits `ldap.AuthenticateAsync(username, password, ct)` with no timeout wrapper; the `ct` it receives from `HandleImpersonation` is `None`. +- `LdapOptions.cs:94-107` — `ToLibraryOptions()` projects Server/Port/Transport/… but **no timeout field**; `LdapOptions` has no `TimeoutMs`/`ConnectTimeout` property at all. +- No connection pooling: each `AuthenticateAsync` is a full connect+bind+search+bind cycle (shared-lib behavior, per report). + +**Root cause:** The timeout responsibility was pushed to the authenticator (per `HandleImpersonation`'s doc comment) but no layer actually enforces one; the shared library's default connect timeout isn't surfaced or bounded per-call, and the call is synchronous under the SDK thread. + +**Proposed design — hard timeout at the OtOpcUa boundary + surfaced option.** The clean fix lives in `LdapOpcUaUserAuthenticator` (OtOpcUa-owned; the shared library's sync-wrapped `AuthenticateAsync` can't be easily made cooperative). Add a `Task.WhenAny(auth, Task.Delay(timeout))` wrapper that **fails closed** (deny) on timeout, and surface `Security:Ldap:TimeoutMs` on `LdapOptions`. Consider a short **negative cache** (deny cache, e.g. 5-30s TTL keyed by username) to shed load during a sustained outage — during a DC-down storm, repeated activations for the same user return the cached deny instantly instead of each stalling for the full timeout. + +Note: `Task.WhenAny` + `Task.Delay` doesn't *cancel* the underlying blocking LDAP call (it observes `CancellationToken.None` only at entry), so the orphaned auth task runs to completion in the background — but the SDK thread is released at `timeout`, which is the goal. Cap concurrent in-flight auths (a `SemaphoreSlim`) so an outage can't accumulate unbounded orphaned tasks. + +**Alternatives considered:** (a) Make the shared library cooperatively cancellable — cross-repo change to `ZB.MOM.WW.Auth`, larger blast radius, and the report frames the fix as OtOpcUa-side. Track as a shared-lib follow-up but don't block on it. (b) Connection pooling in the library — desirable but out of scope for this repo; note it. The boundary timeout + negative cache is the contained, correct-fail-closed fix. + +**Implementation steps.** +1. `LdapOptions.cs` — add `public int TimeoutMs { get; set; } = 10000;` (and optionally a `NegativeCacheSeconds`). Do **not** project into `ToLibraryOptions()` (it's an OtOpcUa-boundary concern) unless the library later exposes a matching field. +2. `LdapOpcUaUserAuthenticator.cs` — inject `IOptions` (or the timeout value); wrap the `ldap.AuthenticateAsync` call in a `Task.WhenAny(authTask, Task.Delay(TimeoutMs, ct))`; on timeout return `OpcUaUserAuthResult.Deny("Authentication timed out")` and log `Warning`. Bound concurrency with a `SemaphoreSlim(maxConcurrent)`. Add the optional negative cache (a `MemoryCache` or a small time-bucketed dictionary). +3. Optionally pass a real (timeout-derived) `CancellationToken` from `HandleImpersonation` instead of `None` so the entry-check short-circuits — but the authoritative bound is the `WhenAny`. +4. Add a validator note (C2): `TimeoutMs > 0`. + +**Tests.** +- **Unit:** authenticator with a stub `ILdapAuthService` that delays beyond `TimeoutMs` → returns Deny within ~`TimeoutMs` (not the full delay); a fast success still succeeds; the negative cache returns a prior deny without re-hitting the stub within TTL; the semaphore bounds concurrency. The existing LDAP fail-closed tests (deny-on-error, opaque messages, zero-role fallback) must stay green. +- **Live/manual:** point `Security:Ldap:Server` at an unreachable host and drive a Client.CLI `connect` with a UserName token; confirm the activation fails fast (~TimeoutMs) rather than hanging ~10s, and that a burst of activations doesn't serialize. + +**Effort:** M. **Risk/blast-radius:** Low-Medium — fail-closed on timeout preserves the existing deny-on-error posture; the risk is a too-short default timeout denying a legitimately-slow DC (default 10s matches the library default, so behavior is unchanged unless configured down). + +--- + +## S3 — HIGH — HistoryRead block-bridges the gateway per node, sequentially, on SDK request threads + +**Restatement:** All four HistoryRead arms call `.GetAwaiter().GetResult()` per node handle with `CancellationToken.None`, bounded only by the gateway client's 30s `CallTimeout`. A request naming N historized nodes against a slow historian holds one SDK request thread up to N × 30s; with `MaxRequestThreadCount=100`, a few misbehaving history clients can exhaust the request pool and degrade all OPC UA services. + +**Verification — Confirmed.** +- `OtOpcUaNodeManager.cs:1795-1807` (`HistoryReadRawModified`), `:1823-1847` (`HistoryReadProcessed`), `:1858-1868` (`HistoryReadAtTime`), `:1886-1934` (`HistoryReadEvents`) — each iterates `nodesToProcess` in a **`foreach` (sequential)**. +- Block-bridge sites: `:2059` (`ServeNode`: `read(source, tagname).GetAwaiter().GetResult()`), `:2170-2171` and `:2202-2203` (Raw + tie-cluster overfetch), `:1902-1911` (Events) — all `CancellationToken.None`. +- `OpcUaApplicationHost.cs:329-331` — `MinRequestThreadCount = 5, MaxRequestThreadCount = 100, MaxQueuedRequestCount = 200`. +- Comment at `:1777-1780` correctly notes the overrides run *outside* the node-manager `Lock`, so this is a thread-pool-exhaustion risk, not a lock-freeze risk. + +**Root cause:** The per-node reads are serialized and unbounded per request; there is no per-request deadline and no server-side cap on concurrent HistoryRead work. + +**Proposed design — bounded per-node parallelism + per-request deadline + concurrency limiter.** +1. **Parallelize per-node reads within a batch, bounded.** Replace the sequential `foreach` with a bounded-concurrency fan-out (`Parallel.ForEachAsync` with `MaxDegreeOfParallelism`, or `Task.WhenAll` over a `SemaphoreSlim`), then block-bridge **once** on the aggregate at the arm boundary. This turns N × 30s into ~ceil(N/P) × 30s worst case. Keep the per-node try/catch so one node's failure still surfaces `Bad` for that node only. +2. **Per-request deadline token.** Derive a `CancellationTokenSource` from a configurable server-side HistoryRead deadline (e.g. `ServerHistorian:HistoryReadDeadline`, default ~60s) and pass it into the gateway calls instead of `CancellationToken.None`, so a single request can't hold a thread indefinitely regardless of per-node `CallTimeout`. +3. **Server-side concurrent-HistoryRead limiter.** A process-wide `SemaphoreSlim` (configurable max, e.g. 8-16) gating entry to the arms so a flood of history clients can't consume more than a bounded slice of the request pool; excess requests wait (or fail with `BadTooManyOperations` if the queue is deep). + +**Alternatives considered:** (a) Fully async HistoryRead — the SDK's `HistoryRead` override surface is synchronous (`void` returns filling `results`/`errors`), so the block-bridge is structural; parallelizing *within* the sync boundary is the achievable win. (b) Reject multi-node history requests — breaks legitimate clients. Bounded parallelism + deadline + limiter is the right combination. + +**Implementation steps.** +1. `OtOpcUaNodeManager.cs` — refactor each arm's `foreach` into a bounded parallel fan-out. Because the arms fill `results[handle.Index]`/`errors[handle.Index]` by index (thread-safe, disjoint indices), parallel writes are safe. Collect the per-node async tasks, `await` them under one `GetAwaiter().GetResult()` at the arm boundary. Add a `MaxHistoryReadConcurrencyPerBatch` field (set from options, mirror `MaxTieClusterOverfetch` at `OtOpcUaServerHostedService.cs:223`). +2. Thread a per-request `CancellationToken` (from a deadline CTS) into `ServeNode`/`ServeRawPaged`/`HistoryReadEvents` in place of `CancellationToken.None`. +3. Add a process-wide `SemaphoreSlim` limiter (field on the node manager or a small injected service) around the arm bodies; make the max configurable. +4. `ServerHistorianOptions` — add `HistoryReadDeadline` (TimeSpan) + `MaxConcurrentHistoryReads` + `MaxHistoryReadConcurrencyPerBatch`. Wire through `OtOpcUaServerHostedService` (same spot as `MaxTieClusterOverfetch`). + +**Tests.** +- **Unit:** with a stub `IHistorianDataSource` that delays each read, a batch of N nodes completes in ~ceil(N/P) × delay (proves parallelism); a per-node throw still yields `Bad` for that node and `Good` for others (regression on the existing per-node isolation); a batch exceeding the deadline cancels and returns `Bad`/`GoodNoData` rather than hanging. +- **Integration/live:** the env-gated live HistoryRead suite (needs the gateway) plus a stress case with a slow stub asserting the request pool isn't exhausted (measure that concurrent non-history services stay responsive while a slow history batch runs). + +**Effort:** M. **Risk/blast-radius:** Medium — parallelizing gateway calls increases peak load on the historian gateway; the concurrency caps bound it. The index-disjoint result writes are safe, but verify no shared mutable state in `ServeRawPaged`'s paging/continuation-point synthesis is touched concurrently (it operates per-handle, but audit the continuation-point cache). + +--- + +## S5 — MEDIUM — Redundancy NodeId identity string-matched across three sources, silent on mismatch + +**Restatement:** `OpcUaPublishActor` matches `n.NodeId == _localNode.Value`; the snapshot derives `host:port` from the gossiped `Member.Address` while the local side derives it from configured `PublicHostname:Port`. A divergence (DNS vs IP, container advertised name) silently computes ServiceLevel 0 / keeps a stale role — the historical "silently inert delivery" bug shape, with no distinguishing log. + +**Verification — Confirmed.** +- `OpcUaPublishActor.cs:512` — `var entry = _lastSnapshot.Nodes.FirstOrDefault(n => n.NodeId == _localNode.Value);` +- `:516-520` — `if (entry is null || entry.Role == Detached) { ServiceLevelChanged(0); return; }` — a **missing entry** is treated identically to legitimate Detached, silently → 0. No log distinguishes the two. +- `RedundancyStateActor.cs:123-134` — snapshot `NodeId` = `ToNodeId(member.Address)` from the **gossiped** `Member.Address`. +- `:141-146` — `ToNodeId` builds canonical `host:port` "the SAME format `ClusterRoleInfo.LocalNode` uses" — but the local side's host comes from configured `PublicHostname`, the snapshot side's from the gossip. They agree only if `PublicHostname` == the gossiped address host. + +**Root cause:** Two independently-derived string identities are compared for equality with no assertion or diagnostic that they actually match. This is the same class as the historical redundancy-state-delivery bug (`project_redundancy_state_delivery.md`). + +**Proposed design — log-once on absence + startup self-check.** This is diagnostic, not behavioral (the S4 fix consumes the same signal). +1. In `OpcUaPublishActor.RecomputeServiceLevel` (and the `DriverHostActor.OnRedundancyStateChanged`, `ScriptedAlarmHostActor` equivalents), when the received snapshot has ≥1 node but **none** equals `_localNode`, log **once** at `Warning`: `local node {LocalNodeId} absent from redundancy snapshot [{snapshotNodeIds}] — check PublicHostname vs gossiped Address`. Log-once (a bool latch) so it doesn't spam every heartbeat. +2. **Startup self-check:** in the cluster bootstrap or `OpcUaPublishActor.PreStart`, compare the locally-derived `ClusterRoleInfo.LocalNode` against `Cluster.Get(system).SelfMember.Address` projected through `ToNodeId`; if they differ, log `Error` (misconfiguration) — fail-fast-ish, this is a deploy config error. +3. Reset the log-once latch when a snapshot *does* contain the local node (so a transient bootstrap gap that self-heals doesn't leave a stale warning impression). + +**Implementation steps.** Touch `OpcUaPublishActor.cs` (RecomputeServiceLevel + PreStart self-check), `DriverHostActor.OnRedundancyStateChanged` (`:1109`), and factor `ToNodeId(SelfMember.Address)` comparison into a shared helper in `ClusterRoleInfo` so the "canonical local node id" derivation is single-sourced (removes the third independent derivation). + +**Tests.** Unit: snapshot with nodes but not the local node → the log-once fires exactly once and ServiceLevel is 0 (distinguishable in logs from Detached); a subsequent snapshot including the local node resets the latch. Startup self-check: mismatched `PublicHostname` vs self-address → Error logged. + +**Effort:** S. **Risk/blast-radius:** Low (diagnostic + one shared helper). + +--- + +## S6 — MEDIUM — Crashed-and-restarted `DriverInstanceActor` loses desired-subscription state; persistent thrower hot-loops + +**Restatement:** No Runtime actor overrides `SupervisorStrategy`, so a driver-child exception triggers Akka's default one-for-one **unlimited, non-backoff** restart. Restart re-runs `PreStart` but the desired-subscription set arrives post-spawn via `SetDesiredSubscriptions` stored in actor state; restart wipes it and `DriverHostActor` has no restart-detection to re-send (restarts don't fire `Terminated`). A persistent thrower hot-loops with no backoff. + +**Verification — Confirmed.** +- Grep for `SupervisorStrategy|BackoffSupervisor` in `src/Server/…/Runtime/` → **zero** hits (confirmed no override → default `OneForOneStrategy` with unlimited restarts, `Directive.Restart`, no backoff). +- `DriverInstanceActor.cs:84` — `SetDesiredSubscriptions` is a message record; `:181-194` — the desired set is stored in actor fields (`_desiredSubscriptions`); `:299 PreStart` re-inits the driver from Props but does **not** request the subscription set; `:861 StoreDesiredSubscriptions` stores it. A restart clears the fields and nothing re-sends. +- No `PostRestart` override that re-requests subscriptions. + +**Root cause:** Message-delivered actor state (desired subscriptions) is not reconstructible from `Props` after a restart, and the parent has no restart-detection to re-push it; plus unlimited non-backoff restart on a persistent fault. + +**Proposed design — BackoffSupervisor + child pulls state in PreStart.** +1. Wrap each `DriverInstanceActor` in a **`BackoffSupervisor`** (`Akka.Pattern.BackoffSupervisor` / `Backoff.OnFailure`) with exponential backoff (e.g. min 1s, max 30s, jitter) and a `SupervisorStrategy` on the child that restarts on transient faults. This kills the hot-loop. +2. Have the child **request its desired-subscription set from the parent in `PreStart`/`PostRestart`** — add a `RequestDesiredSubscriptions(driverInstanceId)` message the child sends to `DriverHostActor` on (re)start; the parent replies with the current desired set (which it already knows — it pushed it originally). This makes the subscription state reconstructible after any restart. Alternatively, have the parent detect the backoff-restart and re-push; the child-pull is cleaner (self-healing, no parent restart-detection needed). + +**Alternatives considered:** (a) Move desired subscriptions into `Props` — they change per-deploy independently of the actor spec, so this couples deploy churn to actor respawn; rejected. (b) Akka.Persistence for the child — overkill; the parent is the source of truth and can re-supply. (c) Parent watches for restart via a custom decider — restarts don't fire `Terminated`, so this needs a bespoke signal; child-pull avoids it. + +**Implementation steps.** +1. `DriverHostActor.cs` — spawn each driver child under a `BackoffSupervisor` Props (or apply `Backoff.OnFailure`) with exp backoff; keep the `(driver, FullName) ⇄ NodeId` maps keyed the same. Handle a new `RequestDesiredSubscriptions` message by replying the current desired set for that driver id. +2. `DriverInstanceActor.cs` — in `PreStart` (covers first start + restart, since Akka calls `PreStart` after `PostRestart`), send `RequestDesiredSubscriptions` to the parent and adopt the reply via the existing `StoreDesiredSubscriptions` path. Re-apply on the `Connected` entry as it already does (`:187`). +3. Ensure the backoff wrapper doesn't break the health-publish + routing (the parent addresses the *supervisor* ref; verify `_children` maps hold the right ref for `Ask`/`Tell` — with BackoffSupervisor you Tell the supervisor which forwards). + +**Tests.** +- **Unit (TestKit) — the U6 gap:** force a driver child to throw; assert (a) it restarts with backoff (not immediately/unbounded), (b) after restart it re-requests and re-adopts the desired subscriptions, (c) a persistent thrower backs off (increasing delay) rather than hot-looping. This is the "no actor supervision/restart test anywhere" gap. +- **Live:** on the docker rig, kill a driver's backing connection to induce faults and confirm the driver recovers subscriptions after a transient throw without redeploy. + +**Effort:** M. **Risk/blast-radius:** Medium — changes the driver-child supervision topology; verify routing/`Ask` still resolves through the supervisor and that the backoff doesn't delay legitimate fast recovery (tune min backoff low). + +--- + +## S7 — MEDIUM — `DriverInstanceActor.PostStop` blocks shutdown on driver shutdown + +**Restatement:** `_driver.ShutdownAsync(CancellationToken.None).GetAwaiter().GetResult()` in `PostStop` is the only synchronous block in the Runtime actors; a hung protocol stack stalls stop/re-deploy of the whole child set. + +**Verification — Confirmed.** `DriverInstanceActor.cs:950` — `try { _driver.ShutdownAsync(CancellationToken.None).GetAwaiter().GetResult(); } catch (…) { log }`. `CancellationToken.None` → unbounded. + +**Root cause:** Unbounded synchronous wait on driver shutdown inside the actor stop path. + +**Proposed design — bound the wait, log-and-abandon on expiry.** Wrap with a timeout: `_driver.ShutdownAsync(cts.Token).WaitAsync(TimeSpan.FromSeconds(N))` where the CTS is created with the same timeout, so both the token and the outer `WaitAsync` cap it. On `TimeoutException`, log `Warning` and abandon (the actor stops regardless; the orphaned shutdown task runs to completion in the background). A configurable `DriverShutdownTimeout` (default ~5s, matching the write ladder's 5s backend bound) is reasonable. + +**Implementation steps.** `DriverInstanceActor.cs:950` — replace with a bounded wait; pass a real cancellation token (from a timeout CTS) into `ShutdownAsync` so cooperative drivers cancel promptly, and `WaitAsync(timeout)` as the hard bound for uncooperative ones. Optionally surface the timeout in options. + +**Tests.** Unit: a stub driver whose `ShutdownAsync` never completes → `PostStop` returns within ~timeout and logs; a fast driver still shuts down cleanly. This complements S6's supervision tests. + +**Effort:** S. **Risk/blast-radius:** Low. + +--- + +## S8 — MEDIUM — Inbound Part 9 commands / native acks are at-most-once; client sees Good even if lost + +**Restatement:** The alarm-command router (`Publish`) and native-ack router (`Tell`) are fire-and-forget, catch-log-drop; the node manager already returned `Good` and applied local condition state. A mediator hiccup or missing `DriverHostActor` silently strands upstream state. Deliberate (non-blocking under `Lock`) but invisible. + +**Verification — Confirmed.** +- `OtOpcUaServerHostedService.cs:139-155` (alarm-command router) and `:167-194` (native-ack router) — both `try { … Tell/Publish … } catch (Exception ex) { _logger.LogWarning(…) }`, no counter/metric. The native-ack `else` branch (no `DriverHostActor`) also only `LogWarning`s (`:181`). +- `OtOpcUaNodeManager.cs:769-772` (referenced) — the Part 9 condition method returns `Good` before/independent of routing. + +**Root cause:** Best-effort routing with only a Warning log — no operator-visible signal (metric/counter) and no reconciliation. + +**Proposed design — add drop counters/metrics + optional reconciliation.** +1. Add OpenTelemetry counters (the codebase uses `OtOpcUaTelemetry` — e.g. `DriverInstanceLifecycle` at `DriverInstanceActor.cs:952`) for `alarm_command_route_dropped` and `native_ack_route_dropped`, tagged by operation/reason (`mediator_fault`, `no_driver_host`). This makes the silent drop observable in metrics/alerting. +2. (Optional, larger) an engine-side reconciliation sweep: periodically the scripted-alarm engine / driver re-asserts authoritative condition state so a dropped command self-heals on the next sweep. Defer unless operationally needed; the metric is the priority. + +**Implementation steps.** `OtOpcUaServerHostedService.cs` — increment the counters in both catch blocks and the native-ack `else`. Add the counter definitions to `OtOpcUaTelemetry`. Surface in the existing metrics dashboard/health if present. + +**Tests.** Unit: inject a throwing mediator accessor / a registry that returns no `DriverHostActor` → the counter increments and no exception escapes into the SDK path. (These routers are wired in `StartAsync`; testable by extracting the router lambdas or via a small harness.) + +**Effort:** S. **Risk/blast-radius:** Low (additive observability). + +--- + +## S9 — MEDIUM — Server certificate has no renewal or expiry monitoring + +**Restatement:** The app cert is auto-created self-signed with SDK defaults (2048-bit, 12-month lifetime), checked only at boot. Nothing monitors expiry; ~12 months after first deploy, Sign/SignAndEncrypt endpoints and UserName-token encryption fail with no warning. + +**Verification — Confirmed.** `OpcUaApplicationHost.cs:305-317` — `EnsureApplicationCertificateAsync` calls `CheckApplicationInstanceCertificatesAsync(false, null, ct)` with `minimumKeySize/lifetimeInMonths: 0` → SDK defaults (comment at `:308` states 2048-bit, 12-month). Boot-only; no periodic check, no expiry metric. + +**Root cause:** No lifecycle monitoring for the server certificate (the AdminUI cert-actions cover *client* certs; the *server* cert is the gap). + +**Proposed design — startup + periodic expiry check with a metric/health-check + runbook.** +1. At startup, after `EnsureApplicationCertificateAsync`, read the resolved cert's `NotAfter` and log `Warning` if within a threshold (e.g. 30 days) and `Info` with the expiry date otherwise. +2. Register a periodic check (a lightweight hosted service or a timer in the existing health pipeline) that emits a health-check/metric (`server_cert_days_to_expiry`) consumed by `MapOtOpcUaHealth` (same surface S10 targets). Degrade `/health` when under threshold. +3. Document a rotation runbook in `docs/security.md` (delete-and-reissue, or a longer `lifetimeInMonths` at issue). Optionally increase the issued lifetime (pass a non-zero `lifetimeInMonths` to `CheckApplicationInstanceCertificatesAsync`) so the reissue cadence is multi-year. + +**Implementation steps.** `OpcUaApplicationHost.cs` — capture the cert `NotAfter` after the check and expose it (property or event); add the threshold log. Add a health contributor + metric. `docs/security.md` — runbook section. + +**Tests.** Unit: a near-expiry cert → the health/metric reports degraded; a fresh cert → healthy. (Injectable via a test cert with a short `NotAfter`.) + +**Effort:** S/M. **Risk/blast-radius:** Low. + +--- + +## S10 — LOW — SDK start failure swallowed; node runs with Null sinks, no health surface + +**Restatement:** `StartAsync` catches, logs, and returns; the node then no-ops all OPC UA work with nothing surfacing the condition through `/health` or ServiceLevel. + +**Verification — Confirmed.** `OtOpcUaServerHostedService.cs:110-129` — `catch { LogError; return; }` (deliberate: AdminUI stays up). The `NodeManager is null` path (`:124-128`) also just warns and returns. No health flag set. + +**Root cause:** The availability trade-off (don't crash the whole binary) leaves no degraded-state signal. + +**Proposed design — set a health flag consumed by `MapOtOpcUaHealth`.** Introduce an `IOpcUaServerHealth` singleton (a simple thread-safe flag holder) that `OtOpcUaServerHostedService` sets to `Faulted`/`Degraded` on the start-failure and `NodeManager is null` paths, and `Healthy` on success. `MapOtOpcUaHealth` reads it so fleet status shows the degraded node. Pairs naturally with S9's cert health contributor. + +**Implementation steps.** New `IOpcUaServerHealth` + impl in `Host`/`OpcUaServer`; inject into `OtOpcUaServerHostedService`; set on each exit path in `StartAsync` and clear on success/`StopAsync`; wire into the health-check mapping. + +**Tests.** Unit: forcing a start failure sets the health flag to degraded; success sets healthy. Health endpoint reflects it. + +**Effort:** S. **Risk/blast-radius:** Low. + +--- + +## S11 — LOW — `AuditWriterActor` unbounded buffer, drops batches on DB outage (moot — no producers) + +**Restatement:** The audit buffer is unbounded between flushes and drops whole batches on a DB outage. Best-effort by contract; currently moot because the pipeline has no producers (U3). + +**Verification — Confirmed** (`ControlPlane/Audit/AuditWriterActor.cs` per report; and U3 confirms zero producers — `AuditOutcomeMapper.cs:12-18` states "no live structured AuditEvent emit sites"). + +**Root cause:** Best-effort design; no backpressure; and the whole pipeline is dormant. + +**Proposed design.** **Fold into U3's decision.** If the audit pipeline is wired to producers (U3), then bound the buffer (drop-oldest with a metered counter, like `SqliteStoreAndForwardSink`'s `_evictedCount`) and consider a store-and-forward durable queue for compliance-grade audit. If the pipeline is deleted (U3 alternative), this is moot. **Do not fix in isolation** — resolve U3 first. + +**Effort:** S (bounding) — gated on U3. **Risk:** Low. + +--- + +## P2 — MEDIUM — Per-value global-lock write, one actor message per value, no batching + +**Restatement:** Each published value takes the global node-manager `Lock` once and flows through one actor message per value, contending with SDK read/subscription/publish threads. At high tag counts × fast polls this serializes everything through one lock and allocates one record per hop. + +**Verification — Confirmed.** +- `DriverHostActor.cs:561-589` (`ForwardToMux`) — per value: dictionary lookup + up to two `Tell`s. +- `OtOpcUaNodeManager.cs:266-281` (`WriteValue`) — takes `lock (Lock)` per value. +- `DependencyMuxActor.cs:95-108` — lean (early-drop, set fan-out), confirmed not the bottleneck. + +**Root cause:** No batching of the sink write; the lock is acquired per value rather than per driver-publish cycle. + +**Proposed design — batched `WriteValues(IReadOnlyList<…>)` sink call.** Add a batch method to `IOpcUaAddressSpaceSink` that takes one `Lock` hold per driver publish cycle and applies all values in the batch. The driver child already receives values in poll-cycle batches upstream; carry the batch shape through `ForwardToMux` → `OpcUaPublishActor` → `sink.WriteValues`. **Respect the U2 forwarding trap:** the new `WriteValues` must be added to `DeferredAddressSpaceSink`, `NullAddressSpaceSink`, and `SdkAddressSpaceSink`, and exhaustively test-guarded — an un-forwarded batch method ships inert (F10b/PR#423 precedent). + +**Alternatives considered:** (a) Finer-grained locking in the node manager — the SDK's `CustomNodeManager2.Lock` is the contract boundary for address-space consistency; sub-locking risks correctness. Batching under the existing lock is safer. (b) Lock-free value store — too invasive. Batching is the contained seam-level win the report recommends. + +**Implementation steps.** Add `WriteValues` to the sink interface + all impls + the deferred wrapper; thread batch shape through `ForwardToMux`/`OpcUaPublishActor`; keep single `WriteValue` for the individual paths (alarm/vtag) or route them through the batch of one. + +**Tests.** Unit: batched write takes the lock once for N values (assert via a recording sink counting lock acquisitions or write calls); U2 forwarding test must include `WriteValues`. Perf: micro-benchmark N values single vs batched. Live: high-tag-count rig to confirm reduced contention (optional). + +**Effort:** M. **Risk/blast-radius:** Medium (hot path; U2 trap). Do after/with P1 since both touch the sink seam. + +--- + +## P3 — MEDIUM — HistoryRead-Events unbounded, no paging + +**Restatement:** `EventMaxEvents` maps `NumValuesPerNode == 0` to `int.MaxValue`, and the Events arm never issues continuation points. A wide window over a busy alarm source materializes the entire result in memory on both gateway and server. + +**Verification — Confirmed.** `OtOpcUaNodeManager.cs:1944-1954` (`EventMaxEvents` → `int.MaxValue` on 0), `:1919-1921` ("never issue continuation points — full window in one shot"). + +**Root cause:** Spec-conformant "no limit" translated to truly unbounded with no server-side backstop. + +**Proposed design — server-side max mirroring `MaxTieClusterOverfetch`.** Impose a configurable server-side max event count for the Events arm (e.g. `ServerHistorian:MaxHistoryReadEvents`, default 65536 to mirror `MaxTieClusterOverfetch`'s philosophy). When the result would exceed it, either **fail loudly** (`BadHistoryOperationInvalid` / a status telling the client to narrow the window) or **page** (synthesize continuation points as the Raw arm already does — larger effort). Start with the bounded-fail backstop (matches the Raw tie-cluster "loud-fail" design at `:2202`), add paging later if clients need it. + +**Implementation steps.** `OtOpcUaNodeManager.cs` — cap `EventMaxEvents`'s unbounded case at the configured max; on overflow return a loud status. Add the option to `ServerHistorianOptions`, wire via `OtOpcUaServerHostedService` (same spot as `MaxTieClusterOverfetch`). + +**Tests.** Unit: `EventMaxEvents(0)` returns the configured cap (not `int.MaxValue`); a result at the cap returns the loud status. Update the existing `EventMaxEvents` unit tests. + +**Effort:** S. **Risk/blast-radius:** Low. + +--- + +## P4 — LOW — Every deploy re-runs the four Materialise passes over the full composition + +**Verification — Confirmed.** `OpcUaPublishActor.cs:338-354` runs all four passes each deploy; acceptable because `EnsureFolder`/`EnsureVariable` early-return on existing ids, but each pass takes/releases `Lock` per node. + +**Proposed design.** **Fold into P2's batching** — a batched ensure (one lock hold per pass) removes the per-node lock churn. No standalone work; note it when implementing P2. The positive designs (Raw tie-cluster overfetch bound, `HandleDiscoveredNodes` unchanged-plan short-circuit at `DriverHostActor.cs:658-673`) are worth preserving. + +**Effort:** S (subsumed by P2). **Risk:** Low. + +--- + +## C1 — MEDIUM — `ServerHistorian` bound imperatively in five places, no `IOptions` + +**Restatement:** `ServerHistorian` is bound in `Program.cs` (own bind + `Validate()`), `AddServerHistorian`, `AddAlarmHistorian`, `AddHistorianProvisioning`, and `OtOpcUaServerHostedService`'s ctor — five `Get<>` sites that can log warnings twice and drift. + +**Verification — Confirmed.** `OtOpcUaServerHostedService.cs:88-90` binds directly (`configuration.GetSection(ServerHistorianOptions.SectionName).Get()`); `Runtime/ServiceCollectionExtensions.cs:86, 132, 168` each re-`Get<>` the section (confirmed the section is read in `AddServerHistorian`/`AddAlarmHistorian`/`AddHistorianProvisioning`); `Program.cs` binds + `Validate()`s. Five sites. + +**Root cause:** No single validated-options registration; each consumer binds independently. + +**Proposed design — one `AddValidatedOptions` + inject `IOptions<>`.** Mirror the `OpcUa`/`Ldap` pattern already in the tree (`Program.cs:102, 254-255`). Register once with `ValidateOnStart` (folds into C2's validator promotion), then inject `IOptions` into `OtOpcUaServerHostedService` and the three `Add*` extensions. This single-sources the section and runs `Validate()` once. + +**Implementation steps.** Add the registration in `Program.cs`; change the five consumers to take `IOptions` (the `Add*` extensions resolve it from the built provider or accept it as a parameter). Remove the per-site `Get<>` calls. + +**Tests.** Unit: options bound once, validated once (assert single warning emission). Existing consumers still function. + +**Effort:** S/M. **Risk/blast-radius:** Low-Medium (touches five wiring sites; verify the `Add*` extensions run after the options registration). + +--- + +## C2 — MEDIUM — Two-tier options validation; `DevStubMode` only log-warned in production + +**Restatement:** `OpcUa`/`Security:Ldap` get fail-fast `ValidateOnStart` validators; `ServerHistorian`/`ContinuousHistorization`/`AlarmHistorian` get only advisory `Validate()` warnings; `DevStubMode=true` (accept-any-credentials Administrator) is merely log-warned in production. + +**Verification — Confirmed.** `OtOpcUaLdapAuthService.cs:79-88` — `if (_options.DevStubMode) { … LogWarning("DevStubMode bypass …"); accept }` with no environment guard (grep for `IsDevelopment`/`EnvironmentName` in the file → none). The historian sections use advisory `Validate()` (C1). + +**Root cause:** Inconsistent validation posture; a dangerous dev bypass isn't environment-gated. + +**Proposed design.** +1. **Promote the historian sections to the validator pattern** (`AddValidatedOptions` + `IValidateOptions<>` with `ValidateOnStart`), folding in C1's single registration. Convert the current `Validate()` warning lists into validator failures (or keep soft warnings for non-fatal knobs but fail on genuinely-invalid combos, e.g. `Enabled=true` with empty `Endpoint`). +2. **Environment-gate `DevStubMode`:** inject `IHostEnvironment`; if `DevStubMode == true && !env.IsDevelopment()`, **fail startup** (throw in a validator) — or at minimum refuse the bypass at runtime (deny + Error log) outside Development. Fail-fast is preferable (a prod deploy with DevStubMode is a critical misconfiguration). + +**Implementation steps.** Add `ServerHistorianOptionsValidator`/`ContinuousHistorizationOptionsValidator`/`AlarmHistorianOptionsValidator` (mirror `OpcUaApplicationHostOptionsValidator`/`LdapOptionsValidator`). Add an `LdapOptionsValidator` (or extend the existing one) rule: `DevStubMode` requires Development. Wire `IHostEnvironment` into the validator. + +**Tests.** Unit: historian validators fail on invalid combos, `ValidateOnStart` surfaces at boot; `DevStubMode=true` outside Development fails startup; inside Development it's allowed (warned). Existing `OpcUa`/`Ldap` validator tests stay green. + +**Effort:** M. **Risk/blast-radius:** Medium — promoting to fail-fast could break a deploy relying on a currently-tolerated invalid config; validate against real appsettings before merge and document the new hard requirements in the migration note. + +--- + +## C3 — LOW — Library code logs via static Serilog; node manager uses obsolete `Utils.LogError` + +**Verification — Confirmed.** `Runtime/ServiceCollectionExtensions.cs:90,100,136` log via static `Log`; node manager uses `Utils.LogError` with `#pragma warning disable CS0618` (e.g. `:1928-1930`, and the report cites `:449-451` et al.). + +**Proposed design.** For the Runtime static-logger sites, inject `ILogger` where an instance is available; where it isn't (pure `Add*` extension methods), accept an `ILoggerFactory`/`ILogger` parameter. For the node manager, wire the acknowledged `ITelemetryContext`/`ILogger` seam so the six `CS0618` `Utils.LogError` sites route through a real logger instead of the obsolete static trace. This is hygiene; the static coupling works only because `Program.cs:305` assigns `Log.Logger` (ordering-sensitive). + +**Effort:** M (node-manager logger wiring is the bulk). **Risk:** Low. Lower priority than the Criticals/Highs. + +--- + +## C4 — LOW — `IHistorianProvisioning` resolve has no missing-registration warning + +**Verification — Confirmed.** `Runtime/ServiceCollectionExtensions.cs:298` — `var provisioning = resolver.GetService();` with a comment noting the Null default, but **no warning** parallel to the evaluator/recorder-deps warnings (`:211-237`). This is the exact seam that shipped dormant in PR #423. The `dispatched=N, requested=0` tally (`AddressSpaceApplier.cs:274-282`) partially compensates. + +**Proposed design.** Add a startup log line at the resolve site: if `GetService()` returns the `NullHistorianProvisioning` singleton while `ServerHistorian:Enabled == true`, log `Warning` ("provisioning enabled but Null provisioner resolved"). Mirror the sibling missing-registration warnings. This is the "startup log proving the wiring" guard from OVERALL theme #1. + +**Effort:** S. **Risk:** Low. + +--- + +## U1 — HIGH (doc drift) — CLAUDE.md "Known Limitation 2" is stale; recorder ref-feed is wired + +**Restatement:** CLAUDE.md says continuous historization "records no values" because the recorder is seeded empty. In code, the applier feeds the delta via `ActorHistorizedTagSubscriptionSink` → `UpdateHistorizedRefs`. + +**Verification — Confirmed stale.** +- `Runtime/ServiceCollectionExtensions.cs:272` — recorder still spawned with `historizedRefs: Array.Empty()`. +- `:291` — `new ActorHistorizedTagSubscriptionSink(continuousRecorder)` is wired. +- `AddressSpaceApplier.cs:213` (`FeedHistorizedRefs`) + `:355-371` (`HistorizedRef`) feed the add/remove delta. +- Restart convergence works via in-memory `_lastApplied` (`OpcUaPublishActor.cs:326-336`): the first post-boot rebuild diffs against empty and emits the full historized set as Added. +- Confirmed independently by OVERALL cross-cutting theme #5. + +**Proposed design — update the doc + add the convergence test.** +1. Update `CLAUDE.md` — rewrite "KNOWN LIMITATION 2" to state the ref-feed is wired; note the load-bearing chain (empty seed + delta feed + in-memory `_lastApplied` → full-set-as-Added on first post-boot deploy). Keep any genuinely-remaining caveat (e.g. the numeric-analog-only v1 limitation) but drop the "records no values" claim. Propagate to `../scadaproj/CLAUDE.md`'s OtOpcUa entry per the repo's cross-repo rule. +2. Add an **explicit restart-convergence test:** boot → apply a full plan (composition with historized tags) → assert the recorder's dependency-mux interest is registered for exactly the historized set (the chain is only implicitly covered today). This retires the "load-bearing but untested" risk. + +**Effort:** S. **Risk:** Low (doc + test only). High-value because it removes a false "known broken" claim that could misdirect future work. + +--- + +## U2 — MEDIUM — Deferred-sink forwarding correct today but only half test-guarded + +**Restatement:** `DeferredAddressSpaceSink` forwards all 10 members correctly (incl. both surgical methods with capability-sniffing), but tests assert forwarding for only ~5/10 members. The F10b incident proves the failure mode is real. + +**Verification — Confirmed.** `DeferredAddressSpaceSink.cs:15` implements `IOpcUaAddressSpaceSink, ISurgicalAddressSpaceSink`; all members forward (`:26-49` for the 8 base members, `:58-69` for the 2 surgical with `_inner is ISurgicalAddressSpaceSink` fallback). Correct today, but per the report only ~5 have per-member forwarding assertions. + +**Proposed design — reflection-driven exhaustive forwarding test.** Add a test that, via reflection, enumerates **every method** of `IOpcUaAddressSpaceSink` + every optional capability interface the wrapper implements (`ISurgicalAddressSpaceSink`, plus any new ones from P1/P2), invokes each on a `DeferredAddressSpaceSink` wrapping a recording inner sink, and asserts the call **reached** the inner sink. This mechanically guarantees no member ships inert — retiring the whole trap class (OVERALL theme #1/#2). Apply the same treatment to `DeferredServiceLevelPublisher`. **This test becomes the gate for P1's remove methods and P2's `WriteValues`.** + +**Implementation steps.** New test in `tests/Server/…/DeferredAddressSpaceSinkTests.cs` (extend the existing file): reflect over interface methods, build default/dummy args per parameter type, invoke, assert the recording sink saw each. Handle the capability-sniffing members (assert they forward when the inner implements the capability, and no-op/return-false safely when it doesn't). + +**Effort:** S/M. **Risk:** Low. Very high leverage — one test permanently closes the "built-but-never-wired forwarding" class. + +--- + +## U3 — MEDIUM — Structured audit pipeline built/tested with zero producers + +**Verification — Confirmed.** `ControlPlane/Audit/AuditOutcomeMapper.cs:12-18` and `Security/Audit/AuditActor.cs:18-24` state there are no live structured `AuditEvent` emit sites (all production audit flows through the bespoke stored-procedure path). + +**Proposed design — decide: wire or delete.** Either (a) wire producers (route the real audit-worthy events — the write-gate audit at `OtOpcUaNodeManager.cs`, alarm ack/shelve, deploy outcomes — through the structured `AuditEvent` pipeline, replacing/augmenting the stored-proc path), or (b) delete the dormant pipeline (`AuditWriterActor`, `AuditOutcomeMapper`, `AuditActor`, tests) to remove decay risk. **Recommendation:** delete unless compliance-grade structured audit is a near-term requirement — the bespoke stored-proc path already serves production, and dormant tested code is the exact "built-but-never-wired" debt the review flags. S11 is subsumed by whichever path is chosen. This is a product decision; surface it to the owner rather than deciding unilaterally. + +**Effort:** M (either path). **Risk:** Low (deletion) / Medium (wiring — new producers touch hot paths). + +--- + +## U4 — MEDIUM — `FleetStatusBroadcaster.DriverHostStatusHeartbeat` dead code with latent NodeId-mismatch + +**Verification — Confirmed** (per report: `FleetStatusBroadcaster.cs:38, 110-121` no producer; `:152-159` keys nodes by **host only** vs the `host:port` canon everywhere else). + +**Proposed design — wire the producer or remove; canonicalize the key regardless.** If fleet-status freshness is wanted, wire `DriverHostActor` (which knows its applied revision) to send `DriverHostStatusHeartbeat` and **canonicalize the broadcaster key to `host:port`** (else a wired `host:port` NodeId creates phantom records — re-enacting the historical NodeId bug, same class as S5). If not wanted, delete the heartbeat path. Either way, fix the key to `host:port`. Recommendation: remove unless the fleet-status freshness signal is used by the AdminUI; the host-only key + no producer signals it was never completed. + +**Effort:** S (remove) / M (wire + key fix). **Risk:** Low. + +--- + +## U5 — MEDIUM — Native Part 9 conditions support Acknowledge-to-driver only + +**Verification — Confirmed** (`OtOpcUaNodeManager.cs:647-658` — Confirm/AddComment/Shelve on a native condition route to the scripted engine which doesn't own them; `:698-703` — Enable/Disable → `BadNotSupported`). Operators shelving a Galaxy alarm via a Part 9 client get silent upstream no-op. + +**Proposed design — route native condition commands to the driver's `IAlarmSource`, or fail loudly.** Extend the native-ack seam (already present: `NativeAlarmAckRouter` → `DriverHostActor.RouteNativeAlarmAck` → driver `AcknowledgeAsync`) to the other Part 9 operations the driver can support (Confirm/AddComment/Shelve/Unshelve) by adding driver-side methods on `IAlarmSource` (mirroring `AcknowledgeAsync`) and routing them like the ack. For operations no driver backend supports (Enable/Disable upstream), return a **loud** status (`BadNotSupported` is already returned for Enable/Disable — keep, but ensure the *silently-ineffective* Shelve/Confirm cases either route or return a status telling the operator it won't propagate upstream). This is documented H6c scope — scope it as a follow-on feature, not a quick fix. + +**Effort:** M/L (driver-side `IAlarmSource` surface expansion + Galaxy gateway support). **Risk:** Medium (touches the driver alarm contract and Galaxy). Lower priority than the availability/perf cluster. + +--- + +## U6 — MEDIUM — Test-coverage gaps concentrate on the fragile seams + +**Restatement:** DPS delivery of redundancy/ServiceLevel state, actor supervision/restart, outbox durability across process restart, hard-kill failover, and the single `OpcUaServer.IntegrationTests` test are all untested; docker-gated protocol tests self-skip to green. + +**Verification — Confirmed** (per report's tests/Server sweep; corroborated by S1/S6 verification — no supervision test, no hard-kill test; and OVERALL theme #4). + +**Proposed design — targeted tests, delivered alongside the fixes above (not as a separate epic).** +- **DPS delivery of redundancy/ServiceLevel** — a real over-the-mediator delivery test (not a stub) on the 2-node harness; assert a `RedundancyStateChanged` published by `RedundancyStateActor` is *received* by `OpcUaPublishActor`/`DriverHostActor` and drives ServiceLevel/role. This is the known "unit tests can't catch it" blind spot (`project_redundancy_state_delivery.md`). +- **Hard-kill failover** — delivered by **S1**'s integration test. +- **Actor supervision/restart** — delivered by **S6**'s TestKit test. +- **Outbox durability across process restart** — a test that appends to the FasterLog outbox, simulates a restart (new outbox instance over the same directory), and asserts un-acked entries drain (continuous-historization path). +- **`OpcUaServer.IntegrationTests` expansion** — subscription-survival-on-add (from **P1**), a security-mode matrix (None/Sign/SignAndEncrypt over the wire), and a HistoryRead round-trip. +- **Make docker skips visible** — align with OVERALL action #6 (fail-on-skip for the integration job); coordinate with report 07. + +**Effort:** M (spread across the fixes). **Risk:** Low (test-only). Track each sub-test with its parent finding so coverage lands with the fix, not after. + +--- + +## U7 / U8 / U9 — LOW — Batched hygiene items + +- **U7** — `IHistoryWriter` permanently Null-wired (`Runtime/ServiceCollectionExtensions.cs:56-58`, infra-gated on a nonexistent live-write RPC); **H2 HistoryUpdate** unimplemented (`OtOpcUaNodeManager.cs:1797-1801`, `IsReadModified` rejected); `LdapAuthFailure` lacks a `DirectoryUnavailable` value (directory-down overloaded onto `ServiceAccountBindFailed`). **Verification — Confirmed** (all backlog/tracked). **Design:** leave `IHistoryWriter`/H2 as tracked backlog (infra-gated). Add the `DirectoryUnavailable` enum value and map directory-down to it in the LDAP path (pairs with S2's outage handling — a distinct failure reason improves the operator signal). **Effort:** S (enum) + backlog (rest). +- **U8** — `BuildSecurityPolicies` doc claims the empty-profile fallback is "logged and very visible" but logs nothing (`OpcUaApplicationHost.cs:376-418`). Defused in prod by the `MinCount ≥ 1` validator but silent for direct embedders. **Verification — Confirmed** (per report). **Design:** either add the log the comment promises (a `Warning` on empty-profile fallback-to-None) or fix the comment. Adding the log is cheap and correct. **Effort:** S. +- **U9** — `EnsureVariable` silently ignores changed historize-intent on an existing node (`OtOpcUaNodeManager.cs:1345-1349`); correct today because the planner routes such deltas elsewhere, but the invariant lives in two comments. **Verification — Confirmed.** **Design:** turn the invariant into an assert/`Debug.Assert` (or a `Warning` log) when `EnsureVariable` is called with a differing historize-intent for an existing id, so a future planner regression surfaces instead of silently no-op'ing. **Effort:** S. + +**Risk (all three):** Low. + +--- + +## Suggested execution order (this report's slice of the OVERALL list) + +1. **S1** — activate SBR + hard-kill failover test (OVERALL #1). *Do first — nothing else matters if failover is broken.* +2. **U1** — update CLAUDE.md (stale Known Limitation 2) + convergence test (OVERALL #11). *Cheap, removes a misleading claim before others build on it.* +3. **U2** — reflection-exhaustive Deferred-sink forwarding test (OVERALL theme #1/#2). *Prerequisite guard for P1/P2 — land it first so P1's remove methods and P2's `WriteValues` can't ship inert.* +4. **S4** — primary-gate default-deny on multi-node (OVERALL #7). *Small, closes the dual-primary data-plane window.* +5. **P1** — surgical pure-adds (phase 1), scoped removals (phase 2) (OVERALL #8). *Highest operational perf/stability win; gated on U2.* +6. **S2 / S3** — async LDAP timeout + channelized/bounded HistoryRead (OVERALL #9). +7. **S5, S6, S7, S8** — redundancy-mismatch diagnostics, BackoffSupervisor + subscription re-pull, bounded PostStop, drop metrics. +8. **C1 + C2** — single validated `ServerHistorian` options + historian validators + `DevStubMode` env-gate. +9. **P2 (+P4), P3, S9, S10** — batched sink writes, Events cap, cert monitoring, start-failure health flag. +10. **U3, U4, U5, C3, C4, U7-U9** — dormant-code decisions, native Part 9 expansion, and hygiene, as capacity allows. diff --git a/archreview/plans/04-adminui-plan.md b/archreview/plans/04-adminui-plan.md new file mode 100644 index 00000000..5a7d3a0b --- /dev/null +++ b/archreview/plans/04-adminui-plan.md @@ -0,0 +1,292 @@ +# Architecture Review 04 — AdminUI: Design + Implementation Plan + +- **Source report:** `archreview/04-adminui.md` (commit `9cad9ed0`) +- **Plan author date:** 2026-07-08 · verified against current tree at `9cad9ed0` +- **Scope:** AdminUI pages, ScriptAnalysis backend, tag editors, authorization +- **Repo constraint that shapes every test recipe:** there is **no bUnit**. Razor `@code`/binding logic and page-level `[Authorize]` gating are only observable by live-verify on the docker-dev rig (AdminUI `http://localhost:9200`). *And* on docker-dev **login is disabled** (`Security:Auth:DisableLogin=true` → `AutoLoginAuthenticationHandler` grants `DevAuthRoles.All` = every role incl. Administrator/Designer/Operator). **Consequence: a Viewer-denied path cannot be observed on the default rig** — negative authz must be proven by (a) a pure policy-registration unit test, (b) a reflection test over page attributes, and (c) a manual real-login pass with a Viewer LDAP bind. This is called out in every authz recipe below. + +## Verification summary + +| ID | Sev | Status | Evidence | +|---|---|---|---| +| C-1 | High | **Confirmed** | 30+ pages carry fully-qualified bare `@attribute [Microsoft.AspNetCore.Authorization.Authorize]`; only `Deployments`/`Scripts`/`ScriptEdit` use `Roles="Administrator,Designer"`, `RoleGrants` uses `Policy="FleetAdmin"`, `Certificates`/`Alerts`/`DriverStatusPanel` use imperative `"DriverOperator"`/`"FleetAdmin"`. `FallbackPolicy` is only `RequireAuthenticatedUser()`. | +| C-2 | Med | **Confirmed** | `ScriptAnalysisEndpoints.cs:17-18` = `Roles="Administrator,Designer"`; `CLAUDE.md:223` + `docs/plans/2026-06-11-adminui-disable-login-design.md:141` say "FleetAdmin". | +| C-5 | Med | **Confirmed** | `ScriptEdit.razor` / `Deployments.razor` / `Fleet.razor` / `Hosts.razor` inject `IDbContextFactory` and run EF + `SaveChangesAsync` in `@code`; UNS pages use `IUnsTreeService`. | +| S-1 | Med | **Confirmed** | `Fleet.razor:171` + `AlarmsHistorian.razor:90` sync `Dispose() => _timer?.Dispose()`; `DriverTestConnectButton.razor:69` `System.Timers.Timer` with `async (_,_)` `Elapsed` + sync `Dispose`. (`DriverStatusPanel`/`Alerts`/`Hosts` already correct via `System.Threading.Timer`+`DisposeAsync`.) | +| S-2 | Med | **Confirmed** | `Fleet.LoadAsync` (119-159) `try/finally` no `catch`; `AlarmsHistorian` (73-78) bare `catch {}` leaving "Loading…". | +| S-3 | Med | **Confirmed** | `grep ErrorBoundary` = none; `MainLayout.razor:53` `@Body` unwrapped. | +| S-4 | Med | **Confirmed** | `GlobalUns.ConfirmDeleteAsync` re-`Load…Async` then passes fresh `.RowVersion`; `EquipmentPage.DeleteTag/VirtualTag/Alarm` (370/415/460) same ("Load … fresh to capture its current RowVersion"). | +| S-5 | Med | **Confirmed** | `EquipmentPage.razor:172/217/263` single-click `@onclick="() => DeleteTag(...)"`; `ScriptEdit.DeleteAsync` single-click. `GlobalUns` has a confirm modal; these don't. | +| S-6 | Low | **Partial** | `DriverStatusPanel` *does* drain via `DisposeAsync` (287-310), but `ShowOpResult` replaces the chip timer with a sync `Dispose()` and no token guard → an already-queued callback can clear a newer message. Race is real but narrow. | +| S-7 | Low | **Confirmed** | `AdminOperationsClient.cs:58-59` `AskAsync` forwards only caller `ct`; typed methods double-guard with `AskTimeout`. | +| P-1 | Med | **Confirmed** | `monaco-init.js:205-207` `onDidChangeModelContent` → `invokeMethodAsync("OnValueChanged", editor.getValue())` no debounce; diagnostics ARE debounced (500 ms). | +| P-2 | Med | **Confirmed** | `ScriptAnalysisService.Analyze` (60-67) re-`ParseText` + `CSharpCompilation.Create` per call; refs/preamble static (good), no `(text→tree)` memo. | +| P-3/P-4/P-5 | Low | **Confirmed** (spot-checked) | Per-circuit polling / `SnapshotAndFlatten` per Deployments render / no `@key` on `Alerts` rows — accepted as bounded. | +| P-6 | Low | **Confirmed** | `monaco-init.js:99-117` registers inlay-hints provider that POSTs every change; `InlayHints` endpoint always empty. | +| C-3 / C-4 | Good | **Confirmed good** | Tag-editor map covers 7 drivers + Galaxy raw path; all driver pages carry `JsonStringEnumConverter`, pinned by `*FormSerializationTests`. No action. | +| U-1..U-4 | — | **Confirmed** | No bUnit; documented stubs; unvalidated raw-JSON fallback; thin a11y. | + +**Nothing in this report is stale/already-fixed.** All actionable findings reproduce at `9cad9ed0`. One nuance correction: the task brief mentioned a `DriverAdmin` policy — **no such policy exists**; only `FleetAdmin` and `DriverOperator` are registered (`Security/ServiceCollectionExtensions.cs:155-160`). + +--- + +## Priority order + +1. **C-1** (High, overall action-item #5) — gate the mutating surface; standardize on one policy idiom with constants. +2. **C-2** (Med) — folds into C-1 (converge ScriptAnalysis gate + fix the doc). +3. **S-4 + S-5** (Med) — delete concurrency + confirmation. +4. **S-3** (Med) — `ErrorBoundary`. +5. **S-1 + S-2** (Med) — timer/error-handling convergence. +6. **P-1** (Med) — debounce Monaco value push. +7. **P-2** (Med) — ScriptAnalysis compilation memo. +8. **C-5** (Med) — declare service-seam canonical; migrate `ScriptEdit`. +9. **Batch Lows** — S-6, S-7, P-3/4/5, P-6, C-6, U-2, U-3, U-4. + +--- + +## 1. C-1 (High) — Standardize authorization; gate the mutating surface + +**Restated:** Three authz idioms coexist and the largest mutating surface (UNS, equipment, cluster/node/namespace/ACL editors, all 8 driver pages, Reservations) carries only bare `[Authorize]`, so any authenticated user — incl. a read-only Viewer — can create/edit/delete config. Only the deploy/scripts pages are role-gated. + +**Verification (confirmed):** +- Policies defined: `Security/ServiceCollectionExtensions.cs:143-161` — `FleetAdmin = RequireRole("Administrator")`, `DriverOperator = RequireRole("Operator","Administrator")`, `FallbackPolicy = RequireAuthenticatedUser()`. No "write" policy exists. +- Roles: `AdminRole` enum = `Viewer`, `Designer`, `Administrator` (`Configuration/Enums/AdminRole.cs`); `Operator` is an appsettings-only control-plane role (`DevAuthRoles.cs:14`). +- Idiom census (all string literals, no constants): + - `Roles="Administrator,Designer"`: `Deployments.razor:12`, `Scripts.razor:2`, `ScriptEdit.razor:5`, `ScriptAnalysisEndpoints.cs:18`. + - `Policy="FleetAdmin"`: `RoleGrants.razor:2` (page), `Certificates.razor:90` (`AuthorizeView`), `Certificates.razor:186` (imperative). + - Imperative `"DriverOperator"`: `Alerts.razor:165`, `DriverStatusPanel.razor:166`, `GalaxyAddressPickerBody.razor:124`, `OpcUaClientAddressPickerBody.razor:75`. + - **Bare `[Authorize]`** (= FallbackPolicy, any authenticated user): `GlobalUns`, `EquipmentPage`, `ClusterEdit`, `NodeEdit`, `NamespaceEdit`, `NewCluster`, `AclEdit`, `ClusterAcls/Namespaces/Drivers/Overview/Audit/Redundancy`, `ClustersList`, all 8 driver pages + `DriverEditRouter` + `DriverTypePicker`, `Reservations`, plus the read-only dashboards. + +**Root cause:** Pages were authored by copying the nearest neighbor; the fused Host's `FallbackPolicy` makes bare `[Authorize]` *look* protective (you must be logged in) while providing zero role separation. No policy-name constants exist to make the correct gate discoverable, so drift is the path of least resistance. Project posture (memory: roles are global, "simplest authz") was never actually applied to the mutating pages. + +### Proposed design + +**Standardize on the policy idiom with named constants**, and introduce one write policy. + +1. **Add a policy-constants type** in the Security project (co-located with the definitions), e.g. `Security/Auth/AdminUiPolicies.cs`: + ```csharp + public static class AdminUiPolicies + { + public const string FleetAdmin = "FleetAdmin"; // RequireRole(Administrator) + public const string DriverOperator = "DriverOperator"; // RequireRole(Operator, Administrator) + public const string ConfigEditor = "ConfigEditor"; // NEW: RequireRole(Administrator, Designer) + } + ``` + Keep the string *values* identical to today's literals so nothing else has to change atomically. + +2. **Register the new `ConfigEditor` policy** in `AddOtOpcUaAuth` (`ServiceCollectionExtensions.cs:143`): + ```csharp + o.AddPolicy(AdminUiPolicies.ConfigEditor, p => p.RequireRole("Administrator", "Designer")); + ``` + `ConfigEditor` is exactly the semantic already spelled out four times as `Roles="Administrator,Designer"` — so `Deployments`/`Scripts`/`ScriptEdit`/`ScriptAnalysisEndpoints` converge onto it too, collapsing idiom #1 into idiom #2. + +3. **Apply `[Authorize(Policy = AdminUiPolicies.ConfigEditor)]`** to every *mutation-dedicated* page. Read-only dashboards keep the FallbackPolicy (bare `[Authorize]`). + + **Gate with `ConfigEditor` (mutation pages):** + `GlobalUns`, `EquipmentPage`, `ClusterEdit`, `NodeEdit`, `NamespaceEdit`, `NewCluster`, `AclEdit`, `ClusterRedundancy`, `DriverEditRouter`, `DriverTypePicker`, all 8 driver pages (`Modbus/S7/AbCip/AbLegacy/TwinCAT/Focas/OpcUaClient/Galaxy`), `Reservations`, plus converge `Deployments`/`Scripts`/`ScriptEdit`. + + **Keep FleetAdmin:** `RoleGrants` (already), `Certificates` mutating actions (already per-action). + + **Leave read-only (FallbackPolicy):** `Home`, `Fleet`, `Hosts`, `Alerts` (view; the Ack/Shelve buttons stay imperative-`DriverOperator`), `ScriptLog`, `AlarmsHistorian`, `Account`, `ClusterOverview`, `ClustersList`, `ClusterAudit`, `Certificates` (view). + +4. **Mixed list pages** (`ClustersList`, `ClusterAcls`, `ClusterNamespaces`, `ClusterDrivers`) contain both read content and destructive buttons (New/Edit/Delete). Two options: + - **(A, recommended) Keep the page viewable, wrap destructive controls in ``** so a Viewer sees the fleet but can't mutate. Preserves read access, matches least-surprise. Slightly more than one attribute per page. + - **(B, minimal) Page-level `ConfigEditor`** — a Viewer loses read access to those lists entirely. Cheaper, but hides visibility a Viewer arguably should have. + + Recommend **(A)** for the four list pages, **page-level `ConfigEditor`** for the pure edit/create forms and driver pages. This is the correct read/write split for a global-role model. + +5. **Convert the imperative/AuthorizeView literals to the constants** (`Alerts`, `DriverStatusPanel`, both pickers, `Certificates`, `RoleGrants`) — mechanical: `"DriverOperator"` → `AdminUiPolicies.DriverOperator`, etc. In razor markup use `Policy="@AdminUiPolicies.ConfigEditor"`. + +**Alternatives considered / rejected:** +- *Keep role-strings everywhere* — rejected: the canonical role rename (`ConfigEditor→Designer`, `FleetAdmin→Administrator`) already shows role strings are brittle; policies decouple the gate from the role vocabulary in one place. +- *Fine-grained per-action policies (WriteTag/WriteCluster/…)* — rejected as over-engineering against the deliberate global-role posture (memory: "roles are global; simplest authz that works"). +- *Move gating into the `IUnsTreeService`/EF layer* — good defense-in-depth but out of scope here and doesn't fix the surface-level page problem; note as a future hardening. + +### Implementation steps + +1. Add `Security/Auth/AdminUiPolicies.cs` (constants). +2. `ServiceCollectionExtensions.cs`: register `ConfigEditor`; switch the two existing `AddPolicy` names to the constants. +3. Edit page `@attribute` lines (one line each) for the mutation pages listed above → `[Authorize(Policy = AdminUiPolicies.ConfigEditor)]`. Add `@using ZB.MOM.WW.OtOpcUa.Security.Auth` to `Components/_Imports.razor` so the constant resolves in every razor file. +4. Wrap destructive controls in the four list pages with `` (option A). +5. Replace imperative/AuthorizeView literals with constants (`Alerts`, `DriverStatusPanel`, pickers, `Certificates`, `RoleGrants`, `ScriptAnalysisEndpoints`). +6. Confirm `AutoLoginAuthenticationHandler` still satisfies `ConfigEditor` (it grants `DevAuthRoles.All` incl. `Administrator`+`Designer` — yes, no dev-rig regression). + +### Tests + live-verify + +- **Unit (Security.Tests, CI-runnable, cheap):** build a `ServiceProvider` via `AddOtOpcUaAuth`, resolve `IAuthorizationService`, and assert the `ConfigEditor` policy: + - `ClaimsPrincipal` with role `Viewer` only → **denied**. + - with `Designer` → allowed; with `Administrator` → allowed; with `Operator` only → denied. + This is the only automated proof of the *policy semantics* (mirrors the existing `Configuration.Tests/AuthorizationTests.cs` style). +- **Reflection guard (AdminUI.Tests, CI-runnable) — closes the "built-but-never-wired" gap for authz:** enumerate all page component types (`typeof(App).Assembly` types with a `[Route]` attribute), and assert every type in a hard-coded `MutatingPages` set carries an `AuthorizeAttribute` whose `Policy == AdminUiPolicies.ConfigEditor` (and that no mutating page has a bare `[Authorize]`). This is the cheapest possible substitute for bUnit and catches a new page copying the wrong idiom — directly implements OVERALL cross-cutting theme #1 ("assert production wiring"). +- **Live-verify (docker-dev, manual, negative path needs real login):** + 1. Positive smoke on the default rig (auto-admin): confirm the gated pages still load and mutate — deploy a config, open `/uns`, add/delete a tag, open a driver page. Auto-login grants all roles so everything must still work (regression check that the gate didn't over-block admins). + 2. **Negative path requires disabling auto-login.** Set `Security:Auth:DisableLogin=false`, bring the AdminUI up against the shared GLAuth (`10.100.0.35:3893`), bind a **Viewer** LDAP user (group→`Viewer` via `Security:Ldap:GroupToRole`), and verify: `/uns`, `/uns/equipment/*`, `/clusters/*/edit`, driver pages, `/reservations` all render the "You do not have permission" `NotAuthorized` slot (from `Routes.razor:19`); the four list pages render but hide New/Edit/Delete. Then bind a **Designer** user and confirm access is restored. Document this recipe in the PR — it is the only way to observe the deny, because the default rig can't. + +**Effort:** M. **Risk/blast-radius:** Medium — touches ~25 page files but each change is one line; the real risk is *over-gating a read-only dashboard* (hiding it from Viewers) or *under-gating a mutation page*. The reflection guard + the manual Viewer pass are the mitigations. No runtime/data-plane impact (OPC UA data-plane auth is independent LDAP). + +--- + +## 2. C-2 (Med) — ScriptAnalysis gate doc/code drift + +**Restated:** `CLAUDE.md` says `/api/script-analysis/*` is "gated by the `FleetAdmin` policy"; code uses `Roles="Administrator,Designer"`. + +**Verification (confirmed):** `ScriptAnalysisEndpoints.cs:17-18` `RequireAuthorization(new AuthorizeAttribute { Roles = "Administrator,Designer" })`; `CLAUDE.md:223` + design plan `…adminui-disable-login-design.md:141` claim FleetAdmin. The original monaco plan (`2026-06-09-monaco-script-editor.md:309`) *intended* `RequireAuthorization("FleetAdmin")` but the implementation diverged to match the Scripts page. + +**Root cause:** implementation deliberately matched the Scripts/ScriptEdit page gate (which is `Administrator,Designer`) but the docs were written from the original plan and never reconciled. + +**Design/fix:** fold into C-1. When `ScriptAnalysisEndpoints` converges onto `AdminUiPolicies.ConfigEditor` (= `Administrator,Designer`), update the doc to state the truth: **script-analysis and the Scripts/ScriptEdit/Deployments pages are gated by the `ConfigEditor` policy (Administrator or Designer)**, distinct from `FleetAdmin` (Administrator-only). This also fixes OVERALL cross-cutting theme #5 for this claim. + +**Implementation:** edit `CLAUDE.md:223`; the design-plan files are historical (leave, or add a one-line "superseded" note). **Tests:** the C-1 policy unit test covers the endpoint's effective gate; add one endpoint-level assertion in `ScriptAnalysis` tests if a `WebApplicationFactory` is already in use there (check first — the suite is service-level today). + +**Effort:** S. **Risk:** trivial (doc + one converged literal). + +--- + +## 3. S-4 + S-5 (Med) — Delete concurrency + confirmation + +**Restated:** Delete paths re-fetch a *fresh* RowVersion at click time (defeating optimistic concurrency → last-writer-wins), and `EquipmentPage`/`ScriptEdit` deletes fire on a single click with no confirmation. + +**Verification (confirmed):** +- S-4: `GlobalUns.ConfirmDeleteAsync` (Area/Line/Equipment branches) `Load…Async` → `Delete…Async(id, loaded.RowVersion)`; `EquipmentPage.DeleteTag/DeleteVirtualTag/DeleteAlarm` (370/415/460) each comment "Load … fresh to capture its current RowVersion". *Update* paths correctly carry the modal-load RowVersion (`EquipmentPage.razor:555`), so the contract is genuinely inconsistent. Note `ScriptEdit.DeleteAsync` is the exception — it uses `_form.RowVersion` (the page-load value), so it is *not* a fresh-refetch; S-4 applies to GlobalUns + EquipmentPage only. +- S-5: `EquipmentPage.razor:172/217/263` `@onclick="() => DeleteTag(...)"` immediate; `ScriptEdit` delete button immediate. `GlobalUns` has a confirm modal (`GlobalUns.razor:71-98`); `DriverStatusPanel` has an inline Restart confirm. + +**Root cause:** the list DTOs the rows render from don't carry RowVersion, so the delete handlers re-load to get *a* RowVersion — silently making delete unconditional. Confirmation was added to GlobalUns/DriverStatusPanel but not propagated to the per-equipment tables or ScriptEdit. + +### Proposed design + +- **S-4 — carry the rendered RowVersion:** add `RowVersion` (byte[]/base64) to the tag/vtag/alarm *list* DTOs returned by `IUnsTreeService` (they already exist for the edit path; extend the list projections). The delete handlers pass the row's RowVersion straight into `Delete…Async` — no re-load. Then a concurrent edit between render and click yields the existing `DbUpdateConcurrencyException` → surfaced as "changed by another user", matching the update contract (first-writer-wins). Same for GlobalUns Area/Line/Equipment (the tree nodes would carry RowVersion, or keep GlobalUns re-load but **document it as intentional** — the tree is a coarser surface). + - *Alternative:* explicitly document delete as unconditional last-writer-wins. Rejected for EquipmentPage (inconsistent with its own updates); acceptable as a documented fallback for the GlobalUns tree if threading RowVersion through the tree DTO is disproportionate. +- **S-5 — shared confirm:** extract a small reusable `ConfirmButton`/`ConfirmModal` component (generalize the `GlobalUns` confirm modal) and use it for `EquipmentPage` tag/vtag/alarm deletes and `ScriptEdit` delete. Reduces duplication and gives every destructive action a consistent guard. + +### Implementation steps + +1. Extend list DTOs in `Uns/UnsTreeService.cs` + `IUnsTreeService.cs` (tag/vtag/alarm list projections) with `RowVersion`. +2. `EquipmentPage` delete handlers: drop the fresh-load, pass `row.RowVersion`. +3. Add `Components/Shared/ConfirmModal.razor` (extract from GlobalUns) or a `ConfirmButton`; wire into `EquipmentPage` (×3) and `ScriptEdit`. +4. Optionally thread RowVersion into the GlobalUns tree node DTO, or add a one-line "delete is unconditional here" doc-comment. + +### Tests + live-verify + +- **Unit (UnsTreeService.Tests — the strong seam):** list projections now include RowVersion; add a test asserting `DeleteTagAsync(id, staleRowVersion)` returns a concurrency failure (the service already has RowVersion tests to mirror). +- **Live-verify:** on docker-dev, open `/uns/equipment/{id}` Tags tab, click Delete → confirm modal appears → cancel leaves the tag → confirm deletes. For concurrency: open the equipment in two tabs, edit+save a tag in tab A, then Delete it in tab B → expect the "changed by another user" error, not a silent delete. Repeat for ScriptEdit delete-confirm. + +**Effort:** M. **Risk:** Low-Medium — DTO shape change ripples to a few call sites; the confirm-modal extraction is additive. + +--- + +## 4. S-3 (Med) — No `ErrorBoundary` + +**Restated:** A single unhandled exception in any handler/render tears down the whole circuit; there's no `ErrorBoundary` anywhere in a 77-component console. + +**Verification (confirmed):** `grep ErrorBoundary` = none; `MainLayout.razor:53` renders `@Body` bare inside `ThemeShell`. + +**Root cause:** never added; the shared `ThemeShell` chassis doesn't provide one. + +**Design:** wrap `@Body` in `MainLayout` in an `` with an error view + a "Reload page" / `Recover()` action; on error, log via the injected logger and show a themed panel instead of the framework's error UI. Consider a second, tighter boundary around the live-tail panels (`Alerts`, `ScriptLog`, `DriverStatusPanel`) so a bridge/render fault there doesn't blank the dashboard — but the single `MainLayout` boundary is the high-value 80%. + +**Implementation:** edit `MainLayout.razor` `ChildContent` to `…` + recover button; add a `_boundary.Recover()` on navigation if desired. + +**Tests + live-verify:** no bUnit — live-verify by temporarily throwing in a page handler (or triggering a known-faulting path) and confirming the boundary panel + Recover works instead of a dead circuit. Leave a note that this can't be unit-tested. + +**Effort:** S. **Risk:** Low (purely additive resilience). + +--- + +## 5. S-1 + S-2 (Med) — Timer disposal + refresh error handling + +**Restated:** Three components don't drain in-flight timer callbacks and `Fleet`/`AlarmsHistorian` mishandle refresh errors. + +**Verification (confirmed):** +- S-1: `Fleet.razor:171` + `AlarmsHistorian.razor:90` sync `Dispose() => _timer?.Dispose()`; `DriverTestConnectButton.razor:66-81` `System.Timers.Timer` + `async (_,_) => {…}` Elapsed (effectively async-void) + sync `Dispose`. The house pattern (`System.Threading.Timer` + `IAsyncDisposable`) is used correctly in `DriverStatusPanel`/`Alerts`/`Hosts`. +- S-2: `Fleet.LoadAsync` `try/finally` no `catch` (a faulted `_ = InvokeAsync(LoadAsync)` is unobserved); `AlarmsHistorian` bare `catch {}` → permanent "Loading…". + +**Root cause:** three stragglers predate the async-dispose convention; `Fleet` copied the timer-schedule but not the `Hosts` catch-log-degrade shape. + +**Design:** converge all three timers onto `System.Threading.Timer` + `public async ValueTask DisposeAsync()` that awaits `_timer.DisposeAsync()` (and unsubscribes first, per the `DriverStatusPanel` template). For `DriverTestConnectButton`, replace `System.Timers.Timer`/`Elapsed` with a `System.Threading.Timer` one-shot (`Timeout.InfiniteTimeSpan` period) — kills the async-void. Give `Fleet.LoadAsync` a `catch (Exception ex) { _error = …; Log … }` (copy `Hosts.LoadConfigAsync`) and add an explicit `_error` surface. Give `AlarmsHistorian` an explicit "no historian on this node role" state instead of infinite "Loading…" (also closes U-2's admitted TODO). + +**Implementation:** edit the three components; make them `@implements IAsyncDisposable`; add error fields + rendering. Reuse the exact `DriverStatusPanel.DisposeAsync` shape. + +**Tests + live-verify:** live-verify on docker-dev — open `/fleet`, `/alarms-historian` on an admin-only vs driver node and confirm graceful states; navigate away rapidly during a refresh tick to confirm no disposed-component `StateHasChanged` warnings in the circuit log. Not unit-testable without bUnit. + +**Effort:** S. **Risk:** Low. + +--- + +## 6. P-1 (Med) — Debounce Monaco value push + +**Restated:** `OnValueChanged(editor.getValue())` crosses the SignalR circuit on every keystroke and re-renders the parent. + +**Verification (confirmed):** `monaco-init.js:205-207` fires `invokeMethodAsync("OnValueChanged", …)` inside `onDidChangeModelContent` with no debounce; diagnostics use a 500 ms `setTimeout` debounce already. + +**Root cause:** the value sync was wired for immediacy; the .NET side only consumes it for Save + the inline problems panel, neither of which needs per-keystroke fidelity. + +**Design:** debounce the value push ~200-300 ms (reuse the existing `diagTimer` debounce shape → add a `valueTimer`), *or* push on the diagnostic tick + on blur/save. Ensure Save reads the latest value (either via a final flush on blur/save-trigger or by having Save request `editor.getValue()` through a JS interop call synchronously). Keep the model-content event driving diagnostics as-is. + +**Implementation:** edit `wwwroot/js/monaco-init.js` (`onDidChangeModelContent`), add a debounced value push + a flush path invoked before Save. Verify `ScriptEdit` + the virtual-tag modal Save both see current text. + +**Tests + live-verify:** live-verify on docker-dev `/scripts/{id}` — type rapidly and confirm (network panel) `OnValueChanged` fires at debounce cadence not per keystroke; save and confirm the persisted source matches the editor. No unit coverage (JS interop). + +**Effort:** S. **Risk:** Low-Medium — the one hazard is a lost final edit if the flush-before-save is missed; the flush path must be verified. + +--- + +## 7. P-2 (Med) — Memoize ScriptAnalysis compilation + +**Restated:** Each of the six endpoints re-parses + creates a fresh `CSharpCompilation` for the same text during a typing burst. + +**Verification (confirmed):** `ScriptAnalysisService.Analyze` (60-67) `CSharpSyntaxTree.ParseText(full)` + `CSharpCompilation.Create(...)` per call; `Sandbox.References`/`Preamble`/`CompileOptions` are static (already cached). + +**Root cause:** no `(normalized text → tree/compilation/model)` memo; a 500 ms diagnostics tick immediately followed by completions/hover/signature-help all recompile identical text. + +**Design:** add a **size-1 LRU** (or a tiny `MemoryCache` keyed on the normalized source hash) inside `ScriptAnalysisService` returning the `(Tree, Compilation, Model, PreambleLength)` tuple, shared by all six endpoints. Size-1 collapses the common completions-after-diagnostics case (same text) at negligible memory. Bound it and note it's single-admin-scale; flag before multi-user. Keep it thread-safe (`lock`/`ConcurrentDictionary`) since the service is a shared registration. + +**Implementation:** wrap `Analyze(userSource)` with a cache check keyed on the source (or a SHA of it) → last-value cache field. No API change; endpoints unchanged. + +**Tests + live-verify:** **unit (ScriptAnalysis.Tests — a strong existing seam):** assert two successive `Analyze`/diagnose+complete calls on identical text produce identical diagnostics (correctness preserved) and that the cache returns the same compilation instance (a memoization assertion via an internal hook). Live-verify: type in the editor, confirm completions/hover latency drops and diagnostics stay correct. + +**Effort:** S. **Risk:** Low — correctness risk is stale cache after an edit; keying on the full text (or hash) eliminates it. + +--- + +## 8. C-5 (Med) — Service-seam vs EF-in-page duality + +**Restated:** UNS pages go through the testable `IUnsTreeService`; `ScriptEdit`/`Scripts`/`Deployments`/`Fleet`/`Hosts` run EF + `SaveChangesAsync` directly in `@code`, untestable without a host. + +**Verification (confirmed):** `ScriptEdit.razor` `DeleteAsync`/save use `DbFactory` + `db.Scripts.Remove` + `SaveChangesAsync` in-page; UNS pages use `Svc`. `IUnsTreeService.UpdateScriptSourceAsync` already exists (`IUnsTreeService.cs:502`). + +**Root cause:** new pages copy the nearest file; both idioms are "blessed" by precedent. + +**Design:** **declare the `IUnsTreeService` service-seam canonical for all config mutation.** Migrate `ScriptEdit`'s save+delete into the service (an `UpdateScriptSourceAsync` already exists; add `CreateScriptAsync`/`DeleteScriptAsync` if missing) so the logic gains the same RowVersion + test coverage the UNS methods have. `Fleet`/`Hosts` are *read-only* projections — leave their EF-in-page (or move to a thin read service later); the priority is the *mutating* EF-in-page (`ScriptEdit`, and the `Deployments` deploy trigger, which already has service seams). This dovetails with S-4 (list DTOs) and OVERALL theme #4 (verification). + +**Implementation:** add/confirm `IUnsTreeService` script CRUD methods; rewrite `ScriptEdit` save/delete to call them; keep read pages as-is with a note. + +**Tests:** the migrated script CRUD gets unit coverage in `UnsTreeService.Tests` (delete-with-stale-RowVersion, create, update-source) — turning previously untestable `@code` into covered seam logic. Live-verify script create/edit/delete on `/scripts`. + +**Effort:** M. **Risk:** Low-Medium — behavior-preserving refactor of one page; regression risk mitigated by the new seam tests. + +--- + +## 9. Batched Lows / Underdeveloped + +Group into one cleanup PR (or fold opportunistically into the above): + +- **S-6 (Low):** `DriverStatusPanel.ShowOpResult` — add the `Alerts`-style stale-fire token compare so a queued chip-clear timer can't wipe a newer result. (`DisposeAsync` drain is already correct.) *Effort S.* +- **S-7 (Low):** `AdminOperationsClient.AskAsync` — wrap with a linked CTS + `AskTimeout` to match the three typed methods' contract. *Effort S.* +- **P-3/P-4 (Low):** per-circuit dashboard polling + `Deployments` `SnapshotAndFlattenAsync` per render. Defer; optionally back with `IMemoryCache` keyed on the deploy event before any multi-viewer rollout. Document as accepted-at-current-scale. +- **P-5 (Low):** add `@key` to `Alerts` rows (it `Insert(0, …)` + renders without `@key`, forcing whole-table diffs). *Effort S.* +- **P-6 (Low):** stop registering the Monaco inlay-hints provider until the endpoint is non-empty (`monaco-init.js:99-117`); it round-trips for a documented no-op (`ScriptAnalysisService.InlayHints` always returns empty). *Effort S.* +- **C-6 (Low):** replace the `"GalaxyMxGateway"` literal (`TagModal.razor:311`) with the driver-type constant; dedupe the `DataTypes` array; normalize `TwinCat`/`FOCAS`/`Focas` casing. *Effort S.* +- **U-2 (stubs):** `AlarmsHistorian` role message (folded into S-2); `MonacoEditor.MarkersChanged` `object[]` — model the DTO; `GlobalUns` default-branch "not yet available" message — reword to not leak not-implemented posture. *Effort S each.* +- **U-3 (raw-JSON fallback):** in `TagModal.SaveAsync`, `JsonDocument.Parse` the fallback textarea before submit so malformed JSON is caught client-side instead of at deploy. `_form.TagConfig` is `[Required]` but never validated for well-formed JSON. *Effort S — good defensive win; add a validator unit test.* +- **U-1 / U-4:** strategic, not this-PR. **U-1:** either adopt bUnit for the modal/table state machines, or keep extracting `@code` into plain classes (`HostsDriverView`/`VirtualTagModalHelpers` precedent) — the reflection authz guard (§1) is a cheap down payment. **U-4:** targeted a11y pass on `UnsTree` expander `aria-expanded`/labels, modal focus-trap/Escape, `aria-live` on `Alerts`, table `scope`/captions — do if operator diversity matters. + +**Effort:** S each; **Risk:** Low across the batch. + +--- + +## Cross-cutting notes for the implementer + +- **The docker-dev auto-login (`DisableLogin=true`) makes every negative-authz test impossible on the default rig** — it grants `DevAuthRoles.All`. Every authz claim in this plan must be proven by the policy unit test + reflection guard (CI) *and* one manual real-login Viewer pass (§1 recipe). Do not accept "it loads for me on docker-dev" as authz verification. +- **No bUnit** means S-1/S-2/S-3/S-5/P-1 are only observable by live `/run`. Prefer, wherever cheap, extracting logic into plain classes (C-5 direction) so the untested `@code` residue stays trivial — this is the repo's stated posture and OVERALL theme #4. +- **The reflection authz guard (§1) is the single highest-leverage test in this report**: it converts "which policy is on which page" from a live-only fact into a CI invariant, directly countering the "built-but-never-wired" house failure mode (OVERALL theme #1). +- Sequencing: land **C-1 + C-2 together** (shared constants/policy), then **S-4/S-5**, **S-3**, **S-1/S-2**, then the perf pair **P-1/P-2**, then **C-5**, then the Lows batch. C-1 and C-5 both touch page files — do C-1 first (attributes) to avoid churn. diff --git a/archreview/plans/05-protocol-drivers-plan.md b/archreview/plans/05-protocol-drivers-plan.md new file mode 100644 index 00000000..63649abb --- /dev/null +++ b/archreview/plans/05-protocol-drivers-plan.md @@ -0,0 +1,808 @@ +# Design + Implementation Plan — 05 Protocol Drivers + +**Source report:** `archreview/05-protocol-drivers.md` +**Review commit:** `9cad9ed0` (== current HEAD, so no drift) +**Verification date:** 2026-07-08 +**Scope:** Modbus, S7, AbCip, AbLegacy, TwinCAT, FOCAS, OpcUaClient (+ Contracts, + Cli) + +--- + +## Verification summary + +Every finding in the report was opened at its cited file:line and checked against +the current tree. **37 actionable findings; 37 CONFIRMED; 0 stale.** (STAB-13 is a +"positive" observation carrying no work.) The line references were accurate to +within 1–2 lines throughout. The zero-drift result is expected: the review commit +is the current HEAD. + +Confirmed counts by section: Stability 12 actionable (STAB-1…12) + 1 positive +(STAB-13); Performance 7 (PERF-1…7); Conventions 8 (CONV-1…8); Underdeveloped 9 +(UNDER-1…9). + +**Top 3 by priority** (align with OVERALL actions #3/#4 and report remediation +order): +1. **STAB-1 — S7 no reconnect path** (Critical). `Plc.OpenAsync` only at + `S7Driver.cs:165`; grep confirms it is the *sole* open site, no `EnsureConnected`. +2. **STAB-2 — TwinCAT native subscriptions orphaned after reconnect** (Critical). + `NativeSubscription` (`TwinCATDriver.cs:517-519`) stores only disposable handles, + not replayable intent; `EnsureConnectedAsync` swaps the client + (`:614-644`) and nothing re-runs `AddNotificationAsync`; `AdsTwinCATClient.Dispose` + clears `_notifications` (`:426`). +3. **STAB-4 + STAB-5 — silent-corruption class.** AbCip runs Read→GetStatus→Decode + on shared libplctag handles with no per-runtime lock (`AbCipDriver.cs:544-575`; + only `GetRmwLock` exists, bit-writes only), while AbLegacy guards the identical + hazard with `GetRuntimeLock` (`AbLegacyDriver.cs:786`, used at `:227`/`:331`). + FOCAS's fixed-tree caches are plain `Dictionary` (`FocasDriver.cs:1184,1190,1192,1194`) + read/written across threads. + +The two systemic threads the report calls out — **theme #2** (fixes don't flow back +to shared templates) and **theme #6** (equipment-tag/TagConfig paths are +second-class) — are both real and are given a dedicated consolidation section +(Part B) so the fleet fixes land in one home rather than being re-patched per driver. + +--- + +# Part A — Per-driver Criticals & Highs + +## A1. STAB-1 (Critical) — S7 has no reconnect path + +**Restatement:** the S7 `Plc` is opened once in `InitializeAsync`; a transient PLC +reboot/network blip permanently kills the instance until redeploy. + +**Verification:** CONFIRMED. `grep OpenAsync S7Driver.cs` → single hit at line 165, +inside `InitializeAsync`. No `EnsureConnected`/`Reconnect` symbol. Reads/writes call +`RequirePlc()` under `_gate` and reuse the dead handle +(`S7Driver.cs:479-483, 995-1001`). The probe loop detects the outage and transitions +`Stopped` but never recycles. `ReinitializeAsync` (`:212`) just re-calls `InitializeAsync` +(externally driven only). Zero S7 reconnect tests (`tests/Drivers/.../S7.Tests` has no +reconnect file; only `Modbus`, `OpcUaClient` do). + +**Root cause:** the `Plc` is a concrete `S7.Net` type `new`-ed inline +(`S7Driver.cs:156`, field at `:122`), with no factory seam and no lazy-connect +gate. The template calls for a lazy `EnsureConnectedAsync` (TwinCAT/AbCip have one; +S7 never got it). + +**Design (mirror TwinCAT + OpcUaClient):** +- Introduce a testable seam: `IS7Plc` (wrapping the members S7Driver uses: + `OpenAsync`, `Close`, `ReadBytesAsync`/`ReadMultiple`, `Write*`, `IsConnected`) + and `IS7PlcFactory` with `Create(cpu, host, port, rack, slot)`. Default impl wraps + `S7.Net.Plc`; the ctor takes an optional `IS7PlcFactory? = null` defaulting to the + real one — exactly TwinCAT's `ITwinCATClientFactory` pattern + (`TwinCATDriver.cs:20,58,64`). +- Add `private async Task EnsureConnectedAsync(CancellationToken ct)` guarded + by the existing `_gate`: fast-path returns a live `Plc`; under the gate, if the + `Plc` is null or a prior socket-level `PlcException` marked it dead, dispose and + re-`OpenAsync` a fresh instance (with the same timeout-linked CTS as `:163-164`). +- Route `ReadAsync`/`WriteAsync`/`PollOnceAsync` through `EnsureConnectedAsync` + instead of `RequirePlc()`. On a socket-level `PlcException` during a read/write, + null the `Plc` (mark dead) so the next call reopens, and set `Degraded` (keep + `LastSuccessfulRead`) rather than `Faulted`. Preserve the existing `Faulted` + escalation for PUT/GET-denied (`S7Driver.cs:921-1006`) — that is a permanent + config fault, not a transient drop. +- Classify socket-level vs protocol-level `PlcException` in a small + `IsS7ConnectionFatal(Exception)` helper (S7.Net surfaces `ErrorCode.ConnectionError` + / `WriteData` / `ReadData` / IO/socket exceptions). Do not reopen on a + data-address/type error. + +**Alternatives considered:** (a) let the probe loop own recycle-on-Stopped — rejected +as sole mechanism because reads between probe ticks would keep failing on the dead +handle; the lazy `EnsureConnectedAsync` repairs on the very next data call and the +probe becomes a backstop. (b) Reconnect handler à la OpcUaClient's +`SessionReconnectHandler` — overkill; S7.Net has no session-transfer semantics, a +plain reopen is correct. + +**Implementation steps:** +1. New file `Driver.S7/IS7Plc.cs` (+ `S7PlcAdapter`, `IS7PlcFactory`, + `S7PlcFactory`). Change `S7Driver.Plc` field type to `IS7Plc?`. +2. Thread `IS7PlcFactory` through the ctor + `S7DriverFactoryExtensions.CreateInstance`. +3. Add `EnsureConnectedAsync` + `IsS7ConnectionFatal`; convert the two `RequirePlc` + call clusters and `PollOnceAsync` (`:1295-1309`). +4. Optionally: on probe transition to `Stopped`, null the `Plc` so the next data + call reconnects even if no read failed in between. + +**Tests:** +- Unit (new `S7DriverReconnectTests.cs`): fake `IS7PlcFactory` whose `IS7Plc` throws + a connection-fatal `PlcException` on read N, succeeds on read N+1; assert the driver + creates a *second* `IS7Plc`, health goes `Degraded`→`Healthy`, and the tag recovers + to Good. Assert a protocol error (illegal address) does **not** trigger a reopen. +- Integration (`S7.IntegrationTests`, skip-gated on `Snap7ServerFixture`): subscribe, + bounce the snap7 container (`lmxopcua-fix down s7 s7_1500` / `up`), assert values + resume without redeploy. Note: no per-test container hook exists, so gate this as a + manual/`[Trait("Category","Reconnect")]` recipe. + +**Effort:** M. **Risk:** Medium — touches the hot read/write path; the `IS7Plc` seam +is mechanical but broad. Mitigated by the fake-factory unit tests (which also close +the TEST-1 gap). + +## A2. STAB-2 (Critical) — TwinCAT native subscriptions orphaned after reconnect + +**Restatement:** native ADS notifications (the *default* mode) silently stop after a +client swap — no Bad status, no error. + +**Verification:** CONFIRMED. `SubscribeAsync` registers `AddNotificationAsync` once +per tag (`TwinCATDriver.cs:472`) and stores the resulting handles in +`NativeSubscription(Handle, Registrations)` (`:495, :517-519`) — **no** record of the +symbol/type/interval/handler needed to replay. `EnsureConnectedAsync` disposes a +stale client and creates a fresh one (`:614-644`); `AdsTwinCATClient.Dispose` clears +`_notifications` (`:426`). Compounding: the fast path keys on `IsConnected` (`:606`), +the local AMS-port state, not wire liveness; and a probe failure only transitions +state, never forces a client recycle (`ProbeLoopAsync :540-547`). + +**Root cause:** subscription state is stored as *opaque handles* rather than +*replayable intent*, and there is no "client changed → replay" hook. Native ADS is +push, so unlike the poll fallback it cannot self-heal by re-reading. + +**Design:** +- Change `NativeSubscription` to store replayable registrations: + `record NativeRegistrationIntent(string DeviceHostAddress, string Reference, + string SymbolName, TwinCATDataType Type, int? BitIndex, TimeSpan Interval, + Action OnChange)` plus the live `ITwinCATNotificationHandle`. +- Give `DeviceState` an epoch/generation counter incremented every time + `EnsureConnectedAsync` installs a *new* client (`:644`). After installing a new + client, replay every `NativeRegistrationIntent` whose `DeviceHostAddress` matches: + call `AddNotificationAsync` again and swap the stored live handle. Do this inside + `EnsureConnectedAsync` under `ConnectGate` so it cannot race a concurrent subscribe. +- Fix the liveness signal: the probe should call a real read (`ProbeAsync` already + does a symbol read) and, on failure, dispose+null the device client so + `EnsureConnectedAsync` rebuilds — i.e. make probe-detected outage force a recycle + (`ProbeLoopAsync` catch block at `:541-545` should null `device.Client`). Consider + keying the fast path (`:606`) additionally on a "known-good since last probe" flag + so a locally-connected-but-wire-dead port still triggers the rebuild-and-replay. + +**Alternatives:** switch the default to the poll fallback (self-healing) — rejected; +native push is the performance-correct default and the report explicitly wants replay, +not abandonment. + +**Implementation steps:** +1. Introduce a per-device `ConcurrentDictionary>` + (or hang intents off `DeviceState`). Populate in `SubscribeAsync`. +2. Extract the per-tag registration into `RegisterNotificationAsync(device, intent)` + used by both initial subscribe and replay. +3. In `EnsureConnectedAsync`, after `device.Client = client` (`:644`), iterate the + device's intents and replay; log a re-registration count. +4. In `ProbeLoopAsync`, on probe failure null the device client (force rebuild). + +**Tests:** +- Unit (new `TwinCATReconnectReplayTests.cs`) using the existing fake + `ITwinCATClientFactory`/`ITwinCATClient`: subscribe a tag, simulate a client that + reports `IsConnected=false`, force `EnsureConnectedAsync` to build a replacement, + assert `AddNotificationAsync` is re-invoked with the same symbol/type/interval and + that a subsequent fake push reaches `OnDataChange`. Assert the old handle is disposed. +- Unit: probe-failure recycles the client (assert a new client instance built). +- Note there is **no** TwinCAT docker fixture (13 integration tests need a real TC3 + XAR); the fake-client unit tests are the only automated coverage — make them + authoritative. + +**Effort:** M. **Risk:** Medium — concurrency around `ConnectGate`/replay; must not +double-register. The fake client makes it fully unit-testable (the report's key gap). + +## A3. STAB-3 (High) — Modbus transport desyncs after a response timeout + +**Restatement:** a per-op timeout or TxId mismatch leaves the single-flight socket +half-read; the next txn reads a stale response and the stream stays desynchronized. + +**Verification:** CONFIRMED. `IsSocketLevelFailure` (`ModbusTcpTransport.cs:257-261`) +matches only `EndOfStreamException`/`IOException`/`SocketException`/ +`ObjectDisposedException`. The per-op deadline (`:226-227` linked CTS `CancelAfter`) +surfaces as `OperationCanceledException`; the TxId guard throws `InvalidDataException` +(`:234-235`); the truncated-length guard throws `InvalidDataException` (`:237`) — none +of which are socket-level, so the reconnect-and-retry catch (`:155-165`) is skipped +and the socket is retained mid-desync. + +**Root cause:** the transport conflates "protocol-layer error, keep the socket" with +"framing/timeout error, the socket is unusable." Timeout and any framing violation +leave unknown bytes in the receive buffer; the connection is fatally desynchronized +even though the exception type is not `IOException`. + +**Design:** treat per-op timeout and any framing violation as **connection-fatal**: +tear the socket down *before* propagating, so the next `SendAsync` reconnects clean. +- In `SendOnceAsync`, wrap the read/write in a try that, on `OperationCanceledException` + where the *caller's* `ct` is NOT cancelled (i.e. the timeout linked-CTS fired), and + on `InvalidDataException`, calls `TearDownAsync()` then rethrows a normalized + `ModbusTransportDesyncException : IOException` (subclassing IOException lets the + existing `:155` catch optionally reconnect-and-retry once). Distinguish caller + cancellation (propagate as OCE, no teardown) from timeout (teardown). +- Simplest correct implementation: after `TearDownAsync`, rethrow as `IOException` + so both the retry classifier and the desync-repair share one path. + +**Alternatives:** add timeout/framing to `IsSocketLevelFailure` directly — works but +must still tear down the socket first (the current retry path does `TearDownAsync` +before reconnect, so adding the types *and* ensuring teardown covers it). Preferred to +keep `IsSocketLevelFailure` about exception *type* and add an explicit teardown at the +throw site for framing/timeout, since those need teardown even when auto-reconnect is +off. + +**Implementation steps:** +1. In `SendOnceAsync`, catch `OperationCanceledException` (timeout branch) and + `InvalidDataException`; `await TearDownAsync()`; rethrow as `IOException` + (`ModbusTransportDesyncException`). +2. Keep the outer `SendAsync` single-retry (`:155-165`) — it now covers the desync. +3. Ensure teardown even when `_autoReconnect` is false (the socket must not be reused + desynced regardless). + +**Tests:** +- Unit (`ModbusTcpReconnectTests.cs` extension): a fake stream that (a) stalls past + the deadline, (b) returns a wrong TxId, (c) returns a truncated header. Assert each + triggers `TearDownAsync` (socket disposed) and the following call reconnects. Assert + a caller-cancellation (external ct) does NOT tear the socket down. +- Integration (`Modbus.IntegrationTests` + `exception_injector`): the existing rig + already forces FC exceptions; add a timeout scenario (non-responding unit) and + assert recovery on the next poll. + +**Effort:** S. **Risk:** Low-Medium — must get the timeout-vs-cancellation distinction +right or a legitimate shutdown could look like a desync (harmless, just an extra +reconnect). + +## A4. STAB-4 (High) — AbCip concurrent ops on shared libplctag handles, no lock + +**Restatement:** AbCip read/group/write run Read→GetStatus→Decode / +Encode→Write→GetStatus on cached `IAbCipTagRuntime` with zero serialization while the +server read path, poll loops, and the alarm-projection loop can hit the same handle. + +**Verification:** CONFIRMED. `ReadSingleAsync` (`AbCipDriver.cs:544-575`), +`ReadGroupAsync` (`:614-637`), `WriteAsync` (`:708-726`) take no runtime lock; the only +locks are `GetRmwLock` (bit-in-DINT writes, `:1243`) and a creation lock. AbLegacy +documents the exact hazard and guards it: `GetRuntimeLock` (`AbLegacyDriver.cs:786`), +applied on both read (`:227`) and write (`:331`), comment "A libplctag `Tag` handle is +not safe for concurrent Read/GetStatus/DecodeValue." `AbCipAlarmProjection.cs:149/228` +reads through the unlocked path. libplctag's `GetStatus` returns the *last* operation's +status, so concurrent Read/GetStatus can cross-attribute results → torn values with +Good status. + +**Root cause:** the AbLegacy fix never propagated to AbCip (theme #2). AbCip caches a +per-tag runtime but never serializes operations on it. + +**Design:** port AbLegacy's per-runtime operation lock verbatim. +- Add `GetRuntimeLock(string tagName)` to AbCip's `DeviceState` (a + `ConcurrentDictionary` keyed by tag name, matching + `AbLegacyDriver.cs:786`). +- Wrap the whole Read/GetStatus/Decode critical section in `ReadSingleAsync` and + `ReadGroupAsync`, and the Encode/Write/GetStatus section in `WriteAsync`, in + `await opLock.WaitAsync(ct)` / `finally Release`. The existing `GetRmwLock` (bit RMW) + must nest *inside or replace-by* the runtime lock consistently — since RMW does + read-modify-write on the same handle, take the runtime lock for the whole RMW too + (this also fixes STAB-11's "direct parent write bypasses the lock" for AbCip: a + direct parent write and an RMW now serialize on the same per-tag runtime lock). + +**Alternatives:** a single per-device lock — rejected; too coarse, serializes +independent tags. Per-runtime lock matches AbLegacy and preserves cross-tag parallelism. + +**Implementation steps:** +1. Add `_runtimeLocks` + `GetRuntimeLock` to AbCip `DeviceState`. +2. Wrap the three op bodies. Ensure eviction-on-failure (`EvictRuntime`, `:554`) also + disposes/removes the tag's lock or leaves it (a fresh runtime under the same name + reuses the lock — fine). +3. Reconcile RMW: make `WriteBitInDIntAsync` acquire the per-runtime lock instead of + (or in addition to) `GetRmwLock`; collapse to one lock domain per parent to also + close STAB-11. + +**Tests:** +- Unit (new `AbCipRuntimeConcurrencyTests.cs`, mirror `AbLegacyRuntimeConcurrencyTests`): + a fake `IAbCipTagRuntime` that asserts no overlapping Read/GetStatus (e.g. a reentrancy + counter that throws if >1); hammer concurrent `ReadAsync`+`WriteAsync`+alarm-read on + one tag; assert serialization and no torn/cross-attributed status. +- Integration (`abcip` fixture on `:44818`): concurrent subscribe + write to overlapping + tags, assert Good values stay coherent. + +**Effort:** S. **Risk:** Low — additive locking, blast radius is AbCip only. + +## A5. STAB-5 (High) — FOCAS fixed-tree caches are plain Dictionaries mutated across threads + +**Restatement:** `LastFixedSnapshots`/`LastServoLoads`/`LastSpindleLoads`/`LastTimers` +are plain `Dictionary`, written by `FixedTreeLoopAsync` and read on `ReadAsync` threads. + +**Verification:** CONFIRMED. Fields at `FocasDriver.cs:1184` (`Dictionary`), +`:1190` (`Dictionary = []`), `:1192` +(`Dictionary`), `:1194` (`Dictionary = []`). Writes at +`:752, :793, :836-839`; reads at `:912, :920, :945`. Contrast the neighbouring +`_parsedAddressesByTagName = new ConcurrentDictionary` (`:40`) — that field got the +treatment, these did not. Concurrent write-during-`Dictionary`-resize is UB (throw or +corruption). + +**Root cause:** partial application of the concurrent-collection fix (theme #2). + +**Design:** convert the four caches to `ConcurrentDictionary<,>`. `TryGetValue` and +indexer-set are the only operations used, both supported. `LastTimers` and +`LastSpindleLoads` collection-initializer `[]` becomes `new()`. +- Alternative (immutable snapshot swap): build a fresh dictionary per loop pass and + `Volatile.Write` the reference; readers `Volatile.Read`. Cleaner isolation but more + churn; `ConcurrentDictionary` is the lower-risk minimal change and matches the + in-tree precedent (`:40`). + +**Implementation steps:** change the four field types (on `DeviceState`) + collection +initializers; no call-site changes needed (indexer + `TryGetValue` are identical). + +**Tests:** +- Unit (`FocasDriver.Tests`): a stress test spinning `FixedTreeLoopAsync`-style writes + while readers call `TryReadFixedTree`; assert no exception over N iterations (would + reliably throw on the plain `Dictionary` today). + +**Effort:** S. **Risk:** Very low — type-only change, FOCAS-local. + +## A6. STAB-6 (High) — AbLegacy probe-loop shutdown race; S7 unsubscribe race + +**Verification:** CONFIRMED both. +- AbLegacy: probe task fire-and-forgot `_ = Task.Run(...)` (`AbLegacyDriver.cs:118`); + `ShutdownAsync` (`:156-170`) cancels then immediately disposes `ProbeCts` + runtimes + without awaiting loop exit — a winding-down loop can raise `OnHostStatusChanged` and + touch a disposed CTS. AbCip already fixed this: retained `ProbeTask` (`:298`) + phased + shutdown with `await probeTask.WaitAsync(10s)` (`:332-365`). +- S7: `UnsubscribeAsync` (`:1185-1193`) cancels + disposes the CTS without draining + `PollTask`; `Task.Delay` on the disposed source can throw `ObjectDisposedException` + the loop only catches as OCE (`:1233-1234`). `ShutdownAsync` *does* drain PollTask + (`:239`), so the bug is confined to the `Unsubscribe` path. + +**Root cause:** cancel-then-dispose-without-await; `PollGroupEngine.StopState` +(`PollGroupEngine.cs:99-112`) already solves this by awaiting the loop (bounded) before +disposing the CTS — neither bespoke path adopted it. + +**Design:** +- AbLegacy: retain the probe task in `DeviceState.ProbeTask` and phase shutdown like + AbCip (`await probeTask.WaitAsync(bounded)` before `ProbeCts.Dispose()`). +- S7: in `UnsubscribeAsync`, `Cancel()`, then `state.PollTask.Wait(bounded)` (or await), + then `Cts.Dispose()` — mirroring `StopState`. Better: once S7 migrates to + `PollGroupEngine` (see B1) this deletes itself. + +**Tests:** unit — subscribe/unsubscribe rapidly in a loop and assert no +`ObjectDisposedException` escapes; shutdown while a probe tick is mid-transition. + +**Effort:** S. **Risk:** Low. + +## A7. UNDER-1 (High) — FOCAS advertises writes it cannot perform + +**Verification:** CONFIRMED. `WireFocasClient.WriteAsync` returns +`FocasStatusMapper.BadNotWritable` for every address (`WireFocasClient.cs:71-73`), yet +`FocasDriverOptions.Writable` defaults `true` (`FocasDriverOptions.cs:168`) and +`FocasEquipmentTagParser` hard-codes `Writable: true` (`FocasEquipmentTagParser.cs:35`). +The driver's write exception-mapping (`FocasDriver.cs:353-381`) is dead code against +this backend. Net: equipment tags surface as Operate-writable OPC UA nodes whose writes +always fail. + +**Root cause:** the writable default was written speculatively (doc cites "write tests") +but the only real backend is read-only. + +**Design (force writable-false at the seams until PMC writes exist):** +- `FocasEquipmentTagParser.cs:35` → `Writable: false`. +- `FocasDriverOptions.Writable` default → `false` (and/or force it false at discovery + materialization). Keep the option field for the future PMC-write feature but do not + advertise writable nodes today. +- Leave the write status-mapping in place (it becomes live if/when PMC writes ship). +- Alternative (larger): implement PMC writes in `WireFocasClient` via `pmc_wrpmcrng` — + a separate feature, out of scope for a hardening pass. + +**Tests:** unit — assert a FOCAS equipment/authored tag materializes without the +`CurrentWrite` access bit; assert a write attempt returns `BadNotWritable` (guards the +regression if someone flips the default back). + +**Effort:** S. **Risk:** Very low (removes a false capability). + +## A8. UNDER-2 / UNDER-3 (High) — OpcUaClient: dead `UnsMappingTable` gate; broken alarm-filter encoding + +**UNDER-2 verification:** CONFIRMED. `ValidateNamespaceKind` hard-fails Equipment-kind +configs lacking `UnsMappingTable` (`OpcUaClientDriver.cs:335-338`) but grep shows the +table is read nowhere except that gate and the options DTO +(`OpcUaClientDriverOptions.cs:169,188`). `DiscoverAsync` renders refs purely from remote +browse via `StableReference` (`:808-809`). Operators must author a table with zero +runtime effect. + +**UNDER-3 verification:** CONFIRMED. Persisted refs are stable `nsu=` strings +(`:808-809`), but the alarm source filter is an ordinal `HashSet` of caller refs +(`:1337`) matched against `EventFields[SourceNode].ToString()` — a session-relative +`ns=N;…` rendering (`:1538-1539`). The two encodings never match, so any non-empty +filter drops every event; only empty-filter subscriptions work. + +**Root cause:** UNDER-2 is a validation gate for an unbuilt feature; UNDER-3 is an +encoding mismatch between the stored `nsu=` form and the runtime `ns=N` form. + +**Design:** +- UNDER-2: **delete the gate** (remove the Equipment-kind `UnsMappingTable` mandatory + check) — the honest minimal change, since the feature does not exist. Keep the option + DTO field (forward-compatible) but stop forcing operators to author dead config. + Alternative (build the feature): have `DiscoverAsync` translate remote refs → UNS refs + via the table — a real feature, larger, defer. +- UNDER-3: normalize both sides to the same encoding before compare. Convert the + incoming `EventFields[SourceNode]` `NodeId` to the stable `nsu=` reference via the + driver's `NamespaceMap.ToStableReference` (the same function used at `:808-809`) before + the `HashSet.Contains` check; build the filter set from stable refs. Then a non-empty + filter works. + +**Implementation steps:** remove `ValidateNamespaceKind`'s Equipment branch (or make it +a no-op warning); in `OnEventNotification`, map the source `NodeId` through +`_namespaceMap` before comparing (`:1538-1539`); build `sourceFilter` from stable refs +(`:1337`). + +**Tests:** unit — event with a known source arrives, non-empty stable-ref filter +matches and the event is delivered (today it is dropped); Equipment config without a +mapping table no longer throws at validation. + +**Effort:** S (both). **Risk:** Low — OpcUaClient-local; UNDER-3 fix is a behavior +change that *enables* previously-dead filtering (verify no consumer depended on the +"filter silently drops all" bug). + +## A9. STAB-13 (Positive) — no action + +Write-outcome surfacing is honoured by every protocol driver (per-item `WriteResult` +with typed exception→status), and OpcUaClient's reconnect machine is exemplary +(double-arm guard, re-arm on exhaustion, session swap under `_probeLock`, +`:1908-2058`). CONFIRMED. Use OpcUaClient's machine as the reference for A1/A2. One +caveat surfaced: `Session.TransferSubscriptionsOnReconnect` is **left unset** +(mentioned only in the doc at `:1948`), so subscription transfer rests on the SDK +default — worth explicitly setting it (folds into UNDER-9 cleanup). + +--- + +# Part B — Shared-template consolidation (themes #2 and #6) + +The report's cross-cutting message: *every* High is a fix present in one sibling and +absent in another. These four items give the fleet a single home for the fixes so they +stop being re-patched per driver. Do these **after** the per-driver criticals but treat +them as the highest-leverage structural work. + +## B1. PollGroupEngine v2 — absorb S7's fork; wire `onError`; add backoff (CONV-1, STAB-9, STAB-8, PERF-3, STAB-6-S7) + +**Verification of the pieces:** +- `PollGroupEngine` already has the `onError` sink (`PollGroupEngine.cs:36,55-67,164-169`), + `StopState` draining (`:99-112`), and structural array compare (`:203-209`). It does + **not** have failure backoff — `PollLoopAsync` uses a fixed `Task.Delay(state.Interval)` + (`:131`). +- **STAB-9 CONFIRMED:** no consumer passes `onError` — the five constructions + (`ModbusDriver.cs:115`, `AbCipDriver.cs:133`, `AbLegacyDriver.cs:65`, + `TwinCATDriver.cs:69`, `FocasDriver.cs:72`) omit it, so the sink is dead fleet-wide. +- **CONV-1 / PERF-3 / STAB-6-S7 CONFIRMED:** S7 re-ships the loop (`S7Driver.cs:1165-1307`), + which *has* backoff (`ComputeBackoffDelay :1284-1293`) and failure-health + (`HandlePollFailure :1258-1274`) the engine lacks, but *lacks* the structural array + diff (uses `Equals(lastSeen?.Value, current.Value)` `:1304` → arrays fire every tick) + and the drain-before-dispose (`UnsubscribeAsync :1189-1190`). + +**Design (single home for the poll loop):** +1. **Add optional backoff to `PollGroupEngine`.** Port S7's `ComputeBackoffDelay` + (capped exponential, ticks-based, saturating at a `PollBackoffCap`) into the engine. + Constructor gains `TimeSpan? backoffCap = null` (null ⇒ current fixed-cadence + behavior, preserving every current consumer). `PollLoopAsync` tracks + `consecutiveFailures`, resets on success, and delays `ComputeBackoffDelay(interval, + failures)`. +2. **Route failures to health.** The engine already forwards to `onError`; make each of + the five drivers pass an `onError` that degrades `_health` (keeping + `LastSuccessfulRead`) + logs — i.e. lift S7's `HandlePollFailure` into a small shared + `Action` per driver. Consider making `onError` **required** (the report + suggests it) once all five pass it. +3. **Migrate S7 onto the engine and delete the fork.** Replace `S7Driver`'s + `_subscriptions` dict + `PollLoopAsync`/`PollOnceAsync`/`ComputeBackoffDelay`/ + `HandlePollFailure` (`:1162-1310`) with a `PollGroupEngine` instance (reader = + `ReadAsync`, onChange = forward `OnDataChange`, onError = degrade health, backoffCap + = 30 s). This deletes PERF-3 (structural compare now inherited) and STAB-6-S7 + (StopState drain now inherited) for free. + +**Implementation steps:** +1. Extend `PollGroupEngine` ctor + `PollLoopAsync` with backoff; add unit tests for + `ComputeBackoffDelay` parity with S7's (move the existing S7 test asserts over). +2. Add an `onError` handler in Modbus/AbCip/AbLegacy/TwinCAT/FOCAS constructions that + degrades health; assert via unit test that a throwing reader degrades health. +3. S7: construct the engine, delete the fork, keep the 100 ms floor and the + `PollFailureHealthThreshold`/`Degraded`-not-`Faulted` nuance in the onError handler. + +**Tests:** +- Engine unit: backoff schedule (0 failures = interval, N failures = capped + exponential); onError invoked once per caught reader exception; structural array + equality still suppresses no-change arrays; StopState drains before CTS dispose. +- S7 unit: array tag no longer fires every tick (regression for PERF-3); rapid + subscribe/unsubscribe no `ObjectDisposedException` (STAB-6); sustained poll failure + degrades health with backoff. + +**Effort:** M (engine change is S; S7 migration + 5 onError wirings is M). **Risk:** +Medium — S7 migration touches its subscription surface; the array-diff change alters +publish cadence (intended). Gate behind the existing S7 subscribe tests + a new +array-cadence test. + +## B2. Shared strict `ReadEnum` for equipment-tag parsers (CONV-2, UNDER-6) + +**Verification:** CONFIRMED. An identical private generic `ReadEnum(…, TEnum +fallback)` with **silent fallback** is copy-pasted six times — a typo'd `dataType` +quietly becomes the default type and reads/writes the wrong width with Good status: +`ModbusEquipmentTagParser.cs:65-67`, `S7EquipmentTagParser.cs:59-61`, +`AbCipEquipmentTagParser.cs:86-88`, `AbLegacyEquipmentTagParser.cs:60-62`, +`TwinCATEquipmentTagParser.cs:47-49`, `FocasEquipmentTagParser.cs:43-45` (all +byte-identical bodies). Contrast the factory's fail-loud `ParseEnum` +(`ModbusDriverFactoryExtensions.cs:191-201`, lists valid names). Probe-parity is also +broken for Modbus: `ModbusDriverProbe` binds the *runtime* `ModbusDriverOptions` +(`ModbusDriverProbe.cs:36`) not the factory DTO, so `timeoutMs` is silently dropped. + +**Root cause:** duplication-by-convention (theme #2); the three parse tiers (factory +strict / probe permissive / parser silent) diverged. + +**Design:** +- **Hoist one strict helper into `Core.Abstractions`**, living beside + `EquipmentTagRefResolver` (theme #6's home): e.g. + `TagConfigJson.ReadEnum(JsonElement o, string name, TEnum @default)` with the + rule **present-but-invalid ⇒ throw** (parse failure), **absent ⇒ default**. The parser + wraps the throw so the resolver's `_parseRef` returns null on a bad enum, and the + driver surfaces `BadNodeIdUnknown` instead of silently reading the wrong width. +- Delete all six private copies; the parsers call the shared helper. Enums stay + per-driver (the helper is generic over `TEnum`), so no coupling of driver vocab. +- **Probe parity (Modbus):** make `ModbusDriverProbe` deserialize the same factory DTO + shape the factory uses (or share a `TryParseConfig` used by both), so `timeoutMs` + binds identically. OpcUaClient already documents this rule + (`OpcUaClientDriverProbe.cs:22`); Modbus should follow. + +**Writable/capability parity (UNDER-6, folds in here):** Modbus/AbLegacy/FOCAS parsers +hard-code `Writable: true` (`ModbusEquipmentTagParser.cs:54-57`, +`AbLegacyEquipmentTagParser.cs:52`, `FocasEquipmentTagParser.cs:35`), so read-only +equipment tags are unauthorable on three drivers (AbCip honours a `writable` key, +`AbCipEquipmentTagParser.cs:53-54`). FOCAS additionally must force writable-false +(A7). Add a shared `ReadBool(o, "writable", default)` helper and honour it in all six +parsers; FOCAS overrides to false until PMC writes exist. FOCAS equipment tags also +bypass the `FocasCapabilityMatrix` gate authored tags get (`FocasDriver.cs:243-247` vs +`:115-119`) — route equipment-tag resolution through the same matrix validate. + +**Implementation steps:** +1. New `Core.Abstractions/TagConfigJson.cs` with strict `ReadEnum` + `ReadBool`. +2. Replace the six `ReadEnum` copies; wrap parse failure so `_parseRef` returns null. +3. Honour `writable` in the three hard-coded parsers; FOCAS → false + capability gate. +4. Fix `ModbusDriverProbe` to parse the factory DTO shape. + +**Tests:** +- **Shared conformance test** (the report explicitly wants one): a parameterized suite + run per driver asserting (a) unknown `dataType` ⇒ resolve failure ⇒ `BadNodeIdUnknown` + (not silent default); (b) `writable:false` honoured; (c) absent enum ⇒ documented + default. Put it in a shared test helper consumed by each driver's `.Tests`. +- Modbus probe: a config with `timeoutMs` round-trips the timeout. + +**Effort:** M. **Risk:** Medium — changing silent-default to hard-fail is a *behavior +change*: a deployment with a currently-silently-defaulted typo will start returning Bad. +That is the correct outcome, but call it out in the migration notes and verify live on +the rig (author a typo'd tag, confirm Bad rather than wrong-width Good). + +## B3. `ResolveHost` via the resolver — fix per-host breaker isolation (CONV-4, theme #6) + +**Verification:** CONFIRMED. Every multi-device driver's `ResolveHost` consults only +`_tagsByName` then falls back to the first device: `AbCipDriver.cs:485-487`, +`AbLegacyDriver.cs:498-500`, `TwinCATDriver.cs:587-591`, `FocasDriver.cs:1082-1087` +(Modbus `:125-131` same shape, but per-tag `UnitId` isn't in its equipment parser). +An equipment-tag reference (raw TagConfig JSON, resolved via `_resolver`, not in +`_tagsByName`) misses and returns the first device's host — so the per-host Polly +bulkhead/circuit-breaker key is wrong for equipment tags, and device B's outage can trip +device A's breaker. Since equipment tags are the primary post-Galaxy authoring model, +the `IPerCallHostResolver` is effectively inert for the flagship path. + +**Root cause:** `ResolveHost` never learned about the equipment-tag path; the resolver +resolves defs but exposes nothing host-specific (`EquipmentTagRefResolver` is generic +over `TDef` and never inspects it — confirmed). + +**Design:** route `ResolveHost` through `EquipmentTagRefResolver.TryResolve`, then +derive the host from the parsed def. The per-driver `TDef` records already carry the +host: `AbCipTagDefinition.DeviceHostAddress`, `AbLegacyTagDefinition.DeviceHostAddress`, +`TwinCATTagDefinition.DeviceHostAddress`, `FocasTagDefinition.DeviceHostAddress` (all +parsed by the equipment parsers). So: +- Each driver's `ResolveHost(ref)` becomes: `if (_resolver.TryResolve(ref, out var def)) + return def.DeviceHostAddress; return `. This covers both + authored (`_byName`) and equipment (`_parseRef`) paths through one call and reuses the + parse cache. +- Optional shared sugar: add a `string? ResolveHost(string ref, Func host)` + overload to `EquipmentTagRefResolver` so the pattern is one call. Keeps the host + extraction driver-specific (the resolver stays generic) while giving theme #6 a single + home. + +**Implementation steps:** rewrite the four `ResolveHost` bodies to call +`_resolver.TryResolve`; add the resolver overload if desired. Modbus: note `UnitId` +isn't in the equipment parser — either add `unitId` to `ModbusEquipmentTagParser` +(UNDER-6) or document that Modbus per-tag host resolution is out of scope. + +**Tests:** unit per driver — an equipment-tag ref for device B resolves to device B's +host (today returns device A/first); assert the Polly breaker key differs per device. + +**Effort:** S-M. **Risk:** Low — corrects a mis-key; verify the breaker-key wiring +downstream consumes the corrected host. + +## B4. Shared reconnect/backoff primitive (STAB-8, supports A1/A2) + +**Verification:** CONFIRMED. No reconnect backoff exists except the Modbus transport's +geometric backoff (`ModbusTcpTransport.cs:179-208`) and S7's poll-loop backoff. Dead +devices pay full reconnect cost per tick: AbCip full Tag create + Forward Open per tag +per tick (`AbCipDriver.cs:891-897` evict-recreate), FOCAS full two-socket reconnect per +tick of each of three loops (`FocasDriver.cs:1089-1120`), TwinCAT connect-attempt per +call (`:602-651`). + +**Design:** a small shared `ConnectionBackoff` primitive in `Core.Abstractions` +(capped exponential, the same `ComputeBackoffDelay` math B1 adds to the poll engine — +factor it out so both use it). Lazy-reconnect drivers gate their +`EnsureConnected`/evict-recreate behind a per-device throttle: record +`_lastConnectAttemptUtc` + a growing backoff, and short-circuit a reconnect attempt that +is inside the backoff window (return the current Bad/Degraded state instead of hammering). + +**Implementation steps:** +1. Extract `ComputeBackoffDelay` into `Core.Abstractions/ConnectionBackoff.cs` + (shared by PollGroupEngine v2 and the lazy-reconnect drivers). +2. Add a per-device backoff gate to TwinCAT `EnsureConnectedAsync`, FOCAS + `EnsureConnectedAsync`, and AbCip's evict-recreate path (skip the recreate if within + the backoff window; reset on success). + +**Tests:** unit — a device that fails to connect is not retried more often than the +backoff schedule; success resets the schedule. + +**Effort:** M. **Risk:** Low-Medium — must not *delay* recovery when the device comes +back (reset promptly on the first success). + +--- + +# Part C — Remaining Mediums & Lows (batched) + +All CONFIRMED. Grouped for efficient sweeps. + +## C1. One-liner conventions sweep (CONV-3, CONV-5) +- **CONV-3** (Medium): S7/TwinCAT/AbCip/FOCAS `Register` take no `ILoggerFactory` + (`S7DriverFactoryExtensions.cs:19`, `TwinCATDriverFactoryExtensions.cs:18`, + `AbCipDriverFactoryExtensions.cs:21`, `FocasDriverFactoryExtensions.cs:36`), so + `DriverFactoryBootstrap.Register` (`:100,102,106,107`) can't pass one and all four log + to `NullLogger` in production — their poll-failure/prohibition/live-risk warnings are + invisible. AbLegacy/Galaxy/Modbus/OpcUaClient already have the overload + (`:101,103,104,105`). Fix: add the `ILoggerFactory? = null` param + thread to the + driver ctor in the four factories; update the four bootstrap call sites. **Effort S, + risk very low.** (Also unblocks A1/A2/B1 logging.) +- **CONV-5** (Medium): four health-idioms (`Volatile` vs `volatile` vs plain field); + AbLegacy no health-refresh on successful write (`AbLegacyDriver.cs:344-346` vs AbCip + `:725`); AbLegacy silently clobbers duplicate tag names (`:91`) where AbCip fails fast + (`:249-256`); TwinCAT declares `Healthy` at init with zero wire contact (`:115`) and + reads host state without the probe lock (`:524-525`). Fix: standardize on + `Volatile.Read/Write` (or `volatile` record ref) across the fleet; add write-success + health refresh to AbLegacy; make AbLegacy fail-fast on duplicate names; make TwinCAT + prove the connection before `Healthy`. **Effort S-M, risk low.** + +## C2. Per-protocol read batching (PERF-1, PERF-2) — the scalability lever +- **PERF-1** (High): five of seven drivers do one tag per round-trip. Priority order per + the report: TwinCAT ADS **sum-read** (`TwinCATDriver.cs:203-248`), S7 **multi-var** + (`ReadMultipleVarsAsync`, unused — S7 loops per tag `:443-484` under one `_gate`), + FOCAS **PMC range reads** (`WireFocasClient.cs:296-334` widens to one value width), + AbLegacy **file-element spans** (N7:0..N7:2). Each is a separate protocol feature. +- **PERF-2** (High): AbCip's whole-UDT batched read is half-built — the planner collapses + N members via *declaration-order* layout, off by default (safe-but-unused, + `AbCipDriverOptions.cs:72` default `false`), while true member offsets are already + fetched via the CIP Template Object (`AbCipDriver.cs:151-182`, + `AbCipTemplateCache`) but consulted only by discovery, never the planner + (`AbCipUdtReadPlanner.cs:69`). **Highest-ROI batching win** (design already in-tree): + feed `AbCipUdtShape` template offsets into the planner, retire the unsafe + declaration-order mode. **Effort M-L, risk Medium (correctness of offset mapping — + test against the abcip fixture with a known UDT).** + + Sequencing: do PERF-2 first (design exists), then TwinCAT sum-read + S7 multi-var. Each + is independently shippable. **Effort L overall.** + +## C3. OpcUaClient performance + monolith (PERF-4, PERF-5, UNDER-9) +- **PERF-4** (Medium): all traffic serializes on one `SemaphoreSlim(1,1)` `_gate` + (12 `WaitAsync` sites, `OpcUaClientDriver.cs:78,626,713,836,1186,1377,1451,1653,1825`). + OPC UA sessions multiplex natively; a multi-second HistoryRead/re-discovery blocks + every read/write. The gate is only needed for session-swap consistency, which + `_probeLock` + re-read-in-critical-section already provide. Fix: scope the gate to + session-swap only; let reads/writes/history run concurrently on the session. **Effort + M, risk Medium — must preserve session-swap safety; test the reconnect race.** +- **PERF-5** (Medium): `EnrichAndRegisterVariablesAsync` sends `pending.Count*4` + ReadValueIds in one call (`:1029-1046`); a server enforcing `MaxNodesPerRead` returns + `BadTooManyOperations` and the blanket catch (`:1049-1057`) silently registers *every* + var as Int32/ViewOnly/non-historized with no log/health. Fix: chunk client-side + (respect `MaxNodesPerRead` / a conservative cap); log + health-signal the fallback. + **Effort S, risk low.** +- **UNDER-9** (Low): 2154-line monolith; extract the reconnect state machine for isolated + testability; set `Session.TransferSubscriptionsOnReconnect` explicitly (`:1946-1949` + — currently unset, relies on SDK default); `BuildCertificateIdentity` loads only + PKCS#12 (`:483-484`) while the option doc promises PEM + (`OpcUaClientDriverOptions.cs:68-75`) — either implement PEM load or fix the doc. + **Effort M (extraction) / S (the two nits), risk low.** + +## C4. Remaining stability/correctness Mediums (STAB-10, STAB-11, STAB-12) +- **STAB-10** (Medium): AbLegacy never evicts failed tag handles on the data path + (`AbLegacyDriver.cs:246-259,269-274`) though AbCip does (`AbCipDriver.cs:891-897`) and + AbLegacy's own probe loop does (`:456-461`). Fix: evict-recreate on non-zero + status/transport exception in the data path (port AbCip's `EvictRuntime`). Pairs with + A4/A6 as an AbLegacy/AbCip parity pass. **Effort S, risk low.** +- **STAB-11** (Medium): RMW locks don't cover direct parent writes (AbCip/AbLegacy/ + TwinCAT) → lost update when a direct parent write interleaves an in-flight RMW; AbCip + RMW hard-codes DINT while `AbCipTagPath` accepts `.N` on any parent + (`AbCipTagPath.cs:17-19`); TwinCAT RMW reads the parent as 4-byte `UDInt` + (`AdsTwinCATClient.cs:96-107`), an unverified assumption that will fail + `DeviceInvalidSize` on WORD/BYTE parents. Fix: make direct parent writes take the same + per-parent/per-runtime lock (AbCip's is subsumed by A4's runtime lock); width-aware RMW + for TwinCAT (read the parent as its actual declared width). **Effort M, risk Medium + (TwinCAT width needs real-hardware verification — flag under UNDER-7).** +- **STAB-12** (Medium): FOCAS sync `Dispose()` does a close-PDU exchange (`SendPdu`+ + `ReadPdu`) on a `NetworkStream` with no `ReadTimeout`/token + (`FocasWireClient.cs:164-175`) — a stalled CNC wedges `ShutdownAsync`. TwinCAT + `ConnectAsync` calls the blocking SDK `Connect` ignoring token+timeout + (`AdsTwinCATClient.cs:73-80`). Fix: bound the FOCAS close read with a `ReadTimeout`; + run TwinCAT `Connect` on a worker with the timeout/token honoured (or set + `_client.Timeout` and wrap in `Task.Run` + `WaitAsync(ct)`). **Effort S, risk low.** + +## C5. Perf Lows (PERF-6, PERF-7) +- **PERF-6** (Low): TwinCAT re-parses `TwinCATSymbolPath` per call + (`TwinCATDriver.cs:220,284,468`) — cache like S7's `_parsedByName`; Modbus fresh + `DriverHealth` per read (`:285`); AbCip `BuildArray` boxes per element. None urgent. +- **PERF-7** (Low): deadband + `WriteOnChangeOnly` exist only in Modbus + (`ModbusDriver.cs:136-159`); generalize into/beside `PollGroupEngine` (driver-agnostic + publish suppression) so noisy analog signals on the other poll drivers don't publish + every jitter. Natural PollGroupEngine-v2 add-on (B1). **Effort M, risk low.** + +## C6. Underdeveloped Mediums (UNDER-4, UNDER-5, UNDER-7, UNDER-8) +- **UNDER-4** (Medium, S7): `S7TagDto` has no `ArrayCount` and `BuildTag` never sets it + (`S7DriverFactoryExtensions.cs:80-90,137-151`) → driver-config array tags silently + scalar (arrays only work as equipment tags, `S7EquipmentTagParser.cs:73-87`); array + writes throw `NotSupportedException` (`S7Driver.cs:1014-1017`); `UInt32` surfaces + lossily as `Int32` (`:1149-1152`). Fix: add `ArrayCount` to the DTO + `BuildTag`; + implement array writes; map `UInt32`→`UInt32`/`Int64`. **Effort M.** +- **UNDER-5** (Medium, AbCip): `AcknowledgeAsync` fire-and-forgets write results + (`AbCipAlarmProjection.cs:149`) → a failed ack is invisible to operator/health/Part 9 + caller; with `EnableControllerBrowse` on, one unreachable PLC faults the entire + multi-device `DiscoverAsync` (uncaught `await foreach`, `AbCipDriver.cs:970-1037`); + browsed tags default writable/Operate (`CipSymbolObjectDecoder.cs:91`). Fix: surface + ack outcomes to health + the caller; per-device try/catch in discovery so one dead PLC + doesn't fault all; default browsed tags read-only. **Effort M.** +- **UNDER-7** (Medium): aggregate the honest "unverified against real hardware" blocks + into a **"needs bench time" checklist** — TwinCAT array-read shape + (`AdsTwinCATClient.cs:109-116`), Flat-mode `SubSymbols` population (`:263-277`), bit-RMW + width (STAB-11); AbLegacy array-decode's five unproven layout assumptions + (`LibplctagLegacyTagRuntime.cs:64-88`); FOCAS `cnc_getfigure` id + servo-load scaling + (`FocasWireClient.cs:391-395,943-947`). Deliverable: a doc checklist, not code. + **Effort S (doc).** +- **UNDER-8** (Medium): test coverage is codec-heavy, failure-path-light. The concrete + gaps this plan closes with new tests: S7 reconnect (A1), TwinCAT replay (A2), AbCip + concurrency (A4), FOCAS cache concurrency (A5), Modbus desync (A3), shared + parser-strictness conformance (B2). Also: `Cli.Common.DriverCommandBase` has no direct + tests — add a small suite. + +## C7. Lows — hygiene (CONV-6, CONV-7, CONV-8) +- **CONV-6** (Low/Medium): `Driver.OpcUaClient.Contracts` drags the full OPC UA SDK + because `NamespaceMap` lives there (`OpcUaClient.Contracts.csproj:9` + `OPCFoundation.NetStandard.Opc.Ua.Client`); every options-DTO consumer transitively + loads the SDK. This is the previously-deferred OpcUaClient.Contracts-002 finding — move + `NamespaceMap` into the driver project (or a new `.Namespace` project), leaving Contracts + POCO-only like the other six. **Effort M (new project boundary), risk low.** +- **CONV-7** (Low/Medium): CLI divergences — `ParseValue`/`ParseBool` copy-pasted six + times (`*/Commands/WriteCommand.cs`); option `Validate()` absent in AbLegacy + (`AbLegacyCommandBase.cs` — `--timeout-ms 0` reaches the driver); `Timeout` init-setter + throws in AbCip (`AbCipCommandBase.cs:43`) but no-ops in five others; only TwinCAT + exposes `browse` though AbCip supports controller browse + (`AbCipCommandBase.cs:63` hardcodes `EnableControllerBrowse=false`); OpcUaClient has no + CLI. Fix: hoist value parsing + a standard `Validate()` into `Cli.Common`; add an + OpcUaClient CLI or document the gap. **Effort M, risk low.** +- **CONV-8** (Low): dead/no-op knobs + stale docs — `DisableFC23` no-op + (`ModbusDriverOptions.cs:83-89`); AbCip `ConnectionSize` plumbed-never-applied + (`:97-103`); AbLegacy class doc says read/write "ship in PRs 2 and 3" + (`AbLegacyDriver.cs:9-11`); OpcUaClient header says "PR 66 ships the scaffold: IDriver + only" (`:12-14`). No options class has `Validate()`. Fix: delete/wire the dead knobs; + refresh the stale headers; optionally add per-driver options `Validate()` (S7's init + guards `:314-385` are the best-of-breed template). **Effort S, risk very low.** + +--- + +# Sequencing & effort roll-up + +Aligns with the report's "Recommended remediation order" and OVERALL actions #3/#4. + +| # | Item | Findings | Effort | Risk | Notes | +|---|---|---|---|---|---| +| 1 | S7 reconnect (`IS7Plc` seam + `EnsureConnected`) + reconnect test | STAB-1, UNDER-8 | M | Med | Critical; template = TwinCAT/OpcUaClient | +| 2 | TwinCAT native-sub replay on client swap + probe recycle | STAB-2 | M | Med | Critical; fake-client unit tests are the only automated coverage | +| 3 | AbCip per-runtime op lock + FOCAS ConcurrentDictionary caches | STAB-4, STAB-5 | S each | Low | Silent-corruption class; port AbLegacy's `GetRuntimeLock` | +| 4 | Modbus timeout/framing ⇒ connection teardown | STAB-3 | S | Low-Med | Distinguish timeout vs caller-cancel | +| 5 | Shutdown/unsub race fixes | STAB-6 | S | Low | Deleted for S7 by item 8 | +| 6 | FOCAS writable-false; OpcUaClient dead gate + alarm-filter encoding | UNDER-1, UNDER-2, UNDER-3 | S each | Low | Kill false capabilities | +| 7 | Factory `ILoggerFactory` on 4 drivers | CONV-3 | S | V.Low | Unblocks item 1/2/8 logging | +| 8 | **PollGroupEngine v2**: backoff + onError-health + migrate S7 off fork | CONV-1, STAB-8, STAB-9, PERF-3, STAB-6-S7 | M | Med | Single home for poll fixes | +| 9 | **Shared strict `ReadEnum`/`ReadBool` + probe parity + writable parity** | CONV-2, UNDER-6 | M | Med | Behavior change: typo ⇒ Bad; live-verify | +| 10 | **`ResolveHost` via resolver** (per-host breaker isolation) | CONV-4 | S-M | Low | Flagship equipment-tag path | +| 11 | Shared reconnect/backoff primitive for lazy drivers | STAB-8 | M | Low-Med | Extract `ConnectionBackoff`; don't delay recovery | +| 12 | Per-protocol batching: AbCip planner⇄template-cache, TwinCAT sum-read, S7 multi-var | PERF-1, PERF-2 | L | Med | Biggest scalability lever; AbCip design in-tree | +| 13 | OpcUaClient gate-scoping + chunk discovery + monolith split | PERF-4, PERF-5, UNDER-9 | M | Med | Preserve session-swap safety | +| 14 | Remaining mediums: evict-on-fail, RMW parent lock/width, blocking teardown | STAB-10, STAB-11, STAB-12 | M | Med | TwinCAT width needs bench time | +| 15 | Underdeveloped: S7 arrays/UInt32, AbCip acks/discovery, bench checklist | UNDER-4, UNDER-5, UNDER-7, UNDER-8 | M | Low-Med | | +| 16 | Lows: Contracts SDK split, CLI hoist, dead knobs/docs, perf lows | CONV-6/7/8, PERF-6/7 | M | Low | | + +**Where shared primitives live:** `Core.Abstractions` — `PollGroupEngine` (v2 with +backoff, item 8), `ConnectionBackoff` (item 11, shared with the engine), +`TagConfigJson.ReadEnum/ReadBool` (item 9, beside `EquipmentTagRefResolver`), and the +`ResolveHost`-via-resolver pattern (item 10). This is the concrete answer to themes #2 +and #6: fleet fixes get one home instead of six copies. + +**Live-verify recipes (docker rig on `10.100.0.35`):** Modbus `:5020` + +`exception_injector` for STAB-3 (item 4) and write-failure paths; S7 `:1102` +(`Snap7ServerFixture`, bounce via `lmxopcua-fix down/up s7 s7_1500`) for STAB-1 (item 1); +AbCip `:44818` for STAB-4 concurrency + PERF-2 UDT (items 3, 12). TwinCAT has **no docker +fixture** (needs a real TC3 XAR) — the fake-`ITwinCATClient` unit tests are authoritative +for STAB-2/item 2. diff --git a/archreview/plans/06-gateway-integrations-plan.md b/archreview/plans/06-gateway-integrations-plan.md new file mode 100644 index 00000000..9e2e1ffe --- /dev/null +++ b/archreview/plans/06-gateway-integrations-plan.md @@ -0,0 +1,353 @@ +# Plan 06 — Gateway Integrations (Galaxy Driver + Historian Gateway Driver) + +- **Source report:** `archreview/06-gateway-integrations.md` +- **Scope:** `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Galaxy` (+ `.Contracts`, `.Browser`), `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway` +- **Commit reviewed against:** current tree (report cut at `9cad9ed0`) +- **Verification method:** every cited file:line opened and checked against the current tree. + +## Verification summary + +| Finding | Sev | Status | Note | +|---|---|---|---| +| S-1 Galaxy write optimistic-Good | High | **Confirmed** | `GatewayGalaxyDataWriter.cs` 234-243 (advise-fail proceeds), 281-302 (empty statuses → Good), 162-165 (silent-loss comment) all present verbatim. | +| S-2 EventPump drops newest | Medium | **Confirmed** | `EventPump.cs:136-139` `TryWrite` → count drop, newest-dropped by construction. | +| S-3 Read can hang on non-cancellable token | Medium | **Confirmed** | `GalaxyDriver.cs:738-753`: BadTimeout fill only via `cancellationToken.Register`; awaits at 753 block forever with `CancellationToken.None`. | +| S-4 Transport-fault detection is EventPump-only | Medium | **Confirmed** | `ReportTransportFailure` sole prod caller is `OnEventPumpStreamFault` (`GalaxyDriver.cs:943`); `WriteAsync` (line 803) and subscribe path never feed the supervisor. | +| S-5 Outbox `RemoveAsync` prefix truncation silent | Medium | **Confirmed** | `FasterLogHistorizationOutbox.cs:158-182`: removes target + all older live entries, no `_droppedCount` bump on non-target removals. | +| S-6 Historian health flags dormant | Medium | **Confirmed (worse)** | `RefreshConnectionStateAsync` has only test callers; **`GetHealthSnapshot()` itself has zero production callers** — the whole snapshot is currently unconsumed in-process. | +| S-7 Alarm feed reconnect no backoff | Low | **Confirmed** | `GatewayGalaxyAlarmFeed.cs:141` fixed `_reconnectDelay` (default 5 s), no exponential/jitter. | +| S-8 `GalaxyMxSession` unsynchronized state | Low | **Confirmed** | `GalaxyMxSession.cs:28-32` plain fields; `RecreateAsync` (112-117) tears down without a lock. Emergent safety accepted. | +| S-9 Store-and-forward discipline | Positive | Confirmed positive | No action. | +| S-10 TLS / API-key posture | Positive | Confirmed positive | No action. | +| P-1 Read = 3 round-trips | Medium | **Confirmed** | `ReadViaSubscribeOnceAsync` 658-780; MxAccess-forced. Doc + optional cache. | +| P-2 Serial `SendEvent` per event | Medium | Confirmed (accepted) | Gateway-constrained; track batched `SendEvents` on sidecar. | +| P-3 Channel count | Low | Confirmed (accepted) | Documented trade-off; no action. | +| P-4 Indexed fan-in/out | Positive | Confirmed positive | No action. | +| P-5 HistoryRead buffers; `maxEvents<=0` unbounded | Low | **Confirmed** | `GatewayHistorianDataSource.cs:142-168` `hasCap = maxEvents>0`; when false collects unbounded. | +| P-6 Peek holds lock over disk I/O | Low | **Confirmed** | `FasterLogHistorizationOutbox.cs:145-152`. Negligible at 64-batch. | +| C-1 Secret resolver | Positive | Confirmed positive | No action. | +| C-2 Seam quality | Positive | Confirmed positive | No action. | +| C-3 Historian seam leaks wire types | Medium | **Confirmed** | `IHistorianGatewayClient.cs:1-11` proto-typed by design. Doc-guard only. | +| C-4 Options-validation idioms diverge | Low | **Confirmed** | Galaxy throw-on-construct vs Historian warnings-list. Convention note. | +| C-5 Retired Wonderware dirs on disk | Low | **Confirmed** | 3 dirs present, **0** `slnx` references. Delete. | +| C-6 EventPump drops unknown families unmetered | Low | **Confirmed** | `EventPump.cs:199-206` `default: return;` no counter. | +| U-1 CLAUDE.md KNOWN LIMITATION 2 stale | High (doc) | **Confirmed stale** | Feed fully wired: `AddressSpaceApplier.cs:343` `UpdateHistorizedRefs` → `ActorHistorizedTagSubscriptionSink` → recorder `OnUpdateHistorizedRefs` (`ContinuousHistorizationRecorder.cs:186/311`). Doc must be corrected; path never live-verified. | +| U-2 Live-validation gate partially run | Medium | Confirmed | Infra complete; needs a full documented run + CLAUDE.md status. | +| U-3 WriteSecured user id stubbed at 0 | Medium | **Confirmed** | `GatewayGalaxyDataWriter.cs:263-264` `CurrentUserId=0, VerifierUserId=0`. | +| U-4 Dormant paths inventory | Medium | **Confirmed** | `HistorianGatewayClientAdapter.cs:88` drops `maxEvents` on the wire; `GalaxyDriver.cs:314-316` replay note; `FlushOptionalCachesAsync` no-op. | +| U-5 Outbox serializer has no version byte | Low | **Confirmed** | `HistorizationOutboxEntrySerializer.cs:14-16` fixed positional layout, no prefix. | +| U-6 Test-coverage thin spots | — | Confirmed | S-1 failure branch, S-5 out-of-order, S-3 hang, C-6 filter all untested. Folded into each finding's test section. | + +**Totals: 28 findings — 23 actionable, 5 positive (no action). Of the 23 actionable, all 23 confirmed against the current tree; 0 stale-as-already-fixed (U-1 is stale *documentation* against correct code).** + +Cross-cutting alignment with `00-OVERALL.md`: S-1 is part of OVERALL action #7 (failure visibility); S-6 + U-1 are instances of OVERALL theme #1 ("built-but-never-wired"); U-1 + U-2 are OVERALL theme #5 (doc drift); U-4/C-3 are theme #4 (gateway-gap workarounds accrue client-side). + +--- + +## Priority-ordered plan + +Priority key: **P0** ship first (data-integrity / visibility), **P1** next (fault resilience), **P2** hygiene/docs, **P3** nice-to-have. + +--- + +### P0-A — S-1: Galaxy write success is optimistic; a committed-write failure can never surface + +**Restatement:** Empty MX status arrays translate to `Good`; a failed supervisory-advise logs a warning and lets the raw `Write` proceed even though the file's own comment says such a write "never reaches the galaxy"; write-outcome self-correction (#5) structurally cannot fire because Galaxy `ExecuteWrite` is fire-and-forget. + +**Verification:** Confirmed. `GatewayGalaxyDataWriter.cs`: +- 234-243: on `AdviseSupervisory` protocol failure → `TryRemove` the handle, log warning, **fall through** to `WriteRawAsync`. +- 281-302: `TranslateReply` → protocol-status check, then first MX status row, else `new WriteResult(StatusCodeMap.Good)`. +- 162-165: in-source comment confirms a non-advised raw Write "doesn't throw (reply looks OK) but the value never reaches the galaxy." + +**Root cause:** Two independent optimistic defaults. (1) The gateway reply's `Statuses` array reflects *command acceptance at the worker*, not the eventual COM-side `WriteComplete`; an empty array is treated as success. (2) The supervisory-advise precondition is best-effort — when it fails the code still issues a Write it knows won't commit, and reports the Write's (empty/OK) reply as `Good`. + +**Design.** + +*In-repo (this PR) — three layers, cheap and independent:* + +1. **Fail-closed on advise failure.** Change `EnsureSupervisoryAdvisedAsync` to return a bool (advised? true/false). In `WriteOneAsync`, when a non-secured write's advise returns false, **do not issue the raw Write** — return `new WriteResult(StatusCodeMap.BadWaitingForInitialData /* or Uncertain */)`. Rationale: the code already *knows* the value can't commit; sending it and reporting Good is the exact silent-loss the comment describes. Preferred status: `Uncertain_NoCommunicationLastUsableValue` is wrong (implies a read); use `Bad` family so #5 self-correction *could* fire (`BadNotConnected`/`BadWaitingForInitialData`). Decision: return **`StatusCodeMap.BadCommunicationError`** — it is already the writer's "couldn't reach the device" status, so the node-write router / self-correction path treats it consistently with a transport failure. (The report suggests `Uncertain`; `Bad` is stronger and lets self-correction revert the phantom-Good node — align with OVERALL theme #3 "the failure must surface *somewhere*".) + +2. **Empty-statuses is not Good for a data write.** `TranslateReply`: when `reply.Statuses.Count == 0` **and** the reply is a data-write reply, return `StatusCodeMap.UncertainNoCommunicationLastUsableValue`-class or, more defensibly, keep `Good` **only** when the gateway contract guarantees a status row on success. This needs a gateway-contract check (see cross-repo). Interim: keep `Good` but **meter** it (layer 3) so the unconfirmed rate is observable, and add an XML-doc note that empty-array-Good is provisional pending the `WriteComplete` correlation. + +3. **`galaxy.writes.unconfirmed` counter.** Add a `Counter` on the existing `GalaxyTelemetry`/pump meter (`ZB.MOM.WW.OtOpcUa.Driver.Galaxy`), incremented whenever `TranslateReply` returns Good off an *empty* statuses array, and a `galaxy.writes.advise_failed` counter on the advise-failure branch. Gives ops the signal the report asks for. + +*Cross-repo (mxaccessgw) — the real fix, tracked follow-on:* pursue a gateway-side `WriteComplete` correlation so the `MxCommandReply.Statuses` row carries the real COM commit outcome. The `OnWriteComplete` event family already exists in the proto and is currently *filtered out* by `EventPump.Dispatch` (`EventPump.cs:199-206`). Design: gateway correlates the async `WriteComplete` back to the originating command's `ClientCorrelationId` and folds the MX status into the unary reply's `Statuses` before returning (making Write synchronously authoritative), **or** emits `OnWriteComplete` on the event stream keyed by correlation id and the driver awaits it with a deadline. The synchronous-fold option is cleaner for the OPC UA write contract (write is a blocking service call). This is a sibling-repo change; file it on the mxaccessgw backlog and link from the in-source comment at `GatewayGalaxyDataWriter.cs:162-165`. + +**Implementation steps (in-repo):** +- `GatewayGalaxyDataWriter.cs`: change `EnsureSupervisoryAdvisedAsync` → `Task`; branch in `WriteOneAsync` to short-circuit to `BadCommunicationError` when advise failed. +- Add `WriteUnconfirmed` / `WriteAdviseFailed` counters (new static `Counter` fields; reuse `EventPump.MeterName` string or a shared `GalaxyTelemetry` meter — check `GalaxyTelemetry.cs` for the existing meter name to avoid a second Meter). +- `TranslateReply`: increment `WriteUnconfirmed` on the empty-statuses branch; add doc note. + +**Tests (unit):** +- New `GatewayGalaxyDataWriterTests` cases (writer already has an internal ctor + `SeedHandleCachesForTest`): advise-fails → assert returned status is `BadCommunicationError` and **no** `WriteRawAsync` was issued (fake session records calls). +- Empty-statuses reply → assert Good returned **and** the unconfirmed counter incremented (via a `MeterListener`). +- Protocol-status non-OK → `BadCommunicationError` (regression guard). + +**Tests (live gateway):** extend the skip-gated `GatewayGalaxyLiveReopenAndWriteTests` with a case that forces an advise-failure (or asserts the unconfirmed counter is emitted on a normal write). Recipe: +``` +KEY=$(docker exec otopcua-dev-central-1-1 printenv GALAXY_MXGW_API_KEY) +MXGW_ENDPOINT=http://10.100.0.48:5120 GALAXY_MXGW_API_KEY="$KEY" \ + dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Tests \ + --filter FullyQualifiedName~GatewayGalaxyLiveReopenAndWrite +``` + +**Effort:** S (in-repo layers 1+3), M (layer 2 + cross-repo correlation). **Risk/blast radius:** Low in-repo — the change makes a currently-silent-loss visible; the only behavioral change is that advise-failed writes now return Bad instead of phantom-Good, which is strictly more correct and feeds #5 revert. Live-verify to confirm normal writes still return Good. + +--- + +### P0-B — S-6: Historian health snapshot's connection flags are dormant (and the snapshot is unconsumed) + +**Restatement:** `RefreshConnectionStateAsync` has no production caller, so `ProcessConnectionOpen`/`EventConnectionOpen` are permanently false; deeper: `GetHealthSnapshot()` itself is never called in production. + +**Verification:** Confirmed and worse than reported. `RefreshConnectionStateAsync` callers = 3 test lines only. `GetHealthSnapshot()` invocation grep across `src` returns only the interface declaration — no production consumer. So the entire health surface (`GetHealthSnapshot` + the refresh) is dead weight in a deployed host today. + +**Root cause:** The "register-AND-pass-into-consumer" trap (same class as PR #423 provisioner, OVERALL theme #1): the component was built and unit-tested, but neither the periodic refresh nor a snapshot consumer was wired. + +**Design.** Two decisions: + +1. **Is the snapshot meant to be surfaced?** Check `HistorianHealthSnapshot`'s intended consumer (AdminUI historian-status panel / a health endpoint). If yes → wire it. If no (the flags predate a since-abandoned dashboard) → **remove** `ProcessConnectionOpen`/`EventConnectionOpen` + `RefreshConnectionStateAsync` to kill the misleading-false surface. The report offers both; pick based on whether any dashboard is planned. Recommendation: **wire it**, because the other integrations (Galaxy `HostStatusAggregator`) *do* surface connectivity and a historian-down indicator is genuinely useful. + +2. **How to drive the refresh.** Add a lightweight `IHostedService` (`HistorianHealthRefreshHostedService`) that, when `ServerHistorian:Enabled`, resolves the registered `IHistorianDataSource`, and if it is a `GatewayHistorianDataSource`, calls `RefreshConnectionStateAsync` on a timer (reuse the existing health-probe cadence if one exists — check `OtOpcUaServerHostedService` for an existing periodic loop to fold into rather than spawning a second timer). Also wire a consumer: project the snapshot into whatever the AdminUI/health endpoint reads. + +**Implementation steps:** +- New `HistorianHealthRefreshHostedService` in `ZB.MOM.WW.OtOpcUa.Runtime` (or Host), registered in the historian DI block alongside `AddServerHistorian`. Guard on `Enabled`. Interval from a new `ServerHistorianOptions.HealthRefreshSeconds` (default 15). +- Cast/seam: `RefreshConnectionStateAsync` is on the concrete `GatewayHistorianDataSource`, not `IHistorianDataSource`. Either (a) add an optional `IHistorianConnectionProbe { Task RefreshConnectionStateAsync(ct) }` capability interface the gateway source implements and the hosted service checks for, or (b) resolve the concrete type. Prefer (a) — matches the capability-interface pattern used elsewhere and keeps `NullHistorianDataSource` clean. +- Wire the snapshot consumer (AdminUI panel or `/health`); if none is desired, execute decision-branch (1) removal instead. + +**Tests:** +- Unit: hosted service calls `RefreshConnectionStateAsync` on tick against a fake source; disabled → never resolves. +- Unit: `GatewayHealthSnapshotTests` already covers the refresh math — add an assertion that after a successful refresh the snapshot flags are true. +- Live: after wiring, run the historian live suite (U-2) and confirm the snapshot flips to connected against the real sidecar. + +**Effort:** S (wire refresh) / M (if adding a consumer surface). **Risk:** Low. **Note:** live-`/run` required per OVERALL theme #1 — do not close on unit tests alone. + +--- + +### P0-C — U-1: CLAUDE.md KNOWN LIMITATION 2 is stale; continuous-historization value-capture never live-verified + +**Restatement:** CLAUDE.md says the recorder is spawned with an empty ref-set and "historizes nothing"; the code has closed that gap but the end-to-end path was never live-verified. + +**Verification:** Confirmed. The feed is fully wired: `ServiceCollectionExtensions.cs:272` still spawns with `Array.Empty()`, **but** wraps in `ActorHistorizedTagSubscriptionSink` (line 290-292) and hands it to the applier; `AddressSpaceApplier.cs:310-343` `FeedHistorizedRefs` computes the per-deploy add/remove delta and calls `_historizedSubscriptions.UpdateHistorizedRefs`; `ContinuousHistorizationRecorder.cs:186/311` receives `UpdateHistorizedRefs` and re-registers. CLAUDE.md's "Known Limitation 2" text is therefore factually wrong. + +**Root cause:** Documentation drift — the feed landed after the CLAUDE.md note was written and the note was never reconciled. + +**Design/steps:** +1. **Doc fix (this PR):** rewrite CLAUDE.md "KNOWN LIMITATION 2" to state the ref-feed is wired (applier `FeedHistorizedRefs` → `ActorHistorizedTagSubscriptionSink.UpdateHistorizedRefs` → recorder `OnUpdateHistorizedRefs`), and downgrade it from "records no values" to "wired; pending a live end-to-end verification." Propagate the same correction to the `../scadaproj/CLAUDE.md` OtOpcUa index entry (per the repo's cross-repo propagation rule). +2. **Live-verify the value-capture path (the real close-out):** deploy a config with a historized numeric tag against a real HistorianGateway (VPN to the sidecar / `wonder-sql-vd03`, gateway with `RuntimeDb:Enabled=true`), drive a value change, and confirm `WriteLiveValues` lands (outbox drains, gateway shows the sample). This is the OVERALL theme #1 "prove production wiring" gate. +3. **Add a restart-convergence integration test** (unit-level with a fake gateway value-writer + real outbox): deploy delta adds refs → mux value → outbox append → drain calls writer; assert the writer received the value. This closes the "wired but inert" risk with a repeatable test even before the live run. + +**Effort:** S (doc) + M (live-verify + convergence test). **Risk:** Doc-only change is zero-risk; the live run may surface a real inertness bug (exactly what OVERALL warns about) — budget for a fix if it does. + +--- + +### P1-A — S-4: Transport-failure detection is EventPump-only; a write-only/idle driver never degrades + +**Restatement:** `ReportTransportFailure` is only called from the EventPump stream-fault callback; a driver that only writes or is idle after discovery keeps `GetHealth()==Healthy` through a gateway restart, and each write fails per-request forever with no reopen/replay. + +**Verification:** Confirmed. Sole prod caller `GalaxyDriver.cs:943` (`OnEventPumpStreamFault`). `WriteAsync` (803) delegates straight to `_dataWriter.WriteAsync` with no supervisor hand-off; failed unary RPCs (SubscribeBulk, Write, AcknowledgeAlarm) don't feed the supervisor. + +**Root cause:** Fault detection was built around the long-lived StreamEvents pump only; unary RPC failures were treated as per-call and never escalated to the recovery state machine. + +**Design.** `ReportTransportFailure` is idempotent by design (the supervisor single-flights recovery), so reporting from more sites is cheap and safe. Add classified transport-failure reporting from the unary paths: +- In `GatewayGalaxyDataWriter.WriteOneAsync`'s catch (currently maps to `BadCommunicationError`), and in the subscribe path, classify the exception (RpcException with transport status codes: Unavailable, DeadlineExceeded, Internal, etc.) and, when transport-fatal, invoke a supplied `Action? reportTransportFailure` callback (mirror the pump's `onStreamFault` pattern) that routes to `_supervisor.ReportTransportFailure`. +- Wire the callback from `GalaxyDriver` when it constructs the writer/subscriber, guarded like `OnEventPumpStreamFault` (null supervisor = skeleton path, no-op). +- Reuse the existing transport-classification helper if one exists (grep `StatusCodeMap`/`ReconnectSupervisor` for a classifier); otherwise add a small `IsTransportFatal(Exception)` in the driver. + +**Implementation steps:** +- Add a `Func`/callback seam to `GatewayGalaxyDataWriter` (optional ctor arg, like `subscribedHandleSource`) or wrap the call at the `GalaxyDriver.WriteAsync` boundary (cleaner — keep the writer pure). Prefer wrapping at `GalaxyDriver`: catch transport-classified failures returned/thrown from the writer and call `supervisor.ReportTransportFailure`. +- Same for `SubscribeAsync`/`ReadViaSubscribeOnceAsync` SubscribeBulk failures and `GatewayGalaxyAlarmAcknowledger`. + +**Tests (unit):** fake writer that throws an Unavailable RpcException → assert `GalaxyDriver.WriteAsync` reports the failure to a fake supervisor and returns Bad; non-transport ArgumentException → not reported. Idempotency: two failures → supervisor sees N reports but single-flights (already covered by `ReconnectSupervisorTests` — add a driver-level report test). + +**Effort:** M. **Risk:** Low-Medium — must avoid escalating *application-level* write rejections (bad value, secured-tag denial) to transport recovery; the classifier must be conservative (only RpcException transport codes). + +--- + +### P1-B — S-3: `ReadViaSubscribeOnceAsync` can wait forever on a non-cancellable token + +**Restatement:** BadTimeout fill is only via `cancellationToken.Register`; a `CancellationToken.None` caller whose tag subscribes OK but never publishes hangs at the await, holding the subscription open. + +**Verification:** Confirmed. `GalaxyDriver.cs:738-748` registers the fill on the CT; 750-755 awaits each pending TCS unconditionally. No internal deadline. + +**Root cause:** The read deadline is delegated entirely to the caller's token; an infinite/None token has no fallback. + +**Design.** Race each pending await against an internal deadline derived from options regardless of the caller's token. Compute `deadline = max(PublishingIntervalMs, DefaultCallTimeoutSeconds)` × a small factor (e.g. 3× publishing interval, floored at the call timeout). Implement by combining tokens: create a linked CTS `CreateLinkedTokenSource(cancellationToken)` + `CancelAfter(deadline)`, and register the BadTimeout fill on the *linked* token instead of the caller's. Then every path (caller cancel, internal deadline) fires the fill. Ensure the linked CTS is disposed in the finally. + +**Implementation steps:** +- In `ReadViaSubscribeOnceAsync`: build `using var deadlineCts = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken); deadlineCts.CancelAfter(TimeSpan.FromMilliseconds(readDeadlineMs));` and change `cancellationToken.Register(...)` → `deadlineCts.Token.Register(...)`. +- Derive `readDeadlineMs` from `_options.MxAccess.PublishingIntervalMs` and `_options.Gateway.DefaultCallTimeoutSeconds` (or equivalent). Add an option `ReadSynthesisTimeoutMs` if a dedicated knob is warranted; default derived. + +**Tests (unit):** the driver has injectable seams (`IGalaxySubscriber` fake). Fake that returns a successful SubscribeBulk but never raises OnDataChange → call `ReadViaSubscribeOnceAsync` with `CancellationToken.None` → assert it returns BadTimeout snapshots within the deadline (not hang) and that UnsubscribeBulk was called (subscription not leaked). + +**Effort:** S. **Risk:** Low — pure addition of an internal deadline; existing caller-cancel behavior preserved via the linked token. + +--- + +### P1-C — S-5: FasterLog outbox `RemoveAsync` truncates the FIFO prefix; out-of-order acks silently drop unacked entries + +**Restatement:** `RemoveAsync(id)` removes the target plus every older live entry and truncates to the target's successor; if a mid-batch id is acked first, older unacked values are durably discarded without incrementing `DroppedCount`. + +**Verification:** Confirmed. `FasterLogHistorizationOutbox.cs:158-182`: while-loop removes `_live.First` repeatedly until it hits the target and breaks; `_droppedCount` not touched for the collateral removals. Contract "acks in FIFO order" enforced by comment only. + +**Root cause:** FasterLog is a truncate-to-address log; removing an arbitrary id necessarily discards everything before it. The API accepted a single id while the underlying primitive is prefix-truncation, so the invariant lives only in the caller's discipline. + +**Design.** Two options: +1. **Count + warn (minimal):** in `RemoveAsync`, when the removal drops entries whose id != the target, increment `_droppedCount` per collateral entry and log a warning naming the count. Makes the silent loss observable. Cheap, no API change. +2. **Make semantics explicit at the call site (preferred):** add `RemoveThroughAsync(Guid throughId)` (or `RemoveBatchAsync(IReadOnlyList)`) to `IHistorizationOutbox`, documenting that it truncates the FIFO prefix through the id; keep single-id `RemoveAsync` but have it assert the target is `_live.First` (FIFO) and throw/log if not. This surfaces the invariant at compile/call time. + +Recommendation: **do both** — rename the operation to `RemoveThroughAsync` to match reality, and add the collateral-drop counter as a safety net for any future non-FIFO caller. + +**Implementation steps:** +- `IHistorizationOutbox`: add `RemoveThroughAsync` (or annotate `RemoveAsync` contract). Update `ContinuousHistorizationRecorder` drain call site to use it. +- `FasterLogHistorizationOutbox.RemoveAsync`: count collateral removals into `_droppedCount` (or a new `_ackPrefixDropped` counter for clarity) and `_logger.LogWarning` when > 0. + +**Tests (unit):** append A,B,C; `RemoveAsync(C.Id)` (skipping A,B) → assert A,B are gone, `DroppedCount == 2` (or the new counter), and a warning was logged. Regression: FIFO ack A,B,C in order → `DroppedCount == 0`. + +**Effort:** S. **Risk:** Low — additive counter; API rename is mechanical (single recorder caller). + +--- + +### P1-D — S-7: Alarm feed reconnect has no backoff + +**Restatement:** `GatewayGalaxyAlarmFeed.RunAsync` re-opens on a fixed 5 s delay forever; `DeployWatcher` and `ReconnectSupervisor` both use capped exponential + jitter. + +**Verification:** Confirmed. `GatewayGalaxyAlarmFeed.cs:141` `await Task.Delay(_reconnectDelay, ct)` with a fixed `_reconnectDelay`. + +**Root cause:** Inconsistency — the alarm feed predates or diverged from the shared backoff pattern. + +**Design.** Replace the fixed delay with capped exponential backoff + jitter, resetting to the floor on a successful stream open (i.e., reset the attempt counter after the first message or after the `snapshot_complete`). Reuse the existing backoff helper `DeployWatcher`/`ReconnectSupervisor` use — grep for a shared `ExponentialBackoff`/`Jitter` type; if none is shared, extract one into the Galaxy `Runtime/` folder and use it in all three (aligns with OVERALL theme #2 "fixes flow to shared templates"). + +**Implementation steps:** +- Introduce/reuse a `Backoff` primitive (min, max, jitter). Wire into `GatewayGalaxyAlarmFeed` ctor (min/max from `GalaxyReconnectOptions`, defaults 1 s→30 s). Reset on successful open. + +**Tests (unit):** the feed uses an injected `_streamFactory` and a settable delay; assert successive faults produce increasing delays capped at max and reset after a successful open (drive via a fake factory + a virtual clock or by asserting the computed delay sequence). + +**Effort:** S. **Risk:** Low. + +--- + +### P2-A — C-5: delete retired Wonderware project directories + +**Verification:** Confirmed — `Historian.Wonderware`, `.Client`, `.Client.Contracts` dirs present; `grep -c Wonderware slnx == 0`. + +**Design/steps:** `git rm -r` the three `src/Drivers/*Wonderware*` dirs and their `tests/` counterparts; confirm no `ProjectReference`/`using` survives (grep). This overlaps OVERALL action #11 and Report 07 hygiene — coordinate so it's deleted once. History preserves the code. + +**Effort:** S. **Risk:** Very low (0 references). Verify with a full `dotnet build ZB.MOM.WW.OtOpcUa.slnx` after removal. + +--- + +### P2-B — C-6: EventPump silently drops unknown event families with no metric + +**Verification:** Confirmed. `EventPump.cs:199-206` `default: return;` — no counter. + +**Design/steps:** add a `galaxy.events.filtered` (or `.unknown_family`) counter incremented in the `default` branch of `Dispatch`, tagged with the family value. Mirrors the alarm feed's decode-drop metering. Keep the filtering behavior — only add observability, so a future gateway family (`OnBufferedDataChange`) surfaces instead of vanishing. + +**Tests (unit):** feed the pump an `MxEvent` with a non-`OnDataChange` family via the injected subscriber fake → assert the filtered counter incremented (MeterListener) and no `OnDataChange` fired. + +**Effort:** S. **Risk:** Very low. + +--- + +### P2-C — U-3: WriteSecured / VerifiedWrite user identity stubbed at zero + +**Verification:** Confirmed. `GatewayGalaxyDataWriter.cs:263-264` hardcodes `CurrentUserId=0, VerifierUserId=0`. + +**Root cause:** No mapping from the OPC UA session's LDAP-authenticated principal to an ArchestrA user id; the secured-write path was stubbed to unblock the common (non-secured, supervisory) path. + +**Design.** Two-part, and honest about the gap: +1. **Fail-fast interim (this PR):** when a tag classifies `SecuredWrite`/`VerifiedWrite` and no ArchestrA user-id mapping is configured, return a clear Bad status (`BadUserAccessDenied` / `BadNotSupported`) and log, rather than silently sending user 0 (which a real galaxy will reject or mis-attribute). Prevents a confusing "commit succeeded as user 0" or opaque rejection. Also surfaces to the operator that secured-classification tags are unsupported today. +2. **Real fix (cross-cutting, larger):** thread the OPC UA session identity down to the writer. The LDAP principal is available server-side (see `RoleCarryingUserIdentity` used by alarm ack). Add an ArchestrA-user-id mapping (config `GroupToArchestraUserId` or a resolver) and pass `CurrentUserId`/`VerifierUserId` into `InvokeWriteSecuredAsync`. This spans the node-write router → driver `WriteRequest` (needs a user-id/identity field on the request contract) → writer. Coordinate with Report 03 (session identity) and the node-write router. VerifiedWrite additionally needs a *second* verifier identity (dual-authorization) — that is a genuine feature, not a plumbing fix; scope it separately. + +**Implementation steps (interim):** +- `WriteOneAsync`: when `NeedsSecuredWrite(classification)` and no user-id source is wired → return `BadNotSupported`/`BadUserAccessDenied`, log, do not send. Document in `docs/` (Galaxy driver / Historian split) that secured/verified writes require the identity-mapping feature. + +**Tests (unit):** classification resolver returns SecuredWrite + no identity → assert Bad status and no `InvokeAsync` call. + +**Effort:** S (interim), L (real identity threading — cross-report). **Risk:** interim is Low and strictly safer than sending user 0. + +--- + +### P2-D — U-4 / P-5: gateway-gap client-side workarounds; clamp unbounded events read + +**Verification:** Confirmed. +- `HistorianGatewayClientAdapter.cs:88` calls `_inner.ReadEventsAsync(startUtc, endUtc, filter, ct)` — `maxEvents` not on the wire (client-side cap only). +- `GatewayHistorianDataSource.cs:142-168` — `maxEvents<=0` collects unbounded (only the gateway's `RuntimeDb:EventReadMaxRows` bounds it). +- Client-side source filter re-applied defensively at 152-157. +- `GalaxyDriver.cs:314-316` replay still per-subscription (batched `ReplaySubscriptionsCommand` note). + +**Design/steps:** +1. **In-repo:** clamp `maxEvents<=0` to a server-side default cap (new `ServerHistorianOptions.DefaultMaxEventsPerRead`, default e.g. 10 000) in `GatewayHistorianDataSource.ReadEventsAsync` so a 0 from the SDK can't drain unboundedly. This is the P-5 fix and is fully in-repo. +2. **Cross-repo (sidecar backlog):** file tracked items for (a) a wire-level `maxEvents` on `ReadEventsRequest`, (b) a server-side source filter so the defensive client filter can be removed, (c) a batched `SendEvents` (P-2), and (d) a batched `ReplaySubscriptionsCommand` (mxaccessgw). Link each from the corresponding in-source comment. These are *not* in-repo fixes — they belong on HistorianGateway / mxaccessgw backlogs; the driver's workarounds stay until they land. + +**Tests (unit):** `FakeHistorianGatewayClient` streaming > cap events with `maxEvents=0` → assert the data source stops at `DefaultMaxEventsPerRead` and flags truncation. + +**Effort:** S (clamp). **Risk:** Low. **Note:** OVERALL theme #4 — track the gateway gaps on the sister repos so they don't rot. + +--- + +### P3-A — S-2: EventPump saturation drops the newest events (inverted staleness bias) + +**Verification:** Confirmed. Newest-dropped by construction (`TryWrite` on a full bounded channel). Metered (`galaxy.events.dropped`). + +**Root cause:** Bounded channel with `FullMode.Wait` + `TryWrite` preserves the oldest queued events and drops the just-arrived one — wrong for last-value-wins telemetry. + +**Design.** Two candidates: +1. **Per-item-handle conflation (best correctness):** keep newest-per-handle. Replace the raw `Channel` with a small conflating structure: a `ConcurrentDictionary` (handle→latest) + a signal channel; the dispatch loop drains the dict. On overflow the newest value for a handle overwrites the older one — no data staleness, bounded by tag count not event rate. More code; changes the pump's core. +2. **Drop-oldest ring (cheap):** switch the channel to `BoundedChannelFullMode.DropOldest`. But the in-source comment explicitly rejected `DropWrite`/`DropOldest` because they discard *inside* Channel without surfacing on the counter. To keep metering, wrap: on a full channel, `TryRead` one (count it as dropped-oldest) then `TryWrite` the new one. Simpler than conflation, keeps recent data, preserves the counter. + +Recommendation: **option 2** (drop-oldest-with-metering) as the low-risk win; option 1 only if soak testing shows a hot-handle storm still loses fresh values. Given P-4 notes the fan-out is already O(1) and the channel default is 50 000, saturation is an edge case — option 2 is proportionate. + +**Implementation steps:** +- In `RunAsync`, replace the `if (!TryWrite) count` block with: `while (!TryWrite(ev)) { if (TryRead(out _)) EventsDroppedOldest.Add(1,tag); }` (bounded loop). Add a `galaxy.events.dropped_oldest` counter (or reuse `dropped`). Update the FullMode comment to match. + +**Tests (unit):** fill the channel (tiny capacity via ctor), push N more → assert the newest are retained (read the dispatched values) and the dropped counter reflects the discarded oldest. + +**Effort:** S-M. **Risk:** Medium — touches the hot pump loop; guard with the existing pump bounded-channel tests + a soak run. Only pursue if S-2 is judged worth the churn (report rates it Medium; it's already metered, so it's a correctness-under-stress refinement). + +--- + +### P3-B — S-8: `GalaxyMxSession` state unsynchronized across reopen (document the invariant) + +**Verification:** Confirmed. Plain `_session`/`_connected` fields; `RecreateAsync` tears down without a lock. Report rates it acceptable (emergent safety via single-flight supervisor + per-call try/catch). + +**Design/steps:** lowest-effort resolution the report endorses — **document the invariant** on the `Session`/`IsConnected` members: "single-flight recovery (ReconnectSupervisor) is the only writer of session state; concurrent readers tolerate a torn read because every per-call failure maps to BadCommunicationError and caches are invalidated post-reopen." If a defensive change is wanted, mark `_connected` `volatile` and read `_session` into a local once per operation (already the pattern in `WriteAsync`). Do **not** add a lock (would serialize the hot path for no proven benefit). + +**Effort:** XS (doc) / S (volatile). **Risk:** None (doc) / Low. + +--- + +### P3-C — U-5: outbox serializer has no version/format byte + +**Verification:** Confirmed. `HistorizationOutboxEntrySerializer.cs:14-16` fixed positional layout, no prefix. + +**Design/steps:** prepend a single version byte (`0x01`) in `Serialize`; in `Deserialize` read it first and branch (today: only v1). One reserved byte now makes any future field addition a clean migration instead of an undiagnosable recovery failure mid-`RecoverState`. **Migration caveat:** on-disk entries written before this change have no version byte — deploying this reader against a non-empty pre-existing outbox would misread. Since the outbox is drained-and-truncated continuously and is dev-only so far, gate the change on an empty-outbox deploy, or make the reader tolerant (peek: if the first byte isn't a known version AND the buffer length matches the legacy layout, treat as v0). Prefer the tolerant reader for safety. + +**Tests (unit):** round-trip v1; legacy-layout bytes (no version) still deserialize under the tolerant reader; unknown version → clear exception. + +**Effort:** S. **Risk:** Low-Medium (the on-disk migration is the only sharp edge — the tolerant reader defuses it). + +--- + +### P3-D — C-3, C-4: convention notes (documentation only) + +- **C-3:** add a one-line guard note to `IHistorianGatewayClient`'s XML doc: "any consumer beyond `GatewayHistorianDataSource` must sit above the `Mapping/` anti-corruption boundary, never on this proto-typed seam." No code change. **Effort:** XS. +- **C-4:** pick one options-validation idiom for *new* driver-adjacent sections — the report (and I) recommend the Historian `Validate()`-returns-warnings style as the friendlier operational contract. Document the choice in CLAUDE.md / a conventions doc; do not churn the existing Galaxy throw-on-construct code. **Effort:** XS (doc). + +--- + +### Accepted / no-action + +- **S-9, S-10, P-4, C-1, C-2** — positives; preserve as-is (don't regress the store-and-forward discipline, the O(1) fan-out indexes, or the secret resolver). +- **P-1** — MxAccess-forced 3-round-trip read; action is documentation ("subscribe, don't poll") + optional read-through cache only if polling clients appear. Fold the doc line into `docs/` Galaxy section. **Effort:** XS. +- **P-2, P-3, P-6** — accepted, gateway-constrained or negligible; P-2 tracked on the sidecar backlog (see U-4). + +--- + +## Suggested execution order (batched) + +1. **Batch 1 (P0, failure-visibility):** S-1 (in-repo layers) + S-6 (refresh wiring) + U-1 (doc fix + convergence test). Ship together; all three are OVERALL theme #1/#3. Live-verify S-1 write and U-1 value-capture on the real gateways. +2. **Batch 2 (P1, fault resilience):** S-4 + S-3 + S-5 + S-7. Cohesive "recovery hardening" set, all unit-testable with existing fakes. +3. **Batch 3 (P2, hygiene/interim):** C-5 (delete dirs) + C-6 (metric) + U-3 (fail-fast) + U-4/P-5 (clamp) + C-3/C-4 (doc). Coordinate C-5 with Reports 03/07. +4. **Batch 4 (P3, refinements):** S-2 (drop-oldest) + S-8 (doc/volatile) + U-5 (version byte) + P-1 doc. Optional / as-capacity. + +Cross-repo follow-ons to file (not in-repo): mxaccessgw `WriteComplete` correlation (S-1) and batched `ReplaySubscriptionsCommand` (U-4); HistorianGateway wire-level `maxEvents`, server-side source filter, batched `SendEvents` (U-4/P-2). Link each from the existing in-source comment. diff --git a/archreview/plans/07-client-tooling-engineering-plan.md b/archreview/plans/07-client-tooling-engineering-plan.md new file mode 100644 index 00000000..c07a81c4 --- /dev/null +++ b/archreview/plans/07-client-tooling-engineering-plan.md @@ -0,0 +1,521 @@ +# Plan 07 — Client Tooling, Analyzers & Cross-Cutting Engineering System + +| | | +|---|---| +| **Source report** | `archreview/07-client-tooling-engineering.md` | +| **Overall context** | `archreview/00-OVERALL.md` (theme #4 "verification system", action #6/#11) | +| **Base commit** | `9cad9ed0` (master); verified against current working tree 2026-07-08 | +| **Domain** | Client.CLI/Shared/UI, Analyzers, build/CPM, CI, test architecture, repo hygiene | + +## Verification summary + +26 findings in the report (S-1..S-8, P-1..P-4, C-1..C-6, U-1..U-8). Every actionable +finding was re-verified against the live tree; **all confirmed, none stale**. Four items are +explicitly "no-action / positive" in the report (P-3, P-4, C-3, U-1) and are recorded but not +planned. Key verifications performed: + +- **C-1 CONFIRMED** — repo-wide grep for `ZB.MOM.WW.OtOpcUa.Analyzers` in `.csproj/.props/.targets` + returns only the analyzer's own csproj and its test project's `ProjectReference`. Zero + `OutputItemType="Analyzer"` anywhere in the tree (the only hit is inside the review markdown). +- **S-1/S-2 CONFIRMED** — `v2-ci.yml:47-52` gates exactly 5 unit projects; slnx contains 47 + `*.Tests.csproj` (10 of them `*.IntegrationTests`). Integration job (`:61-76`) runs + fixture-less on `ubuntu-latest` with no `services:`. Category traits are declared via + `[Trait("Category", "…")]`, so a `Category!=E2E&Category!=LiveIntegration` filter is valid. +- **S-3 CONFIRMED** — grep for `Category…E2E` across `tests/**/*.cs` returns zero matches; the + nightly workflow's own header admits the no-op. +- **S-7 CONFIRMED** — no `global.json` at root (nor anywhere); `v2-ci.yml:10-12` claims one exists. +- **U-3 CONFIRMED** — `docs/Client.CLI.md` has zero sections for `confirm`, `shelve`, `disable` + (and only substring hits for `ack`/`enable`); all 5 alarm commands ship as command files. +- **U-4 CONFIRMED** — `git ls-files` shows `pending.md`, `current.md`, `looseends.md`, + `stillpending.md`, `HISTORIAN-GATEWAY-INTEGRATION-ISSUES.md` all tracked; `pending.md:3` itself + declares "HARD RULE: never stage `pending.md` / `current.md`". +- **U-5 CONFIRMED** — `git ls-files lib/` lists 7 tracked proprietary AVEVA DLLs; zero `HintPath` + hits reference `lib/` anywhere. +- **U-6 CONFIRMED** — `sql_login.txt` is `.gitignore:47`, `git check-ignore` matches, `git log` + empty (never committed). The retired `Driver.Historian.Wonderware*` dirs under `src/`+`tests/` + are untracked `bin/obj` husks (NOT tracked projects; not in slnx). Tracked Wonderware residue is + only `docs/drivers/Historian.Wonderware.md` (an intentional retired-stub) and 3 + `code-reviews/Driver.Historian.Wonderware*/findings.md`. +- **C-2 CONFIRMED** — none of the 4 client/tooling src csprojs set `TreatWarningsAsErrors`. +- **C-5 CONFIRMED** — no `.editorconfig`; `StyleGuide.md:3` says "for all **ScadaBridge** + documentation". +- **S-5/S-6/U-8 CONFIRMED** — `OpcUaClientService.cs` lock gaps at cited lines; `CommandBase.cs:123` + `Log.CloseAndFlush()` + `:133` global `Log.Logger` swap; `CommandBase.cs:91` + `AutoAcceptCertificates = true`. + +--- + +## Priority ordering + +**Tier A — restore meaning to green CI (do first):** C-1, S-1, S-2, S-3, S-7. +**Tier B — repo hygiene:** U-5, U-4, U-6, C-5 (StyleGuide title). +**Tier C — docs:** U-3. +**Tier D — engineering debt (as capacity allows):** C-2, P-1, U-2, S-5, S-6, S-4, C-4, C-6, S-8, U-8, P-2. +No-action (recorded only): P-3, P-4, C-3, U-1, U-7. + +--- + +## TIER A — Restore meaning to green CI + +### C-1 (High) — Wire the OTOPCUA0001 analyzer into every project + +**Restatement:** the sole custom analyzer enforces its CapabilityInvoker-wrapping rule against +nothing but its own 31 unit tests; no consuming project references it. + +**Verification:** confirmed — only self + test `ProjectReference`; zero `OutputItemType="Analyzer"`. + +**Root cause:** the "built-but-never-wired" house failure mode (theme #1). The analyzer project was +authored and unit-tested but the DI/build-graph wiring step was never done. + +**Design:** inject the analyzer as an analyzer-type `ProjectReference` from `Directory.Build.props`, +conditioned to exclude the analyzer project itself (and its own build, to avoid a self-reference +cycle). Analyzer projects target `netstandard2.0`, so `ReferenceOutputAssembly="false"` + +`OutputItemType="Analyzer"` is the correct incantation — the analyzer DLL is loaded into the +compiler host, not linked. This makes OTOPCUA0001 run on every `src/` and `tests/` compilation. + +Add to `Directory.Build.props` (after the existing `ItemGroup`): +```xml + + + +``` +(The `.Analyzers.Tests` exclusion is belt-and-suspenders: it already `ProjectReference`s the analyzer +directly; letting `Directory.Build.props` add a second analyzer-typed reference is harmless but noisy.) + +**Triage sub-step (required, do NOT skip):** OTOPCUA0001 is a diagnostic. Its default severity must +be checked in `UnwrappedCapabilityCallAnalyzer` — if it is `Warning` and a src project has TWE on, +the wired-in analyzer can **break the build** on the first unwrapped call it finds. So the wiring +must be done in this order: +1. Wire the `ProjectReference` (above). +2. `dotnet build ZB.MOM.WW.OtOpcUa.slnx` and capture every OTOPCUA0001 hit. +3. For each hit, decide: real violation → route the call through `CapabilityInvoker`; intentional + (test doubles, the invoker's own internals) → add a scoped `#pragma warning disable OTOPCUA0001` + or a per-project `OTOPCUA0001` with a comment. The analyzer's own tests + demonstrate the intended pattern. +4. Only after the tree is clean should any project promote OTOPCUA0001 to error (out of scope here). + +**Files to touch:** `Directory.Build.props` (add ItemGroup); plus any `.csproj` needing `NoWarn` +and any source files with genuine unwrapped calls surfaced by step 2. + +**Verify:** `dotnet build ZB.MOM.WW.OtOpcUa.slnx` shows OTOPCUA0001 diagnostics on real code (proving +it is live); a deliberately-unwrapped call in a scratch file trips it; the analyzer's 31 tests still +pass. Confirm the build graph does not regress (`dotnet build` clean after triage). + +**Effort:** M (wiring is S; the triage wave is the unknown — could be S or M depending on how many +unwrapped calls exist). **Risk:** Medium — a Warning-severity analyzer on TWE projects can fail the +build; the ordered triage above contains it. Blast radius is every compilation, so build once +locally before pushing. + +--- + +### S-1 (High) — CI gates ~15% of the test matrix + +**Restatement:** `v2-ci.yml` runs 5 of 47 unit-test projects; Client (388 tests), Analyzers (31), +all driver and most Core suites never run in CI. + +**Verification:** confirmed (5 enumerated, 47 total). + +**Root cause:** hand-maintained matrix predates most projects and was never widened; new projects +drift out of coverage silently. + +**Design:** replace the hand-maintained `unit-tests` matrix with a single solution-wide leg that +excludes only the env-gated tiers. The skip-gated `*.IntegrationTests` fixtures already tolerate +unreachable endpoints (they `Assert.Skip`), so running them here is safe; but to keep the unit leg +fast and deterministic, split responsibilities: + +- **`unit-tests` job:** one step — + `dotnet test ZB.MOM.WW.OtOpcUa.slnx --configuration Release --no-restore + --filter "Category!=E2E&Category!=LiveIntegration" --logger "trx;LogFileName=unit.trx"` + after an explicit `dotnet restore` + `dotnet build --no-restore` (share build via `--no-build` + where possible; see P-1). This runs all 47 projects in one invocation — no matrix to drift. + +Rationale for the whole-solution filter over regenerating a matrix from the slnx: it is +self-maintaining (new project = automatically covered), matches CLAUDE.md's own +`dotnet test ZB.MOM.WW.OtOpcUa.slnx` guidance, and the trx output feeds S-2's skip-gate. The +integration split (Host.IntegrationTests / OpcUaServer.IntegrationTests as a distinct matrix leg) +stays, but is folded into the fail-on-skip work below (S-2). + +**Files to touch:** `.github/workflows/v2-ci.yml` — replace the `unit-tests` job body. + +**Verify:** push to a branch; the `unit-tests` job log shows Client.CLI/Shared/UI + Analyzers + +driver + Core suites executing (not skipped); wall-clock is acceptable (pair with P-1 caching). + +**Effort:** S. **Risk:** Medium — first full-solution CI run will surface latent failures / +flakes that were never gated (esp. the fixed-sleep timing tests, S-4). That is the point, but budget +time to stabilize. Blast radius: CI only. + +--- + +### S-2 (High) — "green" integration CI means "skipped"; skip is indistinguishable from pass + +**Restatement:** the integration job runs fixture-less against `10.100.0.35` defaults → probe fails +→ `Assert.Skip` → job green. A real fixture outage cannot be told from a real pass. + +**Verification:** confirmed — no `services:` in the integration job; fixtures hard-default to +`10.100.0.35` (Modbus/AbServer/Snap7/OpcPlc fixtures + `DriverTestConnectE2eTests`). + +**Root cause:** the probe-once skip pattern is correct for dev boxes, but CI has no signal that the +tier ran nothing. + +**Design — two complementary changes:** + +1. **Make skips visible/failing.** Emit `--logger "trx"` from the integration leg and add a post-test + step that parses the trx and **fails the job when skipped-count exceeds a threshold** (e.g. any + integration test skipped when CI was supposed to have fixtures, or > N total). A small inline shell + or a tiny `scripts/ci/assert-not-all-skipped.ps1`/`.sh` reading the ``/`outcome="NotExecuted"` + counts. This is the minimal, always-applicable guard. + +2. **(Optional, higher value) Actually start the reachable fixtures as workflow `services:`.** The + Modbus sim and opc-plc images run on any hosted runner; add them as `services:` and set + `MODBUS_SIM_ENDPOINT=localhost:5020` / `OPCUA_SIM_ENDPOINT=opc.tcp://localhost:50000` so those + probes connect and the tests actually run. S7/AbCip/Galaxy fixtures have no hosted-runner image and + stay legitimately skipped (guard #1 excludes them from the threshold, or scopes the threshold to + the modbus/opc-plc subset). + +Recommend shipping **guard #1 first** (cheap, universal), then #2 as a follow-up. + +**Files to touch:** `.github/workflows/v2-ci.yml` (integration job: add trx logger, post-step, and +optionally `services:` + endpoint env); optionally a new `scripts/ci/assert-not-all-skipped.*`. + +**Verify:** deliberately point an endpoint at an unreachable host in a branch → integration job goes +**red** (not green). With services up, the modbus/opc-plc tests show as passed, not skipped. + +**Effort:** S (guard #1) / M (with services). **Risk:** Low-Medium — a too-tight threshold could +red-flag legitimately-unavailable fixtures; scope the threshold to the images CI can actually run. + +--- + +### S-3 (Medium) — nightly E2E workflow is a permanent green no-op + +**Restatement:** `v2-e2e.yml` filters `Category=E2E`, which matches zero tests; it boots the full +docker-dev fleet for nothing and trains people to ignore a nightly green. + +**Verification:** confirmed — zero `Category=E2E` in the tree; the workflow header admits it. + +**Root cause:** the E2E project (`tests/Server/ZB.MOM.WW.OtOpcUa.E2ETests`) was scoped but never +landed. + +**Design (pick one):** +- **(a) Disable the schedule until the project exists** — comment out the `schedule:` trigger, keep + `workflow_dispatch`, and add a guard step that fails if `Category=E2E` matches zero tests (so the + no-op can't silently return). Lowest effort; removes the false-green. +- **(b) Land a minimal E2E round-trip** — a single `[Trait("Category","E2E")]` test that, against the + booted docker-dev fleet, connects via `Client.Shared`/`Client.CLI` to `opc.tcp://localhost:4840`, + browses to a known seeded node, reads a value, and asserts Good. The `scripts/e2e/test-*.ps1` + harnesses show exactly what to assert. This gives the nightly job real meaning and exercises the + otherwise-uncovered Client stack end-to-end. + +Recommend **(a) now** (removes the false signal immediately) with **(b) as the tracked follow-on** — +option (a)'s zero-match guard also protects (b) from regressing back to a no-op. + +**Files to touch:** `.github/workflows/v2-e2e.yml`; for (b) a new `tests/Server/…E2ETests` project + +slnx entry. + +**Verify:** (a) manual dispatch fails-fast on zero E2E tests; (b) the round-trip test passes against +the fleet and fails if the seeded node is absent. + +**Effort:** S (a) / M-L (b). **Risk:** Low. + +--- + +### S-7 (Low) — no `global.json` though the build depends on an exact SDK band + +**Restatement:** `v2-ci.yml:10-12` claims a repo-root `global.json` pins the SDK; none exists. +Meanwhile `Directory.Packages.props:44-49` pins Roslyn 5.0.0 to match SDK 10.0.105's compiler. + +**Verification:** confirmed — no `global.json`; the CI comment is false. + +**Root cause:** the pin comment describes intended state that was never created; the Roslyn pin note +implicitly depends on SDK version stability the repo doesn't enforce. + +**Design:** add a root `global.json`: +```json +{ + "sdk": { + "version": "10.0.105", + "rollForward": "latestFeature" + } +} +``` +`latestFeature` allows patch/feature SDK rolls within the 10.0.1xx band but keeps the major/minor +stable, so the Roslyn 5.0.0 pin's stated condition ("until the SDK rolls to 10.0.110+") becomes a +deliberate, visible decision rather than a silent drift. Cross-check the exact installed SDK version +before committing the `version` value. Update the Roslyn-pin comment in `Directory.Packages.props` to +reference `global.json` as the coupling point. + +**Files to touch:** new `global.json`; comment tweak in `Directory.Packages.props`. + +**Verify:** `dotnet --version` in-repo resolves within the band; CI `setup-dotnet` honors it (the +workflow comment becomes true). Build + tests unchanged. + +**Effort:** S. **Risk:** Low — a too-strict `version`/`rollForward` could fail a runner lacking that +exact SDK; `latestFeature` mitigates. + +--- + +## TIER B — Repo hygiene + +### U-5 (Medium) — orphaned proprietary AVEVA DLLs tracked in `lib/` + +**Restatement:** 7 committed vendor binaries in `lib/`, referenced by nothing — a +redistribution/licence risk in every clone. + +**Verification:** confirmed — `git ls-files lib/` lists all 7; zero `HintPath` hits repo-wide. + +**Root cause:** leftovers from the retired in-process MXAccess / Wonderware sidecar era; the +bitness/COM story now lives entirely in the mxaccessgw repo (per CLAUDE.md). + +**Design:** `git rm -r lib/`. Before deleting, re-confirm nothing references them: +``` +grep -rn "aahClient\|ArchestrA\|Historian.CBE\|Historian.DPAPI\|HintPath.*lib" \ + --include='*.csproj' --include='*.props' --include='*.targets' . +``` +(expected: no hits outside the review docs). Optionally add `lib/` to `.gitignore` to prevent +re-introduction, and note in the commit that the DLLs remain retrievable from history / the AVEVA SDK +if ever needed. Purging them from *history* is a separate, heavier decision (BFG/filter-repo) — out +of scope; the immediate licence-hygiene win is removing them from the tip. + +**Files to touch:** delete `lib/*` (7 files); optional `.gitignore` entry. + +**Verify:** `git status` clean; `dotnet build ZB.MOM.WW.OtOpcUa.slnx` still succeeds (proves nothing +linked them); `git ls-files lib/` empty. + +**Effort:** S. **Risk:** Low — verified zero references; blast radius nil. + +--- + +### U-4 (Medium) — root planning-file sprawl contradicts its own "never stage" rule + +**Restatement:** `pending.md`, `current.md`, `looseends.md`, `stillpending.md`, +`HISTORIAN-GATEWAY-INTEGRATION-ISSUES.md` are committed at root; `pending.md:3` itself says never +stage `pending.md`/`current.md`. Content is stale snapshots now contradicting `docs/` + CLAUDE.md. + +**Verification:** confirmed — all 5 tracked; the hard-rule text present. + +**Root cause:** working-notes files that were staged despite the rule, and never cleaned up as the +backlog shipped (~95% per the memory index). + +**Design — make the file's own rule true:** +1. **`pending.md` + `current.md`** — untrack but keep locally (they're active working notes): + `git rm --cached pending.md current.md`, then add both to `.gitignore`. This exactly satisfies + the "never stage" rule. +2. **`looseends.md`, `stillpending.md`, `HISTORIAN-GATEWAY-INTEGRATION-ISSUES.md`** — these are + point-in-time snapshots of shipped work. Preferred: move to `docs/plans/` with dated names (the + established convention), e.g. `docs/plans/2026-05-18-looseends.md`, so the history is preserved + as an archived plan rather than a live-looking root file. Acceptable alternative: `git rm` them + outright since the memory index records the backlog as shipped and `docs/`+CLAUDE.md now carry the + authoritative state. Recommend **archive-move** for `stillpending.md`/`HISTORIAN-GATEWAY-…` (they + have referenced history) and **delete** for `looseends.md` if fully superseded — confirm with a + quick read before choosing per-file. + +**Files to touch:** `.gitignore` (+2 entries); `git rm --cached` pending/current; move-or-delete the +3 snapshots; `docs/plans/` if archiving. + +**Verify:** `git ls-files | grep -E '^(pending|current|looseends|stillpending|HISTORIAN)'` returns +only any intentionally-archived `docs/plans/*` paths; `git status` shows pending.md/current.md now +untracked (ignored). `pending.md`'s hard-rule is now consistent with reality. + +**Effort:** S. **Risk:** Low — pure VCS hygiene; keep local copies (`--cached`) so no working data is +lost. + +--- + +### U-6 (Low) — secrets hygiene + retired-dir husks + +**Restatement:** `sql_login.txt` is NOT committed (gitignored) but is a plaintext credential on +Desktop; retired `Driver.Historian.Wonderware*` dirs linger on disk as bin/obj husks. + +**Verification:** confirmed — `sql_login.txt` `.gitignore:47`, `git log` empty; the 8 +`Driver.Historian.Wonderware*` dirs under `src/`+`tests/` are untracked (only `docs/` stub + 3 +`code-reviews/…/findings.md` are tracked residue). + +**Root cause:** local dev debris; the credential file is a convenience left in the repo dir. + +**Design:** +- **`sql_login.txt`** — no tracking action needed (already ignored). Recommend moving the credential + to `dotnet user-secrets` or an env file *outside* the repo directory, and audit + `export-clean-copy.bat` at root to confirm it excludes `sql_login.txt` (and `lib/`, pki, planning + files) from any export bundle. Low-priority, local-only. +- **Wonderware husks** — delete the untracked `src/Drivers/…Wonderware*` and + `tests/Drivers/…Wonderware*` directories from disk (`rm -rf`, they're pure `bin/obj`). Decide + separately whether to also remove the 3 tracked `code-reviews/Driver.Historian.Wonderware*/` + findings and keep `docs/drivers/Historian.Wonderware.md` as the intentional retired stub (the doc + is deliberately retained per CLAUDE.md; leave it). + +**Files to touch:** none tracked for the credential; `rm -rf` the untracked husks; audit +`export-clean-copy.bat`. + +**Verify:** `find src tests -type d -iname '*Wonderware*'` returns nothing; `git status` unchanged by +the husk deletion (they were untracked); `export-clean-copy.bat` excludes the sensitive files. + +**Effort:** S. **Risk:** Low. + +--- + +### C-5 (Low) — no `.editorconfig`; StyleGuide.md mis-titled for a sister repo + +**Restatement:** no mechanical code-style enforcement; `StyleGuide.md:3` still says "for all +**ScadaBridge** documentation" (copy-paste from the sibling repo). + +**Verification:** confirmed — no `.editorconfig`; `StyleGuide.md:3` wording present. + +**Root cause:** style held by review culture only; StyleGuide copied from ScadaBridge without +re-titling. + +**Design — two independent, small changes:** +1. **StyleGuide fix (trivial, do now):** replace "ScadaBridge" with "OtOpcUa" in `StyleGuide.md:3` + (and scan the file for other ScadaBridge leftovers). +2. **`.editorconfig` (optional):** add a root `.editorconfig` codifying the *already-consistent* + conventions — file-scoped namespaces, 4-space indent, `_camelCase` private fields, `var` usage — + set as `suggestion`/`warning` (not `error`) to avoid a big-bang reformat. This is documentation-of- + convention, not a reformatting mandate; a `dotnet format` CI step is a further follow-on (U-7), + deliberately deferred so it doesn't churn the tree. + +**Files to touch:** `StyleGuide.md`; optional new `.editorconfig`. + +**Verify:** StyleGuide title reads OtOpcUa; `.editorconfig` doesn't flip existing warnings to errors +(build unchanged). + +**Effort:** S. **Risk:** Low (keep severities non-error). + +--- + +## TIER C — Documentation + +### U-3 (Medium) — `docs/Client.CLI.md` documents 8 of 14 commands + +**Restatement:** `ack`/`acknowledge`, `confirm`, `shelve`, `enable`, `disable` have no doc sections, +though all ship and CLAUDE.md points operators at this doc. + +**Verification:** confirmed — `confirm`/`shelve`/`disable` = 0 mentions; `ack`/`enable` only +substring hits; command files exist for all 5. + +**Root cause:** the inbound operator ack/shelve feature shipped its runtime + CLI but the CLI doc was +never extended. + +**Design:** add five command sections to `docs/Client.CLI.md` mirroring the existing section shape +(synopsis, options, example invocation, expected output). Source the exact options/args from the +command classes: `AcknowledgeCommand.cs`, `ConfirmCommand.cs`, `ShelveCommand.cs`, `EnableCommand.cs`, +`DisableCommand.cs` (and their shared `CommandBase` connection options). Group them under an "Alarm +operator commands" heading since they are the operator-facing ack/shelve workflow. Cross-check against +`docs/ScriptedAlarms.md` §"Inbound operator ack/shelve" for terminology consistency. + +**Files to touch:** `docs/Client.CLI.md`. + +**Verify:** each of the 5 commands has a section; a `-h`/`--help` run of each command matches the +documented options. Optionally add a CI docs-coverage check (U-7 follow-on) asserting every +`Commands/*Command.cs` has a doc section. + +**Effort:** S. **Risk:** Low. + +--- + +## TIER D — Engineering debt (as capacity allows) + +These are real but lower-leverage than restoring CI meaning. Batched with condensed treatment. + +### C-2 (Medium) — TWE never reached the Client/Tooling slice +Add `true` to `Client.CLI.csproj`, +`Client.Shared.csproj`, `Client.UI.csproj`, `Analyzers.csproj` (v2-era code; the legacy xUnit1051 +blocker doesn't apply). **Do this AFTER C-1's triage** — with the analyzer wired in, a Warning-level +OTOPCUA0001 would now fail these builds, so sequence C-1 triage → C-2. Verify each builds clean with +TWE on. Effort S; Risk Low (build once locally first). Burn down the 38 test-project TWE gaps per +module as a tracked follow-on. + +### P-1 (Medium) — CI does 7 uncached restore+build passes +Add `actions/cache` for `~/.nuget/packages` keyed on a hash of `Directory.Packages.props` + +`NuGet.config`; do one explicit `restore`, then `build --no-restore`, then `test --no-build` in the +unit leg (dovetails with S-1's single-leg design). Effort S; Risk Low; biggest CI wall-clock win. + +### U-2 (Medium) — add `Host.Tests` (the one real coverage hole) +Create `tests/Server/ZB.MOM.WW.OtOpcUa.Host.Tests` asserting DI composition — e.g. "when +`ServerHistorian:Enabled`, `IHistorianProvisioning` resolves to `GatewayTagProvisioner` and the +applier receives it" — the exact class of dormant-wiring bug (PR #423, DeferredAddressSpaceSink) that +unit tests would have caught. Add to slnx so S-1's whole-solution CI picks it up. Effort M; Risk Low. + +### S-5 (Medium) — Client.Shared subscription lock-discipline gaps +In `OpcUaClientService.cs`: (a) move the `_dataSubscription` null-check/create inside an async gate +mirroring `_alarmSubscribeSemaphore` (line ~262); (b) take `_subscriptionLock` in `RunFailoverAsync` +when nulling `_dataSubscription`/`_alarmSubscription` (lines ~695-696); (c) route +`UnsubscribeAlarmsAsync` (line ~335) through `_alarmSubscribeSemaphore`; (d) observe exceptions from +the fire-and-forget keep-alive failover. Near-unhittable for the session-per-command CLI; real for the +long-lived Client.UI service. Effort M; Risk Medium (concurrency code — add targeted tests). Do before +Client.UI grows multi-threaded usage. + +### S-6 (Medium) — global `Log.Logger` swap per CLI command +`CommandBase.ConfigureLogging()` (`:120-134`) does `Log.CloseAndFlush()` + replaces static +`Log.Logger` per command — a race under xunit.v3's parallel collections and hostile to embedding. +Give each command an instance `ILogger` from a per-execution `LoggerConfiguration.CreateLogger()`. +The existing `LoggerLifecycleTests` polices this. Effort M; Risk Low-Medium (touches every command's +logging). + +### S-4 (Medium) — fixed-sleep timing tests +Replace start-up `await Task.Delay(100)` sleeps in `Client.CLI.Tests` (`SubscribeCommandTests`, +`AlarmsCommandTests`, `EventHandlerLifecycleTests`) with a readiness signal — a `TaskCompletionSource` +completed on first `SubscribeAsync` exposed by `FakeOpcUaClientService`. Currently masked because CI +never runs them (S-1); will flake once S-1 lands, so **pair with S-1**. Effort M; Risk Low. + +### C-4 (Low) — finish xunit.v3 migration +Migrate the 3 holdouts (`AdminUI.Tests`, `ControlPlane.Tests`, `Runtime.Tests`) off xunit 2.9.2, then +drop the `xunit`/`xunit.runner.visualstudio` 2.x pins from `Directory.Packages.props:108-109`. +Optionally relocate `GatewayLiveIntegrationTests` to a `*.IntegrationTests` sibling. Effort M; Risk +Low. + +### C-6 (Low) — csproj boilerplate duplicates Directory.Build.props +Strip redundant `TargetFramework`/`Nullable`/`ImplicitUsings` from the ~70 csprojs on next sweep so an +SDK bump touches one file. Effort M (mechanical); Risk Low. Low priority. + +### S-8 (Low) — first-caller interval wins for shared data subscription +Document the shared-publish-interval behavior in `docs/Client.CLI.md`/`docs/Client.UI.md`, or create +per-interval subscriptions if UI users mix cadences. Effort S; Risk Low. + +### U-8 (Low) — CLI security posture is dev-tool-grade +Document that `AutoAcceptCertificates = true` (`CommandBase.cs:91`) means `--security signandencrypt` +encrypts but does not authenticate the server (MITM-able), and that `-P` is process-visible. Optionally +add a `--strict-certs` opt-in. Separately, note the Galaxy-specific alarm attribute names baked into +the generic `OpcUaClientService.cs:851-871` as a documented layering compromise. Effort S; Risk Low. + +### P-2 (Low) — recursive browse N+1 +`BrowseAsync` issues an extra `HasChildrenAsync` per Object child (`:230-231`) and +`SubscribeCommand.CollectVariablesAsync` walks serially. Batch/bulk-read references only if `subscribe +-r` start-up becomes a soak-test bottleneck. Effort M; Risk Low. Defer. + +### U-7 (Low) — missing CI stages inventory +Tracking item: analyzer enforcement (C-1), all-suite coverage (S-1), `dotnet format`/style step, +explicit failing NuGet-audit step, running `scripts/check-code-reviews-readme.ps1` in CI, and a docs +freshness/link check. Also the `ci/ab-server.lock.json` references a non-existent "GitHub Actions +step" in `docs/v2/test-data-sources.md` — either land the AB fixture download step or fix the doc. +Address incrementally as the above land. + +--- + +## No-action (verified positive / accurate — recorded only) + +- **P-3** — probe-once collection-fixture pattern is well-engineered; the issue is what CI does with + it (S-2), not the fixtures. +- **P-4** — CLI subscription output uses a channel-serialised writer; allocation-sane. A bounded + `DropOldest` channel is a cosmetic future option. +- **C-3** — CPM discipline exemplary; the `NuGetAuditSuppress` carve-out and the two transitive CVE + pins are still valid and accurately scoped. Keep the removal reminders alive. +- **U-1** — Client.UI is a genuine, adequately-tested product (not vestigial); "maturing", not + "underdeveloped". Its only gap is the same CI/E2E gap (S-1/S-3). + +--- + +## Suggested execution sequence + +1. **C-1** (wire analyzer) + its triage — establishes the enforcement mechanism; must precede C-2. +2. **S-1 + P-1** (whole-solution CI + caching) in one workflow edit — biggest coverage win. +3. **S-2** (fail-on-skip guard, then optional `services:`) — closes the false-green. +4. **S-7** (`global.json`) + **S-3** (disable E2E schedule / zero-match guard) — small CI truth fixes. +5. **U-5, U-4, U-6, C-5** (hygiene batch) — one commit; verified zero-reference deletions. +6. **U-3** (Client.CLI doc) — five sections. +7. **C-2** (TWE, after C-1 triage), then Tier-D items as capacity allows, pairing **S-4 with S-1**. + +Each of steps 1-4 must be validated by an actual CI run on a branch before merge — per the report's +own theme #4, green CI here has been meaningless, so the fixes must be observed to change the badge's +behavior (a broken build/deliberate skip must now go red).