docs(archreview): remediation plans + fix flagged doc drift
Add the 7 per-domain design+implementation plans (archreview/plans/) with an index, produced from the 2026-07-08 architecture review. Fix two confirmed doc drifts the review flagged (theme #5): - CLAUDE.md KNOWN LIMITATION 2: the continuous-historization historized-ref feed IS wired (AddressSpaceApplier.FeedHistorizedRefs -> UpdateHistorizedRefs -> recorder); rewrite to reflect that value-capture is code-complete and only the live end-to-end + restart-convergence verification remains. - CLAUDE.md ScriptAnalysis gating: endpoints use Roles=Administrator,Designer via RequireAuthorization, not the FleetAdmin policy.
This commit is contained in:
Joseph Doherty
@@ -0,0 +1,69 @@
|
||||
# Arch-Review Remediation Plans — Index
|
||||
|
||||
Design + implementation plans for every finding in the 2026-07-08 architecture review
|
||||
(`archreview/*.md`, review commit `9cad9ed0`). Each plan verifies every finding against
|
||||
the current tree, then gives root cause → design (with alternatives) → concrete file-level
|
||||
steps → tests → effort/risk.
|
||||
|
||||
## Plan documents
|
||||
|
||||
| # | Plan | Findings | Verification |
|
||||
|---|---|---|---|
|
||||
| 01 | [Core & composition](01-core-composition-plan.md) | 21 | all confirmed, 0 stale |
|
||||
| 02 | [Scripting & alarms](02-scripting-alarms-plan.md) | 30 | all confirmed, 0 stale |
|
||||
| 03 | [Server & runtime](03-server-runtime-plan.md) | 28 | all confirmed (U1 = stale *doc* only) |
|
||||
| 04 | [AdminUI](04-adminui-plan.md) | 20 | all confirmed (C-3/C-4 good, no action) |
|
||||
| 05 | [Protocol drivers](05-protocol-drivers-plan.md) | 37 + 1 positive | all confirmed, 0 stale |
|
||||
| 06 | [Gateway integrations](06-gateway-integrations-plan.md) | 28 (23 actionable) | all confirmed (U-1 = stale *doc*) |
|
||||
| 07 | [Client, tooling & engineering](07-client-tooling-engineering-plan.md) | 26 | all confirmed, 0 stale |
|
||||
|
||||
**Total: ~190 findings planned. Zero were stale-because-already-fixed; the only "stale" items are
|
||||
two documentation drifts (CLAUDE.md Known Limitation 2) both flagged for correction.**
|
||||
|
||||
## The 4 Criticals (fix first)
|
||||
|
||||
| # | Critical | Plan | Fix shape |
|
||||
|---|---|---|---|
|
||||
| 1 | Split-brain resolver never activated → no hard-crash failover | 03 S1 | `ClusterOptions.SplitBrainResolver = KeepOldestOption` + hard-kill failover test |
|
||||
| 2 | Production VT script timeout is dead code → one loop wedges the actor | 02 U2 (+U3) | Route through `TimedScriptEvaluator` + `CompiledScriptCache` |
|
||||
| 3 | S7 has no reconnect path → PLC reboot kills the driver | 05 STAB-1 | `IS7PlcFactory` seam + lazy `EnsureConnectedAsync` |
|
||||
| 4 | TwinCAT ADS subs orphaned after reconnect → silent data stop | 05 STAB-2 | Store replayable registration intent, replay on client swap |
|
||||
|
||||
## Cross-cutting guardrails (from OVERALL themes, span multiple plans)
|
||||
|
||||
- **Theme #1 built-but-never-wired** — the recurring failure mode. Guardrails proposed across plans:
|
||||
reflection-exhaustive `DeferredAddressSpaceSink` forwarding test (03 U2, sequenced *before* the
|
||||
surgical-add work), wiring the OTOPCUA0001 analyzer repo-wide (07 C-1), a startup log line proving
|
||||
SBR is active (03 S1), and wiring/removing the dormant historian health surface (06 S-6).
|
||||
- **Theme #2/#6 fixes don't flow to shared templates** — Part B of plan 05: `PollGroupEngine` v2
|
||||
(backoff + onError + delete the S7 fork), shared strict `ReadEnum`/`ReadBool`, `ConnectionBackoff`
|
||||
primitive, `ResolveHost`-via-resolver; plus `TagConfigIntent.Parse` in Commons (01 C-1) killing the
|
||||
4-project byte-parity duplication.
|
||||
- **Theme #3 optimistic success** — failure-visibility fixes: `AddressSpaceApplyOutcome` failure field
|
||||
(01 S-1), Galaxy write fail-closed (06 S-1), primary-gate default-deny on unknown role (03 S4).
|
||||
- **Theme #4 verification gap** — CI to all unit suites + fail-on-skip (07 S-1/S-2), fault-injection tier
|
||||
(hard-kill / drop-PLC), the missing bUnit substitute = reflection/policy guards (04 C-1).
|
||||
- **Theme #5 doc drift** — fix CLAUDE.md Known Limitation 2 + scadaproj index (03 U1 / 06 U-1),
|
||||
ScriptAnalysis policy doc, Client.CLI docs (07 U-3).
|
||||
- **Theme #7 repo hygiene** — `git rm -r lib/` AVEVA DLLs, untrack planning files, delete Wonderware
|
||||
husks (07 U-4/U-5/U-6).
|
||||
|
||||
## Suggested execution order
|
||||
|
||||
Front-loaded per the OVERALL prioritized action list — items 1–6 eliminate every Critical plus the two
|
||||
systemic guards (analyzer + CI) that stop the built-but-never-wired pattern recurring:
|
||||
|
||||
1. **Criticals** — 03 S1, 02 U2+U3, 05 STAB-1, 05 STAB-2 (each with its missing test)
|
||||
2. **Systemic guards** — 07 C-1 (analyzer) + 07 S-1/S-2 (CI to all suites, fail-on-skip); 03 U2 (Deferred-sink forwarding test)
|
||||
3. **Silent-corruption Highs** — 05 STAB-4/5 (AbCip lock, FOCAS caches), 05 Modbus timeout-fatal classification
|
||||
4. **Failure-visibility Highs** — 01 S-1, 06 S-1, 03 S4
|
||||
5. **AdminUI authz** — 04 C-1 (ConfigEditor policy + reflection guard)
|
||||
6. **Perf** — 03 P1 (surgical pure-adds, after 03 U2 guard)
|
||||
7. **Consolidation + hygiene + docs** — 01 C-1, 05 Part B, 07 U-4/U-5, doc fixes
|
||||
|
||||
## Open maintainer decision
|
||||
|
||||
- **Prune vs. keep the dormant machinery** (01 U-2 / 02 U-1): the callerless Tier-C recycle machinery +
|
||||
`IDriverSupervisor`, and the ~2.4k-line `Core.VirtualTags` engine that production bypasses. Plans
|
||||
recommend retiring both *after* the live paths are hardened (02 U2/U3 first), but the delete-vs-keep
|
||||
call is the maintainer's.
|
||||
@@ -0,0 +1,533 @@
|
||||
# Design + Implementation Plan — Core & Composition Pipeline
|
||||
|
||||
**Domain report:** `archreview/01-core-composition.md`
|
||||
**Baseline commit reviewed:** `9cad9ed0` (master)
|
||||
**Plan author verification date:** 2026-07-08
|
||||
**Scope:** Core, Core.Abstractions, Commons, Configuration, Cluster; AddressSpace Composer/Planner/Applier.
|
||||
|
||||
## Verification summary
|
||||
|
||||
All 21 findings were opened at their cited file:line against the current tree. **Every finding is CONFIRMED
|
||||
accurate** — none are stale or already-fixed. (One nuance: U-3's *delta feed* half is already implemented and
|
||||
shipped, matching the report's own "delta feed landed, initial set still empty" framing — so U-3 is confirmed as
|
||||
partially-done debt, not stale.)
|
||||
|
||||
Key structural fact that shapes the C-1/C-3 fix: **`Commons` is a pure leaf project** (zero `ProjectReference`s,
|
||||
verified from `Commons.csproj`), and **`Configuration` does not currently reference it** but *can* without creating a
|
||||
cycle (Commons references neither Configuration nor Core). This makes "move shared parsing/scheme into Commons" the
|
||||
correct, low-risk home for the byte-parity duplication class.
|
||||
|
||||
## Priority ordering (this plan)
|
||||
|
||||
Ordered by severity, then by the OVERALL prioritized action list (items #7 and #10 touch this domain directly).
|
||||
|
||||
| Order | ID | Sev | Title | Effort | OVERALL ref |
|
||||
|---|---|---|---|---|---|
|
||||
| 1 | S-1 | High | Applier swallows sink failures; deploy reports success on broken address space | M | Action #7 |
|
||||
| 2 | C-1 | High | TagConfig parsing byte-parity-replicated in 4 projects | M | Action #10 |
|
||||
| 3 | C-3 | Med | DraftValidator re-derives NodeId scheme + hard-codes driver-type string | S | Action #10 |
|
||||
| 4 | P-1 | Med | Compose parses each TagConfig JSON 4× | S (folds into C-1) | — |
|
||||
| 5 | S-2 | Med | `PollGroupEngine.Unsubscribe` blocks caller up to 5 s | M | — |
|
||||
| 6 | P-2 | Med | Per-operation allocations on authorization hot path | S | — |
|
||||
| 7 | P-3 | Med | PollGroupEngine task-per-subscription, no coalescing | L (doc-only now) | — |
|
||||
| 8 | C-2 | Med | Core → Configuration coupling (accepted-risk, document) | S | — |
|
||||
| 9 | U-1 | Med | Dead/dormant code: GDNM, EquipmentNodeWalker, TryParseRelayBody | S | Action #12 |
|
||||
| 10 | U-2 | Med | Tier C recycle machinery has no `IDriverSupervisor` impl | S | Action #12 |
|
||||
| 11 | U-3 | Med | Continuous-historization initial ref set still empty at spawn | M | — |
|
||||
| 12 | P-4 | Low | Per-write allocation in non-idempotent write path | S | — |
|
||||
| 13 | S-3 | Low | `Subscribe`/`RegisterAsync` have no disposed guard | S | — |
|
||||
| 14 | S-4 | Low | ScheduledRecycleScheduler catch-up storm after stall | S | — |
|
||||
| 15 | S-5 | Low | GDNM re-walk teardown not synchronized (moot — dead code) | — (folds into U-1) | — |
|
||||
| 16 | C-4 | Low | Composer violates purity with `Trace.TraceWarning` | S | — |
|
||||
| 17 | C-5 | Low | Inconsistent duplicate-key defensiveness in Compose | S | — |
|
||||
| 18 | C-6 | Low | Vestigial Akka package reference in Commons | S | — |
|
||||
| 19 | C-7 | Low | Stale scaffold-era XML docs on load-bearing classes | S | — |
|
||||
| 20 | U-4 | Low | Additive-only ACL model (no Deny) — documented v2.0 gap | S (doc) | — |
|
||||
| 21 | U-5 | Low | Uneven test coverage (ClusterRoleInfo untested) | M | — |
|
||||
|
||||
---
|
||||
|
||||
## 1. S-1 — Applier swallows every sink failure; deploy reports applied even when address space is broken (High)
|
||||
|
||||
**Verification: CONFIRMED.**
|
||||
- `AddressSpaceApplyOutcome` (`AddressSpaceApplier.cs:691`) carries only `RemovedNodes/AddedNodes/ChangedNodes/RebuildCalled` — **no failure count**.
|
||||
- `SafeRebuild` (`:373-383`) catches every exception from `_sink.RebuildAddressSpace()` and only logs; the caller still sets `rebuilt = true` (`:150-153`).
|
||||
- `SafeEnsureFolder`/`SafeEnsureVariable` (`:612-622`), `SafeWriteAlarmCondition`/`SafeMaterialiseAlarmCondition` (`:677-687`) swallow per-node failures into Warnings.
|
||||
- Consumer confirmed: `OpcUaPublishActor.cs:335` calls `_applier.Apply(plan)` and **only logs** the outcome (`:357`). The whole rebuild path is itself inside a try/catch that logs (`:360-363`). There is no `ApplyAck`/`DeploymentFailed` reply carrying success/failure back to a deploy coordinator — the deploy seals independently of apply health.
|
||||
|
||||
**Root cause.** The "never fail a deploy" posture (correct for the *detached* provisioning/historized-ref hooks) was applied uniformly, including to the structural materialisation that is the deploy's core contract. The outcome record has no channel to express partial/total failure, so the publish actor has nothing to act on even if it wanted to.
|
||||
|
||||
**Proposed design.** Give the applier a truthful failure surface and route it to an operator-visible signal. This is the domain-01 slice of OVERALL cross-cutting theme #3 ("optimistic success").
|
||||
|
||||
Two-part approach:
|
||||
1. **Count failures in the outcome.** Change the `Safe*` helpers to return a `bool` (or increment a private counter) and add `RebuildFailed` + `FailedNodes` to `AddressSpaceApplyOutcome`. `SafeRebuild` returning false must flip `rebuilt`'s meaning: keep `RebuildCalled` (it *was* attempted) but set `RebuildFailed = true` so callers can distinguish "rebuilt OK" from "rebuild threw".
|
||||
2. **Surface it.** In `OpcUaPublishActor`, when `outcome.RebuildFailed || outcome.FailedNodes > 0`, log at `Error` (not Info) **and** emit an operator-visible signal. Reuse the existing telemetry meter `OtOpcUaTelemetry.OpcUaSinkWrite` with a `kind:"apply-failed"` tag (pattern already used at `OpcUaPublishActor.cs:303,356`), and — if a deploy-status/alert channel is reachable from the actor — post a degraded status. Do **not** convert this into a hard deploy abort in v1: a partially-materialised address space is still better than none, and the publish actor is already past the seal decision. The goal is *visibility*, matching the report's "escalate `SafeRebuild` failure to at least a degraded ack."
|
||||
|
||||
**Alternatives considered.**
|
||||
- *Throw from Apply on rebuild failure* — rejected: the actor's catch would swallow it anyway and it removes the counts other callers want; also risks aborting the subsequent `Materialise*` passes that could still succeed.
|
||||
- *Full ApplyAck→DeploymentFailed wiring* — larger blast radius touching ControlPlane deploy coordination; deferred. The OVERALL list scopes item #7 as "add a failure field ... surface"; the deploy-abort decision belongs with report 03's deploy-coordination work.
|
||||
|
||||
**Implementation steps.**
|
||||
- `AddressSpaceApplier.cs`:
|
||||
- Extend record: `AddressSpaceApplyOutcome(int RemovedNodes, int AddedNodes, int ChangedNodes, bool RebuildCalled, bool RebuildFailed, int FailedNodes)`. Keep a 4-arg → 6-arg default or update the two construction sites (`:80` empty-plan, `:215` main return).
|
||||
- `SafeRebuild()` → `private bool SafeRebuild()` returning `false` on catch; caller records `rebuildFailed`.
|
||||
- `SafeEnsureFolder`/`SafeEnsureVariable`/`SafeWriteAlarmCondition`/`SafeMaterialiseAlarmCondition` → return `bool`; the `Materialise*` passes tally a `_failedNodes` counter. Because the `Materialise*` passes run from `OpcUaPublishActor` *after* `Apply` returns (they are separate public methods, `:338-354`), expose the tally via either (a) an out-param/return on each `Materialise*` method, or (b) an instance counter reset at the start of the rebuild sequence and read after. Prefer (a) — each `Materialise*` returns an `int failedNodes`; the actor sums them.
|
||||
- `OpcUaPublishActor.cs` (`:335-358`): capture the failure tally across `Apply` + the four `Materialise*` calls; branch to `_log.Error` + `OtOpcUaTelemetry.OpcUaSinkWrite.Add(1, kind:"apply-failed")` when non-zero.
|
||||
|
||||
**Tests (unit).** `tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests/AddressSpace*`:
|
||||
- A throwing fixture sink (the `NullOpcUaAddressSpaceSink` pattern extended to throw on `RebuildAddressSpace` / `EnsureVariable`): assert `outcome.RebuildFailed == true` when rebuild throws, and `FailedNodes == N` when N `EnsureVariable` calls throw during a materialise pass.
|
||||
- Assert the happy path still reports `RebuildFailed == false, FailedNodes == 0`.
|
||||
|
||||
**Effort: M. Risk: low-medium** — record shape change ripples to every `Apply` caller + test asserting the record; contained to OpcUaServer + Runtime.
|
||||
|
||||
---
|
||||
|
||||
## 2. C-1 — TagConfig parsing byte-parity-replicated in four projects (High)
|
||||
|
||||
**Verification: CONFIRMED.** Four independent copies of the top-level `FullName` (and sibling `alarm`/`isHistorized`/`isArray`/`HostAddress`) parsing:
|
||||
- `AddressSpaceComposer.cs:536-551` (`ExtractTagFullName`) + `:601-686` (`ExtractTagAlarm`/`ExtractTagHistorize`/`ExtractTagArray`) + `:567` (`TryExtractDeviceHost`).
|
||||
- `Runtime/Drivers/DeploymentArtifact.cs` (artifact-decode mirror; comment at `:666` cites `EquipmentNodeWalker.ExtractFullName`).
|
||||
- `Core/OpcUa/EquipmentNodeWalker.cs` `ExtractFullName` (dormant — see U-1).
|
||||
- `Configuration/Validation/DraftValidator.cs:60-71` (`ExtractTagConfigFullName`, "a small local copy").
|
||||
|
||||
Each copy carries a "MUST parse identically (byte-parity)" comment. `EquipmentScriptPaths` (`Commons/Types/`) is the in-repo precedent for de-duplicating exactly this kind of cross-seam parser.
|
||||
|
||||
**Root cause.** OpcUaServer does not reference the Core driver assembly and Configuration does not reference Commons, so no shared home existed — each seam grew its own parser, synced by human-enforced comments. This is the OVERALL cross-cutting theme #2 ("fixes don't flow back to shared templates") in the Core layer.
|
||||
|
||||
**Proposed design.** Create a single `TagConfigIntent` parser in **Commons** (leaf, no EF/SDK deps — verified) that parses the blob **once** and returns all intents, then retrofit every seam onto it. This simultaneously resolves **P-1** (one parse instead of four).
|
||||
|
||||
```csharp
|
||||
// src/Core/ZB.MOM.WW.OtOpcUa.Commons/Types/TagConfigIntent.cs
|
||||
public sealed record TagConfigIntent(
|
||||
string FullName, // top-level "FullName" or raw blob fallback
|
||||
TagAlarmIntent? Alarm, // "alarm" object or null
|
||||
bool IsHistorized, string? HistorianTagname,
|
||||
bool IsArray, uint? ArrayLength)
|
||||
{
|
||||
public static TagConfigIntent Parse(string? tagConfig); // single JsonDocument.Parse, never throws
|
||||
}
|
||||
public sealed record TagAlarmIntent(string AlarmType, int Severity, bool? HistorizeToAveva);
|
||||
```
|
||||
|
||||
Add a companion `DeviceConfigIntent.TryExtractHost(string?)` (or fold `TryExtractDeviceHost` + `NormalizeDeviceHost` into Commons) so the device-host normalization single-source-of-truth also lives there.
|
||||
|
||||
**Why Commons, and why one record.** The report's own recommendation, and the `EquipmentScriptPaths` precedent, both point here. A single record parsed once (a) kills the 4× duplication, (b) collapses P-1's 4 parses/tag → 1, and (c) lets the byte-parity contract be enforced by **one** cross-seam parity test instead of N.
|
||||
|
||||
**Alternatives considered.**
|
||||
- *Put it in Core.Abstractions* — rejected: DeploymentArtifact (Runtime) and DraftValidator (Configuration) would still need it; Commons is the common ancestor all of them can see (or cheaply add).
|
||||
- *Keep separate parsers, add a shared golden-file parity test* — rejected: tests the symptom, not the cause; still 4 code paths to drift.
|
||||
|
||||
**Implementation steps.**
|
||||
1. Add `Commons/Types/TagConfigIntent.cs` (+ `DeviceConfigIntent` helper). Port the *exact* current parse semantics verbatim from `AddressSpaceComposer` (the canonical copy) — especially the raw-blob fallback for `FullName`, the `historizeToAveva` bool? default-on rule, and the `arrayLength`-only-when-`isArray` rule — so no byte-parity behavior changes.
|
||||
2. `AddressSpaceComposer.cs`: replace the per-tag `ExtractTagFullName`/`ExtractTagAlarm`/`ExtractTagHistorize`/`ExtractTagArray` calls (`:418-433`) with a single `var intent = TagConfigIntent.Parse(t.TagConfig);` then project fields off `intent`. Replace `TryExtractDeviceHost`/`NormalizeDeviceHost` usages with the Commons helper. Delete the now-private methods.
|
||||
3. `Runtime/Drivers/DeploymentArtifact.cs`: replace its mirror `Extract*` methods with `TagConfigIntent.Parse`.
|
||||
4. `Configuration/Validation/DraftValidator.cs`: add a `ProjectReference` to Commons (cycle-safe — verified), replace `ExtractTagConfigFullName` (`:60-71`) with `TagConfigIntent.Parse(t.TagConfig).FullName` (note: current DraftValidator returns null for the missing case; preserve that by checking `string.IsNullOrWhiteSpace` against the parsed FullName, being careful that the Commons parser returns the raw blob as fallback — the Galaxy check wants the *explicit* FullName, so either expose a `HasExplicitFullName` bool on the intent or keep DraftValidator's null-on-absent semantics by adding a `TagConfigIntent.TryGetExplicitFullName`).
|
||||
5. `EquipmentNodeWalker.ExtractFullName`: deleted with the walker under U-1 (or repointed at Commons if the walker survives as a test fixture).
|
||||
|
||||
**Tests.**
|
||||
- `Commons.Tests`: exhaustive `TagConfigIntent.Parse` table (object/non-object/blank/malformed/each field present-absent-wrong-type) — this becomes the *single* byte-parity authority.
|
||||
- Keep one cross-seam parity test asserting `AddressSpaceComposer` compose output and `DeploymentArtifact` decode output agree on a fixture set (this already exists in the OpcUaServer.Tests byte-parity suite — repoint it at the shared parser).
|
||||
|
||||
**Effort: M. Risk: medium** — touches the byte-parity contract, the crown-jewel invariant. Mitigation: port semantics verbatim, land behind the existing byte-parity test suite, no behavior change intended.
|
||||
|
||||
---
|
||||
|
||||
## 3. C-3 — DraftValidator re-derives NodeId scheme + hard-codes driver-type string (Medium)
|
||||
|
||||
**Verification: CONFIRMED.**
|
||||
- `DraftValidator.cs:75-97` (`ValidateNoEquipmentSignalNameCollision`) hand-rolls the folder-scoped key `"{eq}/{folder}/{name}"` via a local `Key(...)` because Configuration can't reference `Commons.OpcUa.EquipmentNodeIds` (`Commons/OpcUa/EquipmentNodeIds.cs` confirmed to exist).
|
||||
- `DraftValidator.cs:48` hard-codes `dtype != "GalaxyMxGateway"`.
|
||||
|
||||
**Root cause.** Same missing shared-contract home as C-1: dependency direction blocked Configuration from seeing the Commons scheme/constants.
|
||||
|
||||
**Proposed design.** Once C-1 adds the `Configuration → Commons` ProjectReference, this becomes trivial:
|
||||
- Replace the local `Key(...)` in `ValidateNoEquipmentSignalNameCollision` with the same key `EquipmentNodeIds` produces (add a public `EquipmentNodeIds.Variable(eq, folder, name)` — it already exists per `AddressSpaceApplier.cs:177` usage — and call it, so the collision check keys on the *exact* materialiser scheme, eliminating drift risk).
|
||||
- Replace `"GalaxyMxGateway"` with a shared driver-type constant. Add `DriverTypes.GalaxyMxGateway` (a `const string`) in Commons (or reuse an existing driver-type-names home if one exists — grep `DriverType` constants first). Point DraftValidator at it.
|
||||
|
||||
**Implementation steps.**
|
||||
1. (Depends on C-1's `Configuration → Commons` reference.)
|
||||
2. Ensure `EquipmentNodeIds.Variable` is public and matches DraftValidator's `null-folder ⇒ "{eq}/{name}"` behavior (verify against `AddressSpaceApplier.cs:177`). If the VirtualTag branch (which passes `folder=null`) needs the no-folder overload, add it.
|
||||
3. Add `Commons` driver-type constant; replace the literal at `DraftValidator.cs:48`.
|
||||
4. Grep for other `"GalaxyMxGateway"` literals (composer comment references it; `DriverFactoryBootstrap` registers it) and repoint the *logic* sites (not doc comments) at the constant.
|
||||
|
||||
**Tests.** Existing `Configuration.Tests` DraftValidator collision + Galaxy-FullName tests should pass unchanged (behavior-preserving). Add one asserting the collision key equals `EquipmentNodeIds.Variable(...)` for a folder + no-folder case, so the shared-scheme binding is guarded.
|
||||
|
||||
**Effort: S. Risk: low.**
|
||||
|
||||
---
|
||||
|
||||
## 4. P-1 — Compose parses each TagConfig JSON four times (Medium)
|
||||
|
||||
**Verification: CONFIRMED.** `AddressSpaceComposer.cs:418-433` invokes `ExtractTagHistorize`, `ExtractTagArray`, `ExtractTagFullName`, `ExtractTagAlarm` — each doing its own `JsonDocument.Parse` (`:541,606,638,670`). 4 parses/tag × 2 seams (compose + artifact-decode).
|
||||
|
||||
**Root cause / fix.** This is fully resolved by C-1's `TagConfigIntent.Parse` (single parse per tag per seam). No separate work item — **fold into C-1**. Note in the C-1 PR that P-1 is closed by the same change.
|
||||
|
||||
**Effort: S (subsumed). Risk: as C-1.**
|
||||
|
||||
---
|
||||
|
||||
## 5. S-2 — `PollGroupEngine.Unsubscribe` blocks the calling thread up to 5 s per subscription (Medium)
|
||||
|
||||
**Verification: CONFIRMED.** `StopState` (`PollGroupEngine.cs:99-112`) does synchronous `task.Wait(TimeSpan.FromSeconds(5))` (`:109`). `Unsubscribe` (`:89-97`) calls it inline. `DisposeAsync` (`:213-236`) does the correct parallel-cancel + awaited `WhenAll(...).WaitAsync(5s)`.
|
||||
|
||||
**Root cause.** The single-subscription teardown path was written synchronously to preserve the "no `_onChange` after teardown" guarantee, but blocks the caller (an OPC UA subscription-management callback / actor handler) while a reader is mid-slow-call. N serial unsubscribes = N × up-to-5 s.
|
||||
|
||||
**Proposed design.** Add `ValueTask UnsubscribeAsync(ISubscriptionHandle)` that cancels the CTS, then awaits `LoopTask.WaitAsync(timeout)` (mirroring `DisposeAsync`'s discipline) rather than blocking. Keep the synchronous `Unsubscribe` for compatibility but re-implement it to **cancel + hand the drain to a background continuation** that disposes the CTS after the loop actually exits — preserving the "CTS not disposed while `Task.Delay` holds the token" ordering the current comment (`:102-106`) protects.
|
||||
|
||||
Because the poll loop already treats `ObjectDisposedException` from a disposed CTS as normal cancellation (`:133-136`), the background-drain variant is safe: worst case the loop observes cancellation, the continuation disposes the CTS, no callback fires post-teardown (the loop's `while (!ct.IsCancellationRequested)` + `Task.Delay(ct)` exits before any further `_onChange`).
|
||||
|
||||
**Alternatives considered.**
|
||||
- *Just shorten the 5 s timeout* — rejected: still blocks, just less; and a short timeout risks disposing the CTS while the loop runs.
|
||||
- *Fire-and-forget cancel with no drain* — rejected: loses the "no callback after teardown" guarantee that the current design deliberately holds.
|
||||
|
||||
**Implementation steps.**
|
||||
- `PollGroupEngine.cs`: add `public async ValueTask UnsubscribeAsync(ISubscriptionHandle handle)` doing `TryRemove` + `state.Cts.Cancel()` + `await (state.LoopTask ?? Task.CompletedTask).WaitAsync(5s)` (swallow) + `state.Cts.Dispose()`.
|
||||
- Re-implement `Unsubscribe` to remove + cancel synchronously, then schedule `_ = DrainAndDisposeAsync(state)` (a private async method that awaits the loop then disposes the CTS) instead of blocking. Return `true` immediately.
|
||||
- Audit the driver callers (Modbus/S7/AbCip/FOCAS subscription-management callbacks) — where the callback is `async`, prefer `await UnsubscribeAsync`. Grep `\.Unsubscribe(` across `src/Drivers`.
|
||||
|
||||
**Tests (Core.Abstractions.Tests).**
|
||||
- `UnsubscribeAsync` completes promptly even when the reader delegate blocks (inject a reader that awaits a `TaskCompletionSource`); assert no `_onChange` fires after `UnsubscribeAsync` returns.
|
||||
- Synchronous `Unsubscribe` returns without a multi-second stall (assert wall-clock under a small bound with a slow reader) and the CTS is eventually disposed (no leak) — the loop count drops to zero.
|
||||
|
||||
**Effort: M. Risk: medium** — teardown ordering is subtle; the existing `DisposeAsync` is the proven template to mirror. Blast radius: all poll-based drivers.
|
||||
|
||||
---
|
||||
|
||||
## 6. P-2 — Per-operation allocations on the authorization hot path (Medium)
|
||||
|
||||
**Verification: CONFIRMED.** `PermissionTrie.CollectMatches` (`PermissionTrie.cs:42`) does `ldapGroups.ToHashSet(OrdinalIgnoreCase)` per call, plus `new List<MatchedGrant>()` (`:45`). Called per node-operation by `TriePermissionEvaluator.Authorize` (verified consumer). A recursive browse / large CreateMonitoredItems batch = per-node garbage.
|
||||
|
||||
**Root cause.** The group set is rebuilt from the session's LDAP groups on every authorize even though `UserAuthorizationState.LdapGroups` is immutable per membership refresh.
|
||||
|
||||
**Proposed design.** Compute the case-insensitive `HashSet<string>` **once per session-version** and pass it in. Two clean options:
|
||||
1. **Overload `CollectMatches(NodeScope, IReadOnlySet<string> groups)`** and have the evaluator hold a cached `HashSet` on `UserAuthorizationState` (or a small per-session wrapper) built when the membership/version changes. Keep the `IEnumerable` overload for callers that don't have a prebuilt set.
|
||||
2. Cache the set on `UserAuthorizationState` itself (lazy, `Lazy<IReadOnlySet<string>>` keyed by the version field).
|
||||
|
||||
Prefer (1): it keeps `PermissionTrie` allocation-free on the hot path and puts the cache where the immutability actually lives (the session state). The `List<MatchedGrant>` is small and bounded by trie depth (≤ 6 levels × grants); leave it unless profiling flags it — or return via a pooled/stackalloc'd span if the evaluator only needs the OR-ed bitmask (it does: the caller "OR-s the flag bits"). A tighter design: add `CollectMask(scope, groups) -> NodePermissions` that OR-s inline and allocates nothing, keeping `CollectMatches` for diagnostics.
|
||||
|
||||
**Implementation steps.**
|
||||
- `Core/Authorization/UserAuthorizationState.cs`: add a cached `IReadOnlySet<string> LdapGroupSet` built once per version (verify the version/refresh field name).
|
||||
- `PermissionTrie.cs`: add `CollectMatches(NodeScope, IReadOnlySet<string>)` overload; optionally add `NodePermissions CollectMask(NodeScope, IReadOnlySet<string>)` that avoids the `List`.
|
||||
- `TriePermissionEvaluator.Authorize`: pass the cached set / call `CollectMask`.
|
||||
|
||||
**Tests.** Existing authorization tests must pass unchanged (behavior identical). Add a test asserting the same grants resolve via the new set overload as the enumerable overload. A micro-bench is optional (no bench harness in repo).
|
||||
|
||||
**Effort: S. Risk: low** — additive overload; existing path preserved.
|
||||
|
||||
---
|
||||
|
||||
## 7. P-3 — PollGroupEngine: task-per-subscription, no cross-subscription batching (Medium)
|
||||
|
||||
**Verification: CONFIRMED.** `Subscribe` (`:73-84`) spawns a dedicated `Task.Run(PollLoopAsync)` per subscription with its own interval timer. No interval-bucketed scheduler or read coalescing.
|
||||
|
||||
**Root cause.** The engine was extracted from ModbusDriver as a per-subscription loop; fine at dozens of subscriptions, a scaling wall at hundreds/driver-instance.
|
||||
|
||||
**Proposed design.** This is a **structural fix disproportionate to current need** — the report itself says "Worth a note in the driver-facing docs at minimum." Recommend:
|
||||
- **v1 (this pass): documentation only.** Add a remarks paragraph to `PollGroupEngine`'s XML doc stating the task-per-subscription cost model and the scaling ceiling, and note it in the driver-authoring doc (`docs/` driver guide) so integrators map poll groups, not per-tag subscriptions.
|
||||
- **v2 (tracked follow-on, only if tag counts grow):** an interval-bucketed shared scheduler — one loop per distinct clamped interval, reading the union of that bucket's references once and fanning results to each subscription's diff state. This is the real fix and pairs naturally with OVERALL theme #2's "treat PollGroupEngine as the single home for fleet fixes."
|
||||
|
||||
**Implementation steps (v1).** Doc-only edit to `PollGroupEngine.cs` remarks + driver-authoring doc. No code/test change.
|
||||
|
||||
**Effort: S now (doc), L later (bucketed scheduler). Risk: none (doc); high-ish later (touches every poll driver's timing).**
|
||||
|
||||
---
|
||||
|
||||
## 8. C-2 — Core depends on Configuration; EF entities are the domain model (Medium)
|
||||
|
||||
**Verification: CONFIRMED.** `Core.csproj` references `Configuration` (`:16`). `PermissionTrie.cs:1` consumes `Configuration.Enums`; `EquipmentNodeWalker` consumes `Configuration.Entities`; composer plan records project EF entities.
|
||||
|
||||
**Root cause / posture.** Deliberate, consistent choice — persistence types *are* the model; plan records decouple the applier. The report flags this as **accepted-risk to document, not refactor.**
|
||||
|
||||
**Proposed design.** No refactor. Add an ADR / architecture-note entry documenting: (a) Configuration is the domain model by design; (b) the ripple cost (EF entity changes reach authorization + composition); (c) the mitigation guidance already in practice ("new code keeps preferring the plan-record boundary the composer established"). Cross-link from `CLAUDE.md`'s Architecture Overview.
|
||||
|
||||
**Implementation steps.** Doc-only: add to `docs/` (architecture/decisions) + a one-line note in the composer/PermissionTrie XML docs pointing at it.
|
||||
|
||||
**Effort: S. Risk: none.**
|
||||
|
||||
---
|
||||
|
||||
## 9. U-1 — Dead/dormant code retained in Core (Medium)
|
||||
|
||||
**Verification: CONFIRMED — all three are test-only:**
|
||||
- `GenericDriverNodeManager` — only `tests/Core/.../GenericDriverNodeManagerTests.cs` references it (grep: zero production refs).
|
||||
- `EquipmentNodeWalker` (+ `IdentificationFolderBuilder`) — only `tests/Core/.../OpcUa/EquipmentNodeWalkerTests.cs` + doc-comment mentions; self-declares "retained for unit-test support only."
|
||||
- `EquipmentScriptPaths.TryParseRelayBody` (`Commons/Types/EquipmentScriptPaths.cs:164`) — only `Commons.Tests/EquipmentScriptPathsTests.cs` calls it (relay→alias converter deleted 2026-06-12).
|
||||
|
||||
**Root cause.** Code retained after its production caller was removed (Phase7→AddressSpace rename retired the walker; Galaxy-standard-driver retired the relay parser; GDNM superseded by composer→applier→sink).
|
||||
|
||||
**Proposed design.** **Delete all three + their test files** (git preserves history). This is OVERALL cross-cutting theme #1's remediation ("dormant code with green tests is worse than deleted code"). Order the deletion to also close **S-5** and part of **C-1**:
|
||||
- Deleting `EquipmentNodeWalker` removes one of C-1's four `ExtractFullName` copies for free.
|
||||
- Deleting `GenericDriverNodeManager` makes **S-5** (its non-synchronized re-walk teardown) moot — no separate fix needed.
|
||||
|
||||
**Implementation steps.**
|
||||
1. Delete `src/Core/.../OpcUa/GenericDriverNodeManager.cs` + `tests/.../GenericDriverNodeManagerTests.cs`.
|
||||
2. Delete `src/Core/.../OpcUa/EquipmentNodeWalker.cs` (+ `IdentificationFolderBuilder` if co-located and unused) + `tests/.../OpcUa/EquipmentNodeWalkerTests.cs`. Update the doc-comment references in `DeploymentArtifact.cs:666`, `AddressSpaceComposer.cs:287,530`, `OpcUaClientTagConfigModel.cs:11`, `IScriptTagCatalog.cs:56,182`, `OtOpcUaNodeManager.cs:34` to point at `TagConfigIntent.Parse` (they currently cite `EquipmentNodeWalker.ExtractFullName` as the canonical parser — that citation moves to Commons under C-1).
|
||||
3. Delete `TryParseRelayBody` from `EquipmentScriptPaths.cs:164-172` + its two test cases in `EquipmentScriptPathsTests.cs:215-236`.
|
||||
4. Verify the `Core/OpcUa/` folder is empty afterward; if so, remove it.
|
||||
|
||||
**Tests.** Removal only — the deleted tests go with the code. Full `dotnet build` + `dotnet test` to confirm no lingering references.
|
||||
|
||||
**Effort: S. Risk: low** (pure deletion; the compiler proves no production caller). Coordinate ordering with C-1 so the doc-comment repointing is consistent.
|
||||
|
||||
---
|
||||
|
||||
## 10. U-2 — Tier C recycle machinery has no `IDriverSupervisor` implementation (Medium)
|
||||
|
||||
**Verification: CONFIRMED.** `IDriverSupervisor` refs are exactly: the interface (`Core.Abstractions/IDriverSupervisor.cs`), two Stability consumers (`MemoryRecycle.cs`, `ScheduledRecycleScheduler.cs`), and their tests. **Zero concrete implementations.** Out-of-process drivers moved to external mxaccessgw/HistorianGateway sidecars, so no in-repo driver is Tier C — the recycle path is compiled, tested, unreachable.
|
||||
|
||||
**Root cause.** Speculative infrastructure for a tier-migration workflow that the external-gateway architecture superseded.
|
||||
|
||||
**Proposed design.** Two defensible options; recommend **prune** consistent with U-1's philosophy and OVERALL Action #12 ("delete `MemoryRecycle`/`IDriverSupervisor` dead machinery"):
|
||||
- **Option A (recommended): delete** `MemoryRecycle`, `ScheduledRecycleScheduler`, `IDriverSupervisor`, `DriverResilienceStatusTracker.RecordRecycle` (verify it has no other caller first), and their tests. Removes a fully-built-but-unreachable subsystem.
|
||||
- **Option B (if there's a real near-term tier-migration plan): keep + document** with an explicit "speculative — no production caller today; wired when in-process Tier C returns" note on each class, plus a tracking issue. The report leans toward pruning ("prune it alongside U-1").
|
||||
|
||||
**Decision input needed:** confirm with the maintainer that no tier-migration workflow is imminent. Absent that, prune.
|
||||
|
||||
**Implementation steps (Option A).**
|
||||
1. Grep `RecordRecycle` — if only Stability + tests, remove it from `DriverResilienceStatusTracker`.
|
||||
2. Delete `MemoryRecycle.cs`, `ScheduledRecycleScheduler.cs`, `IDriverSupervisor.cs`, `WedgeDetector`? (No — `WedgeDetector` is demand-aware stall detection, separate; keep unless it also proves callerless — grep first).
|
||||
3. Delete `MemoryRecycleTests.cs`, `ScheduledRecycleSchedulerTests.cs`.
|
||||
4. Note: `MemoryTracking` (median-baseline breach) may feed `MemoryRecycle` — verify `MemoryTracking` has an independent purpose (it likely feeds meters/health) before keeping; keep it if it has other consumers.
|
||||
|
||||
This also closes **S-4** (the ScheduledRecycleScheduler catch-up storm) by deletion. If Option B is chosen instead, apply the S-4 fast-forward fix below.
|
||||
|
||||
**Effort: S. Risk: low** (deletion, compiler-verified). **Blocked on maintainer confirmation** of no imminent tier-migration.
|
||||
|
||||
---
|
||||
|
||||
## 11. U-3 — Continuous-historization initial ref set still empty at spawn (Medium — known-limitation debt)
|
||||
|
||||
**Verification: CONFIRMED (partially-done, as the report frames it).**
|
||||
- The **delta feed is implemented**: `AddressSpaceApplier.FeedHistorizedRefs` (`:310-353`) → `IHistorizedTagSubscriptionSink.UpdateHistorizedRefs` → `ActorHistorizedTagSubscriptionSink` → `ContinuousHistorizationRecorder.UpdateHistorizedRefs` (all present).
|
||||
- The **initial set is still empty**: `Runtime/ServiceCollectionExtensions.cs:272` spawns the recorder with `historizedRefs: Array.Empty<string>()`, with a comment acknowledging it. So after a **process restart** with no subsequent deploy, the recorder historizes nothing (CLAUDE.md KNOWN LIMITATION 2). **Note:** OVERALL theme #5 flags KNOWN LIMITATION 2's *first half* (delta feed) as stale-in-docs but the *restart-convergence* gap this finding names is real.
|
||||
|
||||
**Root cause.** The deployed address space (and thus the historized-ref set) is materialised at deploy time, not at actor-spawn time, so there's no clean set to resolve at wiring time. Only *subsequent* deploys feed deltas.
|
||||
|
||||
**Proposed design.** **Full-set replay on rebuild.** The convergent fix is: whenever the applier does a full `RebuildCalled` rebuild (which re-materialises the entire composition), feed the recorder the **complete current historized-ref set** (an absolute "set to exactly this" message), not just the diff. On a fresh process, the first deploy/rebuild after boot then converges the recorder to the full set even with no prior state; on steady-state edits the delta feed continues to apply.
|
||||
|
||||
Design detail — add a `SetHistorizedRefs(IReadOnlyList<HistorizedTagRef> full)` absolute-set operation to `IHistorizedTagSubscriptionSink` (the seam contract already supports the fix per the report), and have `AddressSpaceApplier.Apply` call it (instead of / in addition to the delta) **when `rebuilt == true`**, computing the full historized set from `plan`... but `plan` is a *diff*, not the full composition. So the full set must come from the **composition**, which the applier's `Materialise*` methods receive (`MaterialiseEquipmentTags(composition)`). Cleanest: after the rebuild's materialise passes in `OpcUaPublishActor` (`:349`), compute the full historized-ref set from `composition.EquipmentTags` (filter `IsHistorized && Alarm is null`, resolve override-or-FullName exactly as `HistorizedRef` does) and call `SetHistorizedRefs`. Recorder's `OnUpdateHistorizedRefs` already "holds the full set and re-registers it" — add an `OnSetHistorizedRefs` that replaces the set wholesale.
|
||||
|
||||
**Alternatives considered.**
|
||||
- *Feed the full set from `Apply` using the plan* — rejected: the plan is a diff; a fresh boot's first deploy against `_lastApplied = null` would produce an all-adds plan that *happens* to equal the full set, but that's fragile and breaks for a restart mid-stream where `_lastApplied` was restored. Deriving from `composition` is robust.
|
||||
- *Persist the recorder's set across restarts* — larger; the FasterLog outbox is for values, not interest. Rebuild-replay is simpler and already aligned with the deploy lifecycle.
|
||||
|
||||
**Implementation steps.**
|
||||
1. `Core.Abstractions/Historian/IHistorizedTagSubscriptionSink.cs`: add `void SetHistorizedRefs(IReadOnlyList<HistorizedTagRef> full)`. Update `NullHistorizedTagSubscriptionSink` no-op.
|
||||
2. `Runtime/Historian/ActorHistorizedTagSubscriptionSink.cs`: forward to a new `ContinuousHistorizationRecorder.SetHistorizedRefs` message.
|
||||
3. `ContinuousHistorizationRecorder.cs`: `Receive<SetHistorizedRefs>(OnSetHistorizedRefs)` — replace `_historizedRefs` wholesale and re-register dependency-mux interest.
|
||||
4. `OpcUaPublishActor.cs` (after `:354`, when a rebuild ran): derive the full historized set from `composition` and call the sink's `SetHistorizedRefs`. (Or add an applier method `FeedFullHistorizedRefs(composition)` symmetric with `FeedHistorizedRefs(plan)`.)
|
||||
5. Update CLAUDE.md KNOWN LIMITATION 2 to reflect the closed restart-convergence gap (coordinate with OVERALL Action #11).
|
||||
|
||||
**Tests.**
|
||||
- `Runtime.Tests`: recorder converges to the full set on `SetHistorizedRefs` (registers interest in exactly the historized tags), and a fresh-spawn → rebuild → `SetHistorizedRefs` path historizes the full set.
|
||||
- Restart-convergence integration test (the one OVERALL theme #5 says the closure "deserves"): spawn recorder empty, apply a composition with historized tags, assert values flow without a prior delta.
|
||||
|
||||
**Effort: M. Risk: medium** — touches the historization actor + the seam contract; live value-capture is still gated by the separate ContinuousHistorization live-validation (CLAUDE.md), so verify offline via unit/actor tests plus the env-gated live suite when a gateway is reachable.
|
||||
|
||||
---
|
||||
|
||||
## 12. P-4 — Per-write allocation in the non-idempotent write path (Low)
|
||||
|
||||
**Verification: CONFIRMED.** `CapabilityInvoker.ExecuteWriteAsync` (`:135-152`) builds `noRetryOptions = snapshot with { CapabilityPolicies = new Dictionary<...>{...} }` per non-idempotent write, then `GetOrCreate(id, "{host}::non-idempotent", Write, noRetryOptions)` — and `GetOrCreate` keys only on `(id, host, capability)`, ignoring the options after first build (comment cites the "1% pipeline budget").
|
||||
|
||||
**Root cause.** The no-retry options snapshot is rebuilt every call even though the resolved pipeline is cached and the options are only consumed on first build.
|
||||
|
||||
**Proposed design.** Since the cache key already ignores the options after first build, the allocation is pure waste on the hot path. Fastest fix: **only build `noRetryOptions` on a cache miss.** Add a `GetOrCreate` overload that takes an options *factory* (`Func<DriverResilienceOptions>`) invoked lazily only when the pipeline isn't cached. Then `ExecuteWriteAsync` passes a factory closing over `snapshot`; the closure (small) is built but the record+Dictionary only allocate on miss.
|
||||
|
||||
**Alternatives considered.**
|
||||
- *Cache the no-retry pipeline separately keyed on options generation* — more state; the factory-on-miss approach reuses the existing cache and is minimal.
|
||||
- *Precompute a per-host non-idempotent pipeline at options-change time* — most efficient but requires an options-change hook; overkill for a Low.
|
||||
|
||||
**Implementation steps.**
|
||||
- `DriverResiliencePipelineBuilder.cs`: add `GetOrCreate(id, host, capability, Func<DriverResilienceOptions> optionsFactory)` overload that only calls the factory on a `ConcurrentDictionary` miss (use `GetOrAdd` with the factory).
|
||||
- `CapabilityInvoker.ExecuteWriteAsync`: still snapshot `_optionsAccessor()` once (needed for correctness per the existing comment), but move the `with { ... }` construction into the factory lambda so it only runs on miss. (Minor: the snapshot itself is cheap; the Dictionary is the cost.)
|
||||
|
||||
**Tests.** `Core.Tests` resilience: assert a non-idempotent write on a warm cache does not rebuild options (spy the factory invocation count == 0 on the second call), and retries are still disabled.
|
||||
|
||||
**Effort: S. Risk: low.**
|
||||
|
||||
---
|
||||
|
||||
## 13. S-3 — `Subscribe`/`RegisterAsync` have no disposed guard (Low)
|
||||
|
||||
**Verification: CONFIRMED.**
|
||||
- `PollGroupEngine.Subscribe` (`:73-84`) inserts into `_subscriptions` with no `_disposed` check; `DisposeAsync` (`:213-236`) snapshots `.Values` then clears — a `Subscribe` winning the race after the snapshot leaks a live loop.
|
||||
- `DriverHost.RegisterAsync` (`:53-71`) vs `DisposeAsync` (`:92-106`): a registration after the snapshot+clear is initialized but never shut down.
|
||||
|
||||
**Root cause.** Shutdown-window races; no disposed flag.
|
||||
|
||||
**Proposed design.** Add a `volatile bool _disposed` to each, set at the top of `DisposeAsync` (inside the lock for `DriverHost`), and check it before insert:
|
||||
- `PollGroupEngine.Subscribe`: if `_disposed`, either throw `ObjectDisposedException` (strict) or return a no-op handle. Given callers may race legitimately during teardown, prefer throwing `ObjectDisposedException` (the SDK convention) so callers don't silently believe they subscribed.
|
||||
- `DriverHost.RegisterAsync`: check `_disposed` inside the existing `_lock` before adding; throw `ObjectDisposedException` if set.
|
||||
|
||||
Note `DisposeAsync` for `PollGroupEngine` isn't lock-guarded (it's `ConcurrentDictionary`-based); set `_disposed = true` first, then the snapshot loop, and have `Subscribe` check `_disposed` *after* `Interlocked.Increment` but *before* `_subscriptions[id] = state` — and if disposed-after-insert, remove+stop. Simplest robust form: check `_disposed` before starting the loop; if a dispose is concurrent, the double-check pattern (insert, then re-check `_disposed`, and if set, `TryRemove` + cancel) closes the window.
|
||||
|
||||
**Implementation steps.**
|
||||
- `PollGroupEngine.cs`: add `_disposed`; guard `Subscribe` with the insert-then-recheck pattern; set flag in `DisposeAsync`.
|
||||
- `DriverHost.cs`: add `_disposed`; guard `RegisterAsync` inside `_lock`; set flag in `DisposeAsync` inside `_lock`.
|
||||
|
||||
**Tests (Core.Tests / Core.Abstractions.Tests).**
|
||||
- `Subscribe` after `DisposeAsync` throws `ObjectDisposedException` and leaves no live loop (assert `ActiveSubscriptionCount == 0`).
|
||||
- `RegisterAsync` after `DisposeAsync` throws and doesn't leave an initialized-but-unshut driver.
|
||||
|
||||
**Effort: S. Risk: low.**
|
||||
|
||||
---
|
||||
|
||||
## 14. S-4 — ScheduledRecycleScheduler catch-up recycle storm after a stall (Low)
|
||||
|
||||
**Verification: CONFIRMED.** `TickAsync` (`:72-84`) advances `_nextRecycleUtc += _recycleInterval` (`:82`) by exactly one interval per fire. K missed intervals → K back-to-back recycles.
|
||||
|
||||
**Root cause.** Fixed-increment scheduling with no fast-forward past `utcNow`.
|
||||
|
||||
**Proposed design.** **Depends on U-2's decision.** If U-2 prunes this subsystem (recommended), S-4 disappears with it — no work. If U-2 keeps it, apply the report's fix: after a fire, fast-forward `_nextRecycleUtc` past `utcNow` in whole intervals:
|
||||
```csharp
|
||||
// after RecycleAsync:
|
||||
do { _nextRecycleUtc += _recycleInterval; } while (_nextRecycleUtc <= utcNow);
|
||||
```
|
||||
(or compute the number of elapsed intervals arithmetically). Fires exactly once for any K-interval gap.
|
||||
|
||||
**Tests.** `ScheduledRecycleSchedulerTests`: feed a `utcNow` K intervals past `_nextRecycleUtc`; assert `RecycleAsync` called exactly once and `NextRecycleUtc > utcNow`.
|
||||
|
||||
**Effort: S. Risk: low. Gated on U-2 (likely deleted).**
|
||||
|
||||
---
|
||||
|
||||
## 15. S-5 — GDNM re-walk teardown not synchronized (Low — moot)
|
||||
|
||||
**Verification: CONFIRMED but moot.** `GenericDriverNodeManager.BuildAddressSpaceAsync` (`:58-71`) unsubscribes `_alarmForwarder` then `_alarmSinks.Clear()` non-atomically; a concurrent alarm event in the window is dropped. **But GDNM has no production caller (U-1).**
|
||||
|
||||
**Proposed design.** **Fold into U-1 — delete GDNM.** No standalone fix. If (against the recommendation) GDNM is kept, guard the teardown+re-subscribe under a lock.
|
||||
|
||||
**Effort: 0 (subsumed by U-1).**
|
||||
|
||||
---
|
||||
|
||||
## 16. C-4 — Composer violates its own purity contract with `Trace.TraceWarning` (Low)
|
||||
|
||||
**Verification: CONFIRMED.** Class doc says "Same inputs → same outputs, no logging" (`AddressSpaceComposer.cs:281`); the dangling-predicate-script skip emits `Trace.TraceWarning` (`:493-496`) — the only `System.Diagnostics.Trace` use in a Serilog codebase, invisible in production sinks.
|
||||
|
||||
**Root cause.** A warning was needed at a skip site in a static pure method with no logger; `Trace` was the path of least resistance.
|
||||
|
||||
**Proposed design.** **Return skipped-alarm ids in the composition** for the caller (which has an `ILogger`) to log. Add `IReadOnlyList<string> SkippedAlarmIds` (or a richer `SkippedAlarm(id, equipmentId, predicateScriptId)` record list) to `AddressSpaceComposition`; the composer collects skips instead of tracing; `OpcUaPublishActor` (which already logs the apply outcome) logs them at `Warning`. This preserves purity (same inputs → same outputs, including the skip list) and makes the warning visible.
|
||||
|
||||
**Alternatives considered.**
|
||||
- *Inject an `ILogger`* — rejected: breaks the pure-static contract the byte-parity design leans on (and the artifact-decode mirror stays logger-free).
|
||||
- *Just delete the trace* — rejected: loses a genuine operator signal (a dangling predicate reference that the draft validator *should* have caught but is the last line of defense).
|
||||
|
||||
**Implementation steps.**
|
||||
- `AddressSpaceComposition`: add `SkippedAlarmIds` init-only member (default empty).
|
||||
- `AddressSpaceComposer.cs:486-516`: collect skips into a list, drop the `Trace.TraceWarning`, set the member on return.
|
||||
- `OpcUaPublishActor`: after `ParseComposition`/compose, if `SkippedAlarmIds` non-empty, `_log.Warning(...)`. (Note: the actor consumes `DeploymentArtifact.ParseComposition`, not the composer directly — so the artifact-decode mirror must also carry/emit the skip list, or the skip must be surfaced at compose time on the publish side. Verify which seam the runtime uses and thread the skip list through it.)
|
||||
- Fix the class doc if any residual logging remains (it won't).
|
||||
|
||||
**Tests.** `OpcUaServer.Tests`: compose with a scripted alarm whose predicate script is absent → assert the alarm is skipped AND its id appears in `SkippedAlarmIds`.
|
||||
|
||||
**Effort: S. Risk: low.**
|
||||
|
||||
---
|
||||
|
||||
## 17. C-5 — Inconsistent duplicate-key defensiveness in Compose (Low)
|
||||
|
||||
**Verification: CONFIRMED.** `deviceHostById` uses a deliberate last-wins `foreach` with a comment (`AddressSpaceComposer.cs:359-364`), but `driversById`/`namespacesById` (`:401-402`) and `scriptsById` (`:453`) use plain `ToDictionary` — a duplicate logical id (only on DB-constraint bypass) throws and crashes the whole compose.
|
||||
|
||||
**Root cause.** Copy-paste divergence; the defensive posture wasn't applied uniformly.
|
||||
|
||||
**Proposed design.** **Adopt the defensive posture uniformly** (the report: "the defensive one is already argued for"). Replace the three `ToDictionary` calls with last-wins `foreach` builds (or a shared `static Dictionary<string,T> LastWins<T>(IEnumerable<T>, Func<T,string>)` helper in the composer). Rationale: a DB-constraint bypass should degrade gracefully (last-wins, matching the decode side) rather than crash the whole compose — consistent with the byte-parity contract the `deviceHostById` comment already defends.
|
||||
|
||||
**Alternatives considered.** *Make `deviceHostById` throw too (fail-fast everywhere)* — rejected: diverges from the artifact-decode side's last-wins, breaking byte-parity, which the existing comment explicitly warns against.
|
||||
|
||||
**Implementation steps.**
|
||||
- Add a private `LastWins` helper; replace `driversById`, `namespacesById`, `scriptsById` construction. Confirm the decode side (`DeploymentArtifact`) uses last-wins for the same maps and matches.
|
||||
|
||||
**Tests.** `OpcUaServer.Tests`: compose with a duplicate `DriverInstanceId` (or `ScriptId`) → assert no throw and last-wins selection, matching decode.
|
||||
|
||||
**Effort: S. Risk: low.**
|
||||
|
||||
---
|
||||
|
||||
## 18. C-6 — Vestigial Akka package reference in Commons (Low)
|
||||
|
||||
**Verification: CONFIRMED.** `Commons.csproj:9` references `Akka`; the only "Akka" token in Commons source is a **doc comment** in `NodeDiagnosticsSnapshot.cs:15` ("over Akka"). No Akka type is used.
|
||||
|
||||
**Root cause.** Leftover from an earlier design; harmless today (every consumer is Akka-hosted) but contradicts Commons' dependency-light contracts role.
|
||||
|
||||
**Proposed design.** **Remove the `<PackageReference Include="Akka"/>` from `Commons.csproj`.** Build to confirm nothing breaks (grep already proves no type usage).
|
||||
|
||||
**Implementation steps.** Delete the line; `dotnet restore`/`build`. If CPM (`Directory.Packages.props`) versions it, leave the version entry (other projects may use it) but drop the Commons reference.
|
||||
|
||||
**Tests.** Build + full test run.
|
||||
|
||||
**Effort: S. Risk: low.**
|
||||
|
||||
---
|
||||
|
||||
## 19. C-7 — Stale scaffold-era XML docs on load-bearing classes (Low)
|
||||
|
||||
**Verification: CONFIRMED.**
|
||||
- `AddressSpaceApplier.cs:7-26` class doc still describes the F14 scaffold ("For now we record the work", "the SDK adapter that lands in F10b will decide…") though both shipped (the class *does* the surgical-vs-rebuild decision now).
|
||||
- `DriverHost.cs:5-10` doc promises "per-process isolation for Tier C … implemented in Phase 2 via named-pipe RPC" — superseded by the out-of-repo mxaccessgw gateway.
|
||||
|
||||
**Root cause.** Docs not updated when the features shipped / architecture changed.
|
||||
|
||||
**Proposed design.** Rewrite both class docs to describe current reality. For `AddressSpaceApplier`: describe the actual full-rebuild-vs-surgical decision, the two detached hooks, and the SDK-free sink boundary. For `DriverHost`: describe it as the in-process id→`IDriver` lifecycle registry; drop the named-pipe/Tier C narrative (or add a one-line "out-of-process drivers now live in external gateways" pointer). Coordinate with the `fixdocs`/`checkdocs` skill conventions already used in this repo (recent commit `9cad9ed0` was an XML-doc sweep).
|
||||
|
||||
**Implementation steps.** Edit the two class-doc blocks. While there, if U-2 keeps `DriverHost`'s Tier C references anywhere, align them.
|
||||
|
||||
**Tests.** None (doc). Optionally run `checkdocs`/CommentChecker.
|
||||
|
||||
**Effort: S. Risk: none.**
|
||||
|
||||
---
|
||||
|
||||
## 20. U-4 — Additive-only ACL model (no Deny) — documented v2.0 gap (Low)
|
||||
|
||||
**Verification: CONFIRMED.** `PermissionTrie.cs:13-17` explicitly states "pure union (additive grants; no explicit Deny in v2.0)." A broad cluster-root grant can't be carved back at a sub-scope.
|
||||
|
||||
**Root cause.** Deliberate v2.0 simplification aligned with flat, global AdminUI roles.
|
||||
|
||||
**Proposed design.** **No code change — document the boundary in the security docs.** Add a "Authorization model: additive-only, no Deny" subsection to `docs/security.md` stating the bound (fine-grained OT authz needs a trie-walk semantics change to add Deny/most-specific-wins) so the limitation is visible to deployment planners. If/when Deny is needed, it's a scoped follow-on: add a `Deny` flag dimension to `TrieGrant` and change `CollectMatches` from pure-OR to a precedence walk (most-specific-scope Deny wins) — out of scope here.
|
||||
|
||||
**Implementation steps.** Doc-only edit to `docs/security.md`.
|
||||
|
||||
**Effort: S. Risk: none.**
|
||||
|
||||
---
|
||||
|
||||
## 21. U-5 — Uneven test coverage; ClusterRoleInfo untested (Low)
|
||||
|
||||
**Verification: CONFIRMED.** `Cluster.Tests` contains only `RoleParserTests`, `HoconLoaderTests`, `ServiceLevelCalculatorTests` (the three pure classes). `ClusterRoleInfo` — the only concurrency-bearing class in Cluster (lock + subscriber actor + event fan-out) — has **no dedicated unit test** (it appears only in Server-level integration harnesses: `RedundancyStateActorTests`, `MultiClusterScopingTests`, `TwoNodeClusterHarness`, `ServiceCollectionExtensionsTests`). The applier failure surface (S-1) and `DriverHost`/`PollGroupEngine` dispose races (S-3) are untested because those surfaces don't exist yet.
|
||||
|
||||
**Root cause.** Coverage concentrated on pure/deterministic classes; the concurrency-bearing seam and fault surfaces were left to live-verification / integration.
|
||||
|
||||
**Proposed design.** Add an **Akka TestKit harness for `ClusterRoleInfo`** exercising leader-change sequencing — the report's "riskiest untested seam." Cover: role topology snapshot updates under the lock, event fan-out on membership/leader change, and no-lost-update under concurrent snapshot reads during a leader change. The S-1 and S-3 tests are already specified under their own findings (they land as those surfaces are built), which is the bulk of closing U-5's named gaps.
|
||||
|
||||
**Implementation steps.**
|
||||
- `tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests/ClusterRoleInfoTests.cs` using Akka.TestKit (verify TestKit is already a test dep — `Host.IntegrationTests` uses cluster harnesses, so it's available): drive a fake cluster-event stream through the subscriber actor and assert `ClusterRoleInfo`'s snapshot + fan-out.
|
||||
|
||||
**Tests.** This finding *is* a test-addition; no product code changes.
|
||||
|
||||
**Effort: M. Risk: low** (test-only; the risk is flaky async assertions — use TestKit's deterministic probes).
|
||||
|
||||
---
|
||||
|
||||
## Cross-finding sequencing (recommended PR order)
|
||||
|
||||
1. **PR A (C-1 + P-1 + C-3):** land `TagConfigIntent.Parse` + `DeviceConfigIntent` in Commons; add `Configuration → Commons` reference; retrofit composer, DeploymentArtifact, DraftValidator; add shared driver-type constant + `EquipmentNodeIds`-keyed collision check. Behind the byte-parity test suite. *Highest-leverage; unblocks C-3 and the U-1 doc-repointing.*
|
||||
2. **PR B (U-1 + S-5):** delete GDNM, EquipmentNodeWalker, TryParseRelayBody + tests; repoint doc comments at Commons parser (depends on PR A).
|
||||
3. **PR C (S-1):** applier failure surface + publish-actor signal.
|
||||
4. **PR D (U-2 decision):** prune Tier C recycle (closes S-4) — *gated on maintainer confirmation*; else apply S-4 fast-forward.
|
||||
5. **PR E (U-3):** full-set replay on rebuild + CLAUDE.md KNOWN LIMITATION 2 update.
|
||||
6. **PR F (S-2 + S-3):** PollGroupEngine `UnsubscribeAsync` + disposed guards + DriverHost guard.
|
||||
7. **PR G (P-2 + P-4):** authorization set caching + non-idempotent-write factory-on-miss.
|
||||
8. **PR H (Lows/docs):** C-4, C-5, C-6, C-7, C-2 note, U-4 doc, P-3 doc, U-5 ClusterRoleInfo tests.
|
||||
|
||||
---
|
||||
|
||||
## Notes for the executor
|
||||
|
||||
- **Byte-parity is the crown jewel.** Any change to composer/DeploymentArtifact/DraftValidator parsing (C-1/C-3/C-5) must keep the existing cross-seam parity test green; port semantics verbatim, no behavior change intended.
|
||||
- **`Commons` is a verified pure leaf** — moving shared code there is cycle-safe and the correct home per the `EquipmentScriptPaths` precedent.
|
||||
- **Deletions (U-1/U-2) are compiler-verified** — no production caller exists for GDNM, EquipmentNodeWalker, TryParseRelayBody, or any `IDriverSupervisor` impl. Git preserves history.
|
||||
- **U-2 and the U-2-dependent S-4 need one maintainer decision** (prune vs. keep-and-document the Tier C recycle machinery). Everything else is unblocked.
|
||||
- Live-verify surfaces in this domain are thin (compose/authorize/teardown are unit-testable); the only item needing a real gateway is U-3's continuous-historization value capture, which stays behind the existing env-gated live suite.
|
||||
@@ -0,0 +1,362 @@
|
||||
# Design + Implementation Plan — 02 Scripting, Virtual Tags, Scripted Alarms, Alarm Historian
|
||||
|
||||
- **Source report:** `archreview/02-scripting-alarms.md`
|
||||
- **Review commit:** `9cad9ed0` · **Plan verified against tree at:** `9cad9ed0` (master)
|
||||
- **Scope:** `Core.Scripting`(+`Abstractions`), `Core.VirtualTags`, `Core.ScriptedAlarms`, `Core.AlarmHistorian`, plus the live consumers in `Host`/`Runtime`.
|
||||
|
||||
## Verification summary
|
||||
|
||||
Every finding in the report was opened at the cited file:line and checked against the current tree. **All findings CONFIRMED — none stale.** Line numbers in the report match the current source. Key confirmations:
|
||||
|
||||
- **U2** — `RoslynVirtualTagEvaluator.Evaluate` (`Host/Engines/RoslynVirtualTagEvaluator.cs:101-108`) builds a `CancellationTokenSource(_runTimeout)` and calls `evaluator.RunAsync(context, cts.Token).GetAwaiter().GetResult()`. `ScriptEvaluator.RunAsync` (`Core.Scripting/ScriptEvaluator.cs:178-190`) runs the compiled delegate **synchronously** (`var result = _func(globals);` line 188) and only checks the token at entry (line 182). The CTS cannot interrupt a running script; the `catch (OperationCanceledException)` branch (line 105-108) is dead. `VirtualTagActor.OnDependencyChanged` (`Runtime/VirtualTags/VirtualTagActor.cs:113`) calls `_evaluator.Evaluate(...)` **inline in the actor's message handler** → a `while(true)` script hangs that actor's message loop forever. Registered live in `Host/Program.cs:217-221`. **Confirmed.**
|
||||
- **U3** — same file: `_cache` is a raw `ConcurrentDictionary<string, ScriptEvaluator<…>>` keyed by expression source (`RoslynVirtualTagEvaluator.cs:25-26`), populated via `GetOrAdd(expression, …Compile)` (line 67), only ever cleared in `Dispose()` (line 132-141). No apply-boundary eviction → cross-publish ALC accretion. `GetOrAdd` value-factory can double-compile under a race and leak the losing ALC. **Confirmed.**
|
||||
- **P1** — `ScriptEvaluator.Compile` calls `ScriptSandbox.Build(typeof(TContext))` (line 74) on every compile; `Build` runs `MetadataReference.CreateFromFile` for every pinned assembly + every `System.*`/netstandard TPA path (`ScriptSandbox.cs:83-87`, `EnumerateBclAssemblyPaths` 100-130). Immutable per `contextType.Assembly`, rebuilt every call. **Confirmed.**
|
||||
- **U1** — `new VirtualTagEngine`, `TimerTriggerScheduler`, `new VirtualTagSource` appear only under `tests/Core/…Core.VirtualTags.Tests`. No production instantiation. `DependencyGraph` (Tarjan/Kahn) has **no production consumer** — repo grep for topo/cycle in the live path returns only unrelated matches (`MemoryRecycle`, resilience). Live cascade fan-out is `DependencyMuxActor` (no topo sort; cross-tag writes dropped). **Confirmed.**
|
||||
- **S1–S6, S8, S10, S11, P2–P6, C1, C3, C7, U4, U5** — all confirmed at cited lines (details in each section below).
|
||||
|
||||
---
|
||||
|
||||
## Priority ordering
|
||||
|
||||
Ordered by the report's Priority Recommendations and the OVERALL prioritized action list (item #2 = U2/U3):
|
||||
|
||||
1. **U2** (Critical) — production VT script timeout ineffective
|
||||
2. **U3** (High) — live path bypasses `CompiledScriptCache`; ALC accretion
|
||||
3. **P1** (High) — `ScriptSandbox.Build` rebuilds full BCL ref set every compile
|
||||
4. **U1** (High) — dormant `Core.VirtualTags` engine stack
|
||||
5. **S2** (High) — no coalescing/backpressure on upstream fan-out
|
||||
6. **S1** (High) — `VirtualTagEngine.Load` un-gated (resolves with U1)
|
||||
7. **S5 + C7** (Medium) — timed-shelve expiry swallows `Unshelved` + doc drift
|
||||
8. **S3 / S4 / S6** (Medium) — reload-swap, `_alarmsReferencing` sync, sink dispose-drain (one hardening pass)
|
||||
9. **U4 / U5 / C1 / C2 / C5** (Medium) — Null Historize, Part 9 conformance doc, dup interface, namespace doc, contract split
|
||||
10. **Lows** (batched) — S7, S8, S9, S10, S11, P2–P6, C3, C4, C6, U6, U7
|
||||
|
||||
---
|
||||
|
||||
## 1. U2 — CRITICAL — Production virtual-tag script timeout is ineffective
|
||||
|
||||
**Restatement:** `RoslynVirtualTagEvaluator` runs scripts on the calling (actor) thread with a token that can never interrupt them; a CPU-bound/infinite-loop virtual-tag script hangs the owning `VirtualTagActor` forever — no timeout, no log, no recovery.
|
||||
|
||||
**Verification:** Confirmed (see summary). The dead code is `RoslynVirtualTagEvaluator.cs:105-108`. Root cause: `ScriptEvaluator.RunAsync` is synchronous by design (line 184-189 documents "TimedScriptEvaluator wraps this in Task.Run…"), but the Host adapter never wraps it — it hand-rolls a `CancellationTokenSource` that a synchronous `_func(globals)` invocation ignores.
|
||||
|
||||
**Root cause:** The one component that evaluates operator-authored VT scripts in production (`RoslynVirtualTagEvaluator`) reimplemented timeout handling incorrectly instead of reusing `TimedScriptEvaluator`, which was built for exactly this and is already used by the live alarm engine (`ScriptedAlarmEngine.cs:227`) and the dormant VT engine (`VirtualTagEngine.cs:117`).
|
||||
|
||||
**Proposed design:** Route the adapter's execution through `TimedScriptEvaluator<VirtualTagContext, object?>`, accepting the documented orphan-thread trade-off (S7) — this is the sanctioned mechanism (`Task.Run` + `WaitAsync`) that makes the wall-clock budget real for CPU-bound scripts. Alternatives considered:
|
||||
- *Inline `Task.Run` + `WaitAsync` in `Evaluate`* — duplicates `TimedScriptEvaluator` logic (the exact anti-pattern U1/theme-#2 warns against). Rejected in favor of reuse.
|
||||
- *Out-of-process runner* — the real fix for CPU budgeting but a v3 concern (per `TimedScriptEvaluator` docs); out of scope.
|
||||
|
||||
Because `TimedScriptEvaluator` wraps a *single* `ScriptEvaluator`, and the adapter caches evaluators per source, the wrapper is constructed per compiled entry. Combine with U3: the cache value should become the wrapped/timed evaluator (or a small record holding both). Simplest coherent shape — cache the `ScriptEvaluator` (from `CompiledScriptCache`, U3) and construct a `TimedScriptEvaluator` per call (cheap — it's a thin struct-like wrapper over the inner evaluator; the inner compiled delegate is what's expensive and that stays cached).
|
||||
|
||||
**Implementation steps:**
|
||||
1. In `RoslynVirtualTagEvaluator.Evaluate` (`Host/Engines/RoslynVirtualTagEvaluator.cs`), replace the block at lines 99-113:
|
||||
- Construct `var timed = new TimedScriptEvaluator<VirtualTagContext, object?>(evaluator, _runTimeout);`
|
||||
- Call `var raw = timed.RunAsync(context).GetAwaiter().GetResult();` (no CTS needed — the wall-clock budget lives in `TimedScriptEvaluator`).
|
||||
- Change the catch to `catch (ScriptTimeoutException) { return VirtualTagEvalResult.Failure($"script timed out after {_runTimeout.TotalSeconds:F1}s"); }` (keep the generic `catch (Exception)` for user-code faults).
|
||||
2. Keep `_runTimeout` default 2 s (existing). Optionally expose per-tag timeout later (out of scope).
|
||||
3. Note the orphan-thread interaction with S7: a hot looping script now orphans one pool thread per evaluation attempt. Track the S7 circuit-breaker as a fast follow (below) — but U2's fix (stop hanging the *actor*) is strictly better than today regardless.
|
||||
|
||||
**Tests (unit + regression):**
|
||||
- Add to the existing `tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/RoslynVirtualTagEvaluatorTests.cs`:
|
||||
- `Evaluate_WithInfiniteLoopScript_ReturnsFailureWithinTimeout` — expression `while(true){} return 0;` (or a spin that exceeds the budget); assert `result.Success == false`, `result.Reason` contains "timed out", and the call returns within ~timeout + slack (wrap in its own `Task.Run` + `WaitAsync(5s)` so a regression *fails the test* instead of hanging the suite).
|
||||
- `Evaluate_WithFastScript_StillReturnsValue` — regression that the wrapping didn't break the happy path or the passthrough fast-path.
|
||||
- Optionally an actor-level test in `Runtime.Tests` proving `VirtualTagActor` continues processing messages after a timed-out evaluation (message loop not wedged).
|
||||
|
||||
**Effort:** S. **Risk/blast-radius:** Low-Medium. Touches the single live VT evaluation path; the happy path is unchanged (passthrough fast-path still short-circuits before compile). The behavioral change is that a runaway script now returns Bad instead of hanging — strictly safer. Combine the edit with U3 (same method).
|
||||
|
||||
---
|
||||
|
||||
## 2. U3 — HIGH — Live path bypasses `CompiledScriptCache`: unbounded ALC accretion
|
||||
|
||||
**Restatement:** `RoslynVirtualTagEvaluator` caches compiled evaluators in a raw `ConcurrentDictionary` never evicted on republish; every edited-script publish roots a new collectible `AssemblyLoadContext` forever, and the `GetOrAdd` factory can double-compile under a race and leak the loser.
|
||||
|
||||
**Verification:** Confirmed (`RoslynVirtualTagEvaluator.cs:25-26, 67, 132-141`). This is the same leak `CompiledScriptCache.Clear()` was built to fix (`CompiledScriptCache.cs:34-40, 87-118`) and that both engines already route through (`ScriptedAlarmEngine.cs:66-77`, `VirtualTagEngine.cs:26-38`). The adapter is the lone hold-out.
|
||||
|
||||
**Root cause:** Adapter reimplemented caching with a plain dictionary + value-factory `GetOrAdd` instead of the purpose-built `CompiledScriptCache` (`Lazy` + `ExecutionAndPublication` single-compile, dispose-on-`Clear`). No apply-boundary `Clear()` is called because the adapter doesn't see config-apply — but the host actor does.
|
||||
|
||||
**Proposed design:** Swap the raw dictionary for `CompiledScriptCache<VirtualTagContext, object?>` and give the adapter a `Clear()`/`OnConfigApply()` seam the host actor invokes at each deploy boundary. Two wiring options for the apply-boundary clear:
|
||||
- *(a) Adapter exposes `void ClearCompiledScripts()`; `VirtualTagHostActor` calls it when it (re)loads a plan generation.* Preferred — mirrors how `ScriptedAlarmHostActor` drives `ScriptedAlarmEngine.LoadAsync` at apply, and how the alarm engine's `_compileCache.Clear()` runs per generation.
|
||||
- *(b) `IVirtualTagEvaluator` gains an optional `Clear` member.* Broader surface change; the interface has a `NullVirtualTagEvaluator` and other consumers — only do this if the host actor can't reach the concrete type. Given `Host/Program.cs:217-221` registers the concrete `RoslynVirtualTagEvaluator` as a singleton AND as `IVirtualTagEvaluator`, the host actor can resolve the concrete singleton; prefer (a) but if the actor only has the interface, add a narrow `IScriptCacheOwner { void ClearCompiledScripts(); }` the adapter implements and the actor optionally casts to (theme #1: assert the wiring with a test).
|
||||
|
||||
**Implementation steps:**
|
||||
1. In `RoslynVirtualTagEvaluator`: replace `_cache` with `private readonly CompiledScriptCache<VirtualTagContext, object?> _cache = new();`
|
||||
2. Replace the `GetOrAdd` at line 67 with `_cache.GetOrCompile(expression)` inside the same try/catch (it throws the same `CompilationErrorException`/`ScriptSandboxViolationException`).
|
||||
3. Add `public void ClearCompiledScripts() => _cache.Clear();`
|
||||
4. `Dispose()` → `_cache.Dispose()` (drops the per-evaluator dispose loop; `CompiledScriptCache.Dispose` already disposes every materialised ALC).
|
||||
5. Wire the apply-boundary clear: in `Runtime/VirtualTags/VirtualTagHostActor.cs`, at the point it receives a new plan generation / rebuilds its child `VirtualTagActor` set, call `ClearCompiledScripts()` on the injected evaluator (resolve the concrete type, or via the narrow interface from option (b)). Verify the host actor already has an apply/rebuild message it handles — if it recreates children per generation, that handler is the hook.
|
||||
6. Confirm `CompiledScriptCache.GetOrCompile`'s `Lazy` single-compile removes the double-compile race automatically.
|
||||
|
||||
**Tests:**
|
||||
- Unit (`Host.IntegrationTests` or a new `RoslynVirtualTagEvaluator` unit test): compile source A, then call `ClearCompiledScripts()`, assert the prior evaluator's ALC is reclaimed via `WeakReference` + `GC.Collect()` (mirror `CompiledScriptCacheTests` ALC-reclaim test at `tests/Core/…Core.Scripting.Tests/CompiledScriptCacheTests.cs`).
|
||||
- Wiring assertion (theme #1): a `VirtualTagHostActor` test proving a config-apply/rebuild triggers `ClearCompiledScripts` (spy evaluator or count cache entries before/after).
|
||||
|
||||
**Effort:** S. **Risk:** Low. Behavior-preserving except that stale ALCs are now released. Do in the same PR as U2 (same file). Blast radius = VT evaluation + one host-actor hook.
|
||||
|
||||
---
|
||||
|
||||
## 3. P1 — HIGH — `ScriptSandbox.Build` rebuilds the full BCL reference set on every compile
|
||||
|
||||
**Restatement:** Each compile re-reads 100+ `System.*`/netstandard DLLs into fresh `MetadataReference`s; a 200-script publish pays this ~200×, and the AdminUI ScriptAnalysis endpoints pay it per keystroke-cadence request.
|
||||
|
||||
**Verification:** Confirmed (`ScriptSandbox.cs:47, 83-87, 100-130`; called from `ScriptEvaluator.cs:74`). The reference list is a pure function of `contextType.Assembly` (the only per-call input) and the process TPA set — immutable for the process lifetime.
|
||||
|
||||
**Root cause:** No memoization; `MetadataReference.CreateFromFile` (which loads + parses `AssemblyMetadata`) runs unconditionally per compile.
|
||||
|
||||
**Proposed design:** Memoize the built `SandboxConfig` (references + imports) in a `static readonly ConcurrentDictionary<Assembly, SandboxConfig>` keyed by `contextType.Assembly`. The pinned OtOpcUa assemblies and the BCL path set are stable per key. `MetadataReference` instances are immutable and thread-safe to share across compilations (Roslyn is designed for this). Alternative: cache just the BCL `List<MetadataReference>` (process-global, key-independent) and rebuild only the 4 pinned entries per key — marginal extra win, more code. Prefer caching the whole `SandboxConfig` per `contextType.Assembly` (there are only ~3 distinct context assemblies in practice: VirtualTags, ScriptedAlarms, test contexts).
|
||||
|
||||
**Implementation steps:**
|
||||
1. In `ScriptSandbox`, add `private static readonly ConcurrentDictionary<Assembly, SandboxConfig> _cache = new();`
|
||||
2. Change `Build` body to `return _cache.GetOrAdd(contextType.Assembly, static asm => BuildUncached(asm));` — but note `Build` currently takes `contextType` and validates `ScriptContext`-assignability. Keep the validation in `Build` (before the cache lookup, since it's cheap and guards the contract), then key the cache on `contextType.Assembly`. Move the reference-list construction (lines 54-97) into a private `BuildUncached(Type contextType)` / or key on assembly and pin `typeof(DataValueSnapshot).Assembly` etc. which are constant.
|
||||
- Caveat: the four pinned assemblies at lines 59-70 include `contextType.Assembly` itself — keying on `contextType.Assembly` keeps them correct.
|
||||
3. No API change to callers; `SandboxConfig` is already an immutable record.
|
||||
|
||||
**Tests:**
|
||||
- `ScriptSandboxTests` (`tests/Core/…Core.Scripting.Tests/ScriptSandboxTests.cs`): assert two `Build(sameContextType)` calls return the **same** `SandboxConfig` reference (memoized), and `Build` for two different context types returns configs whose reference sets both resolve `ctx.GetTag`. Assert the existing escape-catalog tests still pass (they exercise the analyzer through `Compile`, which will now hit the cache).
|
||||
- Optional micro-benchmark note in the PR: compile-latency before/after (report predicts 10-100×).
|
||||
|
||||
**Effort:** S. **Risk:** Low. `MetadataReference` sharing across compilations is a supported Roslyn pattern. The only subtlety is the `contextType.Assembly` key correctly capturing the pinned-set variation. Blast radius = all script compilation (VT, alarm, ScriptAnalysis) — but purely a perf/allocation change; output is byte-identical.
|
||||
|
||||
---
|
||||
|
||||
## 4. U1 — HIGH — The entire `Core.VirtualTags` engine stack is dormant in production
|
||||
|
||||
**Restatement:** `VirtualTagEngine` (568 lines), `TimerTriggerScheduler` (162), `VirtualTagSource` (106), `DependencyGraph` (324) have no production instantiation; the live path (`VirtualTagHostActor` + `DependencyMuxActor` + `RoslynVirtualTagEvaluator`) reimplements the same concerns. ~1,160 engine + ~1,260 test lines maintained for dead code that carries latent bugs (S1, S3, P3).
|
||||
|
||||
**Verification:** Confirmed — `new VirtualTagEngine`/`TimerTriggerScheduler`/`new VirtualTagSource` only in `tests/Core/…Core.VirtualTags.Tests`. `DependencyGraph` has no production consumer (grep for Tarjan/topo/cycle in the live path finds only unrelated `MemoryRecycle`/resilience). `RoslynVirtualTagEvaluator`'s own xmldoc acknowledges the split (lines 20-21).
|
||||
|
||||
**Root cause:** The Akka actor pipeline (F8b) superseded the embedded engine, but the engine was never retired — a "built-but-never-wired" instance (OVERALL theme #1).
|
||||
|
||||
**Proposed design — RETIRE (report's preferred option (a)).** The live actor pipeline is the single sanctioned implementation. Once U2/U3 land, the actor path has the timeout + cache defenses the engine held, so nothing of value is lost. Retire:
|
||||
- `VirtualTagEngine.cs`, `TimerTriggerScheduler.cs`, `VirtualTagSource.cs`, `DependencyGraph.cs` (+ `DependencyCycleException`) and their test suites (`VirtualTagEngineTests`, `TimerTriggerSchedulerTests`, `VirtualTagSourceTests`, `DependencyGraphTests`).
|
||||
- **Keep** in `Core.VirtualTags`: `ITagUpstreamSource` (subject of C1 — hoist, don't delete), `VirtualTagDefinition`, `IHistoryWriter`/`NullHistoryWriter` (U4), and anything the composer/actor path references. **Verify before deleting** each type has no non-test consumer (grep each; `VirtualTagContext` lives in Abstractions and is consumed live, per C2).
|
||||
|
||||
*Alternative — sanction the engine as an embedded/non-Akka path and fix S1/S3/P3.* Rejected: doubles the maintenance surface with no consumer, and there is no roadmap need for a non-Akka embedded runtime. If one ever appears, the git history preserves the engine.
|
||||
|
||||
Retiring the engine **subsumes S1, S3, P3** (they are latent bugs in the deleted code) and removes half of C1/C3's duplication pressure.
|
||||
|
||||
**Implementation steps:**
|
||||
1. Grep each candidate type for non-test references (`VirtualTagEngine`, `TimerTriggerScheduler`, `VirtualTagSource`, `DependencyGraph`, `DependencyCycleException`). Confirm zero `src/` consumers (done for the four engine types; re-verify at delete time).
|
||||
2. Delete the four source files + their four test files; remove from any `.csproj`/slnx include if explicitly listed (they use globbing — likely no edit needed).
|
||||
3. If `ITagUpstreamSource` is being hoisted (C1), do that first so the delete doesn't strand the interface.
|
||||
4. Update `docs/VirtualTags.md` (referenced by `VirtualTagEngine` xmldoc) to describe only the live actor pipeline; drop engine-specific sections.
|
||||
5. Build + run the full `Core.VirtualTags.Tests` (now trimmed) and the `Runtime`/`Host` suites.
|
||||
|
||||
**Tests:** No new tests; deletion. Confirm the remaining `Core.VirtualTags.Tests` (definition/`IHistoryWriter`/`VirtualTagSource`-adjacent) still build. Confirm `Runtime.Tests` VT-actor coverage is the surviving safety net.
|
||||
|
||||
**Effort:** M (mechanical but touches ~2.4k lines + verification). **Risk:** Low if the grep confirms no `src/` consumers — but it is a large deletion, so land it **after** U2/U3 (so the live path is provably hardened first) and in its own PR for a clean revert. Blast radius = the dormant subtree only.
|
||||
|
||||
---
|
||||
|
||||
## 5. S2 — HIGH — No coalescing/backpressure on upstream-change fan-out
|
||||
|
||||
**Restatement:** Every upstream change spawns a fire-and-forget task queued on the single eval gate; a fast tag (e.g. 100 Hz) referenced by an alarm accumulates tasks without bound — memory growth, staleness, and wasted re-evaluation against the current cache.
|
||||
|
||||
**Verification:** Confirmed. `ScriptedAlarmEngine.OnUpstreamChange` (`ScriptedAlarmEngine.cs:442-450`) spawns one `ReevaluateAsync` per change via `TrackBackgroundTask`. The `_inFlight` set tracks but does not bound them. (The VT-engine sibling `OnUpstreamChange` at `VirtualTagEngine.cs:263-272` is retired under U1 — S2 applies to the *live* alarm engine only after U1.)
|
||||
|
||||
**Root cause:** Per-event task spawning instead of a dirty-set + single pump; cost scales with event rate, not distinct dirty alarms.
|
||||
|
||||
**Proposed design:** Replace per-event `ReevaluateAsync` spawning with a **dirty-set + single-consumer pump** inside `ScriptedAlarmEngine`:
|
||||
- `OnUpstreamChange` updates `_valueCache[path]` (unchanged), then for each referencing alarm id, `_dirtyAlarmIds.Add(id)` (a `ConcurrentDictionary<string, byte>` used as a set, or a `HashSet` under a lock), and signals a single long-lived pump (e.g. `_pumpSignal.Release()` on a `SemaphoreSlim(0)`, or an `AutoResetEvent`/`Channel<>`).
|
||||
- One pump loop (started in `LoadAsync`, stopped in `Dispose`) waits on the signal, drains the current dirty set under `_evalGate`, and re-evaluates each distinct dirty alarm once against the current cache. New changes during a pass simply re-mark dirty for the next pass — this is the coalescing.
|
||||
- This makes cost proportional to distinct dirty alarms per drain, not to event rate, and naturally bounds memory (the dirty set is bounded by the alarm count).
|
||||
|
||||
Alternatives: *bounded channel with drop-oldest* (simpler but drops evaluations — acceptable since re-eval reads current cache, but a dirty-set is strictly better because it never loses the *fact* that an alarm needs re-eval); *debounce timer per alarm* (more timers, more complexity). Prefer the dirty-set + single pump — it also composes with a future VT need if one arises.
|
||||
|
||||
Reuse the existing `_inFlight`/drain discipline for the pump task (the pump is one tracked task; `Dispose` already drains `_inFlight`). Keep the shelving timer as-is.
|
||||
|
||||
**Implementation steps:**
|
||||
1. Add `_dirtyAlarmIds` (concurrent set) + a signal primitive + a `Task _pumpTask` + a `CancellationTokenSource _pumpCts` to `ScriptedAlarmEngine`.
|
||||
2. `LoadAsync`: after `_loaded = true`, start the pump loop (once per engine lifetime, not per load — or restart per load; simplest is start-once in the constructor and gate work on `_loaded`). Guard: the pump must acquire `_evalGate` and re-check `_disposed`/`_loaded` (mirror `ReevaluateAsync` lines 460-463).
|
||||
3. `OnUpstreamChange`: mark dirty + signal instead of `TrackBackgroundTask(ReevaluateAsync(...))`.
|
||||
4. Pump body reuses the existing `EvaluatePredicateToStateAsync` + persist-before-memory + emit-outside-gate pattern (lines 465-489). Snapshot + clear the dirty set under a short lock, then process under `_evalGate`.
|
||||
5. `Dispose`: cancel `_pumpCts`, signal to unblock, await the pump (fold into the existing `_inFlight` drain or await `_pumpTask` explicitly).
|
||||
|
||||
**Tests:** `tests/Core/…Core.ScriptedAlarms.Tests/ScriptedAlarmEngineTests.cs`:
|
||||
- `RapidUpstreamChanges_CoalesceToBoundedEvaluations` — push N (e.g. 1000) changes to one referenced tag faster than eval drains (controllable clock / instrumented predicate counter); assert the predicate ran far fewer than N times and the final state reflects the last value.
|
||||
- `FanOut_DoesNotLeakTasks` — assert the in-flight/dirty structures return to empty after quiescence.
|
||||
- Regression: existing change-driven activation/clear tests still pass (a single change still promptly re-evaluates).
|
||||
|
||||
**Effort:** M. **Risk:** Medium — reworks the alarm engine's hot path and its concurrency model. The `_evalGate` + persist-before-memory + emit-outside-gate invariants must be preserved exactly. Land standalone with careful review; it is the alarm engine's most-exercised path. Blast radius = scripted-alarm evaluation only.
|
||||
|
||||
---
|
||||
|
||||
## 6. S1 — HIGH — `VirtualTagEngine.Load` mutates shared state with no gate
|
||||
|
||||
**Restatement:** `Load` clears/rebuilds `_tags`, `_graph`, and disposes the compile cache without holding `_evalGate`, racing in-flight `CascadeAsync`/`EvaluateInternalAsync`/timer ticks reading those non-thread-safe collections.
|
||||
|
||||
**Verification:** Confirmed. `Load` (`VirtualTagEngine.cs:78-173`) does `UnsubscribeFromUpstream(); _tags.Clear(); _graph.Clear(); _compileCache.Clear();` at 84-87 with no gate; `EvaluateInternalAsync` reads `_tags` at 293 before taking the gate (295); `CascadeAsync` walks `_graph` at 278.
|
||||
|
||||
**Root cause:** The engine never received the alarm engine's gate-held-load + concurrent-collection discipline.
|
||||
|
||||
**Proposed design:** **Resolved by U1 (retire the engine).** Since S1 is a latent bug in dormant code, deleting `VirtualTagEngine` eliminates it. **No separate fix.** If U1 is deferred or the engine is instead sanctioned (option (b)), then port the alarm engine's discipline: acquire `_evalGate` for the whole `Load`, make `_tags` a `ConcurrentDictionary`, and re-check-after-gate in `EvaluateInternalAsync` (it already reads `_tags` before the gate at 293 — move the read inside). Effort in that case: M; risk: Medium.
|
||||
|
||||
**Recommendation:** Fold into U1's deletion. **Effort:** none (subsumed). **Risk:** none.
|
||||
|
||||
---
|
||||
|
||||
## 7. S5 + C7 — MEDIUM — Timed-shelve expiry via predicate path swallows `Unshelved`; doc drift
|
||||
|
||||
**S5 restatement:** When `ApplyPredicate` expires a timed shelve (via `MaybeExpireShelving`), it returns `Activated`/`Cleared`/`None` and **never** `Unshelved` — that emission only comes from `ApplyShelvingCheck` on the 5 s timer. A predicate re-eval landing between expiry time and the next tick persists the state as unshelved but publishes no `Unshelved` event, so OPC UA clients / `/alerts` never learn the shelve ended.
|
||||
|
||||
**C7 restatement:** `ApplyPredicate`'s xmldoc (`Part9StateMachine.cs:31-34`) promises "branch-stack increment when a new active arrives while prior active is still un-acked" — no such logic exists (re-activation just resets Acked/Confirmed at lines 62-63).
|
||||
|
||||
**Verification:** Both confirmed. `ApplyPredicate` (`Part9StateMachine.cs:39-90`): `MaybeExpireShelving` at 48, expiry produces `stateWithShelving` but the only emissions returned are `Suppressed`/`Activated`/`Cleared`/`None` (67, 84, 89). `ApplyShelvingCheck` (300-316) is the sole `Unshelved` source. Doc-drift at 31-34 confirmed against the whole file (no branch logic anywhere).
|
||||
|
||||
**Root cause:** `TransitionResult` carries a single `EmissionKind`, so a compound "shelve expired AND value transitioned" can only report one. The engine's `EvaluatePredicateToStateAsync` (`ScriptedAlarmEngine.cs:546-552`) fires exactly one emission per predicate result.
|
||||
|
||||
**Proposed design:** Make the shelve-expiry emission explicit. Two options:
|
||||
- *(a) Engine-side detection (minimal, preferred).* In `EvaluatePredicateToStateAsync`, before calling `ApplyPredicate`, note `seed.Shelving.Kind`; after, if the result's `Shelving.Kind` changed from `Timed` to `Unshelved` (i.e. expiry happened inside `ApplyPredicate`), append a synthetic `Unshelved` emission to `pendingEmissions` **before** the value-transition emission. Order matters: `Unshelved` first, then `Activated`/`Cleared`. No state-machine signature change. Add an `AutoUnshelve` audit comment to match `ApplyShelvingCheck` for consistency (or centralize expiry so both paths append the same comment).
|
||||
- *(b) Compound result.* Change `ApplyPredicate` to return a list of emissions (or a nullable secondary `Unshelved`). Cleaner semantically but a wider signature change touching all callers/tests. Prefer (a) — smaller blast radius, keeps `Part9StateMachine` pure-function-per-transition.
|
||||
|
||||
Either way, centralize the expiry so `MaybeExpireShelving` and `ApplyShelvingCheck` don't drift.
|
||||
|
||||
**C7 fix:** Correct the `ApplyPredicate` xmldoc (lines 30-34) to describe actual behavior: "activation, clearing, timed-shelve expiry, and shelving suppression; re-activation resets Acked/Confirmed. No condition-branch / previous-instance stack (see U5 conformance note)."
|
||||
|
||||
**Implementation steps:**
|
||||
1. `Part9StateMachine.cs`: fix the `ApplyPredicate` summary (C7). Optionally factor a private `ExpireTimedShelvingWithAudit` so both entry points append the identical `AutoUnshelve` comment.
|
||||
2. `ScriptedAlarmEngine.EvaluatePredicateToStateAsync` (lines 504-553): capture `seed.Shelving.Kind`; after `ApplyPredicate`, detect Timed→Unshelved and prepend an `Unshelved` `ScriptedAlarmEvent` (via `BuildEmission(state, result.State, EmissionKind.Unshelved)`) to `pendingEmissions` ahead of the value emission.
|
||||
|
||||
**Tests:**
|
||||
- `Part9StateMachineTests.cs`: add `ApplyPredicate_WhenTimedShelveExpired_StatePersistsUnshelved` (state-level assertion — the machine already unshelves; assert `Shelving.Kind == Unshelved` after expiry).
|
||||
- `ScriptedAlarmEngineTests.cs`: `PredicateReeval_AfterTimedShelveExpiry_EmitsUnshelvedBeforeTransition` — set a timed shelve, advance the controllable clock past `UnshelveAtUtc`, drive a predicate change via upstream push (not the shelving timer), assert two emissions in order: `Unshelved` then `Activated`/`Cleared`.
|
||||
- Doc-only assertion not needed for C7.
|
||||
|
||||
**Effort:** S-M. **Risk:** Low-Medium — touches emission ordering; keep it additive (never suppress the existing value emission). Blast radius = scripted-alarm shelving semantics; OPC UA clients gain a correct `Unshelved` event.
|
||||
|
||||
---
|
||||
|
||||
## 8. S3 / S4 / S6 — MEDIUM — Reload-swap, `_alarmsReferencing` sync, sink dispose-drain (one hardening pass)
|
||||
|
||||
### S3 — Failed reload is fail-stop, not fail-back
|
||||
|
||||
**Restatement:** Both engines tear down the prior generation before compiling the new one; a compile OR transient `IAlarmStateStore` failure leaves the engine inert (alarm engine: zero alarms until next publish; VT engine: serving stale Good values).
|
||||
|
||||
**Verification:** Confirmed. `ScriptedAlarmEngine.LoadAsync` clears/unsubscribes at 196-209, compiles, throws at 250-255 on any failure — but the store-restore loop (273-281) can also throw (`_store.LoadAsync`/`SaveAsync`), and that path is **not operator-preventable** (unlike compile errors, which publish-time validation catches). VT-engine equivalent is retired under U1.
|
||||
|
||||
**Design:** For the alarm engine, split the two failure classes:
|
||||
- **Compile failures** — already aggregated (250-255); publish-time validation makes these unlikely. Keep fail-stop but note the operator-preventable nature. Optionally build the new generation into locals and swap under the gate only on success (compile is side-effect-free; subscriptions + timer are the only external effects). This is the cleaner "build-then-swap" the report recommends, but it is a larger rewrite of `LoadAsync`.
|
||||
- **Transient store failures** in the restore loop (273-281) — wrap each `_store.LoadAsync`/`SaveAsync` so a transient failure for one alarm falls back to `state.Condition` (the Fresh seed already compiled) and logs, rather than aborting the entire load. At minimum, distinguish store-failures from compile-failures and don't let a store hiccup zero out every alarm.
|
||||
|
||||
**Recommended scope:** the *targeted* fix (don't abort the whole load on a single-alarm store failure) is S; the full build-then-swap is M. Do the targeted fix now; note build-then-swap as optional follow-up.
|
||||
|
||||
**Implementation:** In `LoadAsync`'s restore loop (273-281), try/catch per alarm around `_store.LoadAsync`/`EvaluatePredicateToStateAsync`/`_store.SaveAsync`; on failure, keep the Fresh-seeded `AlarmState`, log a warning, and continue. The alarm still loads and evaluates live; only its persisted-state restore is skipped.
|
||||
|
||||
### S4 — `_alarmsReferencing` read without synchronization
|
||||
|
||||
**Restatement:** `_alarmsReferencing` is a plain `Dictionary` (`ScriptedAlarmEngine.cs:120-121`) mutated under `_evalGate` in `LoadAsync` (201, 237-242) but read by `OnUpstreamChange` (446) from upstream callback threads with no gate; a callback in flight during reload races `Clear()`/`Add()`.
|
||||
|
||||
**Verification:** Confirmed. `_alarms` was deliberately made `ConcurrentDictionary` for exactly this reader class (comment 42-50); `_alarmsReferencing` was missed. Note: S2's dirty-set rework also reads `_alarmsReferencing` in `OnUpstreamChange` — fix S4 **together with S2** since they touch the same method.
|
||||
|
||||
**Design:** Publish `_alarmsReferencing` as an immutable snapshot swapped atomically: keep a `private volatile IReadOnlyDictionary<string, IReadOnlyCollection<string>> _alarmsReferencing`; `LoadAsync` builds a fresh `Dictionary` under the gate and `Volatile.Write`s it; `OnUpstreamChange` reads the current reference (a coherent snapshot). Alternative: `ConcurrentDictionary` — but the value is a mutable `HashSet` also mutated during build, so the immutable-snapshot swap is cleaner and matches the "rebuilt per load" lifecycle. Prefer the volatile-snapshot swap.
|
||||
|
||||
**Implementation:** Change the field type; build a local `Dictionary<string, HashSet<string>>` in the `LoadAsync` compile loop, freeze it (or keep `HashSet` values but never mutate after publish), and `Volatile.Write`. `OnUpstreamChange` reads via `Volatile.Read` (or just the `volatile` field).
|
||||
|
||||
### S6 — `SqliteStoreAndForwardSink.Dispose` races an in-flight drain
|
||||
|
||||
**Restatement:** `Dispose` (`SqliteStoreAndForwardSink.cs:679-686`) sets `_disposed`, disposes the timer, then immediately disposes `_drainGate` + the writer without waiting for an in-flight `DrainOnceAsync` — the drain's `finally { _drainGate.Release(); }` (line 467) then throws `ObjectDisposedException`, and the writer can be disposed mid-`WriteBatchAsync`.
|
||||
|
||||
**Verification:** Confirmed. `DrainOnceAsync` takes `_drainGate.WaitAsync(0)` (323) and releases in `finally` (467); `Timer.Dispose()` (683) doesn't wait for a running callback (`DrainTimerCallback` is `async void`, 199-203).
|
||||
|
||||
**Design:** Mirror the alarm engine's dispose-drain (`ScriptedAlarmEngine.cs:751-769`). Before disposing `_drainGate`/writer, acquire the gate to guarantee no drain is mid-flight:
|
||||
- `Dispose`: set `_disposed`, dispose `_drainTimer`, then `_drainGate.Wait()` (blocking; sink is disposed from the host shutdown path, no sync-context deadlock) — this waits for any in-flight `DrainOnceAsync` to release — then dispose the writer, then dispose `_drainGate` (or don't re-release; since we hold it, just dispose after). Add a `_disposed` re-check at the top of `DrainOnceAsync` after acquiring the gate (it may already have one — verify around line 323-330) so a callback that passed the timer can bail.
|
||||
|
||||
**Implementation:** Rewrite `Dispose` (679-686) to: `_disposed = true; _drainTimer?.Dispose(); _drainGate.Wait(); try { if (_writer is IDisposable d) d.Dispose(); } finally { _drainGate.Dispose(); }`. Confirm `DrainOnceAsync` returns early when `_disposed` after acquiring the gate.
|
||||
|
||||
**Tests (S3/S4/S6):**
|
||||
- S3: `LoadAsync_WhenStoreThrowsForOneAlarm_OtherAlarmsStillLoad` (fake `IAlarmStateStore` throwing for a specific id) — assert the engine loads and evaluates the rest.
|
||||
- S4: covered indirectly by S2's concurrency test; optionally a targeted `Load_ConcurrentWithUpstreamCallback_NoTornRead` stress test.
|
||||
- S6: `Dispose_DuringInFlightDrain_DoesNotThrow` — a slow fake `IAlarmHistorianWriter.WriteBatchAsync` (blocks on a gate the test controls), start a drain, call `Dispose` on another thread, release the writer, assert no `ObjectDisposedException` surfaces and the writer's dispose ran after the batch completed. Add to `SqliteStoreAndForwardSinkTests.cs`.
|
||||
|
||||
**Effort:** S3 targeted S / S4 S / S6 S — bundle as one "alarm/sink hardening" PR (M total). **Risk:** Low-Medium; all three are fault-path hardening with clear tests. Sequence S4 with S2 (same method). Blast radius = alarm engine load + sink shutdown.
|
||||
|
||||
---
|
||||
|
||||
## 9. Medium conventions / underdeveloped: U4, U5, C1, C2, C5
|
||||
|
||||
### U4 — MEDIUM — `IHistoryWriter` is a permanently-Null surface
|
||||
|
||||
**Restatement:** `VirtualTagDefinition.Historize` routes to `IHistoryWriter.Record`, but DI binds only `NullHistoryWriter` (`Runtime/ServiceCollectionExtensions.cs:58`, `237`); the operator-visible Historize checkbox on virtual tags is a silent no-op.
|
||||
|
||||
**Verification:** Confirmed. `services.TryAddSingleton<IHistoryWriter>(NullHistoryWriter.Instance)` (line 58); no real impl in tree. Separate `ContinuousHistorization` path taps the mux directly (CLAUDE.md — and note the OVERALL report says Known Limitation 2 is *closed*, so continuous historization is now live via `FeedHistorizedRefs`/`UpdateHistorizedRefs`).
|
||||
|
||||
**Design:** Two coherent choices; pick per product intent:
|
||||
- *(a) Wire `IHistoryWriter` to the gateway value-writer* — make the checkbox real by binding a gateway-backed `IHistoryWriter` (the same `WriteLiveValues` path continuous-historization uses). Larger; overlaps the continuous-historization recorder. Only do this if VT-result historization is a distinct requirement from continuous historization.
|
||||
- *(b) Hide/annotate the checkbox until it does something* (preferred interim) — in the AdminUI VirtualTag modal, disable/hide the Historize toggle (or add an inline "not yet wired" note) and add an xmldoc/appsettings note that `IHistoryWriter` is Null unless a deployment overrides it. Avoids operator confusion at near-zero cost.
|
||||
|
||||
**Recommendation:** (b) now; open a tracked follow-up for (a) if VT-result historization is on the roadmap. Cross-reference the OVERALL note that continuous historization is the live historization path.
|
||||
|
||||
**Effort:** S (option b). **Risk:** Low. Touches AdminUI VT modal + a doc line.
|
||||
|
||||
### U5 — MEDIUM — Part 9 surface gaps real but undeclared
|
||||
|
||||
**Restatement:** No condition branches/previous-instances; `Severity` static per definition; `AlarmKind` affects only node typing; `MessageTemplate` has no brace escaping; `Retain` carried but not consulted. Each a defensible v1 cut, but scattered.
|
||||
|
||||
**Verification:** Confirmed against `Part9StateMachine.cs` (no branch logic), `ScriptedAlarmDefinition.cs` (static Severity), `MessageTemplate.cs`, and the engine (Retain unused).
|
||||
|
||||
**Design:** Documentation, not code. Add an explicit **OPC UA Part 9 conformance statement** section to `docs/ScriptedAlarms.md` listing the supported subset and the v1 cuts (no branching/previous-instances, static Severity, kind→node-typing only, no template brace escaping, Retain carried-not-enforced) so client integrators know what to expect. Fold C7's corrected `ApplyPredicate` doc into this.
|
||||
|
||||
**Effort:** S. **Risk:** None.
|
||||
|
||||
### C1 — MEDIUM — `ITagUpstreamSource` defined twice
|
||||
|
||||
**Restatement:** Identical-shape interface in `Core.VirtualTags/ITagUpstreamSource.cs` AND at `ScriptedAlarmEngine.cs:860-872`; two distinct .NET types, invariant enforced only by comment.
|
||||
|
||||
**Verification:** Confirmed — two `interface ITagUpstreamSource` definitions.
|
||||
|
||||
**Design:** Hoist a single `ITagUpstreamSource` into `Core.Scripting.Abstractions` (both projects reference it — verify the reference direction) and retire both duplicates. If `Core.VirtualTags` is being retired (U1), the alarm engine's copy is the survivor to relocate. Sequence: do C1 as part of/just before U1 so the delete doesn't strand the interface. Update `DependencyMuxTagUpstreamSource` and any composing bridge to the single type (they currently must adapt each separately).
|
||||
|
||||
**Implementation:** Add `ITagUpstreamSource.cs` to `Core.Scripting.Abstractions`; delete the copy from `ScriptedAlarmEngine.cs` (843-872 also holds `ScriptedAlarmEvent` — see C3) and the `Core.VirtualTags` file; fix usings. Confirm `Core.ScriptedAlarms` and `Core.VirtualTags` already reference `Core.Scripting.Abstractions`.
|
||||
|
||||
**Effort:** S-M. **Risk:** Low-Medium — a shared contract move; ensure all implementors (`DependencyMuxTagUpstreamSource`) compile against the single type. Blast radius = both engines + the mux bridge.
|
||||
|
||||
### C2 — MEDIUM — `Core.Scripting.Abstractions` declares types across three namespaces
|
||||
|
||||
**Restatement:** `ScriptContext`/`ScriptGlobals`/`PassthroughScript` → ns `Core.Scripting`; `VirtualTagContext` → ns `Core.VirtualTags`; `AlarmPredicateContext` → ns `Core.ScriptedAlarms` — all physically in the Abstractions assembly (deliberate: the sandbox pins `contextType.Assembly`, so concrete contexts must live Roslyn-free). Documented only in a `PassthroughScript` remark; misleads find-by-namespace.
|
||||
|
||||
**Verification:** Confirmed (rationale at `ScriptSandbox.cs:60-70`).
|
||||
|
||||
**Design:** Documentation, not a namespace migration (a `TypeForwardedTo` migration is higher-risk churn for a convention nit). Add a project-level `README.md` (or an `AssemblyInfo.cs` doc comment / a `_Namespaces.md`) in `Core.Scripting.Abstractions` stating the rule: "this assembly is the sandbox-pinned Roslyn-free closure; concrete script-context types keep their **consumer** namespaces deliberately so `ScriptSandbox.Build(contextType)` pins this assembly without dragging Roslyn." Reference `ScriptSandbox.cs:60-70`.
|
||||
|
||||
**Effort:** S. **Risk:** None.
|
||||
|
||||
### C5 — LOW/MEDIUM — `Core.AlarmHistorian` mixes contract + implementation
|
||||
|
||||
**Restatement:** `IAlarmHistorianSink`/`IAlarmHistorianWriter`/status types share the project with `SqliteStoreAndForwardSink`, so the Runtime adapter + gateway driver drag the `Microsoft.Data.Sqlite` dependency to get the interfaces.
|
||||
|
||||
**Verification:** Confirmed by project structure.
|
||||
|
||||
**Design:** Optional. Extract the interfaces + status DTOs into a `Core.AlarmHistorian.Abstractions` (mirror the scripting split) so consumers reference the contract without the SQLite impl. Tolerable at three files — schedule only if the SQLite transitive dependency becomes a problem for the gateway driver. Defer unless bundled with a broader Abstractions cleanup.
|
||||
|
||||
**Effort:** M (new project + reference rewiring). **Risk:** Low. **Recommend: defer** (note only).
|
||||
|
||||
---
|
||||
|
||||
## 10. Lows (batched)
|
||||
|
||||
| ID | Restatement | Verification | Action | Effort |
|
||||
|---|---|---|---|---|
|
||||
| **S7** | Sandbox CPU/mem unbounded; timed-out CPU-bound script leaks a pool thread; hot looping script orphans one thread per upstream change | Confirmed (`ScriptSandbox.cs:30-35`, `TimedScriptEvaluator.cs:17-26`). Note U2's fix *increases* orphan exposure on the VT path | Add a **per-script circuit breaker**: N consecutive timeouts → suspend evaluation + surface to Admin UI (meter/health). Cheap interim before out-of-process runner. Applies to both `RoslynVirtualTagEvaluator` (post-U2) and `ScriptedAlarmEngine` (quarantine the predicate after repeated timeouts — currently holds prior state but keeps re-evaluating, `ScriptedAlarmEngine.cs:535-538`) | M |
|
||||
| **S8** | `ScriptedAlarmEngine.Dispose` `Task.WhenAll(...).GetAwaiter().GetResult()` (`ScriptedAlarmEngine.cs:761`) — deadlock trap off the actor host | Confirmed | Implement `IAsyncDisposable` alongside `IDisposable`; actor host awaits. Low urgency (fine under current host) | S |
|
||||
| **S9** | At-least-once delivery duplicates on crash between `WriteBatchAsync` and outcome commit (`SqliteStoreAndForwardSink.cs:372-441`) | Confirmed (correct choice: dupes over loss) | Document in `docs/Historian.md` that the gateway `SendEvent` side must tolerate replays | S |
|
||||
| **S10** | `VirtualTagSource.SubscribeAsync` emits seed Read before registering observer (`VirtualTagSource.cs:62-73`) → a change between Read and Subscribe is missed | Confirmed | Retired with U1 (VirtualTagSource is dormant). If kept: document the seed-then-subscribe trade-off, or register-first + let idempotent newer-wins consumer dedupe | none (U1) |
|
||||
| **S11** | Capacity eviction drops *oldest* accepted alarm events (`SqliteStoreAndForwardSink.cs:602-636`), `EvictedCount` counter exists | Confirmed | Document the drop-oldest policy as deliberate in `docs/AlarmTracking.md`; note drop-newest/refuse-enqueue as the compliance alternative | S |
|
||||
| **P2** | `Task.Run`+`WaitAsync` per evaluation on the hot path (`TimedScriptEvaluator.cs:78-81`) | Confirmed; required for the timeout to work | No action now; note inline-run-with-watchdog as the escape hatch if profiling shows it. (U2 *adds* this hop to the VT path — accepted) | none |
|
||||
| **P3** | VT engine allocates per-eval what the alarm engine pools (`VirtualTagEngine.cs:298, 313-317`) | Confirmed | Retired with U1 | none (U1) |
|
||||
| **P4** | Single global eval gate per engine | Confirmed; correct + simple | No action (per-alarm/sharded gate is the escape hatch) | none |
|
||||
| **P5** | SQLite per-call PRAGMA + `COUNT(*)` overheads (`SqliteStoreAndForwardSink.cs:244-246, 479-484`) | Confirmed; capacity fast path already removed the hot cost | No action | none |
|
||||
| **P6** | `DependencyGraph` well-optimized | Confirmed | Retired with U1 | none (U1) |
|
||||
| **C3** | Trailing type defs: `ScriptedAlarmEvent` + dup `ITagUpstreamSource` at `ScriptedAlarmEngine.cs:843-872`; `CompilationErrorException`/`ScriptAssemblyLoadContext` at `ScriptEvaluator.cs:391-432`; `DependencyCycleException` in `DependencyGraph.cs` | Confirmed | Move `ScriptedAlarmEvent` (public cross-project contract) to its own file (do with C1's interface hoist). Others are reasonable co-location — leave | S |
|
||||
| **C4** | Repeated raw OPC UA status-code literals (`0x80340000u` etc.) across ≥5 files | Confirmed | Add a `KnownStatusCodes` static in `Core.Abstractions` (these layers avoid the OPC Foundation package deliberately) and replace the hand-rolled literals | S |
|
||||
| **C6** | Plan-era naming residue ("Phase 7 plan Stream A.4/B/C/…") in xmldocs across many files | Confirmed | Sweep to current doc anchors (`docs/ScriptedAlarms.md`/`docs/ScriptEditor.md`) during the next doc pass | S |
|
||||
| **U6** | No TODO/HACK/FIXME markers (gaps in xmldoc remarks) | Confirmed — healthy pattern | No action | none |
|
||||
| **U7** | Broad behavior-focused tests with specific holes: (a) S5 expiry-emission, (b) load-vs-cascade concurrency (S1/S4), (c) sink Dispose-during-drain (S6), (d) `ForbiddenTypeAnalyzer` no dedicated suite, (e) production `RoslynVirtualTagEvaluator` timeout (would've caught U2) | Confirmed — no `*ForbiddenType*` test file exists; `RoslynVirtualTagEvaluatorTests.cs` exists in `Host.IntegrationTests` | Holes (a)/(c)/(e) covered by the tests above (S5/S6/U2). Add a dedicated `ForbiddenTypeAnalyzerTests` suite (analyzer-pass unit tests against `CSharpCompilation`s) so analyzer regressions produce clear failures | S-M |
|
||||
|
||||
---
|
||||
|
||||
## Suggested PR sequencing
|
||||
|
||||
1. **PR-A (Critical, small):** U2 + U3 together (same file `RoslynVirtualTagEvaluator.cs` + one `VirtualTagHostActor` hook) + their tests. Ship first — closes the OVERALL action item #2 Critical.
|
||||
2. **PR-B (perf, small):** P1 sandbox-reference memoization + `ScriptSandboxTests`.
|
||||
3. **PR-C (cleanup, medium):** C1 (hoist `ITagUpstreamSource`) + C3 (`ScriptedAlarmEvent` file) → then U1 retire (`VirtualTagEngine`/`TimerTriggerScheduler`/`VirtualTagSource`/`DependencyGraph` + tests). Subsumes S1, S3-VT, P3, P6, S10. Land after PR-A.
|
||||
4. **PR-D (alarm hardening, medium):** S2 (dirty-set pump) + S4 (`_alarmsReferencing` snapshot, same method) + S6 (sink dispose-drain) + S3 (targeted store-failure fallback) + their tests.
|
||||
5. **PR-E (semantics, small-medium):** S5 + C7 (unshelve emission + doc) with state-machine + engine tests.
|
||||
6. **PR-F (docs/nits):** U4 (annotate checkbox), U5 (Part 9 conformance doc), C2 (Abstractions README), C4 (`KnownStatusCodes`), C6 (plan-era doc sweep), S9/S11 (doc policies), U7 (`ForbiddenTypeAnalyzerTests`). Optional S7 circuit-breaker + S8 `IAsyncDisposable` as tracked follow-ups.
|
||||
@@ -0,0 +1,599 @@
|
||||
# Design & Implementation Plan — Server & Runtime Subsystem (Report 03)
|
||||
|
||||
- **Source report:** `archreview/03-server-runtime.md`
|
||||
- **Review commit:** `9cad9ed0`
|
||||
- **Verification pass performed against tree at:** current `master` (same tree; all cited files present)
|
||||
- **Scope:** `OpcUaServer`, `Host`, `Runtime`, `ControlPlane`, `Security`; redundancy, LDAP, node manager, address-space applier.
|
||||
|
||||
## Verification summary
|
||||
|
||||
| ID | Sev | Status | One-liner |
|
||||
|----|-----|--------|-----------|
|
||||
| S1 | Critical | **Confirmed** | SBR configured in HOCON but never activated → NoDowning; hard-crash failover broken |
|
||||
| S2 | High | **Confirmed** | LDAP auth block-bridges session activation; no timeout, no pooling |
|
||||
| S3 | High | **Confirmed** | HistoryRead block-bridges gateway per node, sequentially, on SDK threads |
|
||||
| S4 | High | **Confirmed** | Primary write/ack gate defaults to allow while role unknown → dual-primary window |
|
||||
| P1 | High | **Confirmed** | Any structural add full-rebuilds address space, kills all subscriptions |
|
||||
| S5 | Medium | **Confirmed** | Redundancy NodeId string-matched across 3 sources, silent on mismatch |
|
||||
| S6 | Medium | **Confirmed** | No `SupervisorStrategy` anywhere in Runtime; restart wipes subscription state, hot-loops |
|
||||
| S7 | Medium | **Confirmed** | `DriverInstanceActor.PostStop` block-bridges `ShutdownAsync(None)` |
|
||||
| S8 | Medium | **Confirmed** | Inbound Part 9 / native-ack routes are at-most-once fire-and-forget, no metric |
|
||||
| S9 | Medium | **Confirmed** | Server cert 12-month lifetime, boot-only check, no expiry monitoring |
|
||||
| S10 | Low | **Confirmed** | SDK start failure swallowed; no health surface |
|
||||
| S11 | Low | **Confirmed** | `AuditWriterActor` unbounded buffer, drops batches (moot — no producers, U3) |
|
||||
| P2 | Medium | **Confirmed** | Per-value global-lock write, one actor msg per value, no batching |
|
||||
| P3 | Medium | **Confirmed** | HistoryRead-Events unbounded (`NumValuesPerNode==0 → int.MaxValue`), no paging |
|
||||
| P4 | Low | **Confirmed** | Every deploy re-runs 4 Materialise passes over full composition |
|
||||
| C1 | Medium | **Confirmed** | `ServerHistorian` bound imperatively in 5 places, no `IOptions` |
|
||||
| C2 | Medium | **Confirmed** | Two-tier options validation; `DevStubMode` only log-warned in prod |
|
||||
| C3 | Low | **Confirmed** | Library code logs via static Serilog; node manager uses obsolete `Utils.LogError` |
|
||||
| C4 | Low | **Confirmed** | `IHistorianProvisioning` resolve has no missing-registration warning |
|
||||
| U1 | High (doc) | **Confirmed stale doc** | CLAUDE.md "Known Limitation 2" stale — recorder ref-feed IS wired |
|
||||
| U2 | Medium | **Confirmed** | Deferred-sink forwarding correct today but only ~5/10 members test-guarded |
|
||||
| U3 | Medium | **Confirmed** | Audit pipeline built/tested, zero producers |
|
||||
| U4 | Medium | **Confirmed** | `FleetStatusBroadcaster.DriverHostStatusHeartbeat` dead code, host-only key |
|
||||
| U5 | Medium | **Confirmed** | Native Part 9 conditions: Acknowledge-to-driver only; Confirm/Shelve/Enable no-op upstream |
|
||||
| U6 | Medium | **Confirmed** | Test gaps concentrate on DPS delivery / supervision / hard-kill failover |
|
||||
| U7 | Low | **Confirmed** | `IHistoryWriter` Null-wired; H2 HistoryUpdate unimplemented; enum gap |
|
||||
| U8 | Low | **Confirmed** | `BuildSecurityPolicies` doc claims a log it doesn't emit |
|
||||
| U9 | Low | **Confirmed** | `EnsureVariable` silently ignores changed historize-intent (invariant in comments) |
|
||||
|
||||
**Nothing in this report is stale-because-already-fixed.** The one "stale" item (U1) is stale *documentation* — the code it describes as broken is actually wired. Every other finding reproduces against the current tree.
|
||||
|
||||
**Top 3 by priority:** (1) **S1** — activate the split-brain resolver (OVERALL action #1); (2) **P1** — surgical pure-adds in the address-space applier (OVERALL action #8, highest operational perf win); (3) **S4 + S2/S3** — the availability/blocking cluster (primary-gate default-deny, async LDAP, channelized HistoryRead; OVERALL actions #7/#9).
|
||||
|
||||
---
|
||||
|
||||
## S1 — CRITICAL — Split-brain resolver configured but never activated
|
||||
|
||||
**Restatement:** `akka.conf` carries a `split-brain-resolver { active-strategy = "keep-oldest" … }` block but nothing registers a downing provider, so the cluster runs Akka's default **NoDowning**. Hard-crashed nodes stay unreachable forever → singletons + `driver` role-leader never fail over; a partition leaves both sides at ServiceLevel 240.
|
||||
|
||||
**Verification — Confirmed.**
|
||||
- `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/Resources/akka.conf:40-46` — the SBR block is present, inside `akka.cluster { … }`.
|
||||
- `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/ServiceCollectionExtensions.cs:80-84` — `WithClustering(new ClusterOptions { SeedNodes = …, Roles = … })` sets **only** those two properties; `SplitBrainResolver` is never assigned.
|
||||
- Repo-wide grep for `downing-provider|SplitBrainResolver|KeepOldest|downing-provider-class` returns **zero** hits (confirmed empty).
|
||||
- Packages: `Akka.Cluster` / `Akka.Cluster.Hosting` = **1.5.62** (`Directory.Packages.props:6-13`). This version ships `Akka.Cluster.SBR.SplitBrainResolverProvider` and the `Akka.Cluster.Hosting` `ClusterOptions.SplitBrainResolver` option surface.
|
||||
- `min-nr-of-members = 1` (`akka.conf:38`) makes each partition side self-sufficient for leader election → the "both sides at 240" partition case is real.
|
||||
|
||||
**Root cause:** In Akka.NET the `split-brain-resolver` HOCON block is *inert configuration* — it only takes effect when `akka.cluster.downing-provider-class` points at `Akka.Cluster.SBR.SplitBrainResolverProvider` (or the Hosting `ClusterOptions.SplitBrainResolver` option, which sets that provider under the hood). The HOCON block was written but the activating registration was never added. Classic "config-in-HOCON is not config-in-effect" (OVERALL cross-cutting theme #1).
|
||||
|
||||
**Proposed design.** Prefer the **Akka.Hosting typed option** (`ClusterOptions.SplitBrainResolver`) over hand-writing `downing-provider-class`, because the rest of the cluster bootstrap already goes through `WithClustering(ClusterOptions)` and Akka.Hosting owns provider wiring — mixing a raw HOCON `downing-provider-class` with Hosting risks the same "Hosting doesn't honor HOCON" foot-gun already documented for `akka.loggers` (`akka.conf:9-13`). Keep the HOCON block as the tuning source (`stable-after`, `keep-oldest.down-if-alone`) — the SBR provider reads those keys once activated — OR move the strategy fully into the typed option and delete the HOCON block to avoid a two-source drift. **Recommendation: use the typed option and keep HOCON only for values the typed option can't express, documenting which wins.**
|
||||
|
||||
Strategy choice: **keep-oldest with `down-if-alone = on`** matches the current HOCON intent and is correct for a 2-node warm-redundancy pair (the oldest — typically the long-running primary — survives an even split; `down-if-alone` downs a singleton that loses quorum). `keep-majority` is wrong for 2 nodes (no majority in a 1-1 split). `static-quorum` would need `quorum-size = 2` which defeats single-node deploys. Keep-oldest is the right call; document the reasoning.
|
||||
|
||||
`stable-after = 15s` must be ≥ the failure detector's `acceptable-heartbeat-pause` (10s) plus margin — current 15s is fine but tie it explicitly to `down-removal-margin` (also 15s) in a comment.
|
||||
|
||||
**Implementation steps.**
|
||||
1. `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/ServiceCollectionExtensions.cs` — in `WithOtOpcUaClusterBootstrap`, set the SBR option on the `ClusterOptions`:
|
||||
```csharp
|
||||
builder.WithClustering(new ClusterOptions
|
||||
{
|
||||
SeedNodes = options.SeedNodes,
|
||||
Roles = options.Roles,
|
||||
SplitBrainResolver = new KeepOldestOption { DownIfAlone = true }, // Akka.Cluster.Hosting
|
||||
});
|
||||
```
|
||||
(Confirm the exact type name against Akka.Cluster.Hosting 1.5.62 — it is `KeepOldestOption : SplitBrainResolverOption`, with `DownIfAlone` and optional `Role`. `StableAfter` is set on the option or left to the HOCON default.)
|
||||
2. Add a **startup assertion log line** proving SBR is active (OVERALL theme #1 recommendation). In the same method after `WithClustering`, register a small `IActorRef`-less startup hook (or log in `WithOtOpcUaClusterBootstrap`) that reads `Context.System.Settings.Config.GetString("akka.cluster.downing-provider-class")` at first `MemberUp` and logs `Info` if it equals the SBR provider, `Error` if empty. A cheap version: a tiny actor subscribed to `ClusterEvent.IMemberEvent` that logs the resolved downing provider once. This converts the silent-inert failure mode into an operator-visible signal.
|
||||
3. Reconcile the HOCON: either delete the `split-brain-resolver` block from `akka.conf` (typed option now owns it) OR keep it and add a comment that `WithClustering(SplitBrainResolver=…)` is what activates it. Do not leave both un-cross-referenced.
|
||||
4. Update `docs/Redundancy.md` and `docs/v2/Architecture.md` — document that SBR is now active, the keep-oldest rationale, and the hard-crash failover expectation.
|
||||
|
||||
**Tests.**
|
||||
- **Unit:** a config test asserting `akka.cluster.downing-provider-class == "Akka.Cluster.SBR.SplitBrainResolverProvider, Akka.Cluster"` after `WithOtOpcUaClusterBootstrap` runs (proves the typed option activated the provider). This is the "assert SBR is active" guard.
|
||||
- **Integration (the load-bearing one):** extend the 2-node harness (`tests/Server/.../TwoNodeClusterHarness`) with a **hard-kill failover test**: bring both nodes up, confirm role-leader on node A, `kill -9`-equivalent (abrupt `ActorSystem` terminate / drop the transport without `CoordinatedShutdown`), then assert within `stable-after + margin` that (a) node B becomes `driver` role-leader, (b) the singletons (`RedundancyStateActor`, `ConfigPublishCoordinator`, `AdminOperationsActor`, `AuditWriterActor`) hand over to B, (c) B's ServiceLevel rises to 240 and the (dead) A demotes. The current harness only exercises graceful `StopAsync` — this is exactly the U6 blind spot. If in-process abrupt-kill is hard to simulate deterministically, use `Cluster.Get(sys).Down(selfAddress)` suppression + transport fault injection, or run it as a nightly docker-host multi-process test.
|
||||
- **Partition test (optional, nightly):** simulate a 1-1 partition (block the transport between the two), assert exactly one side downs the other and only one side keeps 240.
|
||||
|
||||
**Effort:** S (the fix) + M (the failover test harness). **Risk/blast-radius:** Medium. The one-line option change alters cluster downing behavior globally — a mis-tuned `stable-after` could down a node during a transient GC pause. Mitigate by keeping `stable-after ≥ acceptable-heartbeat-pause + margin` and validating on the 2-node rig before merge.
|
||||
|
||||
---
|
||||
|
||||
## P1 — HIGH — Structural deploy full-rebuilds the address space, severing every client subscription
|
||||
|
||||
**Restatement:** `AddressSpaceApplier.Apply` forces `RebuildAddressSpace()` for any added/removed/structurally-changed equipment, alarm, tag, or virtual tag. The rebuild removes and recreates every `NodeState`, so all existing client monitored items go dead server-wide. Adding one tag to one equipment drops every subscription on the server.
|
||||
|
||||
**Verification — Confirmed.**
|
||||
- `AddressSpaceApplier.cs:134-140` — `structuralRebuild` is true whenever `AddedEquipmentTags.Count > 0 || RemovedEquipmentTags.Count > 0 || AddedEquipment/AddedAlarms/… > 0`. Pure adds fall into this branch.
|
||||
- `AddressSpaceApplier.cs:150-154` — `if (structuralRebuild) { SafeRebuild(); rebuilt = true; }`.
|
||||
- `OtOpcUaNodeManager.cs:1690-1736` — `RebuildAddressSpace()` clears `_variables`, `_alarmConditions`, `_folders`, `_notifierFolders`, `_eventNotifierSources` under one `Lock`, detaching/removing every `NodeState`.
|
||||
- `OpcUaPublishActor.cs:338-354` — after `Apply`, the actor *unconditionally* re-runs the four `Materialise*` passes, which recreate all nodes from scratch. Existing subscriptions bind to the old (now-removed) `NodeState` instances → dead.
|
||||
- The surgical path (`AddressSpaceApplier.cs:155-199`) covers only **changed** tags (`TagDeltaIsSurgicalEligible`) and folder renames — **not adds or removes**.
|
||||
- Building blocks already exist and are idempotent: `EnsureFolder` (`:1284`), `EnsureVariable` (`:1369`, early-returns on existing id — U9 confirms), `MaterialiseAlarmCondition` (`:586`, idempotent guard `:57`), and `RaiseNodesAddedModelChange` (`:1592`).
|
||||
|
||||
**Root cause:** The applier's rebuild predicate treats "add" and "remove" identically to "structural change." But a pure add needs no teardown at all — the idempotent `EnsureFolder`/`EnsureVariable`/`MaterialiseAlarmCondition` passes add exactly the new nodes and no-op the existing ones; `RaiseNodesAddedModelChange` already exists to announce them to subscribed clients. The rebuild is unnecessary for the pure-add case and catastrophic for live subscriptions.
|
||||
|
||||
**Proposed design — surgical pure-adds (phase 1), scoped removals (phase 2).** This is the highest-leverage item in the subsystem (OVERALL action #8) and the report's own top recommendation. Do it in two phases so phase 1 (adds) ships value fast and low-risk.
|
||||
|
||||
**Phase 1 — pure-add surgical path.** Split the rebuild predicate: a rebuild is only *required* when there is a **removal** or a **node-affecting change** (the existing non-surgical-eligible change set). Pure adds route to the idempotent Materialise passes + a model-change announcement, no teardown:
|
||||
- New predicate:
|
||||
- `requiresRebuild = RemovedEquipment/RemovedAlarms/RemovedEquipmentTags/RemovedEquipmentVirtualTags any > 0` **OR** `ChangedEquipment.Count > 0` **OR** `ChangedAlarms.Count > 0` **OR** `ChangedEquipmentTags.Any(!surgicalEligible)` **OR** `ChangedEquipmentVirtualTags.Any(!nodeIrrelevant)`.
|
||||
- `pureAdds = !requiresRebuild && (AddedEquipment/AddedAlarms/AddedEquipmentTags/AddedEquipmentVirtualTags any > 0)`.
|
||||
- When `pureAdds` (and not requiresRebuild): **do not** call `RebuildAddressSpace()`. The existing unconditional Materialise passes in `OpcUaPublishActor` already add only the new nodes (idempotent). Add a step that calls `RaiseNodesAddedModelChange` for each newly-added node/folder so subscribed clients get a `GeneralModelChangeEvent` and can re-browse.
|
||||
- Because `OpcUaPublishActor` currently re-runs the Materialise passes regardless, the safest minimal change is: in `Apply`, compute `rebuilt=false` for the pure-add case and rely on the downstream passes; then emit model-change events for the plan's added ids. Confirm the Materialise passes run *after* `Apply` returns (they do — `OpcUaPublishActor.cs:335-354`), so the added nodes exist before the announce; if announce must follow materialise, move the announce into the applier's post-materialise hook or have `OpcUaPublishActor` call a new `applier.AnnounceAdds(plan)` after the passes.
|
||||
|
||||
**Phase 2 — scoped per-equipment removal.** For removals, add surgical `RemoveVariable(nodeId)` / `RemoveFolder(nodeId)` / `RemoveAlarmCondition(nodeId)` methods to the node manager (mirroring the per-node teardown already inside `RebuildAddressSpace`'s loops but scoped to one id), forwarded through `ISurgicalAddressSpaceSink` + `DeferredAddressSpaceSink` (respect the U2 forwarding trap — every new capability method MUST be added to the deferred wrapper and test-guarded). A remove of equipment E only tears down E's subtree, preserving every other equipment's subscriptions. Announce via `RaiseNodesDeletedModelChange` (add if the SDK exposes it; else a `GeneralModelChangeEvent`). Only fall back to a full rebuild for the genuinely-ambiguous changed-topology cases.
|
||||
|
||||
**Alternatives considered:** (a) Leave rebuild but *re-attach* subscriptions to new nodes — not feasible; the SDK binds monitored items to `NodeState` instances, and there's no supported re-home API. (b) Diff-and-patch every deploy generically — larger and riskier than the add/remove split; the plan already carries typed add/remove/change deltas, so use them. The add/remove split reuses the exact pattern F10b established for surgical changes.
|
||||
|
||||
**Implementation steps.**
|
||||
1. `AddressSpaceApplier.cs` — refactor the rebuild predicate into `requiresRebuild` / `pureAdds`; route pure adds away from `SafeRebuild()`. Add `AnnounceAdds(AddressSpacePlan)` calling `_sink.RaiseNodesAddedModelChange(id)` for each added folder/variable/alarm node id (compute ids via `EquipmentNodeIds` as the surgical change path already does at `:177`).
|
||||
2. `OtOpcUaNodeManager.cs` — Phase 2: add `RemoveVariable`/`RemoveFolder`/`RemoveAlarmCondition` (scoped teardown under `Lock`, drop the matching `_historizedTagnames`/`_eventNotifierSources`/`_nativeAlarmNodeIds` registrations); add `RaiseNodesDeletedModelChange` if available.
|
||||
3. `ISurgicalAddressSpaceSink` (+ `SdkAddressSpaceSink`, `NullAddressSpaceSink`, `DeferredAddressSpaceSink`) — Phase 2: add the remove methods with capability-sniffing forwarding, matching the existing `UpdateTagAttributes`/`UpdateFolderDisplayName` pattern (`DeferredAddressSpaceSink.cs:58-69`).
|
||||
4. `OpcUaPublishActor.cs` — ensure the announce runs after the Materialise passes (call `_applier.AnnounceAdds(plan)` after line 354, or fold the announce into `Apply` if ordering allows).
|
||||
|
||||
**Tests.**
|
||||
- **Unit:** applier test — a pure-add plan (one added tag) does **not** call `RebuildAddressSpace` (assert via a recording sink) and **does** call `RaiseNodesAddedModelChange` for the added id. A changed-only plan still routes surgical/rebuild as today. A remove plan (phase 2) calls `RemoveVariable` for the removed id and does not rebuild if no other change.
|
||||
- **Live-verify (the decisive one — F10b/OpcUaServer-001 precedent proves unit-green is insufficient):** on docker-dev (rebuild both central-1/2), subscribe a Client.CLI monitored item to an existing equipment tag (`subscribe -n ns=2;s=…`), then `POST /api/deployments` a config that **adds one new tag** to a *different* equipment. Assert the CLI subscription keeps delivering (no `Bad`/dead notifications) and the new tag becomes browsable. This is the exact regression the whole item exists to kill. Reuse the OpcUaServer-001 live-verify recipe from memory (`project_full_codebase_review_2026-06-19.md`).
|
||||
- **Integration:** `OpcUaServer.IntegrationTests` — over-the-wire subscribe-then-add-tag round-trip asserting subscription survival (also addresses U6's "one test only" gap).
|
||||
|
||||
**Effort:** L (phase 1 M, phase 2 M). **Risk/blast-radius:** High — this touches the address-space mutation core. Mitigate with the recording-sink unit tests + the mandatory subscription-survival live-verify + the U2 forwarding guard (a new remove capability that isn't forwarded through `DeferredAddressSpaceSink` ships inert, exactly like F10b).
|
||||
|
||||
---
|
||||
|
||||
## S4 — HIGH — Primary gate defaults to allow while role unknown (dual-primary data-plane window)
|
||||
|
||||
**Restatement:** `DriverHostActor`'s inbound-write / native-ack / native-alarm-emit gates gate on `_localRole is Secondary or Detached` and default to *allow* until the first redundancy snapshot arrives. A freshly-booted or snapshot-missed secondary services shared-field-device writes and emits alerts as primary for up to the 10s heartbeat interval — indefinitely if S5's NodeId mismatch bites.
|
||||
|
||||
**Verification — Confirmed.**
|
||||
- `DriverHostActor.cs:195` — `private RedundancyRole? _localRole;` (nullable, **null at boot**).
|
||||
- `HandleRouteNodeWrite` (`:1018-1026`): `if (_localRole is RedundancyRole.Secondary or RedundancyRole.Detached) { reject }` — null falls through to allow.
|
||||
- `HandleRouteNativeAlarmAck` (`:1074-1083`): same pattern, drops on Secondary/Detached, null → allow.
|
||||
- Native-alarm emit (`:956-984`): `if (_localRole is Secondary or Detached) continue;` — null → emit.
|
||||
- `OnRedundancyStateChanged` (`:1109-1116`): a snapshot not mentioning this node leaves `_localRole` unchanged → stays null → default-allow.
|
||||
|
||||
**Root cause:** The boot-window default-allow is correct for single-node deploys (where no snapshot ever demotes) but wrong for multi-node clusters where a booting secondary has a real primary peer. The gate can't distinguish "single node, no peer" from "multi-node, snapshot not yet arrived." It uses the *role signal* to gate *shared-device writes*, and the role signal is under-determined at boot.
|
||||
|
||||
**Proposed design — cluster-membership-aware default.** Discover whether this is a multi-driver cluster from Akka cluster state (already available: `OpcUaPublishActor.cs:81` does `Akka.Cluster.Cluster.Get(Context.System)`). Change the gate default per cluster size:
|
||||
- **Single-driver cluster** (≤1 member with the `driver` role): keep default-allow (preserves single-node deploys + boot window).
|
||||
- **Multi-driver cluster** (≥2 `driver` members): **default-deny** the write/ack gates until at least one redundancy snapshot has been received (`_localRole is not null`). Native-alarm *condition writes* stay ungated (the secondary keeps its address space warm — that's `:953-954`, correct); only the *alerts publish* + *device write* + *device ack* gate deny-on-unknown.
|
||||
|
||||
Implementation: `DriverHostActor` subscribes to `ClusterEvent.IMemberEvent` (or reads `_cluster.State.Members` on demand) and tracks `driverMemberCount`. The gate becomes:
|
||||
```csharp
|
||||
bool servicedByThisNode = _localRole switch {
|
||||
RedundancyRole.Secondary or RedundancyRole.Detached => false,
|
||||
RedundancyRole.Primary => true,
|
||||
null => DriverMemberCount() <= 1, // unknown: allow only on a single-node cluster
|
||||
};
|
||||
```
|
||||
Add a `_hasReceivedSnapshot` flag (distinct from role, to handle the "snapshot arrived but didn't mention us" case → that's the S5 mismatch; log-once per S5 and deny on multi-node).
|
||||
|
||||
**Alternatives considered:** (a) Always default-deny — breaks single-node deploys (no snapshot ever arrives to enable writes). Rejected. (b) A fixed boot grace timer that flips to deny after N seconds — fragile, and a slow snapshot on a healthy multi-node cluster would wrongly deny. The membership-count approach is deterministic and uses signal already in hand. (c) Require a lease/fencing token — larger architectural change (the report notes "no lease or fencing token"); out of scope, membership-gate is the pragmatic fix.
|
||||
|
||||
**Implementation steps.**
|
||||
1. `DriverHostActor.cs` — add `private readonly Akka.Cluster.Cluster _cluster = Akka.Cluster.Cluster.Get(Context.System);` (mirror `OpcUaPublishActor`); add `DriverMemberCount()` counting `Up` members carrying the `driver` role from `_cluster.State.Members`.
|
||||
2. Introduce a single private `bool ShouldServiceAsPrimary()` helper encoding the switch above; call it from all three gate sites (`:960`, `:1022`, `:1078`) so the policy lives in one place.
|
||||
3. On a snapshot that omits this node while `DriverMemberCount() > 1`, log-once `Warning` (this is the S5 hook — the two findings share the diagnostic).
|
||||
4. Document the semantics in `docs/Redundancy.md`.
|
||||
|
||||
**Tests.**
|
||||
- **Unit (TestKit):** with `_localRole == null` and a stubbed single-driver membership → write is serviced; with `_localRole == null` and a two-driver membership → write is rejected ("not primary / role unknown on multi-node"); after a Primary snapshot → serviced; after a Secondary snapshot → rejected.
|
||||
- **Integration:** on the 2-node harness, boot the secondary and immediately issue a write before the first snapshot; assert it is rejected (not silently applied to the field device). This pairs with the S1 failover test.
|
||||
|
||||
**Effort:** S/M. **Risk/blast-radius:** Medium — changes write-admission policy on multi-node clusters; a bug could deny legitimate primary writes during a snapshot gap. Mitigate: the count check is conservative (only denies when a real peer exists) and the single-node path is untouched.
|
||||
|
||||
---
|
||||
|
||||
## S2 — HIGH — LDAP authentication block-bridges session activation with a non-cancellable, non-configurable timeout
|
||||
|
||||
**Restatement:** The SDK invokes `ImpersonateUser` synchronously; `OpcUaApplicationHost.HandleImpersonation` block-bridges the authenticator with `CancellationToken.None`. The authenticator adds no timeout; the shared library's connect timeout is a default that `LdapOptions.ToLibraryOptions()` neither projects nor exposes. Each auth opens a fresh TCP connect + service bind + search + user bind. A DC outage stalls SDK threads ~10-20s each.
|
||||
|
||||
**Verification — Confirmed.**
|
||||
- `OpcUaApplicationHost.cs:271-273` — `authenticator.AuthenticateUserNameAsync(…, CancellationToken.None).GetAwaiter().GetResult();` on the SDK impersonation callback thread.
|
||||
- `LdapOpcUaUserAuthenticator.cs:30-48` — awaits `ldap.AuthenticateAsync(username, password, ct)` with no timeout wrapper; the `ct` it receives from `HandleImpersonation` is `None`.
|
||||
- `LdapOptions.cs:94-107` — `ToLibraryOptions()` projects Server/Port/Transport/… but **no timeout field**; `LdapOptions` has no `TimeoutMs`/`ConnectTimeout` property at all.
|
||||
- No connection pooling: each `AuthenticateAsync` is a full connect+bind+search+bind cycle (shared-lib behavior, per report).
|
||||
|
||||
**Root cause:** The timeout responsibility was pushed to the authenticator (per `HandleImpersonation`'s doc comment) but no layer actually enforces one; the shared library's default connect timeout isn't surfaced or bounded per-call, and the call is synchronous under the SDK thread.
|
||||
|
||||
**Proposed design — hard timeout at the OtOpcUa boundary + surfaced option.** The clean fix lives in `LdapOpcUaUserAuthenticator` (OtOpcUa-owned; the shared library's sync-wrapped `AuthenticateAsync` can't be easily made cooperative). Add a `Task.WhenAny(auth, Task.Delay(timeout))` wrapper that **fails closed** (deny) on timeout, and surface `Security:Ldap:TimeoutMs` on `LdapOptions`. Consider a short **negative cache** (deny cache, e.g. 5-30s TTL keyed by username) to shed load during a sustained outage — during a DC-down storm, repeated activations for the same user return the cached deny instantly instead of each stalling for the full timeout.
|
||||
|
||||
Note: `Task.WhenAny` + `Task.Delay` doesn't *cancel* the underlying blocking LDAP call (it observes `CancellationToken.None` only at entry), so the orphaned auth task runs to completion in the background — but the SDK thread is released at `timeout`, which is the goal. Cap concurrent in-flight auths (a `SemaphoreSlim`) so an outage can't accumulate unbounded orphaned tasks.
|
||||
|
||||
**Alternatives considered:** (a) Make the shared library cooperatively cancellable — cross-repo change to `ZB.MOM.WW.Auth`, larger blast radius, and the report frames the fix as OtOpcUa-side. Track as a shared-lib follow-up but don't block on it. (b) Connection pooling in the library — desirable but out of scope for this repo; note it. The boundary timeout + negative cache is the contained, correct-fail-closed fix.
|
||||
|
||||
**Implementation steps.**
|
||||
1. `LdapOptions.cs` — add `public int TimeoutMs { get; set; } = 10000;` (and optionally a `NegativeCacheSeconds`). Do **not** project into `ToLibraryOptions()` (it's an OtOpcUa-boundary concern) unless the library later exposes a matching field.
|
||||
2. `LdapOpcUaUserAuthenticator.cs` — inject `IOptions<LdapOptions>` (or the timeout value); wrap the `ldap.AuthenticateAsync` call in a `Task.WhenAny(authTask, Task.Delay(TimeoutMs, ct))`; on timeout return `OpcUaUserAuthResult.Deny("Authentication timed out")` and log `Warning`. Bound concurrency with a `SemaphoreSlim(maxConcurrent)`. Add the optional negative cache (a `MemoryCache` or a small time-bucketed dictionary).
|
||||
3. Optionally pass a real (timeout-derived) `CancellationToken` from `HandleImpersonation` instead of `None` so the entry-check short-circuits — but the authoritative bound is the `WhenAny`.
|
||||
4. Add a validator note (C2): `TimeoutMs > 0`.
|
||||
|
||||
**Tests.**
|
||||
- **Unit:** authenticator with a stub `ILdapAuthService` that delays beyond `TimeoutMs` → returns Deny within ~`TimeoutMs` (not the full delay); a fast success still succeeds; the negative cache returns a prior deny without re-hitting the stub within TTL; the semaphore bounds concurrency. The existing LDAP fail-closed tests (deny-on-error, opaque messages, zero-role fallback) must stay green.
|
||||
- **Live/manual:** point `Security:Ldap:Server` at an unreachable host and drive a Client.CLI `connect` with a UserName token; confirm the activation fails fast (~TimeoutMs) rather than hanging ~10s, and that a burst of activations doesn't serialize.
|
||||
|
||||
**Effort:** M. **Risk/blast-radius:** Low-Medium — fail-closed on timeout preserves the existing deny-on-error posture; the risk is a too-short default timeout denying a legitimately-slow DC (default 10s matches the library default, so behavior is unchanged unless configured down).
|
||||
|
||||
---
|
||||
|
||||
## S3 — HIGH — HistoryRead block-bridges the gateway per node, sequentially, on SDK request threads
|
||||
|
||||
**Restatement:** All four HistoryRead arms call `.GetAwaiter().GetResult()` per node handle with `CancellationToken.None`, bounded only by the gateway client's 30s `CallTimeout`. A request naming N historized nodes against a slow historian holds one SDK request thread up to N × 30s; with `MaxRequestThreadCount=100`, a few misbehaving history clients can exhaust the request pool and degrade all OPC UA services.
|
||||
|
||||
**Verification — Confirmed.**
|
||||
- `OtOpcUaNodeManager.cs:1795-1807` (`HistoryReadRawModified`), `:1823-1847` (`HistoryReadProcessed`), `:1858-1868` (`HistoryReadAtTime`), `:1886-1934` (`HistoryReadEvents`) — each iterates `nodesToProcess` in a **`foreach` (sequential)**.
|
||||
- Block-bridge sites: `:2059` (`ServeNode`: `read(source, tagname).GetAwaiter().GetResult()`), `:2170-2171` and `:2202-2203` (Raw + tie-cluster overfetch), `:1902-1911` (Events) — all `CancellationToken.None`.
|
||||
- `OpcUaApplicationHost.cs:329-331` — `MinRequestThreadCount = 5, MaxRequestThreadCount = 100, MaxQueuedRequestCount = 200`.
|
||||
- Comment at `:1777-1780` correctly notes the overrides run *outside* the node-manager `Lock`, so this is a thread-pool-exhaustion risk, not a lock-freeze risk.
|
||||
|
||||
**Root cause:** The per-node reads are serialized and unbounded per request; there is no per-request deadline and no server-side cap on concurrent HistoryRead work.
|
||||
|
||||
**Proposed design — bounded per-node parallelism + per-request deadline + concurrency limiter.**
|
||||
1. **Parallelize per-node reads within a batch, bounded.** Replace the sequential `foreach` with a bounded-concurrency fan-out (`Parallel.ForEachAsync` with `MaxDegreeOfParallelism`, or `Task.WhenAll` over a `SemaphoreSlim`), then block-bridge **once** on the aggregate at the arm boundary. This turns N × 30s into ~ceil(N/P) × 30s worst case. Keep the per-node try/catch so one node's failure still surfaces `Bad` for that node only.
|
||||
2. **Per-request deadline token.** Derive a `CancellationTokenSource` from a configurable server-side HistoryRead deadline (e.g. `ServerHistorian:HistoryReadDeadline`, default ~60s) and pass it into the gateway calls instead of `CancellationToken.None`, so a single request can't hold a thread indefinitely regardless of per-node `CallTimeout`.
|
||||
3. **Server-side concurrent-HistoryRead limiter.** A process-wide `SemaphoreSlim` (configurable max, e.g. 8-16) gating entry to the arms so a flood of history clients can't consume more than a bounded slice of the request pool; excess requests wait (or fail with `BadTooManyOperations` if the queue is deep).
|
||||
|
||||
**Alternatives considered:** (a) Fully async HistoryRead — the SDK's `HistoryRead` override surface is synchronous (`void` returns filling `results`/`errors`), so the block-bridge is structural; parallelizing *within* the sync boundary is the achievable win. (b) Reject multi-node history requests — breaks legitimate clients. Bounded parallelism + deadline + limiter is the right combination.
|
||||
|
||||
**Implementation steps.**
|
||||
1. `OtOpcUaNodeManager.cs` — refactor each arm's `foreach` into a bounded parallel fan-out. Because the arms fill `results[handle.Index]`/`errors[handle.Index]` by index (thread-safe, disjoint indices), parallel writes are safe. Collect the per-node async tasks, `await` them under one `GetAwaiter().GetResult()` at the arm boundary. Add a `MaxHistoryReadConcurrencyPerBatch` field (set from options, mirror `MaxTieClusterOverfetch` at `OtOpcUaServerHostedService.cs:223`).
|
||||
2. Thread a per-request `CancellationToken` (from a deadline CTS) into `ServeNode`/`ServeRawPaged`/`HistoryReadEvents` in place of `CancellationToken.None`.
|
||||
3. Add a process-wide `SemaphoreSlim` limiter (field on the node manager or a small injected service) around the arm bodies; make the max configurable.
|
||||
4. `ServerHistorianOptions` — add `HistoryReadDeadline` (TimeSpan) + `MaxConcurrentHistoryReads` + `MaxHistoryReadConcurrencyPerBatch`. Wire through `OtOpcUaServerHostedService` (same spot as `MaxTieClusterOverfetch`).
|
||||
|
||||
**Tests.**
|
||||
- **Unit:** with a stub `IHistorianDataSource` that delays each read, a batch of N nodes completes in ~ceil(N/P) × delay (proves parallelism); a per-node throw still yields `Bad` for that node and `Good` for others (regression on the existing per-node isolation); a batch exceeding the deadline cancels and returns `Bad`/`GoodNoData` rather than hanging.
|
||||
- **Integration/live:** the env-gated live HistoryRead suite (needs the gateway) plus a stress case with a slow stub asserting the request pool isn't exhausted (measure that concurrent non-history services stay responsive while a slow history batch runs).
|
||||
|
||||
**Effort:** M. **Risk/blast-radius:** Medium — parallelizing gateway calls increases peak load on the historian gateway; the concurrency caps bound it. The index-disjoint result writes are safe, but verify no shared mutable state in `ServeRawPaged`'s paging/continuation-point synthesis is touched concurrently (it operates per-handle, but audit the continuation-point cache).
|
||||
|
||||
---
|
||||
|
||||
## S5 — MEDIUM — Redundancy NodeId identity string-matched across three sources, silent on mismatch
|
||||
|
||||
**Restatement:** `OpcUaPublishActor` matches `n.NodeId == _localNode.Value`; the snapshot derives `host:port` from the gossiped `Member.Address` while the local side derives it from configured `PublicHostname:Port`. A divergence (DNS vs IP, container advertised name) silently computes ServiceLevel 0 / keeps a stale role — the historical "silently inert delivery" bug shape, with no distinguishing log.
|
||||
|
||||
**Verification — Confirmed.**
|
||||
- `OpcUaPublishActor.cs:512` — `var entry = _lastSnapshot.Nodes.FirstOrDefault(n => n.NodeId == _localNode.Value);`
|
||||
- `:516-520` — `if (entry is null || entry.Role == Detached) { ServiceLevelChanged(0); return; }` — a **missing entry** is treated identically to legitimate Detached, silently → 0. No log distinguishes the two.
|
||||
- `RedundancyStateActor.cs:123-134` — snapshot `NodeId` = `ToNodeId(member.Address)` from the **gossiped** `Member.Address`.
|
||||
- `:141-146` — `ToNodeId` builds canonical `host:port` "the SAME format `ClusterRoleInfo.LocalNode` uses" — but the local side's host comes from configured `PublicHostname`, the snapshot side's from the gossip. They agree only if `PublicHostname` == the gossiped address host.
|
||||
|
||||
**Root cause:** Two independently-derived string identities are compared for equality with no assertion or diagnostic that they actually match. This is the same class as the historical redundancy-state-delivery bug (`project_redundancy_state_delivery.md`).
|
||||
|
||||
**Proposed design — log-once on absence + startup self-check.** This is diagnostic, not behavioral (the S4 fix consumes the same signal).
|
||||
1. In `OpcUaPublishActor.RecomputeServiceLevel` (and the `DriverHostActor.OnRedundancyStateChanged`, `ScriptedAlarmHostActor` equivalents), when the received snapshot has ≥1 node but **none** equals `_localNode`, log **once** at `Warning`: `local node {LocalNodeId} absent from redundancy snapshot [{snapshotNodeIds}] — check PublicHostname vs gossiped Address`. Log-once (a bool latch) so it doesn't spam every heartbeat.
|
||||
2. **Startup self-check:** in the cluster bootstrap or `OpcUaPublishActor.PreStart`, compare the locally-derived `ClusterRoleInfo.LocalNode` against `Cluster.Get(system).SelfMember.Address` projected through `ToNodeId`; if they differ, log `Error` (misconfiguration) — fail-fast-ish, this is a deploy config error.
|
||||
3. Reset the log-once latch when a snapshot *does* contain the local node (so a transient bootstrap gap that self-heals doesn't leave a stale warning impression).
|
||||
|
||||
**Implementation steps.** Touch `OpcUaPublishActor.cs` (RecomputeServiceLevel + PreStart self-check), `DriverHostActor.OnRedundancyStateChanged` (`:1109`), and factor `ToNodeId(SelfMember.Address)` comparison into a shared helper in `ClusterRoleInfo` so the "canonical local node id" derivation is single-sourced (removes the third independent derivation).
|
||||
|
||||
**Tests.** Unit: snapshot with nodes but not the local node → the log-once fires exactly once and ServiceLevel is 0 (distinguishable in logs from Detached); a subsequent snapshot including the local node resets the latch. Startup self-check: mismatched `PublicHostname` vs self-address → Error logged.
|
||||
|
||||
**Effort:** S. **Risk/blast-radius:** Low (diagnostic + one shared helper).
|
||||
|
||||
---
|
||||
|
||||
## S6 — MEDIUM — Crashed-and-restarted `DriverInstanceActor` loses desired-subscription state; persistent thrower hot-loops
|
||||
|
||||
**Restatement:** No Runtime actor overrides `SupervisorStrategy`, so a driver-child exception triggers Akka's default one-for-one **unlimited, non-backoff** restart. Restart re-runs `PreStart` but the desired-subscription set arrives post-spawn via `SetDesiredSubscriptions` stored in actor state; restart wipes it and `DriverHostActor` has no restart-detection to re-send (restarts don't fire `Terminated`). A persistent thrower hot-loops with no backoff.
|
||||
|
||||
**Verification — Confirmed.**
|
||||
- Grep for `SupervisorStrategy|BackoffSupervisor` in `src/Server/…/Runtime/` → **zero** hits (confirmed no override → default `OneForOneStrategy` with unlimited restarts, `Directive.Restart`, no backoff).
|
||||
- `DriverInstanceActor.cs:84` — `SetDesiredSubscriptions` is a message record; `:181-194` — the desired set is stored in actor fields (`_desiredSubscriptions`); `:299 PreStart` re-inits the driver from Props but does **not** request the subscription set; `:861 StoreDesiredSubscriptions` stores it. A restart clears the fields and nothing re-sends.
|
||||
- No `PostRestart` override that re-requests subscriptions.
|
||||
|
||||
**Root cause:** Message-delivered actor state (desired subscriptions) is not reconstructible from `Props` after a restart, and the parent has no restart-detection to re-push it; plus unlimited non-backoff restart on a persistent fault.
|
||||
|
||||
**Proposed design — BackoffSupervisor + child pulls state in PreStart.**
|
||||
1. Wrap each `DriverInstanceActor` in a **`BackoffSupervisor`** (`Akka.Pattern.BackoffSupervisor` / `Backoff.OnFailure`) with exponential backoff (e.g. min 1s, max 30s, jitter) and a `SupervisorStrategy` on the child that restarts on transient faults. This kills the hot-loop.
|
||||
2. Have the child **request its desired-subscription set from the parent in `PreStart`/`PostRestart`** — add a `RequestDesiredSubscriptions(driverInstanceId)` message the child sends to `DriverHostActor` on (re)start; the parent replies with the current desired set (which it already knows — it pushed it originally). This makes the subscription state reconstructible after any restart. Alternatively, have the parent detect the backoff-restart and re-push; the child-pull is cleaner (self-healing, no parent restart-detection needed).
|
||||
|
||||
**Alternatives considered:** (a) Move desired subscriptions into `Props` — they change per-deploy independently of the actor spec, so this couples deploy churn to actor respawn; rejected. (b) Akka.Persistence for the child — overkill; the parent is the source of truth and can re-supply. (c) Parent watches for restart via a custom decider — restarts don't fire `Terminated`, so this needs a bespoke signal; child-pull avoids it.
|
||||
|
||||
**Implementation steps.**
|
||||
1. `DriverHostActor.cs` — spawn each driver child under a `BackoffSupervisor` Props (or apply `Backoff.OnFailure`) with exp backoff; keep the `(driver, FullName) ⇄ NodeId` maps keyed the same. Handle a new `RequestDesiredSubscriptions` message by replying the current desired set for that driver id.
|
||||
2. `DriverInstanceActor.cs` — in `PreStart` (covers first start + restart, since Akka calls `PreStart` after `PostRestart`), send `RequestDesiredSubscriptions` to the parent and adopt the reply via the existing `StoreDesiredSubscriptions` path. Re-apply on the `Connected` entry as it already does (`:187`).
|
||||
3. Ensure the backoff wrapper doesn't break the health-publish + routing (the parent addresses the *supervisor* ref; verify `_children` maps hold the right ref for `Ask`/`Tell` — with BackoffSupervisor you Tell the supervisor which forwards).
|
||||
|
||||
**Tests.**
|
||||
- **Unit (TestKit) — the U6 gap:** force a driver child to throw; assert (a) it restarts with backoff (not immediately/unbounded), (b) after restart it re-requests and re-adopts the desired subscriptions, (c) a persistent thrower backs off (increasing delay) rather than hot-looping. This is the "no actor supervision/restart test anywhere" gap.
|
||||
- **Live:** on the docker rig, kill a driver's backing connection to induce faults and confirm the driver recovers subscriptions after a transient throw without redeploy.
|
||||
|
||||
**Effort:** M. **Risk/blast-radius:** Medium — changes the driver-child supervision topology; verify routing/`Ask` still resolves through the supervisor and that the backoff doesn't delay legitimate fast recovery (tune min backoff low).
|
||||
|
||||
---
|
||||
|
||||
## S7 — MEDIUM — `DriverInstanceActor.PostStop` blocks shutdown on driver shutdown
|
||||
|
||||
**Restatement:** `_driver.ShutdownAsync(CancellationToken.None).GetAwaiter().GetResult()` in `PostStop` is the only synchronous block in the Runtime actors; a hung protocol stack stalls stop/re-deploy of the whole child set.
|
||||
|
||||
**Verification — Confirmed.** `DriverInstanceActor.cs:950` — `try { _driver.ShutdownAsync(CancellationToken.None).GetAwaiter().GetResult(); } catch (…) { log }`. `CancellationToken.None` → unbounded.
|
||||
|
||||
**Root cause:** Unbounded synchronous wait on driver shutdown inside the actor stop path.
|
||||
|
||||
**Proposed design — bound the wait, log-and-abandon on expiry.** Wrap with a timeout: `_driver.ShutdownAsync(cts.Token).WaitAsync(TimeSpan.FromSeconds(N))` where the CTS is created with the same timeout, so both the token and the outer `WaitAsync` cap it. On `TimeoutException`, log `Warning` and abandon (the actor stops regardless; the orphaned shutdown task runs to completion in the background). A configurable `DriverShutdownTimeout` (default ~5s, matching the write ladder's 5s backend bound) is reasonable.
|
||||
|
||||
**Implementation steps.** `DriverInstanceActor.cs:950` — replace with a bounded wait; pass a real cancellation token (from a timeout CTS) into `ShutdownAsync` so cooperative drivers cancel promptly, and `WaitAsync(timeout)` as the hard bound for uncooperative ones. Optionally surface the timeout in options.
|
||||
|
||||
**Tests.** Unit: a stub driver whose `ShutdownAsync` never completes → `PostStop` returns within ~timeout and logs; a fast driver still shuts down cleanly. This complements S6's supervision tests.
|
||||
|
||||
**Effort:** S. **Risk/blast-radius:** Low.
|
||||
|
||||
---
|
||||
|
||||
## S8 — MEDIUM — Inbound Part 9 commands / native acks are at-most-once; client sees Good even if lost
|
||||
|
||||
**Restatement:** The alarm-command router (`Publish`) and native-ack router (`Tell`) are fire-and-forget, catch-log-drop; the node manager already returned `Good` and applied local condition state. A mediator hiccup or missing `DriverHostActor` silently strands upstream state. Deliberate (non-blocking under `Lock`) but invisible.
|
||||
|
||||
**Verification — Confirmed.**
|
||||
- `OtOpcUaServerHostedService.cs:139-155` (alarm-command router) and `:167-194` (native-ack router) — both `try { … Tell/Publish … } catch (Exception ex) { _logger.LogWarning(…) }`, no counter/metric. The native-ack `else` branch (no `DriverHostActor`) also only `LogWarning`s (`:181`).
|
||||
- `OtOpcUaNodeManager.cs:769-772` (referenced) — the Part 9 condition method returns `Good` before/independent of routing.
|
||||
|
||||
**Root cause:** Best-effort routing with only a Warning log — no operator-visible signal (metric/counter) and no reconciliation.
|
||||
|
||||
**Proposed design — add drop counters/metrics + optional reconciliation.**
|
||||
1. Add OpenTelemetry counters (the codebase uses `OtOpcUaTelemetry` — e.g. `DriverInstanceLifecycle` at `DriverInstanceActor.cs:952`) for `alarm_command_route_dropped` and `native_ack_route_dropped`, tagged by operation/reason (`mediator_fault`, `no_driver_host`). This makes the silent drop observable in metrics/alerting.
|
||||
2. (Optional, larger) an engine-side reconciliation sweep: periodically the scripted-alarm engine / driver re-asserts authoritative condition state so a dropped command self-heals on the next sweep. Defer unless operationally needed; the metric is the priority.
|
||||
|
||||
**Implementation steps.** `OtOpcUaServerHostedService.cs` — increment the counters in both catch blocks and the native-ack `else`. Add the counter definitions to `OtOpcUaTelemetry`. Surface in the existing metrics dashboard/health if present.
|
||||
|
||||
**Tests.** Unit: inject a throwing mediator accessor / a registry that returns no `DriverHostActor` → the counter increments and no exception escapes into the SDK path. (These routers are wired in `StartAsync`; testable by extracting the router lambdas or via a small harness.)
|
||||
|
||||
**Effort:** S. **Risk/blast-radius:** Low (additive observability).
|
||||
|
||||
---
|
||||
|
||||
## S9 — MEDIUM — Server certificate has no renewal or expiry monitoring
|
||||
|
||||
**Restatement:** The app cert is auto-created self-signed with SDK defaults (2048-bit, 12-month lifetime), checked only at boot. Nothing monitors expiry; ~12 months after first deploy, Sign/SignAndEncrypt endpoints and UserName-token encryption fail with no warning.
|
||||
|
||||
**Verification — Confirmed.** `OpcUaApplicationHost.cs:305-317` — `EnsureApplicationCertificateAsync` calls `CheckApplicationInstanceCertificatesAsync(false, null, ct)` with `minimumKeySize/lifetimeInMonths: 0` → SDK defaults (comment at `:308` states 2048-bit, 12-month). Boot-only; no periodic check, no expiry metric.
|
||||
|
||||
**Root cause:** No lifecycle monitoring for the server certificate (the AdminUI cert-actions cover *client* certs; the *server* cert is the gap).
|
||||
|
||||
**Proposed design — startup + periodic expiry check with a metric/health-check + runbook.**
|
||||
1. At startup, after `EnsureApplicationCertificateAsync`, read the resolved cert's `NotAfter` and log `Warning` if within a threshold (e.g. 30 days) and `Info` with the expiry date otherwise.
|
||||
2. Register a periodic check (a lightweight hosted service or a timer in the existing health pipeline) that emits a health-check/metric (`server_cert_days_to_expiry`) consumed by `MapOtOpcUaHealth` (same surface S10 targets). Degrade `/health` when under threshold.
|
||||
3. Document a rotation runbook in `docs/security.md` (delete-and-reissue, or a longer `lifetimeInMonths` at issue). Optionally increase the issued lifetime (pass a non-zero `lifetimeInMonths` to `CheckApplicationInstanceCertificatesAsync`) so the reissue cadence is multi-year.
|
||||
|
||||
**Implementation steps.** `OpcUaApplicationHost.cs` — capture the cert `NotAfter` after the check and expose it (property or event); add the threshold log. Add a health contributor + metric. `docs/security.md` — runbook section.
|
||||
|
||||
**Tests.** Unit: a near-expiry cert → the health/metric reports degraded; a fresh cert → healthy. (Injectable via a test cert with a short `NotAfter`.)
|
||||
|
||||
**Effort:** S/M. **Risk/blast-radius:** Low.
|
||||
|
||||
---
|
||||
|
||||
## S10 — LOW — SDK start failure swallowed; node runs with Null sinks, no health surface
|
||||
|
||||
**Restatement:** `StartAsync` catches, logs, and returns; the node then no-ops all OPC UA work with nothing surfacing the condition through `/health` or ServiceLevel.
|
||||
|
||||
**Verification — Confirmed.** `OtOpcUaServerHostedService.cs:110-129` — `catch { LogError; return; }` (deliberate: AdminUI stays up). The `NodeManager is null` path (`:124-128`) also just warns and returns. No health flag set.
|
||||
|
||||
**Root cause:** The availability trade-off (don't crash the whole binary) leaves no degraded-state signal.
|
||||
|
||||
**Proposed design — set a health flag consumed by `MapOtOpcUaHealth`.** Introduce an `IOpcUaServerHealth` singleton (a simple thread-safe flag holder) that `OtOpcUaServerHostedService` sets to `Faulted`/`Degraded` on the start-failure and `NodeManager is null` paths, and `Healthy` on success. `MapOtOpcUaHealth` reads it so fleet status shows the degraded node. Pairs naturally with S9's cert health contributor.
|
||||
|
||||
**Implementation steps.** New `IOpcUaServerHealth` + impl in `Host`/`OpcUaServer`; inject into `OtOpcUaServerHostedService`; set on each exit path in `StartAsync` and clear on success/`StopAsync`; wire into the health-check mapping.
|
||||
|
||||
**Tests.** Unit: forcing a start failure sets the health flag to degraded; success sets healthy. Health endpoint reflects it.
|
||||
|
||||
**Effort:** S. **Risk/blast-radius:** Low.
|
||||
|
||||
---
|
||||
|
||||
## S11 — LOW — `AuditWriterActor` unbounded buffer, drops batches on DB outage (moot — no producers)
|
||||
|
||||
**Restatement:** The audit buffer is unbounded between flushes and drops whole batches on a DB outage. Best-effort by contract; currently moot because the pipeline has no producers (U3).
|
||||
|
||||
**Verification — Confirmed** (`ControlPlane/Audit/AuditWriterActor.cs` per report; and U3 confirms zero producers — `AuditOutcomeMapper.cs:12-18` states "no live structured AuditEvent emit sites").
|
||||
|
||||
**Root cause:** Best-effort design; no backpressure; and the whole pipeline is dormant.
|
||||
|
||||
**Proposed design.** **Fold into U3's decision.** If the audit pipeline is wired to producers (U3), then bound the buffer (drop-oldest with a metered counter, like `SqliteStoreAndForwardSink`'s `_evictedCount`) and consider a store-and-forward durable queue for compliance-grade audit. If the pipeline is deleted (U3 alternative), this is moot. **Do not fix in isolation** — resolve U3 first.
|
||||
|
||||
**Effort:** S (bounding) — gated on U3. **Risk:** Low.
|
||||
|
||||
---
|
||||
|
||||
## P2 — MEDIUM — Per-value global-lock write, one actor message per value, no batching
|
||||
|
||||
**Restatement:** Each published value takes the global node-manager `Lock` once and flows through one actor message per value, contending with SDK read/subscription/publish threads. At high tag counts × fast polls this serializes everything through one lock and allocates one record per hop.
|
||||
|
||||
**Verification — Confirmed.**
|
||||
- `DriverHostActor.cs:561-589` (`ForwardToMux`) — per value: dictionary lookup + up to two `Tell`s.
|
||||
- `OtOpcUaNodeManager.cs:266-281` (`WriteValue`) — takes `lock (Lock)` per value.
|
||||
- `DependencyMuxActor.cs:95-108` — lean (early-drop, set fan-out), confirmed not the bottleneck.
|
||||
|
||||
**Root cause:** No batching of the sink write; the lock is acquired per value rather than per driver-publish cycle.
|
||||
|
||||
**Proposed design — batched `WriteValues(IReadOnlyList<…>)` sink call.** Add a batch method to `IOpcUaAddressSpaceSink` that takes one `Lock` hold per driver publish cycle and applies all values in the batch. The driver child already receives values in poll-cycle batches upstream; carry the batch shape through `ForwardToMux` → `OpcUaPublishActor` → `sink.WriteValues`. **Respect the U2 forwarding trap:** the new `WriteValues` must be added to `DeferredAddressSpaceSink`, `NullAddressSpaceSink`, and `SdkAddressSpaceSink`, and exhaustively test-guarded — an un-forwarded batch method ships inert (F10b/PR#423 precedent).
|
||||
|
||||
**Alternatives considered:** (a) Finer-grained locking in the node manager — the SDK's `CustomNodeManager2.Lock` is the contract boundary for address-space consistency; sub-locking risks correctness. Batching under the existing lock is safer. (b) Lock-free value store — too invasive. Batching is the contained seam-level win the report recommends.
|
||||
|
||||
**Implementation steps.** Add `WriteValues` to the sink interface + all impls + the deferred wrapper; thread batch shape through `ForwardToMux`/`OpcUaPublishActor`; keep single `WriteValue` for the individual paths (alarm/vtag) or route them through the batch of one.
|
||||
|
||||
**Tests.** Unit: batched write takes the lock once for N values (assert via a recording sink counting lock acquisitions or write calls); U2 forwarding test must include `WriteValues`. Perf: micro-benchmark N values single vs batched. Live: high-tag-count rig to confirm reduced contention (optional).
|
||||
|
||||
**Effort:** M. **Risk/blast-radius:** Medium (hot path; U2 trap). Do after/with P1 since both touch the sink seam.
|
||||
|
||||
---
|
||||
|
||||
## P3 — MEDIUM — HistoryRead-Events unbounded, no paging
|
||||
|
||||
**Restatement:** `EventMaxEvents` maps `NumValuesPerNode == 0` to `int.MaxValue`, and the Events arm never issues continuation points. A wide window over a busy alarm source materializes the entire result in memory on both gateway and server.
|
||||
|
||||
**Verification — Confirmed.** `OtOpcUaNodeManager.cs:1944-1954` (`EventMaxEvents` → `int.MaxValue` on 0), `:1919-1921` ("never issue continuation points — full window in one shot").
|
||||
|
||||
**Root cause:** Spec-conformant "no limit" translated to truly unbounded with no server-side backstop.
|
||||
|
||||
**Proposed design — server-side max mirroring `MaxTieClusterOverfetch`.** Impose a configurable server-side max event count for the Events arm (e.g. `ServerHistorian:MaxHistoryReadEvents`, default 65536 to mirror `MaxTieClusterOverfetch`'s philosophy). When the result would exceed it, either **fail loudly** (`BadHistoryOperationInvalid` / a status telling the client to narrow the window) or **page** (synthesize continuation points as the Raw arm already does — larger effort). Start with the bounded-fail backstop (matches the Raw tie-cluster "loud-fail" design at `:2202`), add paging later if clients need it.
|
||||
|
||||
**Implementation steps.** `OtOpcUaNodeManager.cs` — cap `EventMaxEvents`'s unbounded case at the configured max; on overflow return a loud status. Add the option to `ServerHistorianOptions`, wire via `OtOpcUaServerHostedService` (same spot as `MaxTieClusterOverfetch`).
|
||||
|
||||
**Tests.** Unit: `EventMaxEvents(0)` returns the configured cap (not `int.MaxValue`); a result at the cap returns the loud status. Update the existing `EventMaxEvents` unit tests.
|
||||
|
||||
**Effort:** S. **Risk/blast-radius:** Low.
|
||||
|
||||
---
|
||||
|
||||
## P4 — LOW — Every deploy re-runs the four Materialise passes over the full composition
|
||||
|
||||
**Verification — Confirmed.** `OpcUaPublishActor.cs:338-354` runs all four passes each deploy; acceptable because `EnsureFolder`/`EnsureVariable` early-return on existing ids, but each pass takes/releases `Lock` per node.
|
||||
|
||||
**Proposed design.** **Fold into P2's batching** — a batched ensure (one lock hold per pass) removes the per-node lock churn. No standalone work; note it when implementing P2. The positive designs (Raw tie-cluster overfetch bound, `HandleDiscoveredNodes` unchanged-plan short-circuit at `DriverHostActor.cs:658-673`) are worth preserving.
|
||||
|
||||
**Effort:** S (subsumed by P2). **Risk:** Low.
|
||||
|
||||
---
|
||||
|
||||
## C1 — MEDIUM — `ServerHistorian` bound imperatively in five places, no `IOptions`
|
||||
|
||||
**Restatement:** `ServerHistorian` is bound in `Program.cs` (own bind + `Validate()`), `AddServerHistorian`, `AddAlarmHistorian`, `AddHistorianProvisioning`, and `OtOpcUaServerHostedService`'s ctor — five `Get<>` sites that can log warnings twice and drift.
|
||||
|
||||
**Verification — Confirmed.** `OtOpcUaServerHostedService.cs:88-90` binds directly (`configuration.GetSection(ServerHistorianOptions.SectionName).Get<ServerHistorianOptions>()`); `Runtime/ServiceCollectionExtensions.cs:86, 132, 168` each re-`Get<>` the section (confirmed the section is read in `AddServerHistorian`/`AddAlarmHistorian`/`AddHistorianProvisioning`); `Program.cs` binds + `Validate()`s. Five sites.
|
||||
|
||||
**Root cause:** No single validated-options registration; each consumer binds independently.
|
||||
|
||||
**Proposed design — one `AddValidatedOptions<ServerHistorianOptions>` + inject `IOptions<>`.** Mirror the `OpcUa`/`Ldap` pattern already in the tree (`Program.cs:102, 254-255`). Register once with `ValidateOnStart` (folds into C2's validator promotion), then inject `IOptions<ServerHistorianOptions>` into `OtOpcUaServerHostedService` and the three `Add*` extensions. This single-sources the section and runs `Validate()` once.
|
||||
|
||||
**Implementation steps.** Add the registration in `Program.cs`; change the five consumers to take `IOptions<ServerHistorianOptions>` (the `Add*` extensions resolve it from the built provider or accept it as a parameter). Remove the per-site `Get<>` calls.
|
||||
|
||||
**Tests.** Unit: options bound once, validated once (assert single warning emission). Existing consumers still function.
|
||||
|
||||
**Effort:** S/M. **Risk/blast-radius:** Low-Medium (touches five wiring sites; verify the `Add*` extensions run after the options registration).
|
||||
|
||||
---
|
||||
|
||||
## C2 — MEDIUM — Two-tier options validation; `DevStubMode` only log-warned in production
|
||||
|
||||
**Restatement:** `OpcUa`/`Security:Ldap` get fail-fast `ValidateOnStart` validators; `ServerHistorian`/`ContinuousHistorization`/`AlarmHistorian` get only advisory `Validate()` warnings; `DevStubMode=true` (accept-any-credentials Administrator) is merely log-warned in production.
|
||||
|
||||
**Verification — Confirmed.** `OtOpcUaLdapAuthService.cs:79-88` — `if (_options.DevStubMode) { … LogWarning("DevStubMode bypass …"); accept }` with no environment guard (grep for `IsDevelopment`/`EnvironmentName` in the file → none). The historian sections use advisory `Validate()` (C1).
|
||||
|
||||
**Root cause:** Inconsistent validation posture; a dangerous dev bypass isn't environment-gated.
|
||||
|
||||
**Proposed design.**
|
||||
1. **Promote the historian sections to the validator pattern** (`AddValidatedOptions` + `IValidateOptions<>` with `ValidateOnStart`), folding in C1's single registration. Convert the current `Validate()` warning lists into validator failures (or keep soft warnings for non-fatal knobs but fail on genuinely-invalid combos, e.g. `Enabled=true` with empty `Endpoint`).
|
||||
2. **Environment-gate `DevStubMode`:** inject `IHostEnvironment`; if `DevStubMode == true && !env.IsDevelopment()`, **fail startup** (throw in a validator) — or at minimum refuse the bypass at runtime (deny + Error log) outside Development. Fail-fast is preferable (a prod deploy with DevStubMode is a critical misconfiguration).
|
||||
|
||||
**Implementation steps.** Add `ServerHistorianOptionsValidator`/`ContinuousHistorizationOptionsValidator`/`AlarmHistorianOptionsValidator` (mirror `OpcUaApplicationHostOptionsValidator`/`LdapOptionsValidator`). Add an `LdapOptionsValidator` (or extend the existing one) rule: `DevStubMode` requires Development. Wire `IHostEnvironment` into the validator.
|
||||
|
||||
**Tests.** Unit: historian validators fail on invalid combos, `ValidateOnStart` surfaces at boot; `DevStubMode=true` outside Development fails startup; inside Development it's allowed (warned). Existing `OpcUa`/`Ldap` validator tests stay green.
|
||||
|
||||
**Effort:** M. **Risk/blast-radius:** Medium — promoting to fail-fast could break a deploy relying on a currently-tolerated invalid config; validate against real appsettings before merge and document the new hard requirements in the migration note.
|
||||
|
||||
---
|
||||
|
||||
## C3 — LOW — Library code logs via static Serilog; node manager uses obsolete `Utils.LogError`
|
||||
|
||||
**Verification — Confirmed.** `Runtime/ServiceCollectionExtensions.cs:90,100,136` log via static `Log`; node manager uses `Utils.LogError` with `#pragma warning disable CS0618` (e.g. `:1928-1930`, and the report cites `:449-451` et al.).
|
||||
|
||||
**Proposed design.** For the Runtime static-logger sites, inject `ILogger` where an instance is available; where it isn't (pure `Add*` extension methods), accept an `ILoggerFactory`/`ILogger` parameter. For the node manager, wire the acknowledged `ITelemetryContext`/`ILogger` seam so the six `CS0618` `Utils.LogError` sites route through a real logger instead of the obsolete static trace. This is hygiene; the static coupling works only because `Program.cs:305` assigns `Log.Logger` (ordering-sensitive).
|
||||
|
||||
**Effort:** M (node-manager logger wiring is the bulk). **Risk:** Low. Lower priority than the Criticals/Highs.
|
||||
|
||||
---
|
||||
|
||||
## C4 — LOW — `IHistorianProvisioning` resolve has no missing-registration warning
|
||||
|
||||
**Verification — Confirmed.** `Runtime/ServiceCollectionExtensions.cs:298` — `var provisioning = resolver.GetService<IHistorianProvisioning>();` with a comment noting the Null default, but **no warning** parallel to the evaluator/recorder-deps warnings (`:211-237`). This is the exact seam that shipped dormant in PR #423. The `dispatched=N, requested=0` tally (`AddressSpaceApplier.cs:274-282`) partially compensates.
|
||||
|
||||
**Proposed design.** Add a startup log line at the resolve site: if `GetService<IHistorianProvisioning>()` returns the `NullHistorianProvisioning` singleton while `ServerHistorian:Enabled == true`, log `Warning` ("provisioning enabled but Null provisioner resolved"). Mirror the sibling missing-registration warnings. This is the "startup log proving the wiring" guard from OVERALL theme #1.
|
||||
|
||||
**Effort:** S. **Risk:** Low.
|
||||
|
||||
---
|
||||
|
||||
## U1 — HIGH (doc drift) — CLAUDE.md "Known Limitation 2" is stale; recorder ref-feed is wired
|
||||
|
||||
**Restatement:** CLAUDE.md says continuous historization "records no values" because the recorder is seeded empty. In code, the applier feeds the delta via `ActorHistorizedTagSubscriptionSink` → `UpdateHistorizedRefs`.
|
||||
|
||||
**Verification — Confirmed stale.**
|
||||
- `Runtime/ServiceCollectionExtensions.cs:272` — recorder still spawned with `historizedRefs: Array.Empty<string>()`.
|
||||
- `:291` — `new ActorHistorizedTagSubscriptionSink(continuousRecorder)` is wired.
|
||||
- `AddressSpaceApplier.cs:213` (`FeedHistorizedRefs`) + `:355-371` (`HistorizedRef`) feed the add/remove delta.
|
||||
- Restart convergence works via in-memory `_lastApplied` (`OpcUaPublishActor.cs:326-336`): the first post-boot rebuild diffs against empty and emits the full historized set as Added.
|
||||
- Confirmed independently by OVERALL cross-cutting theme #5.
|
||||
|
||||
**Proposed design — update the doc + add the convergence test.**
|
||||
1. Update `CLAUDE.md` — rewrite "KNOWN LIMITATION 2" to state the ref-feed is wired; note the load-bearing chain (empty seed + delta feed + in-memory `_lastApplied` → full-set-as-Added on first post-boot deploy). Keep any genuinely-remaining caveat (e.g. the numeric-analog-only v1 limitation) but drop the "records no values" claim. Propagate to `../scadaproj/CLAUDE.md`'s OtOpcUa entry per the repo's cross-repo rule.
|
||||
2. Add an **explicit restart-convergence test:** boot → apply a full plan (composition with historized tags) → assert the recorder's dependency-mux interest is registered for exactly the historized set (the chain is only implicitly covered today). This retires the "load-bearing but untested" risk.
|
||||
|
||||
**Effort:** S. **Risk:** Low (doc + test only). High-value because it removes a false "known broken" claim that could misdirect future work.
|
||||
|
||||
---
|
||||
|
||||
## U2 — MEDIUM — Deferred-sink forwarding correct today but only half test-guarded
|
||||
|
||||
**Restatement:** `DeferredAddressSpaceSink` forwards all 10 members correctly (incl. both surgical methods with capability-sniffing), but tests assert forwarding for only ~5/10 members. The F10b incident proves the failure mode is real.
|
||||
|
||||
**Verification — Confirmed.** `DeferredAddressSpaceSink.cs:15` implements `IOpcUaAddressSpaceSink, ISurgicalAddressSpaceSink`; all members forward (`:26-49` for the 8 base members, `:58-69` for the 2 surgical with `_inner is ISurgicalAddressSpaceSink` fallback). Correct today, but per the report only ~5 have per-member forwarding assertions.
|
||||
|
||||
**Proposed design — reflection-driven exhaustive forwarding test.** Add a test that, via reflection, enumerates **every method** of `IOpcUaAddressSpaceSink` + every optional capability interface the wrapper implements (`ISurgicalAddressSpaceSink`, plus any new ones from P1/P2), invokes each on a `DeferredAddressSpaceSink` wrapping a recording inner sink, and asserts the call **reached** the inner sink. This mechanically guarantees no member ships inert — retiring the whole trap class (OVERALL theme #1/#2). Apply the same treatment to `DeferredServiceLevelPublisher`. **This test becomes the gate for P1's remove methods and P2's `WriteValues`.**
|
||||
|
||||
**Implementation steps.** New test in `tests/Server/…/DeferredAddressSpaceSinkTests.cs` (extend the existing file): reflect over interface methods, build default/dummy args per parameter type, invoke, assert the recording sink saw each. Handle the capability-sniffing members (assert they forward when the inner implements the capability, and no-op/return-false safely when it doesn't).
|
||||
|
||||
**Effort:** S/M. **Risk:** Low. Very high leverage — one test permanently closes the "built-but-never-wired forwarding" class.
|
||||
|
||||
---
|
||||
|
||||
## U3 — MEDIUM — Structured audit pipeline built/tested with zero producers
|
||||
|
||||
**Verification — Confirmed.** `ControlPlane/Audit/AuditOutcomeMapper.cs:12-18` and `Security/Audit/AuditActor.cs:18-24` state there are no live structured `AuditEvent` emit sites (all production audit flows through the bespoke stored-procedure path).
|
||||
|
||||
**Proposed design — decide: wire or delete.** Either (a) wire producers (route the real audit-worthy events — the write-gate audit at `OtOpcUaNodeManager.cs`, alarm ack/shelve, deploy outcomes — through the structured `AuditEvent` pipeline, replacing/augmenting the stored-proc path), or (b) delete the dormant pipeline (`AuditWriterActor`, `AuditOutcomeMapper`, `AuditActor`, tests) to remove decay risk. **Recommendation:** delete unless compliance-grade structured audit is a near-term requirement — the bespoke stored-proc path already serves production, and dormant tested code is the exact "built-but-never-wired" debt the review flags. S11 is subsumed by whichever path is chosen. This is a product decision; surface it to the owner rather than deciding unilaterally.
|
||||
|
||||
**Effort:** M (either path). **Risk:** Low (deletion) / Medium (wiring — new producers touch hot paths).
|
||||
|
||||
---
|
||||
|
||||
## U4 — MEDIUM — `FleetStatusBroadcaster.DriverHostStatusHeartbeat` dead code with latent NodeId-mismatch
|
||||
|
||||
**Verification — Confirmed** (per report: `FleetStatusBroadcaster.cs:38, 110-121` no producer; `:152-159` keys nodes by **host only** vs the `host:port` canon everywhere else).
|
||||
|
||||
**Proposed design — wire the producer or remove; canonicalize the key regardless.** If fleet-status freshness is wanted, wire `DriverHostActor` (which knows its applied revision) to send `DriverHostStatusHeartbeat` and **canonicalize the broadcaster key to `host:port`** (else a wired `host:port` NodeId creates phantom records — re-enacting the historical NodeId bug, same class as S5). If not wanted, delete the heartbeat path. Either way, fix the key to `host:port`. Recommendation: remove unless the fleet-status freshness signal is used by the AdminUI; the host-only key + no producer signals it was never completed.
|
||||
|
||||
**Effort:** S (remove) / M (wire + key fix). **Risk:** Low.
|
||||
|
||||
---
|
||||
|
||||
## U5 — MEDIUM — Native Part 9 conditions support Acknowledge-to-driver only
|
||||
|
||||
**Verification — Confirmed** (`OtOpcUaNodeManager.cs:647-658` — Confirm/AddComment/Shelve on a native condition route to the scripted engine which doesn't own them; `:698-703` — Enable/Disable → `BadNotSupported`). Operators shelving a Galaxy alarm via a Part 9 client get silent upstream no-op.
|
||||
|
||||
**Proposed design — route native condition commands to the driver's `IAlarmSource`, or fail loudly.** Extend the native-ack seam (already present: `NativeAlarmAckRouter` → `DriverHostActor.RouteNativeAlarmAck` → driver `AcknowledgeAsync`) to the other Part 9 operations the driver can support (Confirm/AddComment/Shelve/Unshelve) by adding driver-side methods on `IAlarmSource` (mirroring `AcknowledgeAsync`) and routing them like the ack. For operations no driver backend supports (Enable/Disable upstream), return a **loud** status (`BadNotSupported` is already returned for Enable/Disable — keep, but ensure the *silently-ineffective* Shelve/Confirm cases either route or return a status telling the operator it won't propagate upstream). This is documented H6c scope — scope it as a follow-on feature, not a quick fix.
|
||||
|
||||
**Effort:** M/L (driver-side `IAlarmSource` surface expansion + Galaxy gateway support). **Risk:** Medium (touches the driver alarm contract and Galaxy). Lower priority than the availability/perf cluster.
|
||||
|
||||
---
|
||||
|
||||
## U6 — MEDIUM — Test-coverage gaps concentrate on the fragile seams
|
||||
|
||||
**Restatement:** DPS delivery of redundancy/ServiceLevel state, actor supervision/restart, outbox durability across process restart, hard-kill failover, and the single `OpcUaServer.IntegrationTests` test are all untested; docker-gated protocol tests self-skip to green.
|
||||
|
||||
**Verification — Confirmed** (per report's tests/Server sweep; corroborated by S1/S6 verification — no supervision test, no hard-kill test; and OVERALL theme #4).
|
||||
|
||||
**Proposed design — targeted tests, delivered alongside the fixes above (not as a separate epic).**
|
||||
- **DPS delivery of redundancy/ServiceLevel** — a real over-the-mediator delivery test (not a stub) on the 2-node harness; assert a `RedundancyStateChanged` published by `RedundancyStateActor` is *received* by `OpcUaPublishActor`/`DriverHostActor` and drives ServiceLevel/role. This is the known "unit tests can't catch it" blind spot (`project_redundancy_state_delivery.md`).
|
||||
- **Hard-kill failover** — delivered by **S1**'s integration test.
|
||||
- **Actor supervision/restart** — delivered by **S6**'s TestKit test.
|
||||
- **Outbox durability across process restart** — a test that appends to the FasterLog outbox, simulates a restart (new outbox instance over the same directory), and asserts un-acked entries drain (continuous-historization path).
|
||||
- **`OpcUaServer.IntegrationTests` expansion** — subscription-survival-on-add (from **P1**), a security-mode matrix (None/Sign/SignAndEncrypt over the wire), and a HistoryRead round-trip.
|
||||
- **Make docker skips visible** — align with OVERALL action #6 (fail-on-skip for the integration job); coordinate with report 07.
|
||||
|
||||
**Effort:** M (spread across the fixes). **Risk:** Low (test-only). Track each sub-test with its parent finding so coverage lands with the fix, not after.
|
||||
|
||||
---
|
||||
|
||||
## U7 / U8 / U9 — LOW — Batched hygiene items
|
||||
|
||||
- **U7** — `IHistoryWriter` permanently Null-wired (`Runtime/ServiceCollectionExtensions.cs:56-58`, infra-gated on a nonexistent live-write RPC); **H2 HistoryUpdate** unimplemented (`OtOpcUaNodeManager.cs:1797-1801`, `IsReadModified` rejected); `LdapAuthFailure` lacks a `DirectoryUnavailable` value (directory-down overloaded onto `ServiceAccountBindFailed`). **Verification — Confirmed** (all backlog/tracked). **Design:** leave `IHistoryWriter`/H2 as tracked backlog (infra-gated). Add the `DirectoryUnavailable` enum value and map directory-down to it in the LDAP path (pairs with S2's outage handling — a distinct failure reason improves the operator signal). **Effort:** S (enum) + backlog (rest).
|
||||
- **U8** — `BuildSecurityPolicies` doc claims the empty-profile fallback is "logged and very visible" but logs nothing (`OpcUaApplicationHost.cs:376-418`). Defused in prod by the `MinCount ≥ 1` validator but silent for direct embedders. **Verification — Confirmed** (per report). **Design:** either add the log the comment promises (a `Warning` on empty-profile fallback-to-None) or fix the comment. Adding the log is cheap and correct. **Effort:** S.
|
||||
- **U9** — `EnsureVariable` silently ignores changed historize-intent on an existing node (`OtOpcUaNodeManager.cs:1345-1349`); correct today because the planner routes such deltas elsewhere, but the invariant lives in two comments. **Verification — Confirmed.** **Design:** turn the invariant into an assert/`Debug.Assert` (or a `Warning` log) when `EnsureVariable` is called with a differing historize-intent for an existing id, so a future planner regression surfaces instead of silently no-op'ing. **Effort:** S.
|
||||
|
||||
**Risk (all three):** Low.
|
||||
|
||||
---
|
||||
|
||||
## Suggested execution order (this report's slice of the OVERALL list)
|
||||
|
||||
1. **S1** — activate SBR + hard-kill failover test (OVERALL #1). *Do first — nothing else matters if failover is broken.*
|
||||
2. **U1** — update CLAUDE.md (stale Known Limitation 2) + convergence test (OVERALL #11). *Cheap, removes a misleading claim before others build on it.*
|
||||
3. **U2** — reflection-exhaustive Deferred-sink forwarding test (OVERALL theme #1/#2). *Prerequisite guard for P1/P2 — land it first so P1's remove methods and P2's `WriteValues` can't ship inert.*
|
||||
4. **S4** — primary-gate default-deny on multi-node (OVERALL #7). *Small, closes the dual-primary data-plane window.*
|
||||
5. **P1** — surgical pure-adds (phase 1), scoped removals (phase 2) (OVERALL #8). *Highest operational perf/stability win; gated on U2.*
|
||||
6. **S2 / S3** — async LDAP timeout + channelized/bounded HistoryRead (OVERALL #9).
|
||||
7. **S5, S6, S7, S8** — redundancy-mismatch diagnostics, BackoffSupervisor + subscription re-pull, bounded PostStop, drop metrics.
|
||||
8. **C1 + C2** — single validated `ServerHistorian` options + historian validators + `DevStubMode` env-gate.
|
||||
9. **P2 (+P4), P3, S9, S10** — batched sink writes, Events cap, cert monitoring, start-failure health flag.
|
||||
10. **U3, U4, U5, C3, C4, U7-U9** — dormant-code decisions, native Part 9 expansion, and hygiene, as capacity allows.
|
||||
@@ -0,0 +1,292 @@
|
||||
# Architecture Review 04 — AdminUI: Design + Implementation Plan
|
||||
|
||||
- **Source report:** `archreview/04-adminui.md` (commit `9cad9ed0`)
|
||||
- **Plan author date:** 2026-07-08 · verified against current tree at `9cad9ed0`
|
||||
- **Scope:** AdminUI pages, ScriptAnalysis backend, tag editors, authorization
|
||||
- **Repo constraint that shapes every test recipe:** there is **no bUnit**. Razor `@code`/binding logic and page-level `[Authorize]` gating are only observable by live-verify on the docker-dev rig (AdminUI `http://localhost:9200`). *And* on docker-dev **login is disabled** (`Security:Auth:DisableLogin=true` → `AutoLoginAuthenticationHandler` grants `DevAuthRoles.All` = every role incl. Administrator/Designer/Operator). **Consequence: a Viewer-denied path cannot be observed on the default rig** — negative authz must be proven by (a) a pure policy-registration unit test, (b) a reflection test over page attributes, and (c) a manual real-login pass with a Viewer LDAP bind. This is called out in every authz recipe below.
|
||||
|
||||
## Verification summary
|
||||
|
||||
| ID | Sev | Status | Evidence |
|
||||
|---|---|---|---|
|
||||
| C-1 | High | **Confirmed** | 30+ pages carry fully-qualified bare `@attribute [Microsoft.AspNetCore.Authorization.Authorize]`; only `Deployments`/`Scripts`/`ScriptEdit` use `Roles="Administrator,Designer"`, `RoleGrants` uses `Policy="FleetAdmin"`, `Certificates`/`Alerts`/`DriverStatusPanel` use imperative `"DriverOperator"`/`"FleetAdmin"`. `FallbackPolicy` is only `RequireAuthenticatedUser()`. |
|
||||
| C-2 | Med | **Confirmed** | `ScriptAnalysisEndpoints.cs:17-18` = `Roles="Administrator,Designer"`; `CLAUDE.md:223` + `docs/plans/2026-06-11-adminui-disable-login-design.md:141` say "FleetAdmin". |
|
||||
| C-5 | Med | **Confirmed** | `ScriptEdit.razor` / `Deployments.razor` / `Fleet.razor` / `Hosts.razor` inject `IDbContextFactory` and run EF + `SaveChangesAsync` in `@code`; UNS pages use `IUnsTreeService`. |
|
||||
| S-1 | Med | **Confirmed** | `Fleet.razor:171` + `AlarmsHistorian.razor:90` sync `Dispose() => _timer?.Dispose()`; `DriverTestConnectButton.razor:69` `System.Timers.Timer` with `async (_,_)` `Elapsed` + sync `Dispose`. (`DriverStatusPanel`/`Alerts`/`Hosts` already correct via `System.Threading.Timer`+`DisposeAsync`.) |
|
||||
| S-2 | Med | **Confirmed** | `Fleet.LoadAsync` (119-159) `try/finally` no `catch`; `AlarmsHistorian` (73-78) bare `catch {}` leaving "Loading…". |
|
||||
| S-3 | Med | **Confirmed** | `grep ErrorBoundary` = none; `MainLayout.razor:53` `<ChildContent>@Body</ChildContent>` unwrapped. |
|
||||
| S-4 | Med | **Confirmed** | `GlobalUns.ConfirmDeleteAsync` re-`Load…Async` then passes fresh `.RowVersion`; `EquipmentPage.DeleteTag/VirtualTag/Alarm` (370/415/460) same ("Load … fresh to capture its current RowVersion"). |
|
||||
| S-5 | Med | **Confirmed** | `EquipmentPage.razor:172/217/263` single-click `@onclick="() => DeleteTag(...)"`; `ScriptEdit.DeleteAsync` single-click. `GlobalUns` has a confirm modal; these don't. |
|
||||
| S-6 | Low | **Partial** | `DriverStatusPanel` *does* drain via `DisposeAsync` (287-310), but `ShowOpResult` replaces the chip timer with a sync `Dispose()` and no token guard → an already-queued callback can clear a newer message. Race is real but narrow. |
|
||||
| S-7 | Low | **Confirmed** | `AdminOperationsClient.cs:58-59` `AskAsync<T>` forwards only caller `ct`; typed methods double-guard with `AskTimeout`. |
|
||||
| P-1 | Med | **Confirmed** | `monaco-init.js:205-207` `onDidChangeModelContent` → `invokeMethodAsync("OnValueChanged", editor.getValue())` no debounce; diagnostics ARE debounced (500 ms). |
|
||||
| P-2 | Med | **Confirmed** | `ScriptAnalysisService.Analyze` (60-67) re-`ParseText` + `CSharpCompilation.Create` per call; refs/preamble static (good), no `(text→tree)` memo. |
|
||||
| P-3/P-4/P-5 | Low | **Confirmed** (spot-checked) | Per-circuit polling / `SnapshotAndFlatten` per Deployments render / no `@key` on `Alerts` rows — accepted as bounded. |
|
||||
| P-6 | Low | **Confirmed** | `monaco-init.js:99-117` registers inlay-hints provider that POSTs every change; `InlayHints` endpoint always empty. |
|
||||
| C-3 / C-4 | Good | **Confirmed good** | Tag-editor map covers 7 drivers + Galaxy raw path; all driver pages carry `JsonStringEnumConverter`, pinned by `*FormSerializationTests`. No action. |
|
||||
| U-1..U-4 | — | **Confirmed** | No bUnit; documented stubs; unvalidated raw-JSON fallback; thin a11y. |
|
||||
|
||||
**Nothing in this report is stale/already-fixed.** All actionable findings reproduce at `9cad9ed0`. One nuance correction: the task brief mentioned a `DriverAdmin` policy — **no such policy exists**; only `FleetAdmin` and `DriverOperator` are registered (`Security/ServiceCollectionExtensions.cs:155-160`).
|
||||
|
||||
---
|
||||
|
||||
## Priority order
|
||||
|
||||
1. **C-1** (High, overall action-item #5) — gate the mutating surface; standardize on one policy idiom with constants.
|
||||
2. **C-2** (Med) — folds into C-1 (converge ScriptAnalysis gate + fix the doc).
|
||||
3. **S-4 + S-5** (Med) — delete concurrency + confirmation.
|
||||
4. **S-3** (Med) — `ErrorBoundary`.
|
||||
5. **S-1 + S-2** (Med) — timer/error-handling convergence.
|
||||
6. **P-1** (Med) — debounce Monaco value push.
|
||||
7. **P-2** (Med) — ScriptAnalysis compilation memo.
|
||||
8. **C-5** (Med) — declare service-seam canonical; migrate `ScriptEdit`.
|
||||
9. **Batch Lows** — S-6, S-7, P-3/4/5, P-6, C-6, U-2, U-3, U-4.
|
||||
|
||||
---
|
||||
|
||||
## 1. C-1 (High) — Standardize authorization; gate the mutating surface
|
||||
|
||||
**Restated:** Three authz idioms coexist and the largest mutating surface (UNS, equipment, cluster/node/namespace/ACL editors, all 8 driver pages, Reservations) carries only bare `[Authorize]`, so any authenticated user — incl. a read-only Viewer — can create/edit/delete config. Only the deploy/scripts pages are role-gated.
|
||||
|
||||
**Verification (confirmed):**
|
||||
- Policies defined: `Security/ServiceCollectionExtensions.cs:143-161` — `FleetAdmin = RequireRole("Administrator")`, `DriverOperator = RequireRole("Operator","Administrator")`, `FallbackPolicy = RequireAuthenticatedUser()`. No "write" policy exists.
|
||||
- Roles: `AdminRole` enum = `Viewer`, `Designer`, `Administrator` (`Configuration/Enums/AdminRole.cs`); `Operator` is an appsettings-only control-plane role (`DevAuthRoles.cs:14`).
|
||||
- Idiom census (all string literals, no constants):
|
||||
- `Roles="Administrator,Designer"`: `Deployments.razor:12`, `Scripts.razor:2`, `ScriptEdit.razor:5`, `ScriptAnalysisEndpoints.cs:18`.
|
||||
- `Policy="FleetAdmin"`: `RoleGrants.razor:2` (page), `Certificates.razor:90` (`AuthorizeView`), `Certificates.razor:186` (imperative).
|
||||
- Imperative `"DriverOperator"`: `Alerts.razor:165`, `DriverStatusPanel.razor:166`, `GalaxyAddressPickerBody.razor:124`, `OpcUaClientAddressPickerBody.razor:75`.
|
||||
- **Bare `[Authorize]`** (= FallbackPolicy, any authenticated user): `GlobalUns`, `EquipmentPage`, `ClusterEdit`, `NodeEdit`, `NamespaceEdit`, `NewCluster`, `AclEdit`, `ClusterAcls/Namespaces/Drivers/Overview/Audit/Redundancy`, `ClustersList`, all 8 driver pages + `DriverEditRouter` + `DriverTypePicker`, `Reservations`, plus the read-only dashboards.
|
||||
|
||||
**Root cause:** Pages were authored by copying the nearest neighbor; the fused Host's `FallbackPolicy` makes bare `[Authorize]` *look* protective (you must be logged in) while providing zero role separation. No policy-name constants exist to make the correct gate discoverable, so drift is the path of least resistance. Project posture (memory: roles are global, "simplest authz") was never actually applied to the mutating pages.
|
||||
|
||||
### Proposed design
|
||||
|
||||
**Standardize on the policy idiom with named constants**, and introduce one write policy.
|
||||
|
||||
1. **Add a policy-constants type** in the Security project (co-located with the definitions), e.g. `Security/Auth/AdminUiPolicies.cs`:
|
||||
```csharp
|
||||
public static class AdminUiPolicies
|
||||
{
|
||||
public const string FleetAdmin = "FleetAdmin"; // RequireRole(Administrator)
|
||||
public const string DriverOperator = "DriverOperator"; // RequireRole(Operator, Administrator)
|
||||
public const string ConfigEditor = "ConfigEditor"; // NEW: RequireRole(Administrator, Designer)
|
||||
}
|
||||
```
|
||||
Keep the string *values* identical to today's literals so nothing else has to change atomically.
|
||||
|
||||
2. **Register the new `ConfigEditor` policy** in `AddOtOpcUaAuth` (`ServiceCollectionExtensions.cs:143`):
|
||||
```csharp
|
||||
o.AddPolicy(AdminUiPolicies.ConfigEditor, p => p.RequireRole("Administrator", "Designer"));
|
||||
```
|
||||
`ConfigEditor` is exactly the semantic already spelled out four times as `Roles="Administrator,Designer"` — so `Deployments`/`Scripts`/`ScriptEdit`/`ScriptAnalysisEndpoints` converge onto it too, collapsing idiom #1 into idiom #2.
|
||||
|
||||
3. **Apply `[Authorize(Policy = AdminUiPolicies.ConfigEditor)]`** to every *mutation-dedicated* page. Read-only dashboards keep the FallbackPolicy (bare `[Authorize]`).
|
||||
|
||||
**Gate with `ConfigEditor` (mutation pages):**
|
||||
`GlobalUns`, `EquipmentPage`, `ClusterEdit`, `NodeEdit`, `NamespaceEdit`, `NewCluster`, `AclEdit`, `ClusterRedundancy`, `DriverEditRouter`, `DriverTypePicker`, all 8 driver pages (`Modbus/S7/AbCip/AbLegacy/TwinCAT/Focas/OpcUaClient/Galaxy`), `Reservations`, plus converge `Deployments`/`Scripts`/`ScriptEdit`.
|
||||
|
||||
**Keep FleetAdmin:** `RoleGrants` (already), `Certificates` mutating actions (already per-action).
|
||||
|
||||
**Leave read-only (FallbackPolicy):** `Home`, `Fleet`, `Hosts`, `Alerts` (view; the Ack/Shelve buttons stay imperative-`DriverOperator`), `ScriptLog`, `AlarmsHistorian`, `Account`, `ClusterOverview`, `ClustersList`, `ClusterAudit`, `Certificates` (view).
|
||||
|
||||
4. **Mixed list pages** (`ClustersList`, `ClusterAcls`, `ClusterNamespaces`, `ClusterDrivers`) contain both read content and destructive buttons (New/Edit/Delete). Two options:
|
||||
- **(A, recommended) Keep the page viewable, wrap destructive controls in `<AuthorizeView Policy="@AdminUiPolicies.ConfigEditor">`** so a Viewer sees the fleet but can't mutate. Preserves read access, matches least-surprise. Slightly more than one attribute per page.
|
||||
- **(B, minimal) Page-level `ConfigEditor`** — a Viewer loses read access to those lists entirely. Cheaper, but hides visibility a Viewer arguably should have.
|
||||
|
||||
Recommend **(A)** for the four list pages, **page-level `ConfigEditor`** for the pure edit/create forms and driver pages. This is the correct read/write split for a global-role model.
|
||||
|
||||
5. **Convert the imperative/AuthorizeView literals to the constants** (`Alerts`, `DriverStatusPanel`, both pickers, `Certificates`, `RoleGrants`) — mechanical: `"DriverOperator"` → `AdminUiPolicies.DriverOperator`, etc. In razor markup use `Policy="@AdminUiPolicies.ConfigEditor"`.
|
||||
|
||||
**Alternatives considered / rejected:**
|
||||
- *Keep role-strings everywhere* — rejected: the canonical role rename (`ConfigEditor→Designer`, `FleetAdmin→Administrator`) already shows role strings are brittle; policies decouple the gate from the role vocabulary in one place.
|
||||
- *Fine-grained per-action policies (WriteTag/WriteCluster/…)* — rejected as over-engineering against the deliberate global-role posture (memory: "roles are global; simplest authz that works").
|
||||
- *Move gating into the `IUnsTreeService`/EF layer* — good defense-in-depth but out of scope here and doesn't fix the surface-level page problem; note as a future hardening.
|
||||
|
||||
### Implementation steps
|
||||
|
||||
1. Add `Security/Auth/AdminUiPolicies.cs` (constants).
|
||||
2. `ServiceCollectionExtensions.cs`: register `ConfigEditor`; switch the two existing `AddPolicy` names to the constants.
|
||||
3. Edit page `@attribute` lines (one line each) for the mutation pages listed above → `[Authorize(Policy = AdminUiPolicies.ConfigEditor)]`. Add `@using ZB.MOM.WW.OtOpcUa.Security.Auth` to `Components/_Imports.razor` so the constant resolves in every razor file.
|
||||
4. Wrap destructive controls in the four list pages with `<AuthorizeView Policy="@AdminUiPolicies.ConfigEditor">` (option A).
|
||||
5. Replace imperative/AuthorizeView literals with constants (`Alerts`, `DriverStatusPanel`, pickers, `Certificates`, `RoleGrants`, `ScriptAnalysisEndpoints`).
|
||||
6. Confirm `AutoLoginAuthenticationHandler` still satisfies `ConfigEditor` (it grants `DevAuthRoles.All` incl. `Administrator`+`Designer` — yes, no dev-rig regression).
|
||||
|
||||
### Tests + live-verify
|
||||
|
||||
- **Unit (Security.Tests, CI-runnable, cheap):** build a `ServiceProvider` via `AddOtOpcUaAuth`, resolve `IAuthorizationService`, and assert the `ConfigEditor` policy:
|
||||
- `ClaimsPrincipal` with role `Viewer` only → **denied**.
|
||||
- with `Designer` → allowed; with `Administrator` → allowed; with `Operator` only → denied.
|
||||
This is the only automated proof of the *policy semantics* (mirrors the existing `Configuration.Tests/AuthorizationTests.cs` style).
|
||||
- **Reflection guard (AdminUI.Tests, CI-runnable) — closes the "built-but-never-wired" gap for authz:** enumerate all page component types (`typeof(App).Assembly` types with a `[Route]` attribute), and assert every type in a hard-coded `MutatingPages` set carries an `AuthorizeAttribute` whose `Policy == AdminUiPolicies.ConfigEditor` (and that no mutating page has a bare `[Authorize]`). This is the cheapest possible substitute for bUnit and catches a new page copying the wrong idiom — directly implements OVERALL cross-cutting theme #1 ("assert production wiring").
|
||||
- **Live-verify (docker-dev, manual, negative path needs real login):**
|
||||
1. Positive smoke on the default rig (auto-admin): confirm the gated pages still load and mutate — deploy a config, open `/uns`, add/delete a tag, open a driver page. Auto-login grants all roles so everything must still work (regression check that the gate didn't over-block admins).
|
||||
2. **Negative path requires disabling auto-login.** Set `Security:Auth:DisableLogin=false`, bring the AdminUI up against the shared GLAuth (`10.100.0.35:3893`), bind a **Viewer** LDAP user (group→`Viewer` via `Security:Ldap:GroupToRole`), and verify: `/uns`, `/uns/equipment/*`, `/clusters/*/edit`, driver pages, `/reservations` all render the "You do not have permission" `NotAuthorized` slot (from `Routes.razor:19`); the four list pages render but hide New/Edit/Delete. Then bind a **Designer** user and confirm access is restored. Document this recipe in the PR — it is the only way to observe the deny, because the default rig can't.
|
||||
|
||||
**Effort:** M. **Risk/blast-radius:** Medium — touches ~25 page files but each change is one line; the real risk is *over-gating a read-only dashboard* (hiding it from Viewers) or *under-gating a mutation page*. The reflection guard + the manual Viewer pass are the mitigations. No runtime/data-plane impact (OPC UA data-plane auth is independent LDAP).
|
||||
|
||||
---
|
||||
|
||||
## 2. C-2 (Med) — ScriptAnalysis gate doc/code drift
|
||||
|
||||
**Restated:** `CLAUDE.md` says `/api/script-analysis/*` is "gated by the `FleetAdmin` policy"; code uses `Roles="Administrator,Designer"`.
|
||||
|
||||
**Verification (confirmed):** `ScriptAnalysisEndpoints.cs:17-18` `RequireAuthorization(new AuthorizeAttribute { Roles = "Administrator,Designer" })`; `CLAUDE.md:223` + design plan `…adminui-disable-login-design.md:141` claim FleetAdmin. The original monaco plan (`2026-06-09-monaco-script-editor.md:309`) *intended* `RequireAuthorization("FleetAdmin")` but the implementation diverged to match the Scripts page.
|
||||
|
||||
**Root cause:** implementation deliberately matched the Scripts/ScriptEdit page gate (which is `Administrator,Designer`) but the docs were written from the original plan and never reconciled.
|
||||
|
||||
**Design/fix:** fold into C-1. When `ScriptAnalysisEndpoints` converges onto `AdminUiPolicies.ConfigEditor` (= `Administrator,Designer`), update the doc to state the truth: **script-analysis and the Scripts/ScriptEdit/Deployments pages are gated by the `ConfigEditor` policy (Administrator or Designer)**, distinct from `FleetAdmin` (Administrator-only). This also fixes OVERALL cross-cutting theme #5 for this claim.
|
||||
|
||||
**Implementation:** edit `CLAUDE.md:223`; the design-plan files are historical (leave, or add a one-line "superseded" note). **Tests:** the C-1 policy unit test covers the endpoint's effective gate; add one endpoint-level assertion in `ScriptAnalysis` tests if a `WebApplicationFactory` is already in use there (check first — the suite is service-level today).
|
||||
|
||||
**Effort:** S. **Risk:** trivial (doc + one converged literal).
|
||||
|
||||
---
|
||||
|
||||
## 3. S-4 + S-5 (Med) — Delete concurrency + confirmation
|
||||
|
||||
**Restated:** Delete paths re-fetch a *fresh* RowVersion at click time (defeating optimistic concurrency → last-writer-wins), and `EquipmentPage`/`ScriptEdit` deletes fire on a single click with no confirmation.
|
||||
|
||||
**Verification (confirmed):**
|
||||
- S-4: `GlobalUns.ConfirmDeleteAsync` (Area/Line/Equipment branches) `Load…Async` → `Delete…Async(id, loaded.RowVersion)`; `EquipmentPage.DeleteTag/DeleteVirtualTag/DeleteAlarm` (370/415/460) each comment "Load … fresh to capture its current RowVersion". *Update* paths correctly carry the modal-load RowVersion (`EquipmentPage.razor:555`), so the contract is genuinely inconsistent. Note `ScriptEdit.DeleteAsync` is the exception — it uses `_form.RowVersion` (the page-load value), so it is *not* a fresh-refetch; S-4 applies to GlobalUns + EquipmentPage only.
|
||||
- S-5: `EquipmentPage.razor:172/217/263` `@onclick="() => DeleteTag(...)"` immediate; `ScriptEdit` delete button immediate. `GlobalUns` has a confirm modal (`GlobalUns.razor:71-98`); `DriverStatusPanel` has an inline Restart confirm.
|
||||
|
||||
**Root cause:** the list DTOs the rows render from don't carry RowVersion, so the delete handlers re-load to get *a* RowVersion — silently making delete unconditional. Confirmation was added to GlobalUns/DriverStatusPanel but not propagated to the per-equipment tables or ScriptEdit.
|
||||
|
||||
### Proposed design
|
||||
|
||||
- **S-4 — carry the rendered RowVersion:** add `RowVersion` (byte[]/base64) to the tag/vtag/alarm *list* DTOs returned by `IUnsTreeService` (they already exist for the edit path; extend the list projections). The delete handlers pass the row's RowVersion straight into `Delete…Async` — no re-load. Then a concurrent edit between render and click yields the existing `DbUpdateConcurrencyException` → surfaced as "changed by another user", matching the update contract (first-writer-wins). Same for GlobalUns Area/Line/Equipment (the tree nodes would carry RowVersion, or keep GlobalUns re-load but **document it as intentional** — the tree is a coarser surface).
|
||||
- *Alternative:* explicitly document delete as unconditional last-writer-wins. Rejected for EquipmentPage (inconsistent with its own updates); acceptable as a documented fallback for the GlobalUns tree if threading RowVersion through the tree DTO is disproportionate.
|
||||
- **S-5 — shared confirm:** extract a small reusable `ConfirmButton`/`ConfirmModal` component (generalize the `GlobalUns` confirm modal) and use it for `EquipmentPage` tag/vtag/alarm deletes and `ScriptEdit` delete. Reduces duplication and gives every destructive action a consistent guard.
|
||||
|
||||
### Implementation steps
|
||||
|
||||
1. Extend list DTOs in `Uns/UnsTreeService.cs` + `IUnsTreeService.cs` (tag/vtag/alarm list projections) with `RowVersion`.
|
||||
2. `EquipmentPage` delete handlers: drop the fresh-load, pass `row.RowVersion`.
|
||||
3. Add `Components/Shared/ConfirmModal.razor` (extract from GlobalUns) or a `ConfirmButton`; wire into `EquipmentPage` (×3) and `ScriptEdit`.
|
||||
4. Optionally thread RowVersion into the GlobalUns tree node DTO, or add a one-line "delete is unconditional here" doc-comment.
|
||||
|
||||
### Tests + live-verify
|
||||
|
||||
- **Unit (UnsTreeService.Tests — the strong seam):** list projections now include RowVersion; add a test asserting `DeleteTagAsync(id, staleRowVersion)` returns a concurrency failure (the service already has RowVersion tests to mirror).
|
||||
- **Live-verify:** on docker-dev, open `/uns/equipment/{id}` Tags tab, click Delete → confirm modal appears → cancel leaves the tag → confirm deletes. For concurrency: open the equipment in two tabs, edit+save a tag in tab A, then Delete it in tab B → expect the "changed by another user" error, not a silent delete. Repeat for ScriptEdit delete-confirm.
|
||||
|
||||
**Effort:** M. **Risk:** Low-Medium — DTO shape change ripples to a few call sites; the confirm-modal extraction is additive.
|
||||
|
||||
---
|
||||
|
||||
## 4. S-3 (Med) — No `ErrorBoundary`
|
||||
|
||||
**Restated:** A single unhandled exception in any handler/render tears down the whole circuit; there's no `ErrorBoundary` anywhere in a 77-component console.
|
||||
|
||||
**Verification (confirmed):** `grep ErrorBoundary` = none; `MainLayout.razor:53` renders `@Body` bare inside `ThemeShell`.
|
||||
|
||||
**Root cause:** never added; the shared `ThemeShell` chassis doesn't provide one.
|
||||
|
||||
**Design:** wrap `@Body` in `MainLayout` in an `<ErrorBoundary>` with an error view + a "Reload page" / `Recover()` action; on error, log via the injected logger and show a themed panel instead of the framework's error UI. Consider a second, tighter boundary around the live-tail panels (`Alerts`, `ScriptLog`, `DriverStatusPanel`) so a bridge/render fault there doesn't blank the dashboard — but the single `MainLayout` boundary is the high-value 80%.
|
||||
|
||||
**Implementation:** edit `MainLayout.razor` `ChildContent` to `<ErrorBoundary Context="ex">…<ErrorContent>` + recover button; add a `_boundary.Recover()` on navigation if desired.
|
||||
|
||||
**Tests + live-verify:** no bUnit — live-verify by temporarily throwing in a page handler (or triggering a known-faulting path) and confirming the boundary panel + Recover works instead of a dead circuit. Leave a note that this can't be unit-tested.
|
||||
|
||||
**Effort:** S. **Risk:** Low (purely additive resilience).
|
||||
|
||||
---
|
||||
|
||||
## 5. S-1 + S-2 (Med) — Timer disposal + refresh error handling
|
||||
|
||||
**Restated:** Three components don't drain in-flight timer callbacks and `Fleet`/`AlarmsHistorian` mishandle refresh errors.
|
||||
|
||||
**Verification (confirmed):**
|
||||
- S-1: `Fleet.razor:171` + `AlarmsHistorian.razor:90` sync `Dispose() => _timer?.Dispose()`; `DriverTestConnectButton.razor:66-81` `System.Timers.Timer` + `async (_,_) => {…}` Elapsed (effectively async-void) + sync `Dispose`. The house pattern (`System.Threading.Timer` + `IAsyncDisposable`) is used correctly in `DriverStatusPanel`/`Alerts`/`Hosts`.
|
||||
- S-2: `Fleet.LoadAsync` `try/finally` no `catch` (a faulted `_ = InvokeAsync(LoadAsync)` is unobserved); `AlarmsHistorian` bare `catch {}` → permanent "Loading…".
|
||||
|
||||
**Root cause:** three stragglers predate the async-dispose convention; `Fleet` copied the timer-schedule but not the `Hosts` catch-log-degrade shape.
|
||||
|
||||
**Design:** converge all three timers onto `System.Threading.Timer` + `public async ValueTask DisposeAsync()` that awaits `_timer.DisposeAsync()` (and unsubscribes first, per the `DriverStatusPanel` template). For `DriverTestConnectButton`, replace `System.Timers.Timer`/`Elapsed` with a `System.Threading.Timer` one-shot (`Timeout.InfiniteTimeSpan` period) — kills the async-void. Give `Fleet.LoadAsync` a `catch (Exception ex) { _error = …; Log … }` (copy `Hosts.LoadConfigAsync`) and add an explicit `_error` surface. Give `AlarmsHistorian` an explicit "no historian on this node role" state instead of infinite "Loading…" (also closes U-2's admitted TODO).
|
||||
|
||||
**Implementation:** edit the three components; make them `@implements IAsyncDisposable`; add error fields + rendering. Reuse the exact `DriverStatusPanel.DisposeAsync` shape.
|
||||
|
||||
**Tests + live-verify:** live-verify on docker-dev — open `/fleet`, `/alarms-historian` on an admin-only vs driver node and confirm graceful states; navigate away rapidly during a refresh tick to confirm no disposed-component `StateHasChanged` warnings in the circuit log. Not unit-testable without bUnit.
|
||||
|
||||
**Effort:** S. **Risk:** Low.
|
||||
|
||||
---
|
||||
|
||||
## 6. P-1 (Med) — Debounce Monaco value push
|
||||
|
||||
**Restated:** `OnValueChanged(editor.getValue())` crosses the SignalR circuit on every keystroke and re-renders the parent.
|
||||
|
||||
**Verification (confirmed):** `monaco-init.js:205-207` fires `invokeMethodAsync("OnValueChanged", …)` inside `onDidChangeModelContent` with no debounce; diagnostics use a 500 ms `setTimeout` debounce already.
|
||||
|
||||
**Root cause:** the value sync was wired for immediacy; the .NET side only consumes it for Save + the inline problems panel, neither of which needs per-keystroke fidelity.
|
||||
|
||||
**Design:** debounce the value push ~200-300 ms (reuse the existing `diagTimer` debounce shape → add a `valueTimer`), *or* push on the diagnostic tick + on blur/save. Ensure Save reads the latest value (either via a final flush on blur/save-trigger or by having Save request `editor.getValue()` through a JS interop call synchronously). Keep the model-content event driving diagnostics as-is.
|
||||
|
||||
**Implementation:** edit `wwwroot/js/monaco-init.js` (`onDidChangeModelContent`), add a debounced value push + a flush path invoked before Save. Verify `ScriptEdit` + the virtual-tag modal Save both see current text.
|
||||
|
||||
**Tests + live-verify:** live-verify on docker-dev `/scripts/{id}` — type rapidly and confirm (network panel) `OnValueChanged` fires at debounce cadence not per keystroke; save and confirm the persisted source matches the editor. No unit coverage (JS interop).
|
||||
|
||||
**Effort:** S. **Risk:** Low-Medium — the one hazard is a lost final edit if the flush-before-save is missed; the flush path must be verified.
|
||||
|
||||
---
|
||||
|
||||
## 7. P-2 (Med) — Memoize ScriptAnalysis compilation
|
||||
|
||||
**Restated:** Each of the six endpoints re-parses + creates a fresh `CSharpCompilation` for the same text during a typing burst.
|
||||
|
||||
**Verification (confirmed):** `ScriptAnalysisService.Analyze` (60-67) `CSharpSyntaxTree.ParseText(full)` + `CSharpCompilation.Create(...)` per call; `Sandbox.References`/`Preamble`/`CompileOptions` are static (already cached).
|
||||
|
||||
**Root cause:** no `(normalized text → tree/compilation/model)` memo; a 500 ms diagnostics tick immediately followed by completions/hover/signature-help all recompile identical text.
|
||||
|
||||
**Design:** add a **size-1 LRU** (or a tiny `MemoryCache` keyed on the normalized source hash) inside `ScriptAnalysisService` returning the `(Tree, Compilation, Model, PreambleLength)` tuple, shared by all six endpoints. Size-1 collapses the common completions-after-diagnostics case (same text) at negligible memory. Bound it and note it's single-admin-scale; flag before multi-user. Keep it thread-safe (`lock`/`ConcurrentDictionary`) since the service is a shared registration.
|
||||
|
||||
**Implementation:** wrap `Analyze(userSource)` with a cache check keyed on the source (or a SHA of it) → last-value cache field. No API change; endpoints unchanged.
|
||||
|
||||
**Tests + live-verify:** **unit (ScriptAnalysis.Tests — a strong existing seam):** assert two successive `Analyze`/diagnose+complete calls on identical text produce identical diagnostics (correctness preserved) and that the cache returns the same compilation instance (a memoization assertion via an internal hook). Live-verify: type in the editor, confirm completions/hover latency drops and diagnostics stay correct.
|
||||
|
||||
**Effort:** S. **Risk:** Low — correctness risk is stale cache after an edit; keying on the full text (or hash) eliminates it.
|
||||
|
||||
---
|
||||
|
||||
## 8. C-5 (Med) — Service-seam vs EF-in-page duality
|
||||
|
||||
**Restated:** UNS pages go through the testable `IUnsTreeService`; `ScriptEdit`/`Scripts`/`Deployments`/`Fleet`/`Hosts` run EF + `SaveChangesAsync` directly in `@code`, untestable without a host.
|
||||
|
||||
**Verification (confirmed):** `ScriptEdit.razor` `DeleteAsync`/save use `DbFactory` + `db.Scripts.Remove` + `SaveChangesAsync` in-page; UNS pages use `Svc`. `IUnsTreeService.UpdateScriptSourceAsync` already exists (`IUnsTreeService.cs:502`).
|
||||
|
||||
**Root cause:** new pages copy the nearest file; both idioms are "blessed" by precedent.
|
||||
|
||||
**Design:** **declare the `IUnsTreeService` service-seam canonical for all config mutation.** Migrate `ScriptEdit`'s save+delete into the service (an `UpdateScriptSourceAsync` already exists; add `CreateScriptAsync`/`DeleteScriptAsync` if missing) so the logic gains the same RowVersion + test coverage the UNS methods have. `Fleet`/`Hosts` are *read-only* projections — leave their EF-in-page (or move to a thin read service later); the priority is the *mutating* EF-in-page (`ScriptEdit`, and the `Deployments` deploy trigger, which already has service seams). This dovetails with S-4 (list DTOs) and OVERALL theme #4 (verification).
|
||||
|
||||
**Implementation:** add/confirm `IUnsTreeService` script CRUD methods; rewrite `ScriptEdit` save/delete to call them; keep read pages as-is with a note.
|
||||
|
||||
**Tests:** the migrated script CRUD gets unit coverage in `UnsTreeService.Tests` (delete-with-stale-RowVersion, create, update-source) — turning previously untestable `@code` into covered seam logic. Live-verify script create/edit/delete on `/scripts`.
|
||||
|
||||
**Effort:** M. **Risk:** Low-Medium — behavior-preserving refactor of one page; regression risk mitigated by the new seam tests.
|
||||
|
||||
---
|
||||
|
||||
## 9. Batched Lows / Underdeveloped
|
||||
|
||||
Group into one cleanup PR (or fold opportunistically into the above):
|
||||
|
||||
- **S-6 (Low):** `DriverStatusPanel.ShowOpResult` — add the `Alerts`-style stale-fire token compare so a queued chip-clear timer can't wipe a newer result. (`DisposeAsync` drain is already correct.) *Effort S.*
|
||||
- **S-7 (Low):** `AdminOperationsClient.AskAsync<T>` — wrap with a linked CTS + `AskTimeout` to match the three typed methods' contract. *Effort S.*
|
||||
- **P-3/P-4 (Low):** per-circuit dashboard polling + `Deployments` `SnapshotAndFlattenAsync` per render. Defer; optionally back with `IMemoryCache` keyed on the deploy event before any multi-viewer rollout. Document as accepted-at-current-scale.
|
||||
- **P-5 (Low):** add `@key` to `Alerts` rows (it `Insert(0, …)` + renders without `@key`, forcing whole-table diffs). *Effort S.*
|
||||
- **P-6 (Low):** stop registering the Monaco inlay-hints provider until the endpoint is non-empty (`monaco-init.js:99-117`); it round-trips for a documented no-op (`ScriptAnalysisService.InlayHints` always returns empty). *Effort S.*
|
||||
- **C-6 (Low):** replace the `"GalaxyMxGateway"` literal (`TagModal.razor:311`) with the driver-type constant; dedupe the `DataTypes` array; normalize `TwinCat`/`FOCAS`/`Focas` casing. *Effort S.*
|
||||
- **U-2 (stubs):** `AlarmsHistorian` role message (folded into S-2); `MonacoEditor.MarkersChanged` `object[]` — model the DTO; `GlobalUns` default-branch "not yet available" message — reword to not leak not-implemented posture. *Effort S each.*
|
||||
- **U-3 (raw-JSON fallback):** in `TagModal.SaveAsync`, `JsonDocument.Parse` the fallback textarea before submit so malformed JSON is caught client-side instead of at deploy. `_form.TagConfig` is `[Required]` but never validated for well-formed JSON. *Effort S — good defensive win; add a validator unit test.*
|
||||
- **U-1 / U-4:** strategic, not this-PR. **U-1:** either adopt bUnit for the modal/table state machines, or keep extracting `@code` into plain classes (`HostsDriverView`/`VirtualTagModalHelpers` precedent) — the reflection authz guard (§1) is a cheap down payment. **U-4:** targeted a11y pass on `UnsTree` expander `aria-expanded`/labels, modal focus-trap/Escape, `aria-live` on `Alerts`, table `scope`/captions — do if operator diversity matters.
|
||||
|
||||
**Effort:** S each; **Risk:** Low across the batch.
|
||||
|
||||
---
|
||||
|
||||
## Cross-cutting notes for the implementer
|
||||
|
||||
- **The docker-dev auto-login (`DisableLogin=true`) makes every negative-authz test impossible on the default rig** — it grants `DevAuthRoles.All`. Every authz claim in this plan must be proven by the policy unit test + reflection guard (CI) *and* one manual real-login Viewer pass (§1 recipe). Do not accept "it loads for me on docker-dev" as authz verification.
|
||||
- **No bUnit** means S-1/S-2/S-3/S-5/P-1 are only observable by live `/run`. Prefer, wherever cheap, extracting logic into plain classes (C-5 direction) so the untested `@code` residue stays trivial — this is the repo's stated posture and OVERALL theme #4.
|
||||
- **The reflection authz guard (§1) is the single highest-leverage test in this report**: it converts "which policy is on which page" from a live-only fact into a CI invariant, directly countering the "built-but-never-wired" house failure mode (OVERALL theme #1).
|
||||
- Sequencing: land **C-1 + C-2 together** (shared constants/policy), then **S-4/S-5**, **S-3**, **S-1/S-2**, then the perf pair **P-1/P-2**, then **C-5**, then the Lows batch. C-1 and C-5 both touch page files — do C-1 first (attributes) to avoid churn.
|
||||
@@ -0,0 +1,808 @@
|
||||
# Design + Implementation Plan — 05 Protocol Drivers
|
||||
|
||||
**Source report:** `archreview/05-protocol-drivers.md`
|
||||
**Review commit:** `9cad9ed0` (== current HEAD, so no drift)
|
||||
**Verification date:** 2026-07-08
|
||||
**Scope:** Modbus, S7, AbCip, AbLegacy, TwinCAT, FOCAS, OpcUaClient (+ Contracts, + Cli)
|
||||
|
||||
---
|
||||
|
||||
## Verification summary
|
||||
|
||||
Every finding in the report was opened at its cited file:line and checked against
|
||||
the current tree. **37 actionable findings; 37 CONFIRMED; 0 stale.** (STAB-13 is a
|
||||
"positive" observation carrying no work.) The line references were accurate to
|
||||
within 1–2 lines throughout. The zero-drift result is expected: the review commit
|
||||
is the current HEAD.
|
||||
|
||||
Confirmed counts by section: Stability 12 actionable (STAB-1…12) + 1 positive
|
||||
(STAB-13); Performance 7 (PERF-1…7); Conventions 8 (CONV-1…8); Underdeveloped 9
|
||||
(UNDER-1…9).
|
||||
|
||||
**Top 3 by priority** (align with OVERALL actions #3/#4 and report remediation
|
||||
order):
|
||||
1. **STAB-1 — S7 no reconnect path** (Critical). `Plc.OpenAsync` only at
|
||||
`S7Driver.cs:165`; grep confirms it is the *sole* open site, no `EnsureConnected`.
|
||||
2. **STAB-2 — TwinCAT native subscriptions orphaned after reconnect** (Critical).
|
||||
`NativeSubscription` (`TwinCATDriver.cs:517-519`) stores only disposable handles,
|
||||
not replayable intent; `EnsureConnectedAsync` swaps the client
|
||||
(`:614-644`) and nothing re-runs `AddNotificationAsync`; `AdsTwinCATClient.Dispose`
|
||||
clears `_notifications` (`:426`).
|
||||
3. **STAB-4 + STAB-5 — silent-corruption class.** AbCip runs Read→GetStatus→Decode
|
||||
on shared libplctag handles with no per-runtime lock (`AbCipDriver.cs:544-575`;
|
||||
only `GetRmwLock` exists, bit-writes only), while AbLegacy guards the identical
|
||||
hazard with `GetRuntimeLock` (`AbLegacyDriver.cs:786`, used at `:227`/`:331`).
|
||||
FOCAS's fixed-tree caches are plain `Dictionary` (`FocasDriver.cs:1184,1190,1192,1194`)
|
||||
read/written across threads.
|
||||
|
||||
The two systemic threads the report calls out — **theme #2** (fixes don't flow back
|
||||
to shared templates) and **theme #6** (equipment-tag/TagConfig paths are
|
||||
second-class) — are both real and are given a dedicated consolidation section
|
||||
(Part B) so the fleet fixes land in one home rather than being re-patched per driver.
|
||||
|
||||
---
|
||||
|
||||
# Part A — Per-driver Criticals & Highs
|
||||
|
||||
## A1. STAB-1 (Critical) — S7 has no reconnect path
|
||||
|
||||
**Restatement:** the S7 `Plc` is opened once in `InitializeAsync`; a transient PLC
|
||||
reboot/network blip permanently kills the instance until redeploy.
|
||||
|
||||
**Verification:** CONFIRMED. `grep OpenAsync S7Driver.cs` → single hit at line 165,
|
||||
inside `InitializeAsync`. No `EnsureConnected`/`Reconnect` symbol. Reads/writes call
|
||||
`RequirePlc()` under `_gate` and reuse the dead handle
|
||||
(`S7Driver.cs:479-483, 995-1001`). The probe loop detects the outage and transitions
|
||||
`Stopped` but never recycles. `ReinitializeAsync` (`:212`) just re-calls `InitializeAsync`
|
||||
(externally driven only). Zero S7 reconnect tests (`tests/Drivers/.../S7.Tests` has no
|
||||
reconnect file; only `Modbus`, `OpcUaClient` do).
|
||||
|
||||
**Root cause:** the `Plc` is a concrete `S7.Net` type `new`-ed inline
|
||||
(`S7Driver.cs:156`, field at `:122`), with no factory seam and no lazy-connect
|
||||
gate. The template calls for a lazy `EnsureConnectedAsync` (TwinCAT/AbCip have one;
|
||||
S7 never got it).
|
||||
|
||||
**Design (mirror TwinCAT + OpcUaClient):**
|
||||
- Introduce a testable seam: `IS7Plc` (wrapping the members S7Driver uses:
|
||||
`OpenAsync`, `Close`, `ReadBytesAsync`/`ReadMultiple`, `Write*`, `IsConnected`)
|
||||
and `IS7PlcFactory` with `Create(cpu, host, port, rack, slot)`. Default impl wraps
|
||||
`S7.Net.Plc`; the ctor takes an optional `IS7PlcFactory? = null` defaulting to the
|
||||
real one — exactly TwinCAT's `ITwinCATClientFactory` pattern
|
||||
(`TwinCATDriver.cs:20,58,64`).
|
||||
- Add `private async Task<IS7Plc> EnsureConnectedAsync(CancellationToken ct)` guarded
|
||||
by the existing `_gate`: fast-path returns a live `Plc`; under the gate, if the
|
||||
`Plc` is null or a prior socket-level `PlcException` marked it dead, dispose and
|
||||
re-`OpenAsync` a fresh instance (with the same timeout-linked CTS as `:163-164`).
|
||||
- Route `ReadAsync`/`WriteAsync`/`PollOnceAsync` through `EnsureConnectedAsync`
|
||||
instead of `RequirePlc()`. On a socket-level `PlcException` during a read/write,
|
||||
null the `Plc` (mark dead) so the next call reopens, and set `Degraded` (keep
|
||||
`LastSuccessfulRead`) rather than `Faulted`. Preserve the existing `Faulted`
|
||||
escalation for PUT/GET-denied (`S7Driver.cs:921-1006`) — that is a permanent
|
||||
config fault, not a transient drop.
|
||||
- Classify socket-level vs protocol-level `PlcException` in a small
|
||||
`IsS7ConnectionFatal(Exception)` helper (S7.Net surfaces `ErrorCode.ConnectionError`
|
||||
/ `WriteData` / `ReadData` / IO/socket exceptions). Do not reopen on a
|
||||
data-address/type error.
|
||||
|
||||
**Alternatives considered:** (a) let the probe loop own recycle-on-Stopped — rejected
|
||||
as sole mechanism because reads between probe ticks would keep failing on the dead
|
||||
handle; the lazy `EnsureConnectedAsync` repairs on the very next data call and the
|
||||
probe becomes a backstop. (b) Reconnect handler à la OpcUaClient's
|
||||
`SessionReconnectHandler` — overkill; S7.Net has no session-transfer semantics, a
|
||||
plain reopen is correct.
|
||||
|
||||
**Implementation steps:**
|
||||
1. New file `Driver.S7/IS7Plc.cs` (+ `S7PlcAdapter`, `IS7PlcFactory`,
|
||||
`S7PlcFactory`). Change `S7Driver.Plc` field type to `IS7Plc?`.
|
||||
2. Thread `IS7PlcFactory` through the ctor + `S7DriverFactoryExtensions.CreateInstance`.
|
||||
3. Add `EnsureConnectedAsync` + `IsS7ConnectionFatal`; convert the two `RequirePlc`
|
||||
call clusters and `PollOnceAsync` (`:1295-1309`).
|
||||
4. Optionally: on probe transition to `Stopped`, null the `Plc` so the next data
|
||||
call reconnects even if no read failed in between.
|
||||
|
||||
**Tests:**
|
||||
- Unit (new `S7DriverReconnectTests.cs`): fake `IS7PlcFactory` whose `IS7Plc` throws
|
||||
a connection-fatal `PlcException` on read N, succeeds on read N+1; assert the driver
|
||||
creates a *second* `IS7Plc`, health goes `Degraded`→`Healthy`, and the tag recovers
|
||||
to Good. Assert a protocol error (illegal address) does **not** trigger a reopen.
|
||||
- Integration (`S7.IntegrationTests`, skip-gated on `Snap7ServerFixture`): subscribe,
|
||||
bounce the snap7 container (`lmxopcua-fix down s7 s7_1500` / `up`), assert values
|
||||
resume without redeploy. Note: no per-test container hook exists, so gate this as a
|
||||
manual/`[Trait("Category","Reconnect")]` recipe.
|
||||
|
||||
**Effort:** M. **Risk:** Medium — touches the hot read/write path; the `IS7Plc` seam
|
||||
is mechanical but broad. Mitigated by the fake-factory unit tests (which also close
|
||||
the TEST-1 gap).
|
||||
|
||||
## A2. STAB-2 (Critical) — TwinCAT native subscriptions orphaned after reconnect
|
||||
|
||||
**Restatement:** native ADS notifications (the *default* mode) silently stop after a
|
||||
client swap — no Bad status, no error.
|
||||
|
||||
**Verification:** CONFIRMED. `SubscribeAsync` registers `AddNotificationAsync` once
|
||||
per tag (`TwinCATDriver.cs:472`) and stores the resulting handles in
|
||||
`NativeSubscription(Handle, Registrations)` (`:495, :517-519`) — **no** record of the
|
||||
symbol/type/interval/handler needed to replay. `EnsureConnectedAsync` disposes a
|
||||
stale client and creates a fresh one (`:614-644`); `AdsTwinCATClient.Dispose` clears
|
||||
`_notifications` (`:426`). Compounding: the fast path keys on `IsConnected` (`:606`),
|
||||
the local AMS-port state, not wire liveness; and a probe failure only transitions
|
||||
state, never forces a client recycle (`ProbeLoopAsync :540-547`).
|
||||
|
||||
**Root cause:** subscription state is stored as *opaque handles* rather than
|
||||
*replayable intent*, and there is no "client changed → replay" hook. Native ADS is
|
||||
push, so unlike the poll fallback it cannot self-heal by re-reading.
|
||||
|
||||
**Design:**
|
||||
- Change `NativeSubscription` to store replayable registrations:
|
||||
`record NativeRegistrationIntent(string DeviceHostAddress, string Reference,
|
||||
string SymbolName, TwinCATDataType Type, int? BitIndex, TimeSpan Interval,
|
||||
Action<string,object?> OnChange)` plus the live `ITwinCATNotificationHandle`.
|
||||
- Give `DeviceState` an epoch/generation counter incremented every time
|
||||
`EnsureConnectedAsync` installs a *new* client (`:644`). After installing a new
|
||||
client, replay every `NativeRegistrationIntent` whose `DeviceHostAddress` matches:
|
||||
call `AddNotificationAsync` again and swap the stored live handle. Do this inside
|
||||
`EnsureConnectedAsync` under `ConnectGate` so it cannot race a concurrent subscribe.
|
||||
- Fix the liveness signal: the probe should call a real read (`ProbeAsync` already
|
||||
does a symbol read) and, on failure, dispose+null the device client so
|
||||
`EnsureConnectedAsync` rebuilds — i.e. make probe-detected outage force a recycle
|
||||
(`ProbeLoopAsync` catch block at `:541-545` should null `device.Client`). Consider
|
||||
keying the fast path (`:606`) additionally on a "known-good since last probe" flag
|
||||
so a locally-connected-but-wire-dead port still triggers the rebuild-and-replay.
|
||||
|
||||
**Alternatives:** switch the default to the poll fallback (self-healing) — rejected;
|
||||
native push is the performance-correct default and the report explicitly wants replay,
|
||||
not abandonment.
|
||||
|
||||
**Implementation steps:**
|
||||
1. Introduce a per-device `ConcurrentDictionary<subId, List<NativeRegistrationIntent>>`
|
||||
(or hang intents off `DeviceState`). Populate in `SubscribeAsync`.
|
||||
2. Extract the per-tag registration into `RegisterNotificationAsync(device, intent)`
|
||||
used by both initial subscribe and replay.
|
||||
3. In `EnsureConnectedAsync`, after `device.Client = client` (`:644`), iterate the
|
||||
device's intents and replay; log a re-registration count.
|
||||
4. In `ProbeLoopAsync`, on probe failure null the device client (force rebuild).
|
||||
|
||||
**Tests:**
|
||||
- Unit (new `TwinCATReconnectReplayTests.cs`) using the existing fake
|
||||
`ITwinCATClientFactory`/`ITwinCATClient`: subscribe a tag, simulate a client that
|
||||
reports `IsConnected=false`, force `EnsureConnectedAsync` to build a replacement,
|
||||
assert `AddNotificationAsync` is re-invoked with the same symbol/type/interval and
|
||||
that a subsequent fake push reaches `OnDataChange`. Assert the old handle is disposed.
|
||||
- Unit: probe-failure recycles the client (assert a new client instance built).
|
||||
- Note there is **no** TwinCAT docker fixture (13 integration tests need a real TC3
|
||||
XAR); the fake-client unit tests are the only automated coverage — make them
|
||||
authoritative.
|
||||
|
||||
**Effort:** M. **Risk:** Medium — concurrency around `ConnectGate`/replay; must not
|
||||
double-register. The fake client makes it fully unit-testable (the report's key gap).
|
||||
|
||||
## A3. STAB-3 (High) — Modbus transport desyncs after a response timeout
|
||||
|
||||
**Restatement:** a per-op timeout or TxId mismatch leaves the single-flight socket
|
||||
half-read; the next txn reads a stale response and the stream stays desynchronized.
|
||||
|
||||
**Verification:** CONFIRMED. `IsSocketLevelFailure` (`ModbusTcpTransport.cs:257-261`)
|
||||
matches only `EndOfStreamException`/`IOException`/`SocketException`/
|
||||
`ObjectDisposedException`. The per-op deadline (`:226-227` linked CTS `CancelAfter`)
|
||||
surfaces as `OperationCanceledException`; the TxId guard throws `InvalidDataException`
|
||||
(`:234-235`); the truncated-length guard throws `InvalidDataException` (`:237`) — none
|
||||
of which are socket-level, so the reconnect-and-retry catch (`:155-165`) is skipped
|
||||
and the socket is retained mid-desync.
|
||||
|
||||
**Root cause:** the transport conflates "protocol-layer error, keep the socket" with
|
||||
"framing/timeout error, the socket is unusable." Timeout and any framing violation
|
||||
leave unknown bytes in the receive buffer; the connection is fatally desynchronized
|
||||
even though the exception type is not `IOException`.
|
||||
|
||||
**Design:** treat per-op timeout and any framing violation as **connection-fatal**:
|
||||
tear the socket down *before* propagating, so the next `SendAsync` reconnects clean.
|
||||
- In `SendOnceAsync`, wrap the read/write in a try that, on `OperationCanceledException`
|
||||
where the *caller's* `ct` is NOT cancelled (i.e. the timeout linked-CTS fired), and
|
||||
on `InvalidDataException`, calls `TearDownAsync()` then rethrows a normalized
|
||||
`ModbusTransportDesyncException : IOException` (subclassing IOException lets the
|
||||
existing `:155` catch optionally reconnect-and-retry once). Distinguish caller
|
||||
cancellation (propagate as OCE, no teardown) from timeout (teardown).
|
||||
- Simplest correct implementation: after `TearDownAsync`, rethrow as `IOException`
|
||||
so both the retry classifier and the desync-repair share one path.
|
||||
|
||||
**Alternatives:** add timeout/framing to `IsSocketLevelFailure` directly — works but
|
||||
must still tear down the socket first (the current retry path does `TearDownAsync`
|
||||
before reconnect, so adding the types *and* ensuring teardown covers it). Preferred to
|
||||
keep `IsSocketLevelFailure` about exception *type* and add an explicit teardown at the
|
||||
throw site for framing/timeout, since those need teardown even when auto-reconnect is
|
||||
off.
|
||||
|
||||
**Implementation steps:**
|
||||
1. In `SendOnceAsync`, catch `OperationCanceledException` (timeout branch) and
|
||||
`InvalidDataException`; `await TearDownAsync()`; rethrow as `IOException`
|
||||
(`ModbusTransportDesyncException`).
|
||||
2. Keep the outer `SendAsync` single-retry (`:155-165`) — it now covers the desync.
|
||||
3. Ensure teardown even when `_autoReconnect` is false (the socket must not be reused
|
||||
desynced regardless).
|
||||
|
||||
**Tests:**
|
||||
- Unit (`ModbusTcpReconnectTests.cs` extension): a fake stream that (a) stalls past
|
||||
the deadline, (b) returns a wrong TxId, (c) returns a truncated header. Assert each
|
||||
triggers `TearDownAsync` (socket disposed) and the following call reconnects. Assert
|
||||
a caller-cancellation (external ct) does NOT tear the socket down.
|
||||
- Integration (`Modbus.IntegrationTests` + `exception_injector`): the existing rig
|
||||
already forces FC exceptions; add a timeout scenario (non-responding unit) and
|
||||
assert recovery on the next poll.
|
||||
|
||||
**Effort:** S. **Risk:** Low-Medium — must get the timeout-vs-cancellation distinction
|
||||
right or a legitimate shutdown could look like a desync (harmless, just an extra
|
||||
reconnect).
|
||||
|
||||
## A4. STAB-4 (High) — AbCip concurrent ops on shared libplctag handles, no lock
|
||||
|
||||
**Restatement:** AbCip read/group/write run Read→GetStatus→Decode /
|
||||
Encode→Write→GetStatus on cached `IAbCipTagRuntime` with zero serialization while the
|
||||
server read path, poll loops, and the alarm-projection loop can hit the same handle.
|
||||
|
||||
**Verification:** CONFIRMED. `ReadSingleAsync` (`AbCipDriver.cs:544-575`),
|
||||
`ReadGroupAsync` (`:614-637`), `WriteAsync` (`:708-726`) take no runtime lock; the only
|
||||
locks are `GetRmwLock` (bit-in-DINT writes, `:1243`) and a creation lock. AbLegacy
|
||||
documents the exact hazard and guards it: `GetRuntimeLock` (`AbLegacyDriver.cs:786`),
|
||||
applied on both read (`:227`) and write (`:331`), comment "A libplctag `Tag` handle is
|
||||
not safe for concurrent Read/GetStatus/DecodeValue." `AbCipAlarmProjection.cs:149/228`
|
||||
reads through the unlocked path. libplctag's `GetStatus` returns the *last* operation's
|
||||
status, so concurrent Read/GetStatus can cross-attribute results → torn values with
|
||||
Good status.
|
||||
|
||||
**Root cause:** the AbLegacy fix never propagated to AbCip (theme #2). AbCip caches a
|
||||
per-tag runtime but never serializes operations on it.
|
||||
|
||||
**Design:** port AbLegacy's per-runtime operation lock verbatim.
|
||||
- Add `GetRuntimeLock(string tagName)` to AbCip's `DeviceState` (a
|
||||
`ConcurrentDictionary<string,SemaphoreSlim>` keyed by tag name, matching
|
||||
`AbLegacyDriver.cs:786`).
|
||||
- Wrap the whole Read/GetStatus/Decode critical section in `ReadSingleAsync` and
|
||||
`ReadGroupAsync`, and the Encode/Write/GetStatus section in `WriteAsync`, in
|
||||
`await opLock.WaitAsync(ct)` / `finally Release`. The existing `GetRmwLock` (bit RMW)
|
||||
must nest *inside or replace-by* the runtime lock consistently — since RMW does
|
||||
read-modify-write on the same handle, take the runtime lock for the whole RMW too
|
||||
(this also fixes STAB-11's "direct parent write bypasses the lock" for AbCip: a
|
||||
direct parent write and an RMW now serialize on the same per-tag runtime lock).
|
||||
|
||||
**Alternatives:** a single per-device lock — rejected; too coarse, serializes
|
||||
independent tags. Per-runtime lock matches AbLegacy and preserves cross-tag parallelism.
|
||||
|
||||
**Implementation steps:**
|
||||
1. Add `_runtimeLocks` + `GetRuntimeLock` to AbCip `DeviceState`.
|
||||
2. Wrap the three op bodies. Ensure eviction-on-failure (`EvictRuntime`, `:554`) also
|
||||
disposes/removes the tag's lock or leaves it (a fresh runtime under the same name
|
||||
reuses the lock — fine).
|
||||
3. Reconcile RMW: make `WriteBitInDIntAsync` acquire the per-runtime lock instead of
|
||||
(or in addition to) `GetRmwLock`; collapse to one lock domain per parent to also
|
||||
close STAB-11.
|
||||
|
||||
**Tests:**
|
||||
- Unit (new `AbCipRuntimeConcurrencyTests.cs`, mirror `AbLegacyRuntimeConcurrencyTests`):
|
||||
a fake `IAbCipTagRuntime` that asserts no overlapping Read/GetStatus (e.g. a reentrancy
|
||||
counter that throws if >1); hammer concurrent `ReadAsync`+`WriteAsync`+alarm-read on
|
||||
one tag; assert serialization and no torn/cross-attributed status.
|
||||
- Integration (`abcip` fixture on `:44818`): concurrent subscribe + write to overlapping
|
||||
tags, assert Good values stay coherent.
|
||||
|
||||
**Effort:** S. **Risk:** Low — additive locking, blast radius is AbCip only.
|
||||
|
||||
## A5. STAB-5 (High) — FOCAS fixed-tree caches are plain Dictionaries mutated across threads
|
||||
|
||||
**Restatement:** `LastFixedSnapshots`/`LastServoLoads`/`LastSpindleLoads`/`LastTimers`
|
||||
are plain `Dictionary`, written by `FixedTreeLoopAsync` and read on `ReadAsync` threads.
|
||||
|
||||
**Verification:** CONFIRMED. Fields at `FocasDriver.cs:1184` (`Dictionary<string,double>`),
|
||||
`:1190` (`Dictionary<FocasTimerKind,FocasTimer> = []`), `:1192`
|
||||
(`Dictionary<string,double>`), `:1194` (`Dictionary<int,int> = []`). Writes at
|
||||
`:752, :793, :836-839`; reads at `:912, :920, :945`. Contrast the neighbouring
|
||||
`_parsedAddressesByTagName = new ConcurrentDictionary` (`:40`) — that field got the
|
||||
treatment, these did not. Concurrent write-during-`Dictionary`-resize is UB (throw or
|
||||
corruption).
|
||||
|
||||
**Root cause:** partial application of the concurrent-collection fix (theme #2).
|
||||
|
||||
**Design:** convert the four caches to `ConcurrentDictionary<,>`. `TryGetValue` and
|
||||
indexer-set are the only operations used, both supported. `LastTimers` and
|
||||
`LastSpindleLoads` collection-initializer `[]` becomes `new()`.
|
||||
- Alternative (immutable snapshot swap): build a fresh dictionary per loop pass and
|
||||
`Volatile.Write` the reference; readers `Volatile.Read`. Cleaner isolation but more
|
||||
churn; `ConcurrentDictionary` is the lower-risk minimal change and matches the
|
||||
in-tree precedent (`:40`).
|
||||
|
||||
**Implementation steps:** change the four field types (on `DeviceState`) + collection
|
||||
initializers; no call-site changes needed (indexer + `TryGetValue` are identical).
|
||||
|
||||
**Tests:**
|
||||
- Unit (`FocasDriver.Tests`): a stress test spinning `FixedTreeLoopAsync`-style writes
|
||||
while readers call `TryReadFixedTree`; assert no exception over N iterations (would
|
||||
reliably throw on the plain `Dictionary` today).
|
||||
|
||||
**Effort:** S. **Risk:** Very low — type-only change, FOCAS-local.
|
||||
|
||||
## A6. STAB-6 (High) — AbLegacy probe-loop shutdown race; S7 unsubscribe race
|
||||
|
||||
**Verification:** CONFIRMED both.
|
||||
- AbLegacy: probe task fire-and-forgot `_ = Task.Run(...)` (`AbLegacyDriver.cs:118`);
|
||||
`ShutdownAsync` (`:156-170`) cancels then immediately disposes `ProbeCts` + runtimes
|
||||
without awaiting loop exit — a winding-down loop can raise `OnHostStatusChanged` and
|
||||
touch a disposed CTS. AbCip already fixed this: retained `ProbeTask` (`:298`) + phased
|
||||
shutdown with `await probeTask.WaitAsync(10s)` (`:332-365`).
|
||||
- S7: `UnsubscribeAsync` (`:1185-1193`) cancels + disposes the CTS without draining
|
||||
`PollTask`; `Task.Delay` on the disposed source can throw `ObjectDisposedException`
|
||||
the loop only catches as OCE (`:1233-1234`). `ShutdownAsync` *does* drain PollTask
|
||||
(`:239`), so the bug is confined to the `Unsubscribe` path.
|
||||
|
||||
**Root cause:** cancel-then-dispose-without-await; `PollGroupEngine.StopState`
|
||||
(`PollGroupEngine.cs:99-112`) already solves this by awaiting the loop (bounded) before
|
||||
disposing the CTS — neither bespoke path adopted it.
|
||||
|
||||
**Design:**
|
||||
- AbLegacy: retain the probe task in `DeviceState.ProbeTask` and phase shutdown like
|
||||
AbCip (`await probeTask.WaitAsync(bounded)` before `ProbeCts.Dispose()`).
|
||||
- S7: in `UnsubscribeAsync`, `Cancel()`, then `state.PollTask.Wait(bounded)` (or await),
|
||||
then `Cts.Dispose()` — mirroring `StopState`. Better: once S7 migrates to
|
||||
`PollGroupEngine` (see B1) this deletes itself.
|
||||
|
||||
**Tests:** unit — subscribe/unsubscribe rapidly in a loop and assert no
|
||||
`ObjectDisposedException` escapes; shutdown while a probe tick is mid-transition.
|
||||
|
||||
**Effort:** S. **Risk:** Low.
|
||||
|
||||
## A7. UNDER-1 (High) — FOCAS advertises writes it cannot perform
|
||||
|
||||
**Verification:** CONFIRMED. `WireFocasClient.WriteAsync` returns
|
||||
`FocasStatusMapper.BadNotWritable` for every address (`WireFocasClient.cs:71-73`), yet
|
||||
`FocasDriverOptions.Writable` defaults `true` (`FocasDriverOptions.cs:168`) and
|
||||
`FocasEquipmentTagParser` hard-codes `Writable: true` (`FocasEquipmentTagParser.cs:35`).
|
||||
The driver's write exception-mapping (`FocasDriver.cs:353-381`) is dead code against
|
||||
this backend. Net: equipment tags surface as Operate-writable OPC UA nodes whose writes
|
||||
always fail.
|
||||
|
||||
**Root cause:** the writable default was written speculatively (doc cites "write tests")
|
||||
but the only real backend is read-only.
|
||||
|
||||
**Design (force writable-false at the seams until PMC writes exist):**
|
||||
- `FocasEquipmentTagParser.cs:35` → `Writable: false`.
|
||||
- `FocasDriverOptions.Writable` default → `false` (and/or force it false at discovery
|
||||
materialization). Keep the option field for the future PMC-write feature but do not
|
||||
advertise writable nodes today.
|
||||
- Leave the write status-mapping in place (it becomes live if/when PMC writes ship).
|
||||
- Alternative (larger): implement PMC writes in `WireFocasClient` via `pmc_wrpmcrng` —
|
||||
a separate feature, out of scope for a hardening pass.
|
||||
|
||||
**Tests:** unit — assert a FOCAS equipment/authored tag materializes without the
|
||||
`CurrentWrite` access bit; assert a write attempt returns `BadNotWritable` (guards the
|
||||
regression if someone flips the default back).
|
||||
|
||||
**Effort:** S. **Risk:** Very low (removes a false capability).
|
||||
|
||||
## A8. UNDER-2 / UNDER-3 (High) — OpcUaClient: dead `UnsMappingTable` gate; broken alarm-filter encoding
|
||||
|
||||
**UNDER-2 verification:** CONFIRMED. `ValidateNamespaceKind` hard-fails Equipment-kind
|
||||
configs lacking `UnsMappingTable` (`OpcUaClientDriver.cs:335-338`) but grep shows the
|
||||
table is read nowhere except that gate and the options DTO
|
||||
(`OpcUaClientDriverOptions.cs:169,188`). `DiscoverAsync` renders refs purely from remote
|
||||
browse via `StableReference` (`:808-809`). Operators must author a table with zero
|
||||
runtime effect.
|
||||
|
||||
**UNDER-3 verification:** CONFIRMED. Persisted refs are stable `nsu=` strings
|
||||
(`:808-809`), but the alarm source filter is an ordinal `HashSet` of caller refs
|
||||
(`:1337`) matched against `EventFields[SourceNode].ToString()` — a session-relative
|
||||
`ns=N;…` rendering (`:1538-1539`). The two encodings never match, so any non-empty
|
||||
filter drops every event; only empty-filter subscriptions work.
|
||||
|
||||
**Root cause:** UNDER-2 is a validation gate for an unbuilt feature; UNDER-3 is an
|
||||
encoding mismatch between the stored `nsu=` form and the runtime `ns=N` form.
|
||||
|
||||
**Design:**
|
||||
- UNDER-2: **delete the gate** (remove the Equipment-kind `UnsMappingTable` mandatory
|
||||
check) — the honest minimal change, since the feature does not exist. Keep the option
|
||||
DTO field (forward-compatible) but stop forcing operators to author dead config.
|
||||
Alternative (build the feature): have `DiscoverAsync` translate remote refs → UNS refs
|
||||
via the table — a real feature, larger, defer.
|
||||
- UNDER-3: normalize both sides to the same encoding before compare. Convert the
|
||||
incoming `EventFields[SourceNode]` `NodeId` to the stable `nsu=` reference via the
|
||||
driver's `NamespaceMap.ToStableReference` (the same function used at `:808-809`) before
|
||||
the `HashSet.Contains` check; build the filter set from stable refs. Then a non-empty
|
||||
filter works.
|
||||
|
||||
**Implementation steps:** remove `ValidateNamespaceKind`'s Equipment branch (or make it
|
||||
a no-op warning); in `OnEventNotification`, map the source `NodeId` through
|
||||
`_namespaceMap` before comparing (`:1538-1539`); build `sourceFilter` from stable refs
|
||||
(`:1337`).
|
||||
|
||||
**Tests:** unit — event with a known source arrives, non-empty stable-ref filter
|
||||
matches and the event is delivered (today it is dropped); Equipment config without a
|
||||
mapping table no longer throws at validation.
|
||||
|
||||
**Effort:** S (both). **Risk:** Low — OpcUaClient-local; UNDER-3 fix is a behavior
|
||||
change that *enables* previously-dead filtering (verify no consumer depended on the
|
||||
"filter silently drops all" bug).
|
||||
|
||||
## A9. STAB-13 (Positive) — no action
|
||||
|
||||
Write-outcome surfacing is honoured by every protocol driver (per-item `WriteResult`
|
||||
with typed exception→status), and OpcUaClient's reconnect machine is exemplary
|
||||
(double-arm guard, re-arm on exhaustion, session swap under `_probeLock`,
|
||||
`:1908-2058`). CONFIRMED. Use OpcUaClient's machine as the reference for A1/A2. One
|
||||
caveat surfaced: `Session.TransferSubscriptionsOnReconnect` is **left unset**
|
||||
(mentioned only in the doc at `:1948`), so subscription transfer rests on the SDK
|
||||
default — worth explicitly setting it (folds into UNDER-9 cleanup).
|
||||
|
||||
---
|
||||
|
||||
# Part B — Shared-template consolidation (themes #2 and #6)
|
||||
|
||||
The report's cross-cutting message: *every* High is a fix present in one sibling and
|
||||
absent in another. These four items give the fleet a single home for the fixes so they
|
||||
stop being re-patched per driver. Do these **after** the per-driver criticals but treat
|
||||
them as the highest-leverage structural work.
|
||||
|
||||
## B1. PollGroupEngine v2 — absorb S7's fork; wire `onError`; add backoff (CONV-1, STAB-9, STAB-8, PERF-3, STAB-6-S7)
|
||||
|
||||
**Verification of the pieces:**
|
||||
- `PollGroupEngine` already has the `onError` sink (`PollGroupEngine.cs:36,55-67,164-169`),
|
||||
`StopState` draining (`:99-112`), and structural array compare (`:203-209`). It does
|
||||
**not** have failure backoff — `PollLoopAsync` uses a fixed `Task.Delay(state.Interval)`
|
||||
(`:131`).
|
||||
- **STAB-9 CONFIRMED:** no consumer passes `onError` — the five constructions
|
||||
(`ModbusDriver.cs:115`, `AbCipDriver.cs:133`, `AbLegacyDriver.cs:65`,
|
||||
`TwinCATDriver.cs:69`, `FocasDriver.cs:72`) omit it, so the sink is dead fleet-wide.
|
||||
- **CONV-1 / PERF-3 / STAB-6-S7 CONFIRMED:** S7 re-ships the loop (`S7Driver.cs:1165-1307`),
|
||||
which *has* backoff (`ComputeBackoffDelay :1284-1293`) and failure-health
|
||||
(`HandlePollFailure :1258-1274`) the engine lacks, but *lacks* the structural array
|
||||
diff (uses `Equals(lastSeen?.Value, current.Value)` `:1304` → arrays fire every tick)
|
||||
and the drain-before-dispose (`UnsubscribeAsync :1189-1190`).
|
||||
|
||||
**Design (single home for the poll loop):**
|
||||
1. **Add optional backoff to `PollGroupEngine`.** Port S7's `ComputeBackoffDelay`
|
||||
(capped exponential, ticks-based, saturating at a `PollBackoffCap`) into the engine.
|
||||
Constructor gains `TimeSpan? backoffCap = null` (null ⇒ current fixed-cadence
|
||||
behavior, preserving every current consumer). `PollLoopAsync` tracks
|
||||
`consecutiveFailures`, resets on success, and delays `ComputeBackoffDelay(interval,
|
||||
failures)`.
|
||||
2. **Route failures to health.** The engine already forwards to `onError`; make each of
|
||||
the five drivers pass an `onError` that degrades `_health` (keeping
|
||||
`LastSuccessfulRead`) + logs — i.e. lift S7's `HandlePollFailure` into a small shared
|
||||
`Action<Exception>` per driver. Consider making `onError` **required** (the report
|
||||
suggests it) once all five pass it.
|
||||
3. **Migrate S7 onto the engine and delete the fork.** Replace `S7Driver`'s
|
||||
`_subscriptions` dict + `PollLoopAsync`/`PollOnceAsync`/`ComputeBackoffDelay`/
|
||||
`HandlePollFailure` (`:1162-1310`) with a `PollGroupEngine` instance (reader =
|
||||
`ReadAsync`, onChange = forward `OnDataChange`, onError = degrade health, backoffCap
|
||||
= 30 s). This deletes PERF-3 (structural compare now inherited) and STAB-6-S7
|
||||
(StopState drain now inherited) for free.
|
||||
|
||||
**Implementation steps:**
|
||||
1. Extend `PollGroupEngine` ctor + `PollLoopAsync` with backoff; add unit tests for
|
||||
`ComputeBackoffDelay` parity with S7's (move the existing S7 test asserts over).
|
||||
2. Add an `onError` handler in Modbus/AbCip/AbLegacy/TwinCAT/FOCAS constructions that
|
||||
degrades health; assert via unit test that a throwing reader degrades health.
|
||||
3. S7: construct the engine, delete the fork, keep the 100 ms floor and the
|
||||
`PollFailureHealthThreshold`/`Degraded`-not-`Faulted` nuance in the onError handler.
|
||||
|
||||
**Tests:**
|
||||
- Engine unit: backoff schedule (0 failures = interval, N failures = capped
|
||||
exponential); onError invoked once per caught reader exception; structural array
|
||||
equality still suppresses no-change arrays; StopState drains before CTS dispose.
|
||||
- S7 unit: array tag no longer fires every tick (regression for PERF-3); rapid
|
||||
subscribe/unsubscribe no `ObjectDisposedException` (STAB-6); sustained poll failure
|
||||
degrades health with backoff.
|
||||
|
||||
**Effort:** M (engine change is S; S7 migration + 5 onError wirings is M). **Risk:**
|
||||
Medium — S7 migration touches its subscription surface; the array-diff change alters
|
||||
publish cadence (intended). Gate behind the existing S7 subscribe tests + a new
|
||||
array-cadence test.
|
||||
|
||||
## B2. Shared strict `ReadEnum` for equipment-tag parsers (CONV-2, UNDER-6)
|
||||
|
||||
**Verification:** CONFIRMED. An identical private generic `ReadEnum<TEnum>(…, TEnum
|
||||
fallback)` with **silent fallback** is copy-pasted six times — a typo'd `dataType`
|
||||
quietly becomes the default type and reads/writes the wrong width with Good status:
|
||||
`ModbusEquipmentTagParser.cs:65-67`, `S7EquipmentTagParser.cs:59-61`,
|
||||
`AbCipEquipmentTagParser.cs:86-88`, `AbLegacyEquipmentTagParser.cs:60-62`,
|
||||
`TwinCATEquipmentTagParser.cs:47-49`, `FocasEquipmentTagParser.cs:43-45` (all
|
||||
byte-identical bodies). Contrast the factory's fail-loud `ParseEnum`
|
||||
(`ModbusDriverFactoryExtensions.cs:191-201`, lists valid names). Probe-parity is also
|
||||
broken for Modbus: `ModbusDriverProbe` binds the *runtime* `ModbusDriverOptions`
|
||||
(`ModbusDriverProbe.cs:36`) not the factory DTO, so `timeoutMs` is silently dropped.
|
||||
|
||||
**Root cause:** duplication-by-convention (theme #2); the three parse tiers (factory
|
||||
strict / probe permissive / parser silent) diverged.
|
||||
|
||||
**Design:**
|
||||
- **Hoist one strict helper into `Core.Abstractions`**, living beside
|
||||
`EquipmentTagRefResolver` (theme #6's home): e.g.
|
||||
`TagConfigJson.ReadEnum<TEnum>(JsonElement o, string name, TEnum @default)` with the
|
||||
rule **present-but-invalid ⇒ throw** (parse failure), **absent ⇒ default**. The parser
|
||||
wraps the throw so the resolver's `_parseRef` returns null on a bad enum, and the
|
||||
driver surfaces `BadNodeIdUnknown` instead of silently reading the wrong width.
|
||||
- Delete all six private copies; the parsers call the shared helper. Enums stay
|
||||
per-driver (the helper is generic over `TEnum`), so no coupling of driver vocab.
|
||||
- **Probe parity (Modbus):** make `ModbusDriverProbe` deserialize the same factory DTO
|
||||
shape the factory uses (or share a `TryParseConfig` used by both), so `timeoutMs`
|
||||
binds identically. OpcUaClient already documents this rule
|
||||
(`OpcUaClientDriverProbe.cs:22`); Modbus should follow.
|
||||
|
||||
**Writable/capability parity (UNDER-6, folds in here):** Modbus/AbLegacy/FOCAS parsers
|
||||
hard-code `Writable: true` (`ModbusEquipmentTagParser.cs:54-57`,
|
||||
`AbLegacyEquipmentTagParser.cs:52`, `FocasEquipmentTagParser.cs:35`), so read-only
|
||||
equipment tags are unauthorable on three drivers (AbCip honours a `writable` key,
|
||||
`AbCipEquipmentTagParser.cs:53-54`). FOCAS additionally must force writable-false
|
||||
(A7). Add a shared `ReadBool(o, "writable", default)` helper and honour it in all six
|
||||
parsers; FOCAS overrides to false until PMC writes exist. FOCAS equipment tags also
|
||||
bypass the `FocasCapabilityMatrix` gate authored tags get (`FocasDriver.cs:243-247` vs
|
||||
`:115-119`) — route equipment-tag resolution through the same matrix validate.
|
||||
|
||||
**Implementation steps:**
|
||||
1. New `Core.Abstractions/TagConfigJson.cs` with strict `ReadEnum` + `ReadBool`.
|
||||
2. Replace the six `ReadEnum` copies; wrap parse failure so `_parseRef` returns null.
|
||||
3. Honour `writable` in the three hard-coded parsers; FOCAS → false + capability gate.
|
||||
4. Fix `ModbusDriverProbe` to parse the factory DTO shape.
|
||||
|
||||
**Tests:**
|
||||
- **Shared conformance test** (the report explicitly wants one): a parameterized suite
|
||||
run per driver asserting (a) unknown `dataType` ⇒ resolve failure ⇒ `BadNodeIdUnknown`
|
||||
(not silent default); (b) `writable:false` honoured; (c) absent enum ⇒ documented
|
||||
default. Put it in a shared test helper consumed by each driver's `.Tests`.
|
||||
- Modbus probe: a config with `timeoutMs` round-trips the timeout.
|
||||
|
||||
**Effort:** M. **Risk:** Medium — changing silent-default to hard-fail is a *behavior
|
||||
change*: a deployment with a currently-silently-defaulted typo will start returning Bad.
|
||||
That is the correct outcome, but call it out in the migration notes and verify live on
|
||||
the rig (author a typo'd tag, confirm Bad rather than wrong-width Good).
|
||||
|
||||
## B3. `ResolveHost` via the resolver — fix per-host breaker isolation (CONV-4, theme #6)
|
||||
|
||||
**Verification:** CONFIRMED. Every multi-device driver's `ResolveHost` consults only
|
||||
`_tagsByName` then falls back to the first device: `AbCipDriver.cs:485-487`,
|
||||
`AbLegacyDriver.cs:498-500`, `TwinCATDriver.cs:587-591`, `FocasDriver.cs:1082-1087`
|
||||
(Modbus `:125-131` same shape, but per-tag `UnitId` isn't in its equipment parser).
|
||||
An equipment-tag reference (raw TagConfig JSON, resolved via `_resolver`, not in
|
||||
`_tagsByName`) misses and returns the first device's host — so the per-host Polly
|
||||
bulkhead/circuit-breaker key is wrong for equipment tags, and device B's outage can trip
|
||||
device A's breaker. Since equipment tags are the primary post-Galaxy authoring model,
|
||||
the `IPerCallHostResolver` is effectively inert for the flagship path.
|
||||
|
||||
**Root cause:** `ResolveHost` never learned about the equipment-tag path; the resolver
|
||||
resolves defs but exposes nothing host-specific (`EquipmentTagRefResolver` is generic
|
||||
over `TDef` and never inspects it — confirmed).
|
||||
|
||||
**Design:** route `ResolveHost` through `EquipmentTagRefResolver.TryResolve`, then
|
||||
derive the host from the parsed def. The per-driver `TDef` records already carry the
|
||||
host: `AbCipTagDefinition.DeviceHostAddress`, `AbLegacyTagDefinition.DeviceHostAddress`,
|
||||
`TwinCATTagDefinition.DeviceHostAddress`, `FocasTagDefinition.DeviceHostAddress` (all
|
||||
parsed by the equipment parsers). So:
|
||||
- Each driver's `ResolveHost(ref)` becomes: `if (_resolver.TryResolve(ref, out var def))
|
||||
return def.DeviceHostAddress; return <first-device-or-sentinel>`. This covers both
|
||||
authored (`_byName`) and equipment (`_parseRef`) paths through one call and reuses the
|
||||
parse cache.
|
||||
- Optional shared sugar: add a `string? ResolveHost(string ref, Func<TDef,string> host)`
|
||||
overload to `EquipmentTagRefResolver` so the pattern is one call. Keeps the host
|
||||
extraction driver-specific (the resolver stays generic) while giving theme #6 a single
|
||||
home.
|
||||
|
||||
**Implementation steps:** rewrite the four `ResolveHost` bodies to call
|
||||
`_resolver.TryResolve`; add the resolver overload if desired. Modbus: note `UnitId`
|
||||
isn't in the equipment parser — either add `unitId` to `ModbusEquipmentTagParser`
|
||||
(UNDER-6) or document that Modbus per-tag host resolution is out of scope.
|
||||
|
||||
**Tests:** unit per driver — an equipment-tag ref for device B resolves to device B's
|
||||
host (today returns device A/first); assert the Polly breaker key differs per device.
|
||||
|
||||
**Effort:** S-M. **Risk:** Low — corrects a mis-key; verify the breaker-key wiring
|
||||
downstream consumes the corrected host.
|
||||
|
||||
## B4. Shared reconnect/backoff primitive (STAB-8, supports A1/A2)
|
||||
|
||||
**Verification:** CONFIRMED. No reconnect backoff exists except the Modbus transport's
|
||||
geometric backoff (`ModbusTcpTransport.cs:179-208`) and S7's poll-loop backoff. Dead
|
||||
devices pay full reconnect cost per tick: AbCip full Tag create + Forward Open per tag
|
||||
per tick (`AbCipDriver.cs:891-897` evict-recreate), FOCAS full two-socket reconnect per
|
||||
tick of each of three loops (`FocasDriver.cs:1089-1120`), TwinCAT connect-attempt per
|
||||
call (`:602-651`).
|
||||
|
||||
**Design:** a small shared `ConnectionBackoff` primitive in `Core.Abstractions`
|
||||
(capped exponential, the same `ComputeBackoffDelay` math B1 adds to the poll engine —
|
||||
factor it out so both use it). Lazy-reconnect drivers gate their
|
||||
`EnsureConnected`/evict-recreate behind a per-device throttle: record
|
||||
`_lastConnectAttemptUtc` + a growing backoff, and short-circuit a reconnect attempt that
|
||||
is inside the backoff window (return the current Bad/Degraded state instead of hammering).
|
||||
|
||||
**Implementation steps:**
|
||||
1. Extract `ComputeBackoffDelay` into `Core.Abstractions/ConnectionBackoff.cs`
|
||||
(shared by PollGroupEngine v2 and the lazy-reconnect drivers).
|
||||
2. Add a per-device backoff gate to TwinCAT `EnsureConnectedAsync`, FOCAS
|
||||
`EnsureConnectedAsync`, and AbCip's evict-recreate path (skip the recreate if within
|
||||
the backoff window; reset on success).
|
||||
|
||||
**Tests:** unit — a device that fails to connect is not retried more often than the
|
||||
backoff schedule; success resets the schedule.
|
||||
|
||||
**Effort:** M. **Risk:** Low-Medium — must not *delay* recovery when the device comes
|
||||
back (reset promptly on the first success).
|
||||
|
||||
---
|
||||
|
||||
# Part C — Remaining Mediums & Lows (batched)
|
||||
|
||||
All CONFIRMED. Grouped for efficient sweeps.
|
||||
|
||||
## C1. One-liner conventions sweep (CONV-3, CONV-5)
|
||||
- **CONV-3** (Medium): S7/TwinCAT/AbCip/FOCAS `Register` take no `ILoggerFactory`
|
||||
(`S7DriverFactoryExtensions.cs:19`, `TwinCATDriverFactoryExtensions.cs:18`,
|
||||
`AbCipDriverFactoryExtensions.cs:21`, `FocasDriverFactoryExtensions.cs:36`), so
|
||||
`DriverFactoryBootstrap.Register` (`:100,102,106,107`) can't pass one and all four log
|
||||
to `NullLogger` in production — their poll-failure/prohibition/live-risk warnings are
|
||||
invisible. AbLegacy/Galaxy/Modbus/OpcUaClient already have the overload
|
||||
(`:101,103,104,105`). Fix: add the `ILoggerFactory? = null` param + thread to the
|
||||
driver ctor in the four factories; update the four bootstrap call sites. **Effort S,
|
||||
risk very low.** (Also unblocks A1/A2/B1 logging.)
|
||||
- **CONV-5** (Medium): four health-idioms (`Volatile` vs `volatile` vs plain field);
|
||||
AbLegacy no health-refresh on successful write (`AbLegacyDriver.cs:344-346` vs AbCip
|
||||
`:725`); AbLegacy silently clobbers duplicate tag names (`:91`) where AbCip fails fast
|
||||
(`:249-256`); TwinCAT declares `Healthy` at init with zero wire contact (`:115`) and
|
||||
reads host state without the probe lock (`:524-525`). Fix: standardize on
|
||||
`Volatile.Read/Write` (or `volatile` record ref) across the fleet; add write-success
|
||||
health refresh to AbLegacy; make AbLegacy fail-fast on duplicate names; make TwinCAT
|
||||
prove the connection before `Healthy`. **Effort S-M, risk low.**
|
||||
|
||||
## C2. Per-protocol read batching (PERF-1, PERF-2) — the scalability lever
|
||||
- **PERF-1** (High): five of seven drivers do one tag per round-trip. Priority order per
|
||||
the report: TwinCAT ADS **sum-read** (`TwinCATDriver.cs:203-248`), S7 **multi-var**
|
||||
(`ReadMultipleVarsAsync`, unused — S7 loops per tag `:443-484` under one `_gate`),
|
||||
FOCAS **PMC range reads** (`WireFocasClient.cs:296-334` widens to one value width),
|
||||
AbLegacy **file-element spans** (N7:0..N7:2). Each is a separate protocol feature.
|
||||
- **PERF-2** (High): AbCip's whole-UDT batched read is half-built — the planner collapses
|
||||
N members via *declaration-order* layout, off by default (safe-but-unused,
|
||||
`AbCipDriverOptions.cs:72` default `false`), while true member offsets are already
|
||||
fetched via the CIP Template Object (`AbCipDriver.cs:151-182`,
|
||||
`AbCipTemplateCache`) but consulted only by discovery, never the planner
|
||||
(`AbCipUdtReadPlanner.cs:69`). **Highest-ROI batching win** (design already in-tree):
|
||||
feed `AbCipUdtShape` template offsets into the planner, retire the unsafe
|
||||
declaration-order mode. **Effort M-L, risk Medium (correctness of offset mapping —
|
||||
test against the abcip fixture with a known UDT).**
|
||||
|
||||
Sequencing: do PERF-2 first (design exists), then TwinCAT sum-read + S7 multi-var. Each
|
||||
is independently shippable. **Effort L overall.**
|
||||
|
||||
## C3. OpcUaClient performance + monolith (PERF-4, PERF-5, UNDER-9)
|
||||
- **PERF-4** (Medium): all traffic serializes on one `SemaphoreSlim(1,1)` `_gate`
|
||||
(12 `WaitAsync` sites, `OpcUaClientDriver.cs:78,626,713,836,1186,1377,1451,1653,1825`).
|
||||
OPC UA sessions multiplex natively; a multi-second HistoryRead/re-discovery blocks
|
||||
every read/write. The gate is only needed for session-swap consistency, which
|
||||
`_probeLock` + re-read-in-critical-section already provide. Fix: scope the gate to
|
||||
session-swap only; let reads/writes/history run concurrently on the session. **Effort
|
||||
M, risk Medium — must preserve session-swap safety; test the reconnect race.**
|
||||
- **PERF-5** (Medium): `EnrichAndRegisterVariablesAsync` sends `pending.Count*4`
|
||||
ReadValueIds in one call (`:1029-1046`); a server enforcing `MaxNodesPerRead` returns
|
||||
`BadTooManyOperations` and the blanket catch (`:1049-1057`) silently registers *every*
|
||||
var as Int32/ViewOnly/non-historized with no log/health. Fix: chunk client-side
|
||||
(respect `MaxNodesPerRead` / a conservative cap); log + health-signal the fallback.
|
||||
**Effort S, risk low.**
|
||||
- **UNDER-9** (Low): 2154-line monolith; extract the reconnect state machine for isolated
|
||||
testability; set `Session.TransferSubscriptionsOnReconnect` explicitly (`:1946-1949`
|
||||
— currently unset, relies on SDK default); `BuildCertificateIdentity` loads only
|
||||
PKCS#12 (`:483-484`) while the option doc promises PEM
|
||||
(`OpcUaClientDriverOptions.cs:68-75`) — either implement PEM load or fix the doc.
|
||||
**Effort M (extraction) / S (the two nits), risk low.**
|
||||
|
||||
## C4. Remaining stability/correctness Mediums (STAB-10, STAB-11, STAB-12)
|
||||
- **STAB-10** (Medium): AbLegacy never evicts failed tag handles on the data path
|
||||
(`AbLegacyDriver.cs:246-259,269-274`) though AbCip does (`AbCipDriver.cs:891-897`) and
|
||||
AbLegacy's own probe loop does (`:456-461`). Fix: evict-recreate on non-zero
|
||||
status/transport exception in the data path (port AbCip's `EvictRuntime`). Pairs with
|
||||
A4/A6 as an AbLegacy/AbCip parity pass. **Effort S, risk low.**
|
||||
- **STAB-11** (Medium): RMW locks don't cover direct parent writes (AbCip/AbLegacy/
|
||||
TwinCAT) → lost update when a direct parent write interleaves an in-flight RMW; AbCip
|
||||
RMW hard-codes DINT while `AbCipTagPath` accepts `.N` on any parent
|
||||
(`AbCipTagPath.cs:17-19`); TwinCAT RMW reads the parent as 4-byte `UDInt`
|
||||
(`AdsTwinCATClient.cs:96-107`), an unverified assumption that will fail
|
||||
`DeviceInvalidSize` on WORD/BYTE parents. Fix: make direct parent writes take the same
|
||||
per-parent/per-runtime lock (AbCip's is subsumed by A4's runtime lock); width-aware RMW
|
||||
for TwinCAT (read the parent as its actual declared width). **Effort M, risk Medium
|
||||
(TwinCAT width needs real-hardware verification — flag under UNDER-7).**
|
||||
- **STAB-12** (Medium): FOCAS sync `Dispose()` does a close-PDU exchange (`SendPdu`+
|
||||
`ReadPdu`) on a `NetworkStream` with no `ReadTimeout`/token
|
||||
(`FocasWireClient.cs:164-175`) — a stalled CNC wedges `ShutdownAsync`. TwinCAT
|
||||
`ConnectAsync` calls the blocking SDK `Connect` ignoring token+timeout
|
||||
(`AdsTwinCATClient.cs:73-80`). Fix: bound the FOCAS close read with a `ReadTimeout`;
|
||||
run TwinCAT `Connect` on a worker with the timeout/token honoured (or set
|
||||
`_client.Timeout` and wrap in `Task.Run` + `WaitAsync(ct)`). **Effort S, risk low.**
|
||||
|
||||
## C5. Perf Lows (PERF-6, PERF-7)
|
||||
- **PERF-6** (Low): TwinCAT re-parses `TwinCATSymbolPath` per call
|
||||
(`TwinCATDriver.cs:220,284,468`) — cache like S7's `_parsedByName`; Modbus fresh
|
||||
`DriverHealth` per read (`:285`); AbCip `BuildArray<T>` boxes per element. None urgent.
|
||||
- **PERF-7** (Low): deadband + `WriteOnChangeOnly` exist only in Modbus
|
||||
(`ModbusDriver.cs:136-159`); generalize into/beside `PollGroupEngine` (driver-agnostic
|
||||
publish suppression) so noisy analog signals on the other poll drivers don't publish
|
||||
every jitter. Natural PollGroupEngine-v2 add-on (B1). **Effort M, risk low.**
|
||||
|
||||
## C6. Underdeveloped Mediums (UNDER-4, UNDER-5, UNDER-7, UNDER-8)
|
||||
- **UNDER-4** (Medium, S7): `S7TagDto` has no `ArrayCount` and `BuildTag` never sets it
|
||||
(`S7DriverFactoryExtensions.cs:80-90,137-151`) → driver-config array tags silently
|
||||
scalar (arrays only work as equipment tags, `S7EquipmentTagParser.cs:73-87`); array
|
||||
writes throw `NotSupportedException` (`S7Driver.cs:1014-1017`); `UInt32` surfaces
|
||||
lossily as `Int32` (`:1149-1152`). Fix: add `ArrayCount` to the DTO + `BuildTag`;
|
||||
implement array writes; map `UInt32`→`UInt32`/`Int64`. **Effort M.**
|
||||
- **UNDER-5** (Medium, AbCip): `AcknowledgeAsync` fire-and-forgets write results
|
||||
(`AbCipAlarmProjection.cs:149`) → a failed ack is invisible to operator/health/Part 9
|
||||
caller; with `EnableControllerBrowse` on, one unreachable PLC faults the entire
|
||||
multi-device `DiscoverAsync` (uncaught `await foreach`, `AbCipDriver.cs:970-1037`);
|
||||
browsed tags default writable/Operate (`CipSymbolObjectDecoder.cs:91`). Fix: surface
|
||||
ack outcomes to health + the caller; per-device try/catch in discovery so one dead PLC
|
||||
doesn't fault all; default browsed tags read-only. **Effort M.**
|
||||
- **UNDER-7** (Medium): aggregate the honest "unverified against real hardware" blocks
|
||||
into a **"needs bench time" checklist** — TwinCAT array-read shape
|
||||
(`AdsTwinCATClient.cs:109-116`), Flat-mode `SubSymbols` population (`:263-277`), bit-RMW
|
||||
width (STAB-11); AbLegacy array-decode's five unproven layout assumptions
|
||||
(`LibplctagLegacyTagRuntime.cs:64-88`); FOCAS `cnc_getfigure` id + servo-load scaling
|
||||
(`FocasWireClient.cs:391-395,943-947`). Deliverable: a doc checklist, not code.
|
||||
**Effort S (doc).**
|
||||
- **UNDER-8** (Medium): test coverage is codec-heavy, failure-path-light. The concrete
|
||||
gaps this plan closes with new tests: S7 reconnect (A1), TwinCAT replay (A2), AbCip
|
||||
concurrency (A4), FOCAS cache concurrency (A5), Modbus desync (A3), shared
|
||||
parser-strictness conformance (B2). Also: `Cli.Common.DriverCommandBase` has no direct
|
||||
tests — add a small suite.
|
||||
|
||||
## C7. Lows — hygiene (CONV-6, CONV-7, CONV-8)
|
||||
- **CONV-6** (Low/Medium): `Driver.OpcUaClient.Contracts` drags the full OPC UA SDK
|
||||
because `NamespaceMap` lives there (`OpcUaClient.Contracts.csproj:9`
|
||||
`OPCFoundation.NetStandard.Opc.Ua.Client`); every options-DTO consumer transitively
|
||||
loads the SDK. This is the previously-deferred OpcUaClient.Contracts-002 finding — move
|
||||
`NamespaceMap` into the driver project (or a new `.Namespace` project), leaving Contracts
|
||||
POCO-only like the other six. **Effort M (new project boundary), risk low.**
|
||||
- **CONV-7** (Low/Medium): CLI divergences — `ParseValue`/`ParseBool` copy-pasted six
|
||||
times (`*/Commands/WriteCommand.cs`); option `Validate()` absent in AbLegacy
|
||||
(`AbLegacyCommandBase.cs` — `--timeout-ms 0` reaches the driver); `Timeout` init-setter
|
||||
throws in AbCip (`AbCipCommandBase.cs:43`) but no-ops in five others; only TwinCAT
|
||||
exposes `browse` though AbCip supports controller browse
|
||||
(`AbCipCommandBase.cs:63` hardcodes `EnableControllerBrowse=false`); OpcUaClient has no
|
||||
CLI. Fix: hoist value parsing + a standard `Validate()` into `Cli.Common`; add an
|
||||
OpcUaClient CLI or document the gap. **Effort M, risk low.**
|
||||
- **CONV-8** (Low): dead/no-op knobs + stale docs — `DisableFC23` no-op
|
||||
(`ModbusDriverOptions.cs:83-89`); AbCip `ConnectionSize` plumbed-never-applied
|
||||
(`:97-103`); AbLegacy class doc says read/write "ship in PRs 2 and 3"
|
||||
(`AbLegacyDriver.cs:9-11`); OpcUaClient header says "PR 66 ships the scaffold: IDriver
|
||||
only" (`:12-14`). No options class has `Validate()`. Fix: delete/wire the dead knobs;
|
||||
refresh the stale headers; optionally add per-driver options `Validate()` (S7's init
|
||||
guards `:314-385` are the best-of-breed template). **Effort S, risk very low.**
|
||||
|
||||
---
|
||||
|
||||
# Sequencing & effort roll-up
|
||||
|
||||
Aligns with the report's "Recommended remediation order" and OVERALL actions #3/#4.
|
||||
|
||||
| # | Item | Findings | Effort | Risk | Notes |
|
||||
|---|---|---|---|---|---|
|
||||
| 1 | S7 reconnect (`IS7Plc` seam + `EnsureConnected`) + reconnect test | STAB-1, UNDER-8 | M | Med | Critical; template = TwinCAT/OpcUaClient |
|
||||
| 2 | TwinCAT native-sub replay on client swap + probe recycle | STAB-2 | M | Med | Critical; fake-client unit tests are the only automated coverage |
|
||||
| 3 | AbCip per-runtime op lock + FOCAS ConcurrentDictionary caches | STAB-4, STAB-5 | S each | Low | Silent-corruption class; port AbLegacy's `GetRuntimeLock` |
|
||||
| 4 | Modbus timeout/framing ⇒ connection teardown | STAB-3 | S | Low-Med | Distinguish timeout vs caller-cancel |
|
||||
| 5 | Shutdown/unsub race fixes | STAB-6 | S | Low | Deleted for S7 by item 8 |
|
||||
| 6 | FOCAS writable-false; OpcUaClient dead gate + alarm-filter encoding | UNDER-1, UNDER-2, UNDER-3 | S each | Low | Kill false capabilities |
|
||||
| 7 | Factory `ILoggerFactory` on 4 drivers | CONV-3 | S | V.Low | Unblocks item 1/2/8 logging |
|
||||
| 8 | **PollGroupEngine v2**: backoff + onError-health + migrate S7 off fork | CONV-1, STAB-8, STAB-9, PERF-3, STAB-6-S7 | M | Med | Single home for poll fixes |
|
||||
| 9 | **Shared strict `ReadEnum`/`ReadBool` + probe parity + writable parity** | CONV-2, UNDER-6 | M | Med | Behavior change: typo ⇒ Bad; live-verify |
|
||||
| 10 | **`ResolveHost` via resolver** (per-host breaker isolation) | CONV-4 | S-M | Low | Flagship equipment-tag path |
|
||||
| 11 | Shared reconnect/backoff primitive for lazy drivers | STAB-8 | M | Low-Med | Extract `ConnectionBackoff`; don't delay recovery |
|
||||
| 12 | Per-protocol batching: AbCip planner⇄template-cache, TwinCAT sum-read, S7 multi-var | PERF-1, PERF-2 | L | Med | Biggest scalability lever; AbCip design in-tree |
|
||||
| 13 | OpcUaClient gate-scoping + chunk discovery + monolith split | PERF-4, PERF-5, UNDER-9 | M | Med | Preserve session-swap safety |
|
||||
| 14 | Remaining mediums: evict-on-fail, RMW parent lock/width, blocking teardown | STAB-10, STAB-11, STAB-12 | M | Med | TwinCAT width needs bench time |
|
||||
| 15 | Underdeveloped: S7 arrays/UInt32, AbCip acks/discovery, bench checklist | UNDER-4, UNDER-5, UNDER-7, UNDER-8 | M | Low-Med | |
|
||||
| 16 | Lows: Contracts SDK split, CLI hoist, dead knobs/docs, perf lows | CONV-6/7/8, PERF-6/7 | M | Low | |
|
||||
|
||||
**Where shared primitives live:** `Core.Abstractions` — `PollGroupEngine` (v2 with
|
||||
backoff, item 8), `ConnectionBackoff` (item 11, shared with the engine),
|
||||
`TagConfigJson.ReadEnum/ReadBool` (item 9, beside `EquipmentTagRefResolver`), and the
|
||||
`ResolveHost`-via-resolver pattern (item 10). This is the concrete answer to themes #2
|
||||
and #6: fleet fixes get one home instead of six copies.
|
||||
|
||||
**Live-verify recipes (docker rig on `10.100.0.35`):** Modbus `:5020` +
|
||||
`exception_injector` for STAB-3 (item 4) and write-failure paths; S7 `:1102`
|
||||
(`Snap7ServerFixture`, bounce via `lmxopcua-fix down/up s7 s7_1500`) for STAB-1 (item 1);
|
||||
AbCip `:44818` for STAB-4 concurrency + PERF-2 UDT (items 3, 12). TwinCAT has **no docker
|
||||
fixture** (needs a real TC3 XAR) — the fake-`ITwinCATClient` unit tests are authoritative
|
||||
for STAB-2/item 2.
|
||||
@@ -0,0 +1,353 @@
|
||||
# Plan 06 — Gateway Integrations (Galaxy Driver + Historian Gateway Driver)
|
||||
|
||||
- **Source report:** `archreview/06-gateway-integrations.md`
|
||||
- **Scope:** `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Galaxy` (+ `.Contracts`, `.Browser`), `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway`
|
||||
- **Commit reviewed against:** current tree (report cut at `9cad9ed0`)
|
||||
- **Verification method:** every cited file:line opened and checked against the current tree.
|
||||
|
||||
## Verification summary
|
||||
|
||||
| Finding | Sev | Status | Note |
|
||||
|---|---|---|---|
|
||||
| S-1 Galaxy write optimistic-Good | High | **Confirmed** | `GatewayGalaxyDataWriter.cs` 234-243 (advise-fail proceeds), 281-302 (empty statuses → Good), 162-165 (silent-loss comment) all present verbatim. |
|
||||
| S-2 EventPump drops newest | Medium | **Confirmed** | `EventPump.cs:136-139` `TryWrite` → count drop, newest-dropped by construction. |
|
||||
| S-3 Read can hang on non-cancellable token | Medium | **Confirmed** | `GalaxyDriver.cs:738-753`: BadTimeout fill only via `cancellationToken.Register`; awaits at 753 block forever with `CancellationToken.None`. |
|
||||
| S-4 Transport-fault detection is EventPump-only | Medium | **Confirmed** | `ReportTransportFailure` sole prod caller is `OnEventPumpStreamFault` (`GalaxyDriver.cs:943`); `WriteAsync` (line 803) and subscribe path never feed the supervisor. |
|
||||
| S-5 Outbox `RemoveAsync` prefix truncation silent | Medium | **Confirmed** | `FasterLogHistorizationOutbox.cs:158-182`: removes target + all older live entries, no `_droppedCount` bump on non-target removals. |
|
||||
| S-6 Historian health flags dormant | Medium | **Confirmed (worse)** | `RefreshConnectionStateAsync` has only test callers; **`GetHealthSnapshot()` itself has zero production callers** — the whole snapshot is currently unconsumed in-process. |
|
||||
| S-7 Alarm feed reconnect no backoff | Low | **Confirmed** | `GatewayGalaxyAlarmFeed.cs:141` fixed `_reconnectDelay` (default 5 s), no exponential/jitter. |
|
||||
| S-8 `GalaxyMxSession` unsynchronized state | Low | **Confirmed** | `GalaxyMxSession.cs:28-32` plain fields; `RecreateAsync` (112-117) tears down without a lock. Emergent safety accepted. |
|
||||
| S-9 Store-and-forward discipline | Positive | Confirmed positive | No action. |
|
||||
| S-10 TLS / API-key posture | Positive | Confirmed positive | No action. |
|
||||
| P-1 Read = 3 round-trips | Medium | **Confirmed** | `ReadViaSubscribeOnceAsync` 658-780; MxAccess-forced. Doc + optional cache. |
|
||||
| P-2 Serial `SendEvent` per event | Medium | Confirmed (accepted) | Gateway-constrained; track batched `SendEvents` on sidecar. |
|
||||
| P-3 Channel count | Low | Confirmed (accepted) | Documented trade-off; no action. |
|
||||
| P-4 Indexed fan-in/out | Positive | Confirmed positive | No action. |
|
||||
| P-5 HistoryRead buffers; `maxEvents<=0` unbounded | Low | **Confirmed** | `GatewayHistorianDataSource.cs:142-168` `hasCap = maxEvents>0`; when false collects unbounded. |
|
||||
| P-6 Peek holds lock over disk I/O | Low | **Confirmed** | `FasterLogHistorizationOutbox.cs:145-152`. Negligible at 64-batch. |
|
||||
| C-1 Secret resolver | Positive | Confirmed positive | No action. |
|
||||
| C-2 Seam quality | Positive | Confirmed positive | No action. |
|
||||
| C-3 Historian seam leaks wire types | Medium | **Confirmed** | `IHistorianGatewayClient.cs:1-11` proto-typed by design. Doc-guard only. |
|
||||
| C-4 Options-validation idioms diverge | Low | **Confirmed** | Galaxy throw-on-construct vs Historian warnings-list. Convention note. |
|
||||
| C-5 Retired Wonderware dirs on disk | Low | **Confirmed** | 3 dirs present, **0** `slnx` references. Delete. |
|
||||
| C-6 EventPump drops unknown families unmetered | Low | **Confirmed** | `EventPump.cs:199-206` `default: return;` no counter. |
|
||||
| U-1 CLAUDE.md KNOWN LIMITATION 2 stale | High (doc) | **Confirmed stale** | Feed fully wired: `AddressSpaceApplier.cs:343` `UpdateHistorizedRefs` → `ActorHistorizedTagSubscriptionSink` → recorder `OnUpdateHistorizedRefs` (`ContinuousHistorizationRecorder.cs:186/311`). Doc must be corrected; path never live-verified. |
|
||||
| U-2 Live-validation gate partially run | Medium | Confirmed | Infra complete; needs a full documented run + CLAUDE.md status. |
|
||||
| U-3 WriteSecured user id stubbed at 0 | Medium | **Confirmed** | `GatewayGalaxyDataWriter.cs:263-264` `CurrentUserId=0, VerifierUserId=0`. |
|
||||
| U-4 Dormant paths inventory | Medium | **Confirmed** | `HistorianGatewayClientAdapter.cs:88` drops `maxEvents` on the wire; `GalaxyDriver.cs:314-316` replay note; `FlushOptionalCachesAsync` no-op. |
|
||||
| U-5 Outbox serializer has no version byte | Low | **Confirmed** | `HistorizationOutboxEntrySerializer.cs:14-16` fixed positional layout, no prefix. |
|
||||
| U-6 Test-coverage thin spots | — | Confirmed | S-1 failure branch, S-5 out-of-order, S-3 hang, C-6 filter all untested. Folded into each finding's test section. |
|
||||
|
||||
**Totals: 28 findings — 23 actionable, 5 positive (no action). Of the 23 actionable, all 23 confirmed against the current tree; 0 stale-as-already-fixed (U-1 is stale *documentation* against correct code).**
|
||||
|
||||
Cross-cutting alignment with `00-OVERALL.md`: S-1 is part of OVERALL action #7 (failure visibility); S-6 + U-1 are instances of OVERALL theme #1 ("built-but-never-wired"); U-1 + U-2 are OVERALL theme #5 (doc drift); U-4/C-3 are theme #4 (gateway-gap workarounds accrue client-side).
|
||||
|
||||
---
|
||||
|
||||
## Priority-ordered plan
|
||||
|
||||
Priority key: **P0** ship first (data-integrity / visibility), **P1** next (fault resilience), **P2** hygiene/docs, **P3** nice-to-have.
|
||||
|
||||
---
|
||||
|
||||
### P0-A — S-1: Galaxy write success is optimistic; a committed-write failure can never surface
|
||||
|
||||
**Restatement:** Empty MX status arrays translate to `Good`; a failed supervisory-advise logs a warning and lets the raw `Write` proceed even though the file's own comment says such a write "never reaches the galaxy"; write-outcome self-correction (#5) structurally cannot fire because Galaxy `ExecuteWrite` is fire-and-forget.
|
||||
|
||||
**Verification:** Confirmed. `GatewayGalaxyDataWriter.cs`:
|
||||
- 234-243: on `AdviseSupervisory` protocol failure → `TryRemove` the handle, log warning, **fall through** to `WriteRawAsync`.
|
||||
- 281-302: `TranslateReply` → protocol-status check, then first MX status row, else `new WriteResult(StatusCodeMap.Good)`.
|
||||
- 162-165: in-source comment confirms a non-advised raw Write "doesn't throw (reply looks OK) but the value never reaches the galaxy."
|
||||
|
||||
**Root cause:** Two independent optimistic defaults. (1) The gateway reply's `Statuses` array reflects *command acceptance at the worker*, not the eventual COM-side `WriteComplete`; an empty array is treated as success. (2) The supervisory-advise precondition is best-effort — when it fails the code still issues a Write it knows won't commit, and reports the Write's (empty/OK) reply as `Good`.
|
||||
|
||||
**Design.**
|
||||
|
||||
*In-repo (this PR) — three layers, cheap and independent:*
|
||||
|
||||
1. **Fail-closed on advise failure.** Change `EnsureSupervisoryAdvisedAsync` to return a bool (advised? true/false). In `WriteOneAsync`, when a non-secured write's advise returns false, **do not issue the raw Write** — return `new WriteResult(StatusCodeMap.BadWaitingForInitialData /* or Uncertain */)`. Rationale: the code already *knows* the value can't commit; sending it and reporting Good is the exact silent-loss the comment describes. Preferred status: `Uncertain_NoCommunicationLastUsableValue` is wrong (implies a read); use `Bad` family so #5 self-correction *could* fire (`BadNotConnected`/`BadWaitingForInitialData`). Decision: return **`StatusCodeMap.BadCommunicationError`** — it is already the writer's "couldn't reach the device" status, so the node-write router / self-correction path treats it consistently with a transport failure. (The report suggests `Uncertain`; `Bad` is stronger and lets self-correction revert the phantom-Good node — align with OVERALL theme #3 "the failure must surface *somewhere*".)
|
||||
|
||||
2. **Empty-statuses is not Good for a data write.** `TranslateReply`: when `reply.Statuses.Count == 0` **and** the reply is a data-write reply, return `StatusCodeMap.UncertainNoCommunicationLastUsableValue`-class or, more defensibly, keep `Good` **only** when the gateway contract guarantees a status row on success. This needs a gateway-contract check (see cross-repo). Interim: keep `Good` but **meter** it (layer 3) so the unconfirmed rate is observable, and add an XML-doc note that empty-array-Good is provisional pending the `WriteComplete` correlation.
|
||||
|
||||
3. **`galaxy.writes.unconfirmed` counter.** Add a `Counter<long>` on the existing `GalaxyTelemetry`/pump meter (`ZB.MOM.WW.OtOpcUa.Driver.Galaxy`), incremented whenever `TranslateReply` returns Good off an *empty* statuses array, and a `galaxy.writes.advise_failed` counter on the advise-failure branch. Gives ops the signal the report asks for.
|
||||
|
||||
*Cross-repo (mxaccessgw) — the real fix, tracked follow-on:* pursue a gateway-side `WriteComplete` correlation so the `MxCommandReply.Statuses` row carries the real COM commit outcome. The `OnWriteComplete` event family already exists in the proto and is currently *filtered out* by `EventPump.Dispatch` (`EventPump.cs:199-206`). Design: gateway correlates the async `WriteComplete` back to the originating command's `ClientCorrelationId` and folds the MX status into the unary reply's `Statuses` before returning (making Write synchronously authoritative), **or** emits `OnWriteComplete` on the event stream keyed by correlation id and the driver awaits it with a deadline. The synchronous-fold option is cleaner for the OPC UA write contract (write is a blocking service call). This is a sibling-repo change; file it on the mxaccessgw backlog and link from the in-source comment at `GatewayGalaxyDataWriter.cs:162-165`.
|
||||
|
||||
**Implementation steps (in-repo):**
|
||||
- `GatewayGalaxyDataWriter.cs`: change `EnsureSupervisoryAdvisedAsync` → `Task<bool>`; branch in `WriteOneAsync` to short-circuit to `BadCommunicationError` when advise failed.
|
||||
- Add `WriteUnconfirmed` / `WriteAdviseFailed` counters (new static `Counter<long>` fields; reuse `EventPump.MeterName` string or a shared `GalaxyTelemetry` meter — check `GalaxyTelemetry.cs` for the existing meter name to avoid a second Meter).
|
||||
- `TranslateReply`: increment `WriteUnconfirmed` on the empty-statuses branch; add doc note.
|
||||
|
||||
**Tests (unit):**
|
||||
- New `GatewayGalaxyDataWriterTests` cases (writer already has an internal ctor + `SeedHandleCachesForTest`): advise-fails → assert returned status is `BadCommunicationError` and **no** `WriteRawAsync` was issued (fake session records calls).
|
||||
- Empty-statuses reply → assert Good returned **and** the unconfirmed counter incremented (via a `MeterListener`).
|
||||
- Protocol-status non-OK → `BadCommunicationError` (regression guard).
|
||||
|
||||
**Tests (live gateway):** extend the skip-gated `GatewayGalaxyLiveReopenAndWriteTests` with a case that forces an advise-failure (or asserts the unconfirmed counter is emitted on a normal write). Recipe:
|
||||
```
|
||||
KEY=$(docker exec otopcua-dev-central-1-1 printenv GALAXY_MXGW_API_KEY)
|
||||
MXGW_ENDPOINT=http://10.100.0.48:5120 GALAXY_MXGW_API_KEY="$KEY" \
|
||||
dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Tests \
|
||||
--filter FullyQualifiedName~GatewayGalaxyLiveReopenAndWrite
|
||||
```
|
||||
|
||||
**Effort:** S (in-repo layers 1+3), M (layer 2 + cross-repo correlation). **Risk/blast radius:** Low in-repo — the change makes a currently-silent-loss visible; the only behavioral change is that advise-failed writes now return Bad instead of phantom-Good, which is strictly more correct and feeds #5 revert. Live-verify to confirm normal writes still return Good.
|
||||
|
||||
---
|
||||
|
||||
### P0-B — S-6: Historian health snapshot's connection flags are dormant (and the snapshot is unconsumed)
|
||||
|
||||
**Restatement:** `RefreshConnectionStateAsync` has no production caller, so `ProcessConnectionOpen`/`EventConnectionOpen` are permanently false; deeper: `GetHealthSnapshot()` itself is never called in production.
|
||||
|
||||
**Verification:** Confirmed and worse than reported. `RefreshConnectionStateAsync` callers = 3 test lines only. `GetHealthSnapshot()` invocation grep across `src` returns only the interface declaration — no production consumer. So the entire health surface (`GetHealthSnapshot` + the refresh) is dead weight in a deployed host today.
|
||||
|
||||
**Root cause:** The "register-AND-pass-into-consumer" trap (same class as PR #423 provisioner, OVERALL theme #1): the component was built and unit-tested, but neither the periodic refresh nor a snapshot consumer was wired.
|
||||
|
||||
**Design.** Two decisions:
|
||||
|
||||
1. **Is the snapshot meant to be surfaced?** Check `HistorianHealthSnapshot`'s intended consumer (AdminUI historian-status panel / a health endpoint). If yes → wire it. If no (the flags predate a since-abandoned dashboard) → **remove** `ProcessConnectionOpen`/`EventConnectionOpen` + `RefreshConnectionStateAsync` to kill the misleading-false surface. The report offers both; pick based on whether any dashboard is planned. Recommendation: **wire it**, because the other integrations (Galaxy `HostStatusAggregator`) *do* surface connectivity and a historian-down indicator is genuinely useful.
|
||||
|
||||
2. **How to drive the refresh.** Add a lightweight `IHostedService` (`HistorianHealthRefreshHostedService`) that, when `ServerHistorian:Enabled`, resolves the registered `IHistorianDataSource`, and if it is a `GatewayHistorianDataSource`, calls `RefreshConnectionStateAsync` on a timer (reuse the existing health-probe cadence if one exists — check `OtOpcUaServerHostedService` for an existing periodic loop to fold into rather than spawning a second timer). Also wire a consumer: project the snapshot into whatever the AdminUI/health endpoint reads.
|
||||
|
||||
**Implementation steps:**
|
||||
- New `HistorianHealthRefreshHostedService` in `ZB.MOM.WW.OtOpcUa.Runtime` (or Host), registered in the historian DI block alongside `AddServerHistorian`. Guard on `Enabled`. Interval from a new `ServerHistorianOptions.HealthRefreshSeconds` (default 15).
|
||||
- Cast/seam: `RefreshConnectionStateAsync` is on the concrete `GatewayHistorianDataSource`, not `IHistorianDataSource`. Either (a) add an optional `IHistorianConnectionProbe { Task RefreshConnectionStateAsync(ct) }` capability interface the gateway source implements and the hosted service checks for, or (b) resolve the concrete type. Prefer (a) — matches the capability-interface pattern used elsewhere and keeps `NullHistorianDataSource` clean.
|
||||
- Wire the snapshot consumer (AdminUI panel or `/health`); if none is desired, execute decision-branch (1) removal instead.
|
||||
|
||||
**Tests:**
|
||||
- Unit: hosted service calls `RefreshConnectionStateAsync` on tick against a fake source; disabled → never resolves.
|
||||
- Unit: `GatewayHealthSnapshotTests` already covers the refresh math — add an assertion that after a successful refresh the snapshot flags are true.
|
||||
- Live: after wiring, run the historian live suite (U-2) and confirm the snapshot flips to connected against the real sidecar.
|
||||
|
||||
**Effort:** S (wire refresh) / M (if adding a consumer surface). **Risk:** Low. **Note:** live-`/run` required per OVERALL theme #1 — do not close on unit tests alone.
|
||||
|
||||
---
|
||||
|
||||
### P0-C — U-1: CLAUDE.md KNOWN LIMITATION 2 is stale; continuous-historization value-capture never live-verified
|
||||
|
||||
**Restatement:** CLAUDE.md says the recorder is spawned with an empty ref-set and "historizes nothing"; the code has closed that gap but the end-to-end path was never live-verified.
|
||||
|
||||
**Verification:** Confirmed. The feed is fully wired: `ServiceCollectionExtensions.cs:272` still spawns with `Array.Empty<string>()`, **but** wraps in `ActorHistorizedTagSubscriptionSink` (line 290-292) and hands it to the applier; `AddressSpaceApplier.cs:310-343` `FeedHistorizedRefs` computes the per-deploy add/remove delta and calls `_historizedSubscriptions.UpdateHistorizedRefs`; `ContinuousHistorizationRecorder.cs:186/311` receives `UpdateHistorizedRefs` and re-registers. CLAUDE.md's "Known Limitation 2" text is therefore factually wrong.
|
||||
|
||||
**Root cause:** Documentation drift — the feed landed after the CLAUDE.md note was written and the note was never reconciled.
|
||||
|
||||
**Design/steps:**
|
||||
1. **Doc fix (this PR):** rewrite CLAUDE.md "KNOWN LIMITATION 2" to state the ref-feed is wired (applier `FeedHistorizedRefs` → `ActorHistorizedTagSubscriptionSink.UpdateHistorizedRefs` → recorder `OnUpdateHistorizedRefs`), and downgrade it from "records no values" to "wired; pending a live end-to-end verification." Propagate the same correction to the `../scadaproj/CLAUDE.md` OtOpcUa index entry (per the repo's cross-repo propagation rule).
|
||||
2. **Live-verify the value-capture path (the real close-out):** deploy a config with a historized numeric tag against a real HistorianGateway (VPN to the sidecar / `wonder-sql-vd03`, gateway with `RuntimeDb:Enabled=true`), drive a value change, and confirm `WriteLiveValues` lands (outbox drains, gateway shows the sample). This is the OVERALL theme #1 "prove production wiring" gate.
|
||||
3. **Add a restart-convergence integration test** (unit-level with a fake gateway value-writer + real outbox): deploy delta adds refs → mux value → outbox append → drain calls writer; assert the writer received the value. This closes the "wired but inert" risk with a repeatable test even before the live run.
|
||||
|
||||
**Effort:** S (doc) + M (live-verify + convergence test). **Risk:** Doc-only change is zero-risk; the live run may surface a real inertness bug (exactly what OVERALL warns about) — budget for a fix if it does.
|
||||
|
||||
---
|
||||
|
||||
### P1-A — S-4: Transport-failure detection is EventPump-only; a write-only/idle driver never degrades
|
||||
|
||||
**Restatement:** `ReportTransportFailure` is only called from the EventPump stream-fault callback; a driver that only writes or is idle after discovery keeps `GetHealth()==Healthy` through a gateway restart, and each write fails per-request forever with no reopen/replay.
|
||||
|
||||
**Verification:** Confirmed. Sole prod caller `GalaxyDriver.cs:943` (`OnEventPumpStreamFault`). `WriteAsync` (803) delegates straight to `_dataWriter.WriteAsync` with no supervisor hand-off; failed unary RPCs (SubscribeBulk, Write, AcknowledgeAlarm) don't feed the supervisor.
|
||||
|
||||
**Root cause:** Fault detection was built around the long-lived StreamEvents pump only; unary RPC failures were treated as per-call and never escalated to the recovery state machine.
|
||||
|
||||
**Design.** `ReportTransportFailure` is idempotent by design (the supervisor single-flights recovery), so reporting from more sites is cheap and safe. Add classified transport-failure reporting from the unary paths:
|
||||
- In `GatewayGalaxyDataWriter.WriteOneAsync`'s catch (currently maps to `BadCommunicationError`), and in the subscribe path, classify the exception (RpcException with transport status codes: Unavailable, DeadlineExceeded, Internal, etc.) and, when transport-fatal, invoke a supplied `Action<Exception>? reportTransportFailure` callback (mirror the pump's `onStreamFault` pattern) that routes to `_supervisor.ReportTransportFailure`.
|
||||
- Wire the callback from `GalaxyDriver` when it constructs the writer/subscriber, guarded like `OnEventPumpStreamFault` (null supervisor = skeleton path, no-op).
|
||||
- Reuse the existing transport-classification helper if one exists (grep `StatusCodeMap`/`ReconnectSupervisor` for a classifier); otherwise add a small `IsTransportFatal(Exception)` in the driver.
|
||||
|
||||
**Implementation steps:**
|
||||
- Add a `Func<Exception,bool>`/callback seam to `GatewayGalaxyDataWriter` (optional ctor arg, like `subscribedHandleSource`) or wrap the call at the `GalaxyDriver.WriteAsync` boundary (cleaner — keep the writer pure). Prefer wrapping at `GalaxyDriver`: catch transport-classified failures returned/thrown from the writer and call `supervisor.ReportTransportFailure`.
|
||||
- Same for `SubscribeAsync`/`ReadViaSubscribeOnceAsync` SubscribeBulk failures and `GatewayGalaxyAlarmAcknowledger`.
|
||||
|
||||
**Tests (unit):** fake writer that throws an Unavailable RpcException → assert `GalaxyDriver.WriteAsync` reports the failure to a fake supervisor and returns Bad; non-transport ArgumentException → not reported. Idempotency: two failures → supervisor sees N reports but single-flights (already covered by `ReconnectSupervisorTests` — add a driver-level report test).
|
||||
|
||||
**Effort:** M. **Risk:** Low-Medium — must avoid escalating *application-level* write rejections (bad value, secured-tag denial) to transport recovery; the classifier must be conservative (only RpcException transport codes).
|
||||
|
||||
---
|
||||
|
||||
### P1-B — S-3: `ReadViaSubscribeOnceAsync` can wait forever on a non-cancellable token
|
||||
|
||||
**Restatement:** BadTimeout fill is only via `cancellationToken.Register`; a `CancellationToken.None` caller whose tag subscribes OK but never publishes hangs at the await, holding the subscription open.
|
||||
|
||||
**Verification:** Confirmed. `GalaxyDriver.cs:738-748` registers the fill on the CT; 750-755 awaits each pending TCS unconditionally. No internal deadline.
|
||||
|
||||
**Root cause:** The read deadline is delegated entirely to the caller's token; an infinite/None token has no fallback.
|
||||
|
||||
**Design.** Race each pending await against an internal deadline derived from options regardless of the caller's token. Compute `deadline = max(PublishingIntervalMs, DefaultCallTimeoutSeconds)` × a small factor (e.g. 3× publishing interval, floored at the call timeout). Implement by combining tokens: create a linked CTS `CreateLinkedTokenSource(cancellationToken)` + `CancelAfter(deadline)`, and register the BadTimeout fill on the *linked* token instead of the caller's. Then every path (caller cancel, internal deadline) fires the fill. Ensure the linked CTS is disposed in the finally.
|
||||
|
||||
**Implementation steps:**
|
||||
- In `ReadViaSubscribeOnceAsync`: build `using var deadlineCts = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken); deadlineCts.CancelAfter(TimeSpan.FromMilliseconds(readDeadlineMs));` and change `cancellationToken.Register(...)` → `deadlineCts.Token.Register(...)`.
|
||||
- Derive `readDeadlineMs` from `_options.MxAccess.PublishingIntervalMs` and `_options.Gateway.DefaultCallTimeoutSeconds` (or equivalent). Add an option `ReadSynthesisTimeoutMs` if a dedicated knob is warranted; default derived.
|
||||
|
||||
**Tests (unit):** the driver has injectable seams (`IGalaxySubscriber` fake). Fake that returns a successful SubscribeBulk but never raises OnDataChange → call `ReadViaSubscribeOnceAsync` with `CancellationToken.None` → assert it returns BadTimeout snapshots within the deadline (not hang) and that UnsubscribeBulk was called (subscription not leaked).
|
||||
|
||||
**Effort:** S. **Risk:** Low — pure addition of an internal deadline; existing caller-cancel behavior preserved via the linked token.
|
||||
|
||||
---
|
||||
|
||||
### P1-C — S-5: FasterLog outbox `RemoveAsync` truncates the FIFO prefix; out-of-order acks silently drop unacked entries
|
||||
|
||||
**Restatement:** `RemoveAsync(id)` removes the target plus every older live entry and truncates to the target's successor; if a mid-batch id is acked first, older unacked values are durably discarded without incrementing `DroppedCount`.
|
||||
|
||||
**Verification:** Confirmed. `FasterLogHistorizationOutbox.cs:158-182`: while-loop removes `_live.First` repeatedly until it hits the target and breaks; `_droppedCount` not touched for the collateral removals. Contract "acks in FIFO order" enforced by comment only.
|
||||
|
||||
**Root cause:** FasterLog is a truncate-to-address log; removing an arbitrary id necessarily discards everything before it. The API accepted a single id while the underlying primitive is prefix-truncation, so the invariant lives only in the caller's discipline.
|
||||
|
||||
**Design.** Two options:
|
||||
1. **Count + warn (minimal):** in `RemoveAsync`, when the removal drops entries whose id != the target, increment `_droppedCount` per collateral entry and log a warning naming the count. Makes the silent loss observable. Cheap, no API change.
|
||||
2. **Make semantics explicit at the call site (preferred):** add `RemoveThroughAsync(Guid throughId)` (or `RemoveBatchAsync(IReadOnlyList<Guid>)`) to `IHistorizationOutbox`, documenting that it truncates the FIFO prefix through the id; keep single-id `RemoveAsync` but have it assert the target is `_live.First` (FIFO) and throw/log if not. This surfaces the invariant at compile/call time.
|
||||
|
||||
Recommendation: **do both** — rename the operation to `RemoveThroughAsync` to match reality, and add the collateral-drop counter as a safety net for any future non-FIFO caller.
|
||||
|
||||
**Implementation steps:**
|
||||
- `IHistorizationOutbox`: add `RemoveThroughAsync` (or annotate `RemoveAsync` contract). Update `ContinuousHistorizationRecorder` drain call site to use it.
|
||||
- `FasterLogHistorizationOutbox.RemoveAsync`: count collateral removals into `_droppedCount` (or a new `_ackPrefixDropped` counter for clarity) and `_logger.LogWarning` when > 0.
|
||||
|
||||
**Tests (unit):** append A,B,C; `RemoveAsync(C.Id)` (skipping A,B) → assert A,B are gone, `DroppedCount == 2` (or the new counter), and a warning was logged. Regression: FIFO ack A,B,C in order → `DroppedCount == 0`.
|
||||
|
||||
**Effort:** S. **Risk:** Low — additive counter; API rename is mechanical (single recorder caller).
|
||||
|
||||
---
|
||||
|
||||
### P1-D — S-7: Alarm feed reconnect has no backoff
|
||||
|
||||
**Restatement:** `GatewayGalaxyAlarmFeed.RunAsync` re-opens on a fixed 5 s delay forever; `DeployWatcher` and `ReconnectSupervisor` both use capped exponential + jitter.
|
||||
|
||||
**Verification:** Confirmed. `GatewayGalaxyAlarmFeed.cs:141` `await Task.Delay(_reconnectDelay, ct)` with a fixed `_reconnectDelay`.
|
||||
|
||||
**Root cause:** Inconsistency — the alarm feed predates or diverged from the shared backoff pattern.
|
||||
|
||||
**Design.** Replace the fixed delay with capped exponential backoff + jitter, resetting to the floor on a successful stream open (i.e., reset the attempt counter after the first message or after the `snapshot_complete`). Reuse the existing backoff helper `DeployWatcher`/`ReconnectSupervisor` use — grep for a shared `ExponentialBackoff`/`Jitter` type; if none is shared, extract one into the Galaxy `Runtime/` folder and use it in all three (aligns with OVERALL theme #2 "fixes flow to shared templates").
|
||||
|
||||
**Implementation steps:**
|
||||
- Introduce/reuse a `Backoff` primitive (min, max, jitter). Wire into `GatewayGalaxyAlarmFeed` ctor (min/max from `GalaxyReconnectOptions`, defaults 1 s→30 s). Reset on successful open.
|
||||
|
||||
**Tests (unit):** the feed uses an injected `_streamFactory` and a settable delay; assert successive faults produce increasing delays capped at max and reset after a successful open (drive via a fake factory + a virtual clock or by asserting the computed delay sequence).
|
||||
|
||||
**Effort:** S. **Risk:** Low.
|
||||
|
||||
---
|
||||
|
||||
### P2-A — C-5: delete retired Wonderware project directories
|
||||
|
||||
**Verification:** Confirmed — `Historian.Wonderware`, `.Client`, `.Client.Contracts` dirs present; `grep -c Wonderware slnx == 0`.
|
||||
|
||||
**Design/steps:** `git rm -r` the three `src/Drivers/*Wonderware*` dirs and their `tests/` counterparts; confirm no `ProjectReference`/`using` survives (grep). This overlaps OVERALL action #11 and Report 07 hygiene — coordinate so it's deleted once. History preserves the code.
|
||||
|
||||
**Effort:** S. **Risk:** Very low (0 references). Verify with a full `dotnet build ZB.MOM.WW.OtOpcUa.slnx` after removal.
|
||||
|
||||
---
|
||||
|
||||
### P2-B — C-6: EventPump silently drops unknown event families with no metric
|
||||
|
||||
**Verification:** Confirmed. `EventPump.cs:199-206` `default: return;` — no counter.
|
||||
|
||||
**Design/steps:** add a `galaxy.events.filtered` (or `.unknown_family`) counter incremented in the `default` branch of `Dispatch`, tagged with the family value. Mirrors the alarm feed's decode-drop metering. Keep the filtering behavior — only add observability, so a future gateway family (`OnBufferedDataChange`) surfaces instead of vanishing.
|
||||
|
||||
**Tests (unit):** feed the pump an `MxEvent` with a non-`OnDataChange` family via the injected subscriber fake → assert the filtered counter incremented (MeterListener) and no `OnDataChange` fired.
|
||||
|
||||
**Effort:** S. **Risk:** Very low.
|
||||
|
||||
---
|
||||
|
||||
### P2-C — U-3: WriteSecured / VerifiedWrite user identity stubbed at zero
|
||||
|
||||
**Verification:** Confirmed. `GatewayGalaxyDataWriter.cs:263-264` hardcodes `CurrentUserId=0, VerifierUserId=0`.
|
||||
|
||||
**Root cause:** No mapping from the OPC UA session's LDAP-authenticated principal to an ArchestrA user id; the secured-write path was stubbed to unblock the common (non-secured, supervisory) path.
|
||||
|
||||
**Design.** Two-part, and honest about the gap:
|
||||
1. **Fail-fast interim (this PR):** when a tag classifies `SecuredWrite`/`VerifiedWrite` and no ArchestrA user-id mapping is configured, return a clear Bad status (`BadUserAccessDenied` / `BadNotSupported`) and log, rather than silently sending user 0 (which a real galaxy will reject or mis-attribute). Prevents a confusing "commit succeeded as user 0" or opaque rejection. Also surfaces to the operator that secured-classification tags are unsupported today.
|
||||
2. **Real fix (cross-cutting, larger):** thread the OPC UA session identity down to the writer. The LDAP principal is available server-side (see `RoleCarryingUserIdentity` used by alarm ack). Add an ArchestrA-user-id mapping (config `GroupToArchestraUserId` or a resolver) and pass `CurrentUserId`/`VerifierUserId` into `InvokeWriteSecuredAsync`. This spans the node-write router → driver `WriteRequest` (needs a user-id/identity field on the request contract) → writer. Coordinate with Report 03 (session identity) and the node-write router. VerifiedWrite additionally needs a *second* verifier identity (dual-authorization) — that is a genuine feature, not a plumbing fix; scope it separately.
|
||||
|
||||
**Implementation steps (interim):**
|
||||
- `WriteOneAsync`: when `NeedsSecuredWrite(classification)` and no user-id source is wired → return `BadNotSupported`/`BadUserAccessDenied`, log, do not send. Document in `docs/` (Galaxy driver / Historian split) that secured/verified writes require the identity-mapping feature.
|
||||
|
||||
**Tests (unit):** classification resolver returns SecuredWrite + no identity → assert Bad status and no `InvokeAsync` call.
|
||||
|
||||
**Effort:** S (interim), L (real identity threading — cross-report). **Risk:** interim is Low and strictly safer than sending user 0.
|
||||
|
||||
---
|
||||
|
||||
### P2-D — U-4 / P-5: gateway-gap client-side workarounds; clamp unbounded events read
|
||||
|
||||
**Verification:** Confirmed.
|
||||
- `HistorianGatewayClientAdapter.cs:88` calls `_inner.ReadEventsAsync(startUtc, endUtc, filter, ct)` — `maxEvents` not on the wire (client-side cap only).
|
||||
- `GatewayHistorianDataSource.cs:142-168` — `maxEvents<=0` collects unbounded (only the gateway's `RuntimeDb:EventReadMaxRows` bounds it).
|
||||
- Client-side source filter re-applied defensively at 152-157.
|
||||
- `GalaxyDriver.cs:314-316` replay still per-subscription (batched `ReplaySubscriptionsCommand` note).
|
||||
|
||||
**Design/steps:**
|
||||
1. **In-repo:** clamp `maxEvents<=0` to a server-side default cap (new `ServerHistorianOptions.DefaultMaxEventsPerRead`, default e.g. 10 000) in `GatewayHistorianDataSource.ReadEventsAsync` so a 0 from the SDK can't drain unboundedly. This is the P-5 fix and is fully in-repo.
|
||||
2. **Cross-repo (sidecar backlog):** file tracked items for (a) a wire-level `maxEvents` on `ReadEventsRequest`, (b) a server-side source filter so the defensive client filter can be removed, (c) a batched `SendEvents` (P-2), and (d) a batched `ReplaySubscriptionsCommand` (mxaccessgw). Link each from the corresponding in-source comment. These are *not* in-repo fixes — they belong on HistorianGateway / mxaccessgw backlogs; the driver's workarounds stay until they land.
|
||||
|
||||
**Tests (unit):** `FakeHistorianGatewayClient` streaming > cap events with `maxEvents=0` → assert the data source stops at `DefaultMaxEventsPerRead` and flags truncation.
|
||||
|
||||
**Effort:** S (clamp). **Risk:** Low. **Note:** OVERALL theme #4 — track the gateway gaps on the sister repos so they don't rot.
|
||||
|
||||
---
|
||||
|
||||
### P3-A — S-2: EventPump saturation drops the newest events (inverted staleness bias)
|
||||
|
||||
**Verification:** Confirmed. Newest-dropped by construction (`TryWrite` on a full bounded channel). Metered (`galaxy.events.dropped`).
|
||||
|
||||
**Root cause:** Bounded channel with `FullMode.Wait` + `TryWrite` preserves the oldest queued events and drops the just-arrived one — wrong for last-value-wins telemetry.
|
||||
|
||||
**Design.** Two candidates:
|
||||
1. **Per-item-handle conflation (best correctness):** keep newest-per-handle. Replace the raw `Channel<MxEvent>` with a small conflating structure: a `ConcurrentDictionary<long,MxEvent>` (handle→latest) + a signal channel; the dispatch loop drains the dict. On overflow the newest value for a handle overwrites the older one — no data staleness, bounded by tag count not event rate. More code; changes the pump's core.
|
||||
2. **Drop-oldest ring (cheap):** switch the channel to `BoundedChannelFullMode.DropOldest`. But the in-source comment explicitly rejected `DropWrite`/`DropOldest` because they discard *inside* Channel<T> without surfacing on the counter. To keep metering, wrap: on a full channel, `TryRead` one (count it as dropped-oldest) then `TryWrite` the new one. Simpler than conflation, keeps recent data, preserves the counter.
|
||||
|
||||
Recommendation: **option 2** (drop-oldest-with-metering) as the low-risk win; option 1 only if soak testing shows a hot-handle storm still loses fresh values. Given P-4 notes the fan-out is already O(1) and the channel default is 50 000, saturation is an edge case — option 2 is proportionate.
|
||||
|
||||
**Implementation steps:**
|
||||
- In `RunAsync`, replace the `if (!TryWrite) count` block with: `while (!TryWrite(ev)) { if (TryRead(out _)) EventsDroppedOldest.Add(1,tag); }` (bounded loop). Add a `galaxy.events.dropped_oldest` counter (or reuse `dropped`). Update the FullMode comment to match.
|
||||
|
||||
**Tests (unit):** fill the channel (tiny capacity via ctor), push N more → assert the newest are retained (read the dispatched values) and the dropped counter reflects the discarded oldest.
|
||||
|
||||
**Effort:** S-M. **Risk:** Medium — touches the hot pump loop; guard with the existing pump bounded-channel tests + a soak run. Only pursue if S-2 is judged worth the churn (report rates it Medium; it's already metered, so it's a correctness-under-stress refinement).
|
||||
|
||||
---
|
||||
|
||||
### P3-B — S-8: `GalaxyMxSession` state unsynchronized across reopen (document the invariant)
|
||||
|
||||
**Verification:** Confirmed. Plain `_session`/`_connected` fields; `RecreateAsync` tears down without a lock. Report rates it acceptable (emergent safety via single-flight supervisor + per-call try/catch).
|
||||
|
||||
**Design/steps:** lowest-effort resolution the report endorses — **document the invariant** on the `Session`/`IsConnected` members: "single-flight recovery (ReconnectSupervisor) is the only writer of session state; concurrent readers tolerate a torn read because every per-call failure maps to BadCommunicationError and caches are invalidated post-reopen." If a defensive change is wanted, mark `_connected` `volatile` and read `_session` into a local once per operation (already the pattern in `WriteAsync`). Do **not** add a lock (would serialize the hot path for no proven benefit).
|
||||
|
||||
**Effort:** XS (doc) / S (volatile). **Risk:** None (doc) / Low.
|
||||
|
||||
---
|
||||
|
||||
### P3-C — U-5: outbox serializer has no version/format byte
|
||||
|
||||
**Verification:** Confirmed. `HistorizationOutboxEntrySerializer.cs:14-16` fixed positional layout, no prefix.
|
||||
|
||||
**Design/steps:** prepend a single version byte (`0x01`) in `Serialize`; in `Deserialize` read it first and branch (today: only v1). One reserved byte now makes any future field addition a clean migration instead of an undiagnosable recovery failure mid-`RecoverState`. **Migration caveat:** on-disk entries written before this change have no version byte — deploying this reader against a non-empty pre-existing outbox would misread. Since the outbox is drained-and-truncated continuously and is dev-only so far, gate the change on an empty-outbox deploy, or make the reader tolerant (peek: if the first byte isn't a known version AND the buffer length matches the legacy layout, treat as v0). Prefer the tolerant reader for safety.
|
||||
|
||||
**Tests (unit):** round-trip v1; legacy-layout bytes (no version) still deserialize under the tolerant reader; unknown version → clear exception.
|
||||
|
||||
**Effort:** S. **Risk:** Low-Medium (the on-disk migration is the only sharp edge — the tolerant reader defuses it).
|
||||
|
||||
---
|
||||
|
||||
### P3-D — C-3, C-4: convention notes (documentation only)
|
||||
|
||||
- **C-3:** add a one-line guard note to `IHistorianGatewayClient`'s XML doc: "any consumer beyond `GatewayHistorianDataSource` must sit above the `Mapping/` anti-corruption boundary, never on this proto-typed seam." No code change. **Effort:** XS.
|
||||
- **C-4:** pick one options-validation idiom for *new* driver-adjacent sections — the report (and I) recommend the Historian `Validate()`-returns-warnings style as the friendlier operational contract. Document the choice in CLAUDE.md / a conventions doc; do not churn the existing Galaxy throw-on-construct code. **Effort:** XS (doc).
|
||||
|
||||
---
|
||||
|
||||
### Accepted / no-action
|
||||
|
||||
- **S-9, S-10, P-4, C-1, C-2** — positives; preserve as-is (don't regress the store-and-forward discipline, the O(1) fan-out indexes, or the secret resolver).
|
||||
- **P-1** — MxAccess-forced 3-round-trip read; action is documentation ("subscribe, don't poll") + optional read-through cache only if polling clients appear. Fold the doc line into `docs/` Galaxy section. **Effort:** XS.
|
||||
- **P-2, P-3, P-6** — accepted, gateway-constrained or negligible; P-2 tracked on the sidecar backlog (see U-4).
|
||||
|
||||
---
|
||||
|
||||
## Suggested execution order (batched)
|
||||
|
||||
1. **Batch 1 (P0, failure-visibility):** S-1 (in-repo layers) + S-6 (refresh wiring) + U-1 (doc fix + convergence test). Ship together; all three are OVERALL theme #1/#3. Live-verify S-1 write and U-1 value-capture on the real gateways.
|
||||
2. **Batch 2 (P1, fault resilience):** S-4 + S-3 + S-5 + S-7. Cohesive "recovery hardening" set, all unit-testable with existing fakes.
|
||||
3. **Batch 3 (P2, hygiene/interim):** C-5 (delete dirs) + C-6 (metric) + U-3 (fail-fast) + U-4/P-5 (clamp) + C-3/C-4 (doc). Coordinate C-5 with Reports 03/07.
|
||||
4. **Batch 4 (P3, refinements):** S-2 (drop-oldest) + S-8 (doc/volatile) + U-5 (version byte) + P-1 doc. Optional / as-capacity.
|
||||
|
||||
Cross-repo follow-ons to file (not in-repo): mxaccessgw `WriteComplete` correlation (S-1) and batched `ReplaySubscriptionsCommand` (U-4); HistorianGateway wire-level `maxEvents`, server-side source filter, batched `SendEvents` (U-4/P-2). Link each from the existing in-source comment.
|
||||
@@ -0,0 +1,521 @@
|
||||
# Plan 07 — Client Tooling, Analyzers & Cross-Cutting Engineering System
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| **Source report** | `archreview/07-client-tooling-engineering.md` |
|
||||
| **Overall context** | `archreview/00-OVERALL.md` (theme #4 "verification system", action #6/#11) |
|
||||
| **Base commit** | `9cad9ed0` (master); verified against current working tree 2026-07-08 |
|
||||
| **Domain** | Client.CLI/Shared/UI, Analyzers, build/CPM, CI, test architecture, repo hygiene |
|
||||
|
||||
## Verification summary
|
||||
|
||||
26 findings in the report (S-1..S-8, P-1..P-4, C-1..C-6, U-1..U-8). Every actionable
|
||||
finding was re-verified against the live tree; **all confirmed, none stale**. Four items are
|
||||
explicitly "no-action / positive" in the report (P-3, P-4, C-3, U-1) and are recorded but not
|
||||
planned. Key verifications performed:
|
||||
|
||||
- **C-1 CONFIRMED** — repo-wide grep for `ZB.MOM.WW.OtOpcUa.Analyzers` in `.csproj/.props/.targets`
|
||||
returns only the analyzer's own csproj and its test project's `ProjectReference`. Zero
|
||||
`OutputItemType="Analyzer"` anywhere in the tree (the only hit is inside the review markdown).
|
||||
- **S-1/S-2 CONFIRMED** — `v2-ci.yml:47-52` gates exactly 5 unit projects; slnx contains 47
|
||||
`*.Tests.csproj` (10 of them `*.IntegrationTests`). Integration job (`:61-76`) runs
|
||||
fixture-less on `ubuntu-latest` with no `services:`. Category traits are declared via
|
||||
`[Trait("Category", "…")]`, so a `Category!=E2E&Category!=LiveIntegration` filter is valid.
|
||||
- **S-3 CONFIRMED** — grep for `Category…E2E` across `tests/**/*.cs` returns zero matches; the
|
||||
nightly workflow's own header admits the no-op.
|
||||
- **S-7 CONFIRMED** — no `global.json` at root (nor anywhere); `v2-ci.yml:10-12` claims one exists.
|
||||
- **U-3 CONFIRMED** — `docs/Client.CLI.md` has zero sections for `confirm`, `shelve`, `disable`
|
||||
(and only substring hits for `ack`/`enable`); all 5 alarm commands ship as command files.
|
||||
- **U-4 CONFIRMED** — `git ls-files` shows `pending.md`, `current.md`, `looseends.md`,
|
||||
`stillpending.md`, `HISTORIAN-GATEWAY-INTEGRATION-ISSUES.md` all tracked; `pending.md:3` itself
|
||||
declares "HARD RULE: never stage `pending.md` / `current.md`".
|
||||
- **U-5 CONFIRMED** — `git ls-files lib/` lists 7 tracked proprietary AVEVA DLLs; zero `HintPath`
|
||||
hits reference `lib/` anywhere.
|
||||
- **U-6 CONFIRMED** — `sql_login.txt` is `.gitignore:47`, `git check-ignore` matches, `git log`
|
||||
empty (never committed). The retired `Driver.Historian.Wonderware*` dirs under `src/`+`tests/`
|
||||
are untracked `bin/obj` husks (NOT tracked projects; not in slnx). Tracked Wonderware residue is
|
||||
only `docs/drivers/Historian.Wonderware.md` (an intentional retired-stub) and 3
|
||||
`code-reviews/Driver.Historian.Wonderware*/findings.md`.
|
||||
- **C-2 CONFIRMED** — none of the 4 client/tooling src csprojs set `TreatWarningsAsErrors`.
|
||||
- **C-5 CONFIRMED** — no `.editorconfig`; `StyleGuide.md:3` says "for all **ScadaBridge**
|
||||
documentation".
|
||||
- **S-5/S-6/U-8 CONFIRMED** — `OpcUaClientService.cs` lock gaps at cited lines; `CommandBase.cs:123`
|
||||
`Log.CloseAndFlush()` + `:133` global `Log.Logger` swap; `CommandBase.cs:91`
|
||||
`AutoAcceptCertificates = true`.
|
||||
|
||||
---
|
||||
|
||||
## Priority ordering
|
||||
|
||||
**Tier A — restore meaning to green CI (do first):** C-1, S-1, S-2, S-3, S-7.
|
||||
**Tier B — repo hygiene:** U-5, U-4, U-6, C-5 (StyleGuide title).
|
||||
**Tier C — docs:** U-3.
|
||||
**Tier D — engineering debt (as capacity allows):** C-2, P-1, U-2, S-5, S-6, S-4, C-4, C-6, S-8, U-8, P-2.
|
||||
No-action (recorded only): P-3, P-4, C-3, U-1, U-7.
|
||||
|
||||
---
|
||||
|
||||
## TIER A — Restore meaning to green CI
|
||||
|
||||
### C-1 (High) — Wire the OTOPCUA0001 analyzer into every project
|
||||
|
||||
**Restatement:** the sole custom analyzer enforces its CapabilityInvoker-wrapping rule against
|
||||
nothing but its own 31 unit tests; no consuming project references it.
|
||||
|
||||
**Verification:** confirmed — only self + test `ProjectReference`; zero `OutputItemType="Analyzer"`.
|
||||
|
||||
**Root cause:** the "built-but-never-wired" house failure mode (theme #1). The analyzer project was
|
||||
authored and unit-tested but the DI/build-graph wiring step was never done.
|
||||
|
||||
**Design:** inject the analyzer as an analyzer-type `ProjectReference` from `Directory.Build.props`,
|
||||
conditioned to exclude the analyzer project itself (and its own build, to avoid a self-reference
|
||||
cycle). Analyzer projects target `netstandard2.0`, so `ReferenceOutputAssembly="false"` +
|
||||
`OutputItemType="Analyzer"` is the correct incantation — the analyzer DLL is loaded into the
|
||||
compiler host, not linked. This makes OTOPCUA0001 run on every `src/` and `tests/` compilation.
|
||||
|
||||
Add to `Directory.Build.props` (after the existing `ItemGroup`):
|
||||
```xml
|
||||
<ItemGroup Condition="'$(MSBuildProjectName)' != 'ZB.MOM.WW.OtOpcUa.Analyzers'
|
||||
and '$(MSBuildProjectName)' != 'ZB.MOM.WW.OtOpcUa.Analyzers.Tests'">
|
||||
<ProjectReference Include="$(MSBuildThisFileDirectory)src/Tooling/ZB.MOM.WW.OtOpcUa.Analyzers/ZB.MOM.WW.OtOpcUa.Analyzers.csproj"
|
||||
OutputItemType="Analyzer"
|
||||
ReferenceOutputAssembly="false"
|
||||
PrivateAssets="all" />
|
||||
</ItemGroup>
|
||||
```
|
||||
(The `.Analyzers.Tests` exclusion is belt-and-suspenders: it already `ProjectReference`s the analyzer
|
||||
directly; letting `Directory.Build.props` add a second analyzer-typed reference is harmless but noisy.)
|
||||
|
||||
**Triage sub-step (required, do NOT skip):** OTOPCUA0001 is a diagnostic. Its default severity must
|
||||
be checked in `UnwrappedCapabilityCallAnalyzer` — if it is `Warning` and a src project has TWE on,
|
||||
the wired-in analyzer can **break the build** on the first unwrapped call it finds. So the wiring
|
||||
must be done in this order:
|
||||
1. Wire the `ProjectReference` (above).
|
||||
2. `dotnet build ZB.MOM.WW.OtOpcUa.slnx` and capture every OTOPCUA0001 hit.
|
||||
3. For each hit, decide: real violation → route the call through `CapabilityInvoker`; intentional
|
||||
(test doubles, the invoker's own internals) → add a scoped `#pragma warning disable OTOPCUA0001`
|
||||
or a per-project `<NoWarn>OTOPCUA0001</NoWarn>` with a comment. The analyzer's own tests
|
||||
demonstrate the intended pattern.
|
||||
4. Only after the tree is clean should any project promote OTOPCUA0001 to error (out of scope here).
|
||||
|
||||
**Files to touch:** `Directory.Build.props` (add ItemGroup); plus any `.csproj` needing `NoWarn`
|
||||
and any source files with genuine unwrapped calls surfaced by step 2.
|
||||
|
||||
**Verify:** `dotnet build ZB.MOM.WW.OtOpcUa.slnx` shows OTOPCUA0001 diagnostics on real code (proving
|
||||
it is live); a deliberately-unwrapped call in a scratch file trips it; the analyzer's 31 tests still
|
||||
pass. Confirm the build graph does not regress (`dotnet build` clean after triage).
|
||||
|
||||
**Effort:** M (wiring is S; the triage wave is the unknown — could be S or M depending on how many
|
||||
unwrapped calls exist). **Risk:** Medium — a Warning-severity analyzer on TWE projects can fail the
|
||||
build; the ordered triage above contains it. Blast radius is every compilation, so build once
|
||||
locally before pushing.
|
||||
|
||||
---
|
||||
|
||||
### S-1 (High) — CI gates ~15% of the test matrix
|
||||
|
||||
**Restatement:** `v2-ci.yml` runs 5 of 47 unit-test projects; Client (388 tests), Analyzers (31),
|
||||
all driver and most Core suites never run in CI.
|
||||
|
||||
**Verification:** confirmed (5 enumerated, 47 total).
|
||||
|
||||
**Root cause:** hand-maintained matrix predates most projects and was never widened; new projects
|
||||
drift out of coverage silently.
|
||||
|
||||
**Design:** replace the hand-maintained `unit-tests` matrix with a single solution-wide leg that
|
||||
excludes only the env-gated tiers. The skip-gated `*.IntegrationTests` fixtures already tolerate
|
||||
unreachable endpoints (they `Assert.Skip`), so running them here is safe; but to keep the unit leg
|
||||
fast and deterministic, split responsibilities:
|
||||
|
||||
- **`unit-tests` job:** one step —
|
||||
`dotnet test ZB.MOM.WW.OtOpcUa.slnx --configuration Release --no-restore
|
||||
--filter "Category!=E2E&Category!=LiveIntegration" --logger "trx;LogFileName=unit.trx"`
|
||||
after an explicit `dotnet restore` + `dotnet build --no-restore` (share build via `--no-build`
|
||||
where possible; see P-1). This runs all 47 projects in one invocation — no matrix to drift.
|
||||
|
||||
Rationale for the whole-solution filter over regenerating a matrix from the slnx: it is
|
||||
self-maintaining (new project = automatically covered), matches CLAUDE.md's own
|
||||
`dotnet test ZB.MOM.WW.OtOpcUa.slnx` guidance, and the trx output feeds S-2's skip-gate. The
|
||||
integration split (Host.IntegrationTests / OpcUaServer.IntegrationTests as a distinct matrix leg)
|
||||
stays, but is folded into the fail-on-skip work below (S-2).
|
||||
|
||||
**Files to touch:** `.github/workflows/v2-ci.yml` — replace the `unit-tests` job body.
|
||||
|
||||
**Verify:** push to a branch; the `unit-tests` job log shows Client.CLI/Shared/UI + Analyzers +
|
||||
driver + Core suites executing (not skipped); wall-clock is acceptable (pair with P-1 caching).
|
||||
|
||||
**Effort:** S. **Risk:** Medium — first full-solution CI run will surface latent failures /
|
||||
flakes that were never gated (esp. the fixed-sleep timing tests, S-4). That is the point, but budget
|
||||
time to stabilize. Blast radius: CI only.
|
||||
|
||||
---
|
||||
|
||||
### S-2 (High) — "green" integration CI means "skipped"; skip is indistinguishable from pass
|
||||
|
||||
**Restatement:** the integration job runs fixture-less against `10.100.0.35` defaults → probe fails
|
||||
→ `Assert.Skip` → job green. A real fixture outage cannot be told from a real pass.
|
||||
|
||||
**Verification:** confirmed — no `services:` in the integration job; fixtures hard-default to
|
||||
`10.100.0.35` (Modbus/AbServer/Snap7/OpcPlc fixtures + `DriverTestConnectE2eTests`).
|
||||
|
||||
**Root cause:** the probe-once skip pattern is correct for dev boxes, but CI has no signal that the
|
||||
tier ran nothing.
|
||||
|
||||
**Design — two complementary changes:**
|
||||
|
||||
1. **Make skips visible/failing.** Emit `--logger "trx"` from the integration leg and add a post-test
|
||||
step that parses the trx and **fails the job when skipped-count exceeds a threshold** (e.g. any
|
||||
integration test skipped when CI was supposed to have fixtures, or > N total). A small inline shell
|
||||
or a tiny `scripts/ci/assert-not-all-skipped.ps1`/`.sh` reading the `<Results>`/`outcome="NotExecuted"`
|
||||
counts. This is the minimal, always-applicable guard.
|
||||
|
||||
2. **(Optional, higher value) Actually start the reachable fixtures as workflow `services:`.** The
|
||||
Modbus sim and opc-plc images run on any hosted runner; add them as `services:` and set
|
||||
`MODBUS_SIM_ENDPOINT=localhost:5020` / `OPCUA_SIM_ENDPOINT=opc.tcp://localhost:50000` so those
|
||||
probes connect and the tests actually run. S7/AbCip/Galaxy fixtures have no hosted-runner image and
|
||||
stay legitimately skipped (guard #1 excludes them from the threshold, or scopes the threshold to
|
||||
the modbus/opc-plc subset).
|
||||
|
||||
Recommend shipping **guard #1 first** (cheap, universal), then #2 as a follow-up.
|
||||
|
||||
**Files to touch:** `.github/workflows/v2-ci.yml` (integration job: add trx logger, post-step, and
|
||||
optionally `services:` + endpoint env); optionally a new `scripts/ci/assert-not-all-skipped.*`.
|
||||
|
||||
**Verify:** deliberately point an endpoint at an unreachable host in a branch → integration job goes
|
||||
**red** (not green). With services up, the modbus/opc-plc tests show as passed, not skipped.
|
||||
|
||||
**Effort:** S (guard #1) / M (with services). **Risk:** Low-Medium — a too-tight threshold could
|
||||
red-flag legitimately-unavailable fixtures; scope the threshold to the images CI can actually run.
|
||||
|
||||
---
|
||||
|
||||
### S-3 (Medium) — nightly E2E workflow is a permanent green no-op
|
||||
|
||||
**Restatement:** `v2-e2e.yml` filters `Category=E2E`, which matches zero tests; it boots the full
|
||||
docker-dev fleet for nothing and trains people to ignore a nightly green.
|
||||
|
||||
**Verification:** confirmed — zero `Category=E2E` in the tree; the workflow header admits it.
|
||||
|
||||
**Root cause:** the E2E project (`tests/Server/ZB.MOM.WW.OtOpcUa.E2ETests`) was scoped but never
|
||||
landed.
|
||||
|
||||
**Design (pick one):**
|
||||
- **(a) Disable the schedule until the project exists** — comment out the `schedule:` trigger, keep
|
||||
`workflow_dispatch`, and add a guard step that fails if `Category=E2E` matches zero tests (so the
|
||||
no-op can't silently return). Lowest effort; removes the false-green.
|
||||
- **(b) Land a minimal E2E round-trip** — a single `[Trait("Category","E2E")]` test that, against the
|
||||
booted docker-dev fleet, connects via `Client.Shared`/`Client.CLI` to `opc.tcp://localhost:4840`,
|
||||
browses to a known seeded node, reads a value, and asserts Good. The `scripts/e2e/test-*.ps1`
|
||||
harnesses show exactly what to assert. This gives the nightly job real meaning and exercises the
|
||||
otherwise-uncovered Client stack end-to-end.
|
||||
|
||||
Recommend **(a) now** (removes the false signal immediately) with **(b) as the tracked follow-on** —
|
||||
option (a)'s zero-match guard also protects (b) from regressing back to a no-op.
|
||||
|
||||
**Files to touch:** `.github/workflows/v2-e2e.yml`; for (b) a new `tests/Server/…E2ETests` project +
|
||||
slnx entry.
|
||||
|
||||
**Verify:** (a) manual dispatch fails-fast on zero E2E tests; (b) the round-trip test passes against
|
||||
the fleet and fails if the seeded node is absent.
|
||||
|
||||
**Effort:** S (a) / M-L (b). **Risk:** Low.
|
||||
|
||||
---
|
||||
|
||||
### S-7 (Low) — no `global.json` though the build depends on an exact SDK band
|
||||
|
||||
**Restatement:** `v2-ci.yml:10-12` claims a repo-root `global.json` pins the SDK; none exists.
|
||||
Meanwhile `Directory.Packages.props:44-49` pins Roslyn 5.0.0 to match SDK 10.0.105's compiler.
|
||||
|
||||
**Verification:** confirmed — no `global.json`; the CI comment is false.
|
||||
|
||||
**Root cause:** the pin comment describes intended state that was never created; the Roslyn pin note
|
||||
implicitly depends on SDK version stability the repo doesn't enforce.
|
||||
|
||||
**Design:** add a root `global.json`:
|
||||
```json
|
||||
{
|
||||
"sdk": {
|
||||
"version": "10.0.105",
|
||||
"rollForward": "latestFeature"
|
||||
}
|
||||
}
|
||||
```
|
||||
`latestFeature` allows patch/feature SDK rolls within the 10.0.1xx band but keeps the major/minor
|
||||
stable, so the Roslyn 5.0.0 pin's stated condition ("until the SDK rolls to 10.0.110+") becomes a
|
||||
deliberate, visible decision rather than a silent drift. Cross-check the exact installed SDK version
|
||||
before committing the `version` value. Update the Roslyn-pin comment in `Directory.Packages.props` to
|
||||
reference `global.json` as the coupling point.
|
||||
|
||||
**Files to touch:** new `global.json`; comment tweak in `Directory.Packages.props`.
|
||||
|
||||
**Verify:** `dotnet --version` in-repo resolves within the band; CI `setup-dotnet` honors it (the
|
||||
workflow comment becomes true). Build + tests unchanged.
|
||||
|
||||
**Effort:** S. **Risk:** Low — a too-strict `version`/`rollForward` could fail a runner lacking that
|
||||
exact SDK; `latestFeature` mitigates.
|
||||
|
||||
---
|
||||
|
||||
## TIER B — Repo hygiene
|
||||
|
||||
### U-5 (Medium) — orphaned proprietary AVEVA DLLs tracked in `lib/`
|
||||
|
||||
**Restatement:** 7 committed vendor binaries in `lib/`, referenced by nothing — a
|
||||
redistribution/licence risk in every clone.
|
||||
|
||||
**Verification:** confirmed — `git ls-files lib/` lists all 7; zero `HintPath` hits repo-wide.
|
||||
|
||||
**Root cause:** leftovers from the retired in-process MXAccess / Wonderware sidecar era; the
|
||||
bitness/COM story now lives entirely in the mxaccessgw repo (per CLAUDE.md).
|
||||
|
||||
**Design:** `git rm -r lib/`. Before deleting, re-confirm nothing references them:
|
||||
```
|
||||
grep -rn "aahClient\|ArchestrA\|Historian.CBE\|Historian.DPAPI\|HintPath.*lib" \
|
||||
--include='*.csproj' --include='*.props' --include='*.targets' .
|
||||
```
|
||||
(expected: no hits outside the review docs). Optionally add `lib/` to `.gitignore` to prevent
|
||||
re-introduction, and note in the commit that the DLLs remain retrievable from history / the AVEVA SDK
|
||||
if ever needed. Purging them from *history* is a separate, heavier decision (BFG/filter-repo) — out
|
||||
of scope; the immediate licence-hygiene win is removing them from the tip.
|
||||
|
||||
**Files to touch:** delete `lib/*` (7 files); optional `.gitignore` entry.
|
||||
|
||||
**Verify:** `git status` clean; `dotnet build ZB.MOM.WW.OtOpcUa.slnx` still succeeds (proves nothing
|
||||
linked them); `git ls-files lib/` empty.
|
||||
|
||||
**Effort:** S. **Risk:** Low — verified zero references; blast radius nil.
|
||||
|
||||
---
|
||||
|
||||
### U-4 (Medium) — root planning-file sprawl contradicts its own "never stage" rule
|
||||
|
||||
**Restatement:** `pending.md`, `current.md`, `looseends.md`, `stillpending.md`,
|
||||
`HISTORIAN-GATEWAY-INTEGRATION-ISSUES.md` are committed at root; `pending.md:3` itself says never
|
||||
stage `pending.md`/`current.md`. Content is stale snapshots now contradicting `docs/` + CLAUDE.md.
|
||||
|
||||
**Verification:** confirmed — all 5 tracked; the hard-rule text present.
|
||||
|
||||
**Root cause:** working-notes files that were staged despite the rule, and never cleaned up as the
|
||||
backlog shipped (~95% per the memory index).
|
||||
|
||||
**Design — make the file's own rule true:**
|
||||
1. **`pending.md` + `current.md`** — untrack but keep locally (they're active working notes):
|
||||
`git rm --cached pending.md current.md`, then add both to `.gitignore`. This exactly satisfies
|
||||
the "never stage" rule.
|
||||
2. **`looseends.md`, `stillpending.md`, `HISTORIAN-GATEWAY-INTEGRATION-ISSUES.md`** — these are
|
||||
point-in-time snapshots of shipped work. Preferred: move to `docs/plans/` with dated names (the
|
||||
established convention), e.g. `docs/plans/2026-05-18-looseends.md`, so the history is preserved
|
||||
as an archived plan rather than a live-looking root file. Acceptable alternative: `git rm` them
|
||||
outright since the memory index records the backlog as shipped and `docs/`+CLAUDE.md now carry the
|
||||
authoritative state. Recommend **archive-move** for `stillpending.md`/`HISTORIAN-GATEWAY-…` (they
|
||||
have referenced history) and **delete** for `looseends.md` if fully superseded — confirm with a
|
||||
quick read before choosing per-file.
|
||||
|
||||
**Files to touch:** `.gitignore` (+2 entries); `git rm --cached` pending/current; move-or-delete the
|
||||
3 snapshots; `docs/plans/` if archiving.
|
||||
|
||||
**Verify:** `git ls-files | grep -E '^(pending|current|looseends|stillpending|HISTORIAN)'` returns
|
||||
only any intentionally-archived `docs/plans/*` paths; `git status` shows pending.md/current.md now
|
||||
untracked (ignored). `pending.md`'s hard-rule is now consistent with reality.
|
||||
|
||||
**Effort:** S. **Risk:** Low — pure VCS hygiene; keep local copies (`--cached`) so no working data is
|
||||
lost.
|
||||
|
||||
---
|
||||
|
||||
### U-6 (Low) — secrets hygiene + retired-dir husks
|
||||
|
||||
**Restatement:** `sql_login.txt` is NOT committed (gitignored) but is a plaintext credential on
|
||||
Desktop; retired `Driver.Historian.Wonderware*` dirs linger on disk as bin/obj husks.
|
||||
|
||||
**Verification:** confirmed — `sql_login.txt` `.gitignore:47`, `git log` empty; the 8
|
||||
`Driver.Historian.Wonderware*` dirs under `src/`+`tests/` are untracked (only `docs/` stub + 3
|
||||
`code-reviews/…/findings.md` are tracked residue).
|
||||
|
||||
**Root cause:** local dev debris; the credential file is a convenience left in the repo dir.
|
||||
|
||||
**Design:**
|
||||
- **`sql_login.txt`** — no tracking action needed (already ignored). Recommend moving the credential
|
||||
to `dotnet user-secrets` or an env file *outside* the repo directory, and audit
|
||||
`export-clean-copy.bat` at root to confirm it excludes `sql_login.txt` (and `lib/`, pki, planning
|
||||
files) from any export bundle. Low-priority, local-only.
|
||||
- **Wonderware husks** — delete the untracked `src/Drivers/…Wonderware*` and
|
||||
`tests/Drivers/…Wonderware*` directories from disk (`rm -rf`, they're pure `bin/obj`). Decide
|
||||
separately whether to also remove the 3 tracked `code-reviews/Driver.Historian.Wonderware*/`
|
||||
findings and keep `docs/drivers/Historian.Wonderware.md` as the intentional retired stub (the doc
|
||||
is deliberately retained per CLAUDE.md; leave it).
|
||||
|
||||
**Files to touch:** none tracked for the credential; `rm -rf` the untracked husks; audit
|
||||
`export-clean-copy.bat`.
|
||||
|
||||
**Verify:** `find src tests -type d -iname '*Wonderware*'` returns nothing; `git status` unchanged by
|
||||
the husk deletion (they were untracked); `export-clean-copy.bat` excludes the sensitive files.
|
||||
|
||||
**Effort:** S. **Risk:** Low.
|
||||
|
||||
---
|
||||
|
||||
### C-5 (Low) — no `.editorconfig`; StyleGuide.md mis-titled for a sister repo
|
||||
|
||||
**Restatement:** no mechanical code-style enforcement; `StyleGuide.md:3` still says "for all
|
||||
**ScadaBridge** documentation" (copy-paste from the sibling repo).
|
||||
|
||||
**Verification:** confirmed — no `.editorconfig`; `StyleGuide.md:3` wording present.
|
||||
|
||||
**Root cause:** style held by review culture only; StyleGuide copied from ScadaBridge without
|
||||
re-titling.
|
||||
|
||||
**Design — two independent, small changes:**
|
||||
1. **StyleGuide fix (trivial, do now):** replace "ScadaBridge" with "OtOpcUa" in `StyleGuide.md:3`
|
||||
(and scan the file for other ScadaBridge leftovers).
|
||||
2. **`.editorconfig` (optional):** add a root `.editorconfig` codifying the *already-consistent*
|
||||
conventions — file-scoped namespaces, 4-space indent, `_camelCase` private fields, `var` usage —
|
||||
set as `suggestion`/`warning` (not `error`) to avoid a big-bang reformat. This is documentation-of-
|
||||
convention, not a reformatting mandate; a `dotnet format` CI step is a further follow-on (U-7),
|
||||
deliberately deferred so it doesn't churn the tree.
|
||||
|
||||
**Files to touch:** `StyleGuide.md`; optional new `.editorconfig`.
|
||||
|
||||
**Verify:** StyleGuide title reads OtOpcUa; `.editorconfig` doesn't flip existing warnings to errors
|
||||
(build unchanged).
|
||||
|
||||
**Effort:** S. **Risk:** Low (keep severities non-error).
|
||||
|
||||
---
|
||||
|
||||
## TIER C — Documentation
|
||||
|
||||
### U-3 (Medium) — `docs/Client.CLI.md` documents 8 of 14 commands
|
||||
|
||||
**Restatement:** `ack`/`acknowledge`, `confirm`, `shelve`, `enable`, `disable` have no doc sections,
|
||||
though all ship and CLAUDE.md points operators at this doc.
|
||||
|
||||
**Verification:** confirmed — `confirm`/`shelve`/`disable` = 0 mentions; `ack`/`enable` only
|
||||
substring hits; command files exist for all 5.
|
||||
|
||||
**Root cause:** the inbound operator ack/shelve feature shipped its runtime + CLI but the CLI doc was
|
||||
never extended.
|
||||
|
||||
**Design:** add five command sections to `docs/Client.CLI.md` mirroring the existing section shape
|
||||
(synopsis, options, example invocation, expected output). Source the exact options/args from the
|
||||
command classes: `AcknowledgeCommand.cs`, `ConfirmCommand.cs`, `ShelveCommand.cs`, `EnableCommand.cs`,
|
||||
`DisableCommand.cs` (and their shared `CommandBase` connection options). Group them under an "Alarm
|
||||
operator commands" heading since they are the operator-facing ack/shelve workflow. Cross-check against
|
||||
`docs/ScriptedAlarms.md` §"Inbound operator ack/shelve" for terminology consistency.
|
||||
|
||||
**Files to touch:** `docs/Client.CLI.md`.
|
||||
|
||||
**Verify:** each of the 5 commands has a section; a `-h`/`--help` run of each command matches the
|
||||
documented options. Optionally add a CI docs-coverage check (U-7 follow-on) asserting every
|
||||
`Commands/*Command.cs` has a doc section.
|
||||
|
||||
**Effort:** S. **Risk:** Low.
|
||||
|
||||
---
|
||||
|
||||
## TIER D — Engineering debt (as capacity allows)
|
||||
|
||||
These are real but lower-leverage than restoring CI meaning. Batched with condensed treatment.
|
||||
|
||||
### C-2 (Medium) — TWE never reached the Client/Tooling slice
|
||||
Add `<TreatWarningsAsErrors>true</TreatWarningsAsErrors>` to `Client.CLI.csproj`,
|
||||
`Client.Shared.csproj`, `Client.UI.csproj`, `Analyzers.csproj` (v2-era code; the legacy xUnit1051
|
||||
blocker doesn't apply). **Do this AFTER C-1's triage** — with the analyzer wired in, a Warning-level
|
||||
OTOPCUA0001 would now fail these builds, so sequence C-1 triage → C-2. Verify each builds clean with
|
||||
TWE on. Effort S; Risk Low (build once locally first). Burn down the 38 test-project TWE gaps per
|
||||
module as a tracked follow-on.
|
||||
|
||||
### P-1 (Medium) — CI does 7 uncached restore+build passes
|
||||
Add `actions/cache` for `~/.nuget/packages` keyed on a hash of `Directory.Packages.props` +
|
||||
`NuGet.config`; do one explicit `restore`, then `build --no-restore`, then `test --no-build` in the
|
||||
unit leg (dovetails with S-1's single-leg design). Effort S; Risk Low; biggest CI wall-clock win.
|
||||
|
||||
### U-2 (Medium) — add `Host.Tests` (the one real coverage hole)
|
||||
Create `tests/Server/ZB.MOM.WW.OtOpcUa.Host.Tests` asserting DI composition — e.g. "when
|
||||
`ServerHistorian:Enabled`, `IHistorianProvisioning` resolves to `GatewayTagProvisioner` and the
|
||||
applier receives it" — the exact class of dormant-wiring bug (PR #423, DeferredAddressSpaceSink) that
|
||||
unit tests would have caught. Add to slnx so S-1's whole-solution CI picks it up. Effort M; Risk Low.
|
||||
|
||||
### S-5 (Medium) — Client.Shared subscription lock-discipline gaps
|
||||
In `OpcUaClientService.cs`: (a) move the `_dataSubscription` null-check/create inside an async gate
|
||||
mirroring `_alarmSubscribeSemaphore` (line ~262); (b) take `_subscriptionLock` in `RunFailoverAsync`
|
||||
when nulling `_dataSubscription`/`_alarmSubscription` (lines ~695-696); (c) route
|
||||
`UnsubscribeAlarmsAsync` (line ~335) through `_alarmSubscribeSemaphore`; (d) observe exceptions from
|
||||
the fire-and-forget keep-alive failover. Near-unhittable for the session-per-command CLI; real for the
|
||||
long-lived Client.UI service. Effort M; Risk Medium (concurrency code — add targeted tests). Do before
|
||||
Client.UI grows multi-threaded usage.
|
||||
|
||||
### S-6 (Medium) — global `Log.Logger` swap per CLI command
|
||||
`CommandBase.ConfigureLogging()` (`:120-134`) does `Log.CloseAndFlush()` + replaces static
|
||||
`Log.Logger` per command — a race under xunit.v3's parallel collections and hostile to embedding.
|
||||
Give each command an instance `ILogger` from a per-execution `LoggerConfiguration.CreateLogger()`.
|
||||
The existing `LoggerLifecycleTests` polices this. Effort M; Risk Low-Medium (touches every command's
|
||||
logging).
|
||||
|
||||
### S-4 (Medium) — fixed-sleep timing tests
|
||||
Replace start-up `await Task.Delay(100)` sleeps in `Client.CLI.Tests` (`SubscribeCommandTests`,
|
||||
`AlarmsCommandTests`, `EventHandlerLifecycleTests`) with a readiness signal — a `TaskCompletionSource`
|
||||
completed on first `SubscribeAsync` exposed by `FakeOpcUaClientService`. Currently masked because CI
|
||||
never runs them (S-1); will flake once S-1 lands, so **pair with S-1**. Effort M; Risk Low.
|
||||
|
||||
### C-4 (Low) — finish xunit.v3 migration
|
||||
Migrate the 3 holdouts (`AdminUI.Tests`, `ControlPlane.Tests`, `Runtime.Tests`) off xunit 2.9.2, then
|
||||
drop the `xunit`/`xunit.runner.visualstudio` 2.x pins from `Directory.Packages.props:108-109`.
|
||||
Optionally relocate `GatewayLiveIntegrationTests` to a `*.IntegrationTests` sibling. Effort M; Risk
|
||||
Low.
|
||||
|
||||
### C-6 (Low) — csproj boilerplate duplicates Directory.Build.props
|
||||
Strip redundant `TargetFramework`/`Nullable`/`ImplicitUsings` from the ~70 csprojs on next sweep so an
|
||||
SDK bump touches one file. Effort M (mechanical); Risk Low. Low priority.
|
||||
|
||||
### S-8 (Low) — first-caller interval wins for shared data subscription
|
||||
Document the shared-publish-interval behavior in `docs/Client.CLI.md`/`docs/Client.UI.md`, or create
|
||||
per-interval subscriptions if UI users mix cadences. Effort S; Risk Low.
|
||||
|
||||
### U-8 (Low) — CLI security posture is dev-tool-grade
|
||||
Document that `AutoAcceptCertificates = true` (`CommandBase.cs:91`) means `--security signandencrypt`
|
||||
encrypts but does not authenticate the server (MITM-able), and that `-P` is process-visible. Optionally
|
||||
add a `--strict-certs` opt-in. Separately, note the Galaxy-specific alarm attribute names baked into
|
||||
the generic `OpcUaClientService.cs:851-871` as a documented layering compromise. Effort S; Risk Low.
|
||||
|
||||
### P-2 (Low) — recursive browse N+1
|
||||
`BrowseAsync` issues an extra `HasChildrenAsync` per Object child (`:230-231`) and
|
||||
`SubscribeCommand.CollectVariablesAsync` walks serially. Batch/bulk-read references only if `subscribe
|
||||
-r` start-up becomes a soak-test bottleneck. Effort M; Risk Low. Defer.
|
||||
|
||||
### U-7 (Low) — missing CI stages inventory
|
||||
Tracking item: analyzer enforcement (C-1), all-suite coverage (S-1), `dotnet format`/style step,
|
||||
explicit failing NuGet-audit step, running `scripts/check-code-reviews-readme.ps1` in CI, and a docs
|
||||
freshness/link check. Also the `ci/ab-server.lock.json` references a non-existent "GitHub Actions
|
||||
step" in `docs/v2/test-data-sources.md` — either land the AB fixture download step or fix the doc.
|
||||
Address incrementally as the above land.
|
||||
|
||||
---
|
||||
|
||||
## No-action (verified positive / accurate — recorded only)
|
||||
|
||||
- **P-3** — probe-once collection-fixture pattern is well-engineered; the issue is what CI does with
|
||||
it (S-2), not the fixtures.
|
||||
- **P-4** — CLI subscription output uses a channel-serialised writer; allocation-sane. A bounded
|
||||
`DropOldest` channel is a cosmetic future option.
|
||||
- **C-3** — CPM discipline exemplary; the `NuGetAuditSuppress` carve-out and the two transitive CVE
|
||||
pins are still valid and accurately scoped. Keep the removal reminders alive.
|
||||
- **U-1** — Client.UI is a genuine, adequately-tested product (not vestigial); "maturing", not
|
||||
"underdeveloped". Its only gap is the same CI/E2E gap (S-1/S-3).
|
||||
|
||||
---
|
||||
|
||||
## Suggested execution sequence
|
||||
|
||||
1. **C-1** (wire analyzer) + its triage — establishes the enforcement mechanism; must precede C-2.
|
||||
2. **S-1 + P-1** (whole-solution CI + caching) in one workflow edit — biggest coverage win.
|
||||
3. **S-2** (fail-on-skip guard, then optional `services:`) — closes the false-green.
|
||||
4. **S-7** (`global.json`) + **S-3** (disable E2E schedule / zero-match guard) — small CI truth fixes.
|
||||
5. **U-5, U-4, U-6, C-5** (hygiene batch) — one commit; verified zero-reference deletions.
|
||||
6. **U-3** (Client.CLI doc) — five sections.
|
||||
7. **C-2** (TWE, after C-1 triage), then Tier-D items as capacity allows, pairing **S-4 with S-1**.
|
||||
|
||||
Each of steps 1-4 must be validated by an actual CI run on a branch before merge — per the report's
|
||||
own theme #4, green CI here has been meaningless, so the fixes must be observed to change the badge's
|
||||
behavior (a broken build/deliberate skip must now go red).
|
||||
Reference in New Issue
Block a user