Files
lmxopcua/archreview/01-core-composition.md
T
Joseph Doherty 9fadead6a6 docs(archreview): remediation plans + fix flagged doc drift
Add the 7 per-domain design+implementation plans (archreview/plans/) with
an index, produced from the 2026-07-08 architecture review.

Fix two confirmed doc drifts the review flagged (theme #5):
- CLAUDE.md KNOWN LIMITATION 2: the continuous-historization historized-ref
  feed IS wired (AddressSpaceApplier.FeedHistorizedRefs -> UpdateHistorizedRefs
  -> recorder); rewrite to reflect that value-capture is code-complete and only
  the live end-to-end + restart-convergence verification remains.
- CLAUDE.md ScriptAnalysis gating: endpoints use Roles=Administrator,Designer
  via RequireAuthorization, not the FleetAdmin policy.
2026-07-08 16:14:37 -04:00

373 lines
26 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Architecture Review — Core Composition Pipeline and Core Libraries
**Date:** 2026-07-08
**Commit:** `9cad9ed0` (master)
**Reviewer scope:**
- `src/Core/ZB.MOM.WW.OtOpcUa.Core` (driver hosting, resilience, stability, authorization trie)
- `src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions` (driver capability interfaces, `PollGroupEngine`, historian seams)
- `src/Core/ZB.MOM.WW.OtOpcUa.Commons` (cluster message contracts, deferred sinks, NodeId scheme, telemetry)
- `src/Core/ZB.MOM.WW.OtOpcUa.Configuration` (EF config persistence, generation-sealed local cache, draft validation)
- `src/Core/ZB.MOM.WW.OtOpcUa.Cluster` (Akka bootstrap, role info, ServiceLevel)
- Plus the address-space composition pipeline itself — `AddressSpaceComposer` / `AddressSpacePlanner` / `AddressSpaceApplier`
**Scope note (location drift).** The review brief places the composition pipeline in
`ZB.MOM.WW.OtOpcUa.Core`, but since the Phase7→AddressSpace rename it lives in
`src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/` (`AddressSpaceComposer.cs`, `AddressSpacePlan.cs`,
`AddressSpaceApplier.cs`), with its tests under `tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests/`.
The `Core` project's own `OpcUa/` folder now holds only the retired `EquipmentNodeWalker` and an
unused `GenericDriverNodeManager` (see U-1). The pipeline was reviewed in full regardless.
---
## Architecture Overview
### Composition pipeline (deploy → address space)
The pipeline is a clean three-stage compose/diff/apply design:
1. **`AddressSpaceComposer.Compose`** (`OpcUaServer/AddressSpaceComposer.cs:290`) — a pure static
projection from EF config entities (`UnsArea`, `UnsLine`, `Equipment`, `DriverInstance`, `Tag`,
`VirtualTag`, `Script`, `ScriptedAlarm`, `Device`, `Namespace`) into an
`AddressSpaceComposition`: sorted lists of `UnsAreaProjection`/`UnsLineProjection`/
`EquipmentNode`/`DriverInstancePlan`/`EquipmentTagPlan`/`EquipmentVirtualTagPlan`/
`EquipmentScriptedAlarmPlan`. Tag intent (driver `FullName`, native-alarm object, historize
flags, array shape) is parsed out of the schemaless `Tag.TagConfig` JSON here. VirtualTag
scripts get `{{equip}}` token substitution and dependency-ref extraction via the shared
`Commons.Types.EquipmentScriptPaths` helper. Everything is ordinally sorted so the composition
is deterministic — the foundation of the "byte-parity" contract with the artifact-decode mirror
in `Runtime/Drivers/DeploymentArtifact.cs`.
2. **`AddressSpacePlanner.Compute`** (`OpcUaServer/AddressSpacePlan.cs:96`) — a pure diff of two
compositions keyed on stable logical ids, using record value-equality as the changed check
(with hand-written `Equals`/`GetHashCode` on the plans carrying `IReadOnlyList` members so a
no-op redeploy diffs empty). Emits Added/Removed/Changed sets per entity class plus UNS
folder renames. `AddressSpacePlan.IsEmpty` is the short-circuit gate `OpcUaPublishActor` uses.
3. **`AddressSpaceApplier.Apply`** (`OpcUaServer/AddressSpaceApplier.cs:73`) — the only
side-effecting stage. Decides between a **full structural rebuild** (any topology change) and a
**surgical in-place update** (`ISurgicalAddressSpaceSink.UpdateTagAttributes` /
`UpdateFolderDisplayName` for whitelisted node-irrelevant deltas — the F10b optimization),
falling back to rebuild when the sink lacks the capability or any surgical call fails. After
the address-space work it dispatches two non-blocking hooks: fire-and-forget historian tag
provisioning (`IHistorianProvisioning.EnsureTagsAsync`) and the historized-ref delta feed to
the continuous-historization recorder (`IHistorizedTagSubscriptionSink.UpdateHistorizedRefs`).
Separate `Materialise*` passes (Hierarchy / EquipmentTags / VirtualTags / ScriptedAlarms /
DiscoveredNodes) re-derive nodes from the composition after a rebuild; NodeIds are
folder-scoped (`Commons.OpcUa.EquipmentNodeIds`: `{equipment}/{folder}/{name}`), never the
driver `FullName`, to avoid collisions across identical machines.
The sink boundary (`Commons.OpcUa.IOpcUaAddressSpaceSink`) keeps the whole pipeline SDK-free:
production binds `SdkAddressSpaceSink` via the late-swap `DeferredAddressSpaceSink` (actors resolve
the wrapper at DI time, the OPC UA hosted service swaps the real sink in after `StandardServer`
starts); dev/tests bind the `NullOpcUaAddressSpaceSink` no-op.
### Supporting Core libraries
- **`Core/Hosting`** — `DriverFactoryRegistry` (driver-type → factory), `DriverFactoryRegistryAdapter`
(v1 registry → v2 `IDriverFactory`), `DriverHost` (id → `IDriver` lifecycle registry).
- **`Core/Resilience`** — Polly pipelines cached per `(DriverInstanceId, HostName, Capability)`
(`DriverResiliencePipelineBuilder`), executed via `CapabilityInvoker` (with the
non-idempotent-write no-retry override), fanned out per host by `AlarmSurfaceInvoker`, observed
by `DriverResilienceStatusTracker` for Admin `/hosts`.
- **`Core/Stability`** — `MemoryTracking` (median-baseline soft/hard breach), `MemoryRecycle` and
`ScheduledRecycleScheduler` (Tier C recycle via `IDriverSupervisor`), `WedgeDetector`
(demand-aware stall detection).
- **`Core/Authorization`** — generation-sealed `PermissionTrie` per `(ClusterId, GenerationId)`
cached in `PermissionTrieCache` (CAS-pruned), walked by `TriePermissionEvaluator` against
per-session `UserAuthorizationState` (freshness window + fail-closed staleness ceiling).
- **`Commons`** — immutable record message contracts for the Akka DPS topics (deploy dispatch/ack,
alerts `AlarmTransitionEvent`, redundancy state, admin ops, script log), strongly-typed ids
(`DeploymentId`, `RevisionHash`, `CorrelationId`, `NodeId`), the deferred sink/publisher pair,
`EquipmentScriptPaths` (shared script-path/dependency extraction), and `OtOpcUaTelemetry`
(central Meter/ActivitySource).
- **`Configuration`** — `OtOpcUaConfigDbContext` (26 entities, logical-id unique indexes, JSON
check constraints, RowVersion concurrency), the generation-sealed LiteDB fallback cache
(`GenerationSealedCache` + atomic `CURRENT` pointer, fail-closed on corruption),
`ResilientConfigReader` (timeout→retry→cache-fallback with secret-scrubbed logging),
`DraftValidator` (all-errors-in-one-pass pre-publish rules).
- **`Cluster`** — `AddOtOpcUaCluster` / `WithOtOpcUaClusterBootstrap` (Akka.Hosting bootstrap with
embedded HOCON + Serilog logger wiring), `ClusterRoleInfo` (lock-guarded role topology snapshot
fed by a subscriber actor), `ServiceLevelCalculator` (pure 0255 tiering), `RoleParser`.
Dependency direction: `Core.Abstractions` (leaf) ← `Configuration``Core`; `Commons` (leaf,
Akka+Audit packages) ← `Cluster`. The composer/applier in `OpcUaServer` reference both
`Configuration` (entities) and `Commons` (sink contracts).
---
## Findings
### 1. Stability
#### S-1 — Applier swallows every sink failure; a deploy is reported applied even when the address space is broken — **High**
`AddressSpaceApplier` wraps every sink call in catch-all "Safe" helpers that log and continue:
`SafeEnsureFolder`/`SafeEnsureVariable` (`AddressSpaceApplier.cs:612622`), `SafeRebuild`
(`AddressSpaceApplier.cs:373383`), `SafeWriteAlarmCondition`/`SafeMaterialiseAlarmCondition`
(`AddressSpaceApplier.cs:677687`). `AddressSpaceApplyOutcome` (`AddressSpaceApplier.cs:691`)
carries only Added/Removed/Changed counts and `RebuildCalled`**no failure count**. If
`RebuildAddressSpace()` throws, `rebuilt` is still reported `true` (`AddressSpaceApplier.cs:150153`)
and the outcome flows back to the deploy coordinator as a success; the deployment seals while the
running server holds a partially-materialised (or entirely stale) address space. Individual
`EnsureVariable` failures during a materialise pass likewise vanish into per-node Warnings.
The never-fail-a-deploy posture is defensible for the *detached* hooks (provisioning, historized-ref
feed — both correctly isolated), but structural materialisation failures are the deploy's core
contract.
**Recommendation:** add `FailedNodes`/`RebuildFailed` to `AddressSpaceApplyOutcome`, propagate into
the `ApplyAck`/`DeploymentFailed` decision and the audit log, and escalate `SafeRebuild` failure to
at least a degraded ack. Tests should assert the failure surface (the fixture sink can throw).
#### S-2 — `PollGroupEngine.Unsubscribe` blocks the calling thread up to 5 s per subscription — **Medium**
`StopState` does a synchronous `task.Wait(TimeSpan.FromSeconds(5))`
(`Core.Abstractions/PollGroupEngine.cs:99112`). `Unsubscribe` is the teardown path drivers call
from OPC UA subscription-management callbacks and actor message handlers; a reader stuck in a slow
network call turns every unsubscribe into a 5-second stall of the calling thread (and N stalls when
tearing down N subscriptions serially). `DisposeAsync` gets this right (parallel cancel, awaited
`WhenAll` with one shared timeout, `PollGroupEngine.cs:213236`); the single-subscription path does
not.
**Recommendation:** add an `UnsubscribeAsync` (or make `Unsubscribe` cancel + hand the await to a
background drain), keeping the "no callback after teardown" guarantee via the existing
LoopTask/CTS-dispose ordering.
#### S-3 — `PollGroupEngine.Subscribe` has no disposed guard — **Low**
`Subscribe` racing `DisposeAsync` can insert a fresh `SubscriptionState` after the dispose loop has
snapshotted `_subscriptions.Values` (`PollGroupEngine.cs:7384` vs `213236`), leaking a live poll
loop with no owner. Same pattern in `DriverHost.RegisterAsync` vs `DisposeAsync`
(`Core/Hosting/DriverHost.cs:5371` vs `92106`): a registration that wins the race after the
snapshot+clear is initialized but never shut down. Both are shutdown-window races, unlikely in
practice but cheap to close with a `_disposed` flag checked inside the lock / before insert.
#### S-4 — `ScheduledRecycleScheduler` catch-up recycle storm after a stall — **Low**
`TickAsync` advances `_nextRecycleUtc` by exactly one interval per fire
(`Core/Stability/ScheduledRecycleScheduler.cs:7284`). If the host was suspended (VM pause, long
outage) across K intervals, the next K ticks each trigger a full Tier C process recycle
back-to-back. **Recommendation:** fast-forward `_nextRecycleUtc` past `utcNow` after a fire.
#### S-5 — `GenericDriverNodeManager` re-walk teardown is not synchronized — **Low**
`BuildAddressSpaceAsync` tears down the previous alarm forwarder and clears `_alarmSinks`
non-atomically (`Core/OpcUa/GenericDriverNodeManager.cs:5871`); a concurrent driver alarm event
during the window between unsubscribe and re-subscribe is dropped, and two concurrent Build calls
would interleave badly. Currently moot because the class has no production caller (see U-1) — fold
this into whatever decision is made there.
**Positive observations.** The failure-mode engineering in `Configuration/LocalCache` is exemplary:
`GenerationSealedCache` writes temp-file + atomic `File.Replace` pointer swaps, fails closed on any
corruption rather than silently serving an older generation (`GenerationSealedCache.cs:111155`);
`LiteDbConfigCache` documents and defuses two real LiteDB concurrency traps (global `BsonMapper`
races, find-then-insert upsert races) (`LiteDbConfigCache.cs:1636`). `ResilientConfigReader`
correctly distinguishes SQL-command `TaskCanceledException` from genuine caller cancellation
(`ResilientConfigReader.cs:117138`) and scrubs connection-string secrets from log messages.
`PermissionTrieCache.Prune` is a correct reference-equality CAS loop (`PermissionTrieCache.cs:73102`).
The applier's two historian hooks are properly detached (continuation observes the task, never
`.Result` on a fault — `AddressSpaceApplier.cs:259286`). `PollGroupEngine` classifies fatal
exceptions and hardens the `onError` callback (`PollGroupEngine.cs:153169`).
### 2. Performance
#### P-1 — `Compose` parses each tag's `TagConfig` JSON four times — **Medium**
`ExtractTagFullName`, `ExtractTagAlarm`, `ExtractTagHistorize` and `ExtractTagArray` each do their
own `JsonDocument.Parse` of the same blob per tag (`AddressSpaceComposer.cs:418433` calling
`:536686`). A compose over a 10k-tag fleet performs 40k JSON parses (plus the same again on the
artifact-decode side). Compose runs on every deploy and every diff baseline, not on the data hot
path, so this is a scalability tax rather than a latency bug today.
**Recommendation:** parse once per tag into a small `TagConfigIntent` record (which also collapses
the byte-parity duplication — see C-1).
#### P-2 — Per-operation allocations on the authorization hot path — **Medium**
`PermissionTrie.CollectMatches` allocates a fresh `HashSet<string>` from the session's LDAP groups
plus a `List<MatchedGrant>` on every call (`Core/Authorization/PermissionTrie.cs:4245`), and
`TriePermissionEvaluator.Authorize` runs per node-operation. A recursive browse or a large
CreateMonitoredItems batch turns this into per-node garbage. `UserAuthorizationState.LdapGroups` is
immutable per membership refresh — the case-insensitive set can be computed once per session
version and reused.
#### P-3 — `PollGroupEngine`: one background task + independent timer per subscription, no cross-subscription batching — **Medium**
Each `Subscribe` spawns a dedicated `Task.Run` loop (`PollGroupEngine.cs:7384`). Drivers that map
each OPC UA monitored-item group (or each poll group) to a subscription get linear task/timer
growth, and two subscriptions polling overlapping tag sets at the same interval issue duplicate
protocol reads — the engine has no interval-bucketed scheduler or read coalescing. Fine at dozens
of subscriptions; a scaling wall at hundreds per driver instance. Worth a note in the driver-facing
docs at minimum; an interval-bucketed shared loop is the structural fix if tag counts grow.
#### P-4 — Per-write allocation in the non-idempotent write path — **Low**
`CapabilityInvoker.ExecuteWriteAsync` builds a new options record + `Dictionary` per non-idempotent
write (`Core/Resilience/CapabilityInvoker.cs:135152`) even though `GetOrCreate` keys only on
`(id, host, capability)` and ignores the options after the first build — the comment even cites the
"1% pipeline budget". Cache the no-retry snapshot per options generation, or key the fast path on a
precomputed pipeline.
**Positive observations.** Pipeline resolution is lock-free `ConcurrentDictionary` reads
(`DriverResiliencePipelineBuilder.cs:5871`). The planner's diff is O(N) dictionary passes with
deterministic sorted outputs. The surgical-apply path exists precisely to avoid the expensive full
rebuild + subscription teardown for attribute-only edits, and its whitelist-via-`with`-expression
technique (`AddressSpaceApplier.cs:630663`) is future-field-safe (unknown new fields force the
safe rebuild). Telemetry instruments are no-op until a listener attaches (`OtOpcUaTelemetry.cs`).
### 3. Conventions
#### C-1 — "Byte-parity by convention": TagConfig parsing replicated in four projects with comment-enforced sync — **High**
`ExtractTagFullName` (and friends) exist as deliberate copies in:
- `OpcUaServer/AddressSpaceComposer.cs:536551` (+ `ExtractTagAlarm`/`ExtractTagHistorize`/`ExtractTagArray`/`TryExtractDeviceHost`)
- `Runtime/Drivers/DeploymentArtifact.cs` (the artifact-decode mirror, per its own comments)
- `Core/OpcUa/EquipmentNodeWalker.cs:193208`
- `Configuration/Validation/DraftValidator.cs:6071` (a fourth "small local copy, consistent with
this codebase where the composer keeps its own")
Each copy carries a "MUST parse identically (byte-parity)" comment. The equality of the deployed
address space across the live-compose and artifact-decode seams — and therefore the correctness of
the diff/no-op-redeploy behavior — rests on humans keeping four JSON parsers in sync. The codebase
already demonstrated the fix once: `EquipmentScriptPaths` (`Commons/Types/EquipmentScriptPaths.cs`)
was created exactly to de-duplicate the `ctx.GetTag` extraction "those two seams used to duplicate".
**Recommendation:** move the TagConfig/DeviceConfig intent parsing into `Commons` (it has no EF or
SDK dependency; every current copy's host project already references Commons or could) as a single
`TagConfigIntent.Parse(string)` — also resolving P-1 — and keep one cross-seam parity test instead
of N replicated implementations.
#### C-2 — Core depends on Configuration: EF entities and enums are the domain model — **Medium**
`ZB.MOM.WW.OtOpcUa.Core.csproj` references `Configuration`; `PermissionTrie` consumes
`Configuration.Enums.NodePermissions`/`NodeAclScopeKind` (`Core/Authorization/PermissionTrie.cs:1`),
`EquipmentNodeWalker` consumes `Configuration.Entities` directly, and the composer's plan records
are projections of EF entities. There is no separate domain layer — persistence types *are* the
model. This is a deliberate, consistent choice (and the plan records do decouple the applier), but
it means: (a) any EF-driven entity change ripples straight into authorization and composition
logic; (b) `Configuration` (with EF Core, SqlServer, LiteDB, DataProtection packages) is pulled
into anything referencing Core. Flagging as accepted-risk to document rather than refactor —
new code should keep preferring the plan-record boundary the composer established.
#### C-3 — `DraftValidator` re-derives the NodeId scheme and hard-codes a driver-type string — **Medium**
`ValidateNoEquipmentSignalNameCollision` re-implements the folder-scoped NodeId key by hand
(`Configuration/Validation/DraftValidator.cs:7597`) because `Configuration` cannot reference
`Commons.OpcUa.EquipmentNodeIds` (dependency direction) — a third copy of the
`{equipment}/{folder}/{name}` scheme whose drift would silently break the collision check.
`ValidateGalaxyTagFullName` hard-codes `dtype != "GalaxyMxGateway"`
(`DraftValidator.cs:4155`) rather than a shared driver-type constant. Both are symptoms of the
same missing shared-contract home as C-1; a Commons-level scheme/constants type (or moving
`EquipmentNodeIds` down to a project Configuration can see) fixes both.
#### C-4 — Composer violates its own purity contract with `Trace.TraceWarning` — **Low**
The class doc says "Same inputs → same outputs, no logging" (`AddressSpaceComposer.cs:280283`),
but the dangling-predicate-script skip emits `Trace.TraceWarning` (`AddressSpaceComposer.cs:493496`)
— the only `System.Diagnostics.Trace` usage in a Serilog/`ILogger` codebase, invisible in the
production sinks. Either return skipped-alarm ids in the composition for the caller to log, or
accept an injected logger and fix the doc.
#### C-5 — Inconsistent duplicate-key defensiveness inside `Compose` — **Low**
`deviceHostById` is built with a deliberate last-wins `foreach` and a comment explaining why
`ToDictionary` (throw-on-dupe) would diverge from the decode side (`AddressSpaceComposer.cs:352364`),
yet three lines later `driversById`/`namespacesById`/`scriptsById` use plain `ToDictionary`
(`AddressSpaceComposer.cs:401402, 453`) — a duplicate logical id (only possible on DB-constraint
bypass) crashes the whole compose. Pick one posture; the defensive one is already argued for.
#### C-6 — Vestigial `Akka` package reference in Commons — **Low**
`Commons.csproj` references the `Akka` package but no Commons source uses an Akka type (only doc
comments mention actors). Harmless today because every consumer is Akka-hosted anyway, but it
contradicts Commons' role as the dependency-light contracts layer. Remove it.
#### C-7 — Stale scaffold-era XML docs on load-bearing classes — **Low**
`AddressSpaceApplier`'s class doc still describes the F14 scaffold ("For now we record the work",
"the SDK adapter that lands in F10b will decide…" — `AddressSpaceApplier.cs:726`) although both
features shipped. `DriverHost`'s doc promises "per-process isolation for Tier C … implemented in
Phase 2 via named-pipe RPC" (`Core/Hosting/DriverHost.cs:510`), an architecture superseded by the
out-of-repo mxaccessgw gateway. In a codebase whose XML documentation is otherwise a genuine asset
(and evidently relied on), stale top-of-class narratives are actively misleading.
**Positive observations.** Naming and structure are highly consistent (logical-id + RowVersion
entity pattern, `Null*`/`Deferred*` sink pattern, options records with `Validate()`, message
records per DPS topic). Zero `TODO`/`HACK`/`FIXME`/`Obsolete` markers across all five projects. The
DbContext matches a documented schema with an introspection compliance test
(`SchemaComplianceTests`). The `DeferredAddressSpaceSink` explicitly forwards the surgical
capability with a comment memorializing the F10b prod-inertness bug (`DeferredAddressSpaceSink.cs:5270`)
— institutional memory encoded where the next person will trip.
### 4. Underdeveloped Areas
#### U-1 — Dead/dormant code retained in Core: `GenericDriverNodeManager`, `EquipmentNodeWalker`, `TryParseRelayBody` — **Medium**
- `GenericDriverNodeManager` (`Core/OpcUa/GenericDriverNodeManager.cs`) has **no production
references** — only its own test file. The production path is the composer→applier→sink chain.
- `EquipmentNodeWalker` (`Core/OpcUa/EquipmentNodeWalker.cs:160165`) self-declares "retained for
unit-test support only; real server deployments never invoke Walk" — ~280 lines plus
`IdentificationFolderBuilder` and a full test suite exercising a code path that cannot run in
production. Its `ExtractFullName` copy is also one of the four C-1 duplicates.
- `EquipmentScriptPaths.TryParseRelayBody` (`Commons/Types/EquipmentScriptPaths.cs:164172`)
serviced the relay→alias converter that was deleted when Galaxy became a standard driver
(2026-06-12); only tests call it now.
Dormant code with green tests is worse than deleted code: it passes every gate while silently
diverging from the live path. **Recommendation:** delete the walker + GDNM + relay parser (git
preserves them), or, if the walker must survive as a test fixture, move it into the test project.
#### U-2 — The entire Tier C recycle machinery has no `IDriverSupervisor` implementation — **Medium**
`MemoryRecycle`, `ScheduledRecycleScheduler`, and `DriverResilienceStatusTracker.RecordRecycle`
all drive `IDriverSupervisor.RecycleAsync`, but a repo-wide search finds **zero concrete
implementations** of `IDriverSupervisor` (`Core.Abstractions/IDriverSupervisor.cs` + the two
Stability consumers are the only references). Since the out-of-process driver story moved to the
external mxaccessgw/HistorianGateway sidecars, no in-repo driver is Tier C, so the recycle path is
compiled, tested, and unreachable. Either document it as speculative infrastructure for a future
tier-migration workflow (the `MemoryRecycle` doc hints at one) or prune it alongside U-1.
#### U-3 — Continuous-historization interest set: delta feed landed, initial set still empty — **Medium (known-limitation debt)**
The applier now pushes add/remove historized-ref deltas per deploy
(`AddressSpaceApplier.cs:310353``IHistorizedTagSubscriptionSink`), which is the convergent
design for *subsequent* deploys — but per the documented KNOWN LIMITATION 2 (CLAUDE.md), the
recorder is still spawned with an empty ref set, so after a process restart the recorder historizes
nothing until the next deploy produces a delta. The seam contract
(`Core.Abstractions/Historian/IHistorizedTagSubscriptionSink.cs`) supports the fix; the missing
piece is a full-set replay at spawn/rebuild (e.g. have the applier feed the *complete* current set
on `RebuildCalled`, not just the diff). This is the highest-leverage half-finished feature in the
subsystem.
#### U-4 — Additive-only ACL model (no Deny) is a documented v2.0 gap — **Low**
`PermissionTrie` is explicitly pure-union ("no explicit Deny in v2.0",
`Core/Authorization/PermissionTrie.cs:1317`): a broad cluster-root grant cannot be carved back at
a sub-scope. Acceptable for the current flat-role deployments (AdminUI roles are global by design),
but it bounds how fine-grained OT authorization can get without a trie-walk semantics change —
worth keeping visible in the security docs.
#### U-5 — Test coverage is strong but unevenly distributed — **Low**
Test method counts: Core.Tests 208, Configuration.Tests 94, Core.Abstractions.Tests 65,
Commons.Tests 45, Cluster.Tests 21. The composition pipeline itself is thoroughly covered (10+
dedicated files in `tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests/AddressSpace*`). Gaps:
`ClusterRoleInfo` — the only concurrency-bearing class in the Cluster project (lock + subscriber
actor + event fan-out) — has no test (Cluster.Tests covers only the three pure classes);
`DriverHost` dispose/register races (S-3) and the applier failure surface (S-1) are untested
because the surface doesn't exist. An Akka TestKit harness for `ClusterRoleInfo` leader-change
sequencing would close the riskiest untested seam.
---
## Maturity Ratings
| Dimension | Rating (15) | Justification |
|---|---|---|
| Stability | **4** | Deliberate, well-commented failure-mode engineering throughout (fail-closed caches, atomic pointer swaps, detached hooks, fatal-exception classification); docked for the applier's swallow-everything success reporting (S-1) and the blocking unsubscribe (S-2). |
| Performance | **3** | Deploy-path costs are deterministic and diffed to near-zero for no-op redeploys, and the surgical-apply path avoids rebuild churn; but hot paths carry avoidable per-operation allocations (P-2), redundant JSON parsing scales 4× with tag count (P-1), and the poll engine is task-per-subscription with no coalescing (P-3). |
| Conventions | **4** | Exceptionally consistent naming, patterns, and XML docs with zero TODO debt; docked for the systemic replicate-and-comment "byte-parity" duplication (C-1/C-3) and the Core→Configuration persistence coupling (C-2). |
| Underdeveloped areas | **3** | The live pipeline is feature-complete and well-tested, but Core retains three dead/dormant code paths with green tests (U-1), a fully-built-but-unreachable Tier C recycle subsystem (U-2), and the known continuous-historization initial-set gap (U-3). |