docs(archreview): 2026-07-12 re-review at f6eaa267 + 12 round-2 remediation plans
Re-ran all seven domain reviews at master f6eaa267 (reports rewritten in
place, each with a prior-finding status table): all 4 round-1 Criticals
verified closed; new top findings are the S7 connect-timeout OCE regression
(05/STAB-14), the ResilienceConfig operator-authorable brick (01/S-6), and
a batch of resilience-seam Mediums. 00-OVERALL.md carries the updated
maturity matrix + 12-item action list.
Adds R2-01..R2-12 design/implementation plans (one per action item, house
format + bite-sized TDD task breakdowns + co-located .tasks.json; 193 tasks,
~18-24 dev-days). STATUS.md updated: round-1 topology marked historical
(all merged+pushed), re-review findings table + plan pointers added.
This commit is contained in:
Joseph Doherty
+116
-79
@@ -1,163 +1,200 @@
|
||||
# Architecture Review — Server & Runtime Subsystem
|
||||
|
||||
- **Date:** 2026-07-08
|
||||
- **Commit:** `9cad9ed0`
|
||||
- **Scope:** `src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer`, `.Host`, `.Runtime`, `.ControlPlane`, `.Security`, plus the sink/publisher wrappers in `src/Core/ZB.MOM.WW.OtOpcUa.Commons/OpcUa` (they are the load-bearing seam between Host and Runtime) and `tests/Server` for coverage assessment.
|
||||
- **Date:** 2026-07-12
|
||||
- **Commit:** `f6eaa267` (master, clean tree)
|
||||
- **Updates:** the 2026-07-08 review at `9cad9ed0`. Since then all arch-review remediation branches merged to master (~40 commits), including the S1 SBR work + #11 premise correction, the #9 hard-kill failover test, the U2 deferred-sink reflection guard, and the #10/#13 CapabilityInvoker dispatch wiring + per-instance ResilienceConfig.
|
||||
- **Scope:** `src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer`, `.Host`, `.Runtime`, `.ControlPlane`, `.Security`, plus the sink/publisher wrappers in `src/Core/ZB.MOM.WW.OtOpcUa.Commons/OpcUa`, cluster/SBR behavior in `src/Core/ZB.MOM.WW.OtOpcUa.Cluster`, and `tests/Server` for coverage assessment.
|
||||
- **Dimensions:** Stability · Performance · Conventions · Underdeveloped areas
|
||||
|
||||
---
|
||||
|
||||
## Prior-finding status (9cad9ed0 → f6eaa267)
|
||||
|
||||
Every finding from the 2026-07-08 review, verified against the actual code (not just STATUS.md). Production code in `OpcUaServer`, `ControlPlane`, `Security`, and `Commons/OpcUa` is **byte-identical** since 9cad9ed0 (only `Host/Drivers/DriverFactoryBootstrap.cs`, `Host/Engines/*`, `Runtime/Drivers/*`, `Runtime/ServiceCollectionExtensions.cs`, and `Cluster/*` changed), so most findings carry over verbatim with shifted line numbers noted.
|
||||
|
||||
| ID | One-liner | Verified status |
|
||||
|---|---|---|
|
||||
| S1 | SBR "configured but never activated"; cluster runs NoDowning | **INVALID PREMISE, GAP CLOSED ANYWAY (#11 + #9).** The original Critical was factually wrong: Akka.Cluster.Hosting's `WithClustering` applies `SplitBrainResolverOption.Default` when the typed option is null, which registers the SBR downing provider that reads the pre-existing `akka.conf` `keep-oldest` block — so SBR **was already active on master** and hard-crash failover already worked; the cluster was never NoDowning. Proven by the #9 negative control (only an explicit `downing-provider-class = ""` breaks failover). What *was* real: the strategy was implicit-in-framework-default and hard-kill failover was untested. Both closed: `BuildClusterOptions` now sets `KeepOldestOption { DownIfAlone = true }` explicitly (`Cluster/ServiceCollectionExtensions.cs:108-116`, `a81dea10`) with corrected-premise docs in the XML comment + `akka.conf:40-47` (`eaf78aad`), and `HardKillFailoverTests` + `TwoNodeClusterHarness.HardKillNodeAAsync` (transport shutdown → B downs the alone oldest and takes over as `driver` role-leader, ~35 s, stable-streak polled) guard the failover *outcome* regardless of activation path (`a25c9ed0`). The Failover trait is not excluded by the CI filter (`v2-ci.yml:61`), so it runs in the whole-solution leg. |
|
||||
| S2 | LDAP auth blocks SDK session-activation thread, no timeout | **STILL OPEN.** `OpcUaApplicationHost.cs:271-274` still block-bridges with `CancellationToken.None`; `LdapOpcUaUserAuthenticator.cs:30-48` adds no timeout; `LdapOptions.ToLibraryOptions()` (`Security/Ldap/LdapOptions.cs:94-116`) still projects no timeout option. No remediation branch existed. |
|
||||
| S3 | HistoryRead block-bridges gateway per node, sequentially | **STILL OPEN.** `OtOpcUaNodeManager.cs:1911, 2059, 2171, 2203` unchanged — per-node `.GetAwaiter().GetResult()` with `CancellationToken.None`, bounded only by the 30 s gateway `CallTimeout` against `MaxRequestThreadCount = 100`. |
|
||||
| S4 | Primary-gate default-allow while role unknown (dual-primary window) | **STILL OPEN.** `DriverHostActor.cs:977` (alarm emit), `:1039` (write gate), `:1095` (ack gate) still gate on `_localRole is Secondary or Detached` with null → allow; `OnRedundancyStateChanged` `:1126`. Listed in STATUS.md's "suggested next", not started. |
|
||||
| S5 | Redundancy NodeId string-match, no mismatch diagnostic | **STILL OPEN.** `OpcUaPublishActor.cs:512` unchanged; no log when the local node is absent from a snapshot. |
|
||||
| S6 | Driver-child restart loses desired-subscription state; no backoff supervision | **STILL OPEN.** Repo-wide grep: still zero `SupervisorStrategy` overrides in Runtime/ControlPlane; desired-subs still arrive post-spawn into actor state (`DriverInstanceActor.cs:84, 190-203, 912`). Still no supervision/restart test. |
|
||||
| S7 | `PostStop` blocks on `ShutdownAsync(CancellationToken.None)` | **STILL OPEN.** Now at `DriverInstanceActor.cs:1009`. |
|
||||
| S8 | Part 9 commands / native acks at-most-once, client sees Good | **STILL OPEN.** `OtOpcUaServerHostedService.cs:139-194` routers unchanged; no drop metric. |
|
||||
| S9 | No server-cert renewal/expiry monitoring | **STILL OPEN.** `OpcUaApplicationHost.cs:305-317` unchanged. |
|
||||
| S10 | SDK start failure swallowed; node runs with Null sinks, invisible to /health | **STILL OPEN.** `OtOpcUaServerHostedService.cs:110-129` unchanged. |
|
||||
| S11 | `AuditWriterActor` unbounded buffer, drops batches on outage | **STILL OPEN** (and still moot — see U3). |
|
||||
| P1 | Any structural deploy change → full rebuild severs all monitored items | **STILL OPEN.** `AddressSpaceApplier.cs:134-154` / `OtOpcUaNodeManager.cs:1690-1736` unchanged. The U2 reflection guard (now merged) was the declared prerequisite gate for the surgical pure-adds work; the work itself has not started. |
|
||||
| P2 | Per-value global-lock write, no batching | **STILL OPEN.** `OtOpcUaNodeManager.cs:266-281` unchanged. Same guard-unblocked, not-started status as P1. |
|
||||
| P3 | HistoryRead-Events unpaged, `0` → `int.MaxValue` | **STILL OPEN.** `OtOpcUaNodeManager.cs:1914-1954` unchanged. |
|
||||
| P4 | Deploy re-runs four Materialise passes (low) | **STILL OPEN.** `OpcUaPublishActor.cs:338-354` unchanged. |
|
||||
| C1 | `ServerHistorian` bound imperatively in five places | **STILL OPEN.** `Program.cs:121`, `Runtime/ServiceCollectionExtensions.cs:132, 168` (+ `AddAlarmHistorian`), `OtOpcUaServerHostedService.cs:89` — all unchanged. |
|
||||
| C2 | Two-tier options validation; `DevStubMode` only log-warned in prod | **STILL OPEN.** `OtOpcUaLdapAuthService.cs:79-88` unchanged. |
|
||||
| C3 | Static Serilog logger islands + node-manager CS0618 pragmas | **STILL OPEN.** `Runtime/ServiceCollectionExtensions.cs:90, 100, 136` unchanged. (The *new* resilience code in `DriverFactoryBootstrap` correctly uses `ILoggerFactory` — the convention is honored by new code, not retrofitted.) |
|
||||
| C4 | `IHistorianProvisioning` resolve has no missing-registration warning | **STILL OPEN.** Now `Runtime/ServiceCollectionExtensions.cs:303` (shifted +5). The new invoker-factory resolve (`:219-220`) is likewise silent, but deliberately so (admin nodes legitimately lack it) and is backstopped by the new DI guard test — see positive notes. |
|
||||
| U1 | CLAUDE.md "KNOWN LIMITATION 2" stale (ref-feed gap closed in code) | **FIXED.** Rewritten on the docs branch (`9fadead6`, merged `b67bd9e8`): now reads "value-capture path is code-complete… treat continuous value-capture as unproven-in-production rather than absent". The residual recommendation (explicit restart-convergence test) remains open — folded into U6. |
|
||||
| U2 | Deferred-sink forwarding trap half test-guarded | **FIXED.** `DeferredSinkForwardingReflectionTests` (`tests/Core/ZB.MOM.WW.OtOpcUa.Commons.Tests/OpcUa/`, `a65c2ced`, merged `1d7e7caa`): reflects over every method of every forwarding interface of `DeferredAddressSpaceSink` + `DeferredServiceLevelPublisher`, invokes each against a `DispatchProxy` recording inner (auto-covers future members), asserts arrival, plus a guard-of-the-guard pinning the wrapper's interface set. Negative control verified per STATUS. This retires the F10b/PR#423 trap class and unblocks P1/P2. |
|
||||
| U3 | Structured audit pipeline has zero producers | **STILL OPEN.** `AuditOutcomeMapper.cs:12-18` / `Security/Audit/AuditActor.cs:18-24` unchanged. |
|
||||
| U4 | `FleetStatusBroadcaster` heartbeat dead code + host-only NodeId key | **STILL OPEN.** `FleetStatusBroadcaster.cs:38, 110-121, 152-159` unchanged. |
|
||||
| U5 | Native Part 9 conditions Acknowledge-only to driver | **STILL OPEN.** `OtOpcUaNodeManager.cs:647-658, 698-703` unchanged. |
|
||||
| U6 | Test gaps concentrate on the fragile seams | **PARTIALLY FIXED.** Closed: hard-kill failover (#9, the biggest single blind spot); whole-solution CI leg + fail-on-skip gate + CLI deflake (`10b89830`, `v2-ci.yml:58-61` + `scripts/ci/assert-not-all-skipped.sh`) — green-because-all-skipped is now detected. Still open: DPS delivery of redundancy/ServiceLevel state untested over a real mediator; no actor supervision/restart test (S6); outbox restart-durability untested; recorder restart-convergence untested (from U1); `OpcUaServer.IntegrationTests` still contains exactly one test (`DualEndpointTests`). |
|
||||
| U7 | `IHistoryWriter` Null-wired; H2 HistoryUpdate unimplemented; `LdapAuthFailure` overload | **STILL OPEN** (tracked backlog). |
|
||||
| U8 | `BuildSecurityPolicies` doc claims a log that doesn't exist | **STILL OPEN.** `OpcUaApplicationHost.cs:376-418` unchanged. |
|
||||
| U9 | `EnsureVariable` historize-intent invariant lives in comments | **STILL OPEN.** `OtOpcUaNodeManager.cs:1345-1349` unchanged. |
|
||||
|
||||
---
|
||||
|
||||
## Architecture overview
|
||||
|
||||
### Host composition
|
||||
|
||||
`Program.cs` (`src/Server/ZB.MOM.WW.OtOpcUa.Host/Program.cs`) is a single fused binary whose wiring is driven by the `OTOPCUA_ROLES` env var (`admin`, `driver`, or both). Notable mechanics: per-role appsettings overlays with env-vars/args re-appended so deployment overrides keep precedence (`Program.cs:61-73`); CWD anchored to `AppContext.BaseDirectory` for Windows-service path sanity (`:83`); Serilog via the shared `AddZbSerilog` with the static `Log.Logger` assigned post-build so Akka's `SerilogLogger` routes actor logs (`:305`).
|
||||
|
||||
On `driver` nodes the host registers the historian stack (three config-gated layers: `AddServerHistorian` read client, `AddAlarmHistorian` SQLite store-and-forward, `AddHistorianProvisioning` EnsureTags, plus the ContinuousHistorization outbox/writer pair), the driver-factory registry (8 drivers, `DriverFactoryBootstrap.cs`), the LDAP data-plane authenticator, and — critically — the **late-binding seams**: `DeferredAddressSpaceSink` and `DeferredServiceLevelPublisher` are DI singletons that actors capture at construction; `OtOpcUaServerHostedService` swaps the real SDK-backed implementations in after `StandardServer` start and swaps Nulls back in on stop (`OtOpcUaServerHostedService.cs:131, 226-231, 241-253`).
|
||||
On `driver` nodes the host registers the historian stack (three config-gated layers plus the ContinuousHistorization outbox/writer pair), the driver-factory registry (8 drivers, `DriverFactoryBootstrap.cs`), **the Phase 6.1 resilience infrastructure** (new since 9cad9ed0: process-singleton `DriverResilienceStatusTracker` + `DriverResiliencePipelineBuilder` + the `IDriverCapabilityInvokerFactory` closing over `DriverFactoryRegistry.GetTier`, `DriverFactoryBootstrap.cs:53-71`), the LDAP data-plane authenticator, and the late-binding seams: `DeferredAddressSpaceSink` / `DeferredServiceLevelPublisher` are DI singletons that actors capture at construction; `OtOpcUaServerHostedService` swaps real SDK-backed implementations in after `StandardServer` start and Nulls back on stop (`OtOpcUaServerHostedService.cs:131, 226-231, 241-253`).
|
||||
|
||||
### OPC UA server & node manager
|
||||
|
||||
`OtOpcUaSdkServer` (a `StandardServer`) creates the single `OtOpcUaNodeManager` (`CustomNodeManager2`, 2 424 lines) that owns the entire writable address space: UNS Area/Line/Equipment folders, typed equipment-tag variables, Part 9 `AlarmConditionState` nodes, historized-node registration, and four HistoryRead override arms (Raw with synthesized continuation-point paging + tie-cluster over-fetch, Processed, AtTime, Events). Reverse paths out of the node manager are deliberately Akka-free delegates/interfaces set by the Host at start: `AlarmCommandRouter` (→ DPS `alarm-commands` topic), `NativeAlarmAckRouter` (→ `DriverHostActor.RouteNativeAlarmAck`), `NodeWriteGateway` (→ `ActorNodeWriteGateway` Ask), `HistorianDataSource` (→ gateway read client). Inbound writes are role-gated (`WriteOperate`), dispatched fire-and-forget under the SDK `Lock`, optimistically applied, and self-corrected (Bad-quality blip + revert + Part 8 audit event) on a failed device outcome.
|
||||
Unchanged since 9cad9ed0. `OtOpcUaSdkServer` (a `StandardServer`) creates the single `OtOpcUaNodeManager` (`CustomNodeManager2`, 2 424 lines) owning the entire writable address space: UNS folders, typed equipment-tag variables, Part 9 `AlarmConditionState` nodes, historized-node registration, four HistoryRead override arms. Reverse paths are Akka-free delegates set by the Host at start (`AlarmCommandRouter`, `NativeAlarmAckRouter`, `NodeWriteGateway`, `HistorianDataSource`). Inbound writes are role-gated (`WriteOperate`), dispatched fire-and-forget under the SDK `Lock`, optimistically applied, self-corrected on failed device outcome.
|
||||
|
||||
### Runtime actor topology (per driver node)
|
||||
|
||||
`WithOtOpcUaRuntimeActors` (`Runtime/ServiceCollectionExtensions.cs:194-347`) spawns, in order: `DbHealthProbeActor` → `DependencyMuxActor` → (gated) `ContinuousHistorizationRecorder` → `OpcUaPublishActor` (pinned single-thread dispatcher `opcua-synchronized-dispatcher`, owns the `AddressSpaceApplier`) → `PeerProbeSupervisor` → `DriverHostActor` → `HistorianAdapterActor`. `DriverHostActor` spawns one `DriverInstanceActor` per deployed driver plus the `VirtualTagHostActor` and `ScriptedAlarmHostActor` sub-hosts, maintains the `(driver, FullName) ⇄ NodeId` routing maps, and fans every `AttributeValuePublished` to the mux + the publish actor (`ForwardToMux`, `DriverHostActor.cs:561-589`). **No actor in Runtime overrides `SupervisorStrategy`** — everything runs under Akka's default one-for-one unlimited restart decider.
|
||||
`WithOtOpcUaRuntimeActors` (`Runtime/ServiceCollectionExtensions.cs:196-357`) spawns: `DbHealthProbeActor` → `DependencyMuxActor` → (gated) `ContinuousHistorizationRecorder` → `OpcUaPublishActor` (pinned dispatcher, owns `AddressSpaceApplier`) → `PeerProbeSupervisor` → `DriverHostActor` → `HistorianAdapterActor`. `DriverHostActor` spawns one `DriverInstanceActor` per deployed driver plus the VirtualTag/ScriptedAlarm sub-hosts.
|
||||
|
||||
**New since 9cad9ed0 — the resilience dispatch seam (#10/#13):** `DriverInstanceActor` now routes all **six** of its driver-capability call sites through an injected `IDriverCapabilityInvoker` (Core.Abstractions; Runtime stays deliberately Polly-free — verified: the csproj still references only `Core.Abstractions`, never `Core`): Write via `ExecuteWriteAsync` (`DriverInstanceActor.cs:597`), alarm-ack `AlarmAcknowledge` (`:650`), subscribe/unsubscribe `Subscribe` (`:693, :722`), alarm-subscribe `AlarmSubscribe` (`:803`), discover `Discover` (`:868`). Single-reference calls key the per-host breaker via `IPerCallHostResolver` (`:957`; implemented by Modbus/AbCip/AbLegacy/FOCAS/TwinCAT with a safe fall-back-to-instance-id); bulk calls key on the instance id. Per-instance `ResilienceConfig` from the deploy artifact (`DeploymentArtifact.cs:171-185`) is layered onto tier defaults by `DriverCapabilityInvokerFactory.Create`, which also `Invalidate`s the instance's cached pipelines so a respawn with changed options rebuilds them; `DriverSpawnPlanner` forces stop+respawn on a ResilienceConfig change while a pure DriverConfig change stays an in-place delta (`DriverSpawnPlan.cs:45-56`). Generation-suffixed actor names (`DriverHostActor.cs:1666-1670`, pre-existing) make the stop→respawn name-collision-safe, and the apply path re-pushes desired subscriptions after reconcile (`:1215-1225`), so a respawned child converges. The OTOPCUA0001 analyzer (wired tree-wide via `Directory.Build.props:48`) is the compile-time guard that every guarded call stays wrapped; `ResilienceInvokerFactoryRegistrationTests` guards that production DI binds the *real* factory, not the pass-through.
|
||||
|
||||
**No actor in Runtime overrides `SupervisorStrategy`** — everything still runs under Akka's default one-for-one unlimited restart decider (S6).
|
||||
|
||||
### Redundancy model
|
||||
|
||||
Non-transparent warm redundancy. `RedundancyStateActor` (admin-role cluster singleton, `ControlPlane/Redundancy/RedundancyStateActor.cs`) derives Primary/Secondary purely from Akka's `driver` role-leader, debounces 250 ms, publishes `RedundancyStateChanged` on the DPS `redundancy-state` topic, and re-publishes on a 10 s heartbeat (the late-subscriber fix). Consumers: `OpcUaPublishActor` computes ServiceLevel (`ServiceLevelCalculator` + DB-health probe + peer OPC UA probes; 240/200/100/0 bands, first-publish-always so the SDK's 255 default never stands — `OpcUaPublishActor.cs:418-437, 508-542`); `DriverHostActor`, `ScriptedAlarmHostActor`, and `HistorianAdapterActor` use the same snapshot to Primary-gate device writes, alarm emission, and historization. There is no lease or fencing token — correctness rests entirely on Akka's single-role-leader guarantee within a converged cluster.
|
||||
Non-transparent warm redundancy, unchanged in code; the *understanding* of it is corrected (S1/#11). `RedundancyStateActor` derives Primary/Secondary from Akka's `driver` role-leader, debounces 250 ms, publishes on DPS `redundancy-state`, re-publishes on a 10 s heartbeat. The split-brain resolver is **active and now explicit**: `BuildClusterOptions` sets `KeepOldestOption { DownIfAlone = true }` (correct for a 2-node pair; `keep-majority`/`static-quorum` would be wrong), consistent with the `akka.conf` block (`stable-after = 15s`, which only HOCON can express). Hard-crash failover is proven by `HardKillFailoverTests` (~35 s in-process). There is still no lease or fencing token — correctness rests on Akka's single-role-leader guarantee within a converged cluster, and the boot-window default-allow (S4) remains the data-plane soft spot.
|
||||
|
||||
---
|
||||
|
||||
## Findings
|
||||
|
||||
Severity scale: **Critical** (correctness/availability broken in a realistic scenario) · **High** (significant risk or debt, plan a fix) · **Medium** (real but bounded) · **Low** (hygiene).
|
||||
Severity scale: **Critical** (correctness/availability broken in a realistic scenario) · **High** (significant risk or debt, plan a fix) · **Medium** (real but bounded) · **Low** (hygiene). Fixed/invalidated findings live only in the status table above; still-open findings keep their IDs; new findings continue the scheme.
|
||||
|
||||
### 1. Stability
|
||||
|
||||
**S1 — CRITICAL: The split-brain resolver is configured but never activated; unreachable members are never downed.**
|
||||
`akka.conf` carries a full `split-brain-resolver { active-strategy = "keep-oldest" … }` block (`src/Core/ZB.MOM.WW.OtOpcUa.Cluster/Resources/akka.conf:40-46`), but there is **no `akka.cluster.downing-provider-class` anywhere in the repo**, and `WithClustering` sets only `SeedNodes`/`Roles` — not `ClusterOptions.SplitBrainResolver` (`src/Core/ZB.MOM.WW.OtOpcUa.Cluster/ServiceCollectionExtensions.cs:80-84`; repo-wide grep for `downing-provider|SplitBrainResolver` returns nothing). In Akka.NET the SBR HOCON block is inert without the provider, so the cluster runs the default **NoDowning** provider. Consequences: (a) a *hard-crashed* node stays "unreachable" forever, so **cluster singletons (`ConfigPublishCoordinator`, `AdminOperationsActor`, `RedundancyStateActor`, `AuditWriterActor`) never fail over** and the `driver` role-leader never moves — i.e. redundancy failover works only for graceful shutdowns; (b) a network partition never resolves, and with `min-nr-of-members = 1` each side elects its own role-leader, so **both sides advertise ServiceLevel 240 indefinitely**. The two-node integration harness only exercises graceful `StopAsync`, which is why this is invisible in tests.
|
||||
*Recommendation:* set `ClusterOptions.SplitBrainResolver = SplitBrainResolverOption` (or `downing-provider-class = "Akka.Cluster.SBR.SplitBrainResolverProvider"` in `akka.conf`), then add a two-node kill-9 failover test asserting singleton handover + ServiceLevel demotion.
|
||||
|
||||
**S2 — HIGH: LDAP authentication blocks the OPC UA session-activation thread with an effectively non-cancellable, non-configurable timeout.**
|
||||
The SDK invokes `ImpersonateUser` synchronously; `OpcUaApplicationHost.HandleImpersonation` block-bridges the authenticator (`OpcUaApplicationHost.cs:271-273`) with `CancellationToken.None`. The doc comment shifts timeout responsibility to the authenticator ("callers must enforce their own timeouts"), but `LdapOpcUaUserAuthenticator` adds none (`Host/OpcUa/LdapOpcUaUserAuthenticator.cs:30-48`), and the shared library's `AuthenticateAsync` is a **synchronous method wrapped in `Task.FromResult`** that observes the token only at entry; its 10 s connect timeout is a library default that OtOpcUa's `LdapOptions.ToLibraryOptions()` does not project or expose (`Security/Ldap/LdapOptions.cs:94-107`). Every authentication also opens a fresh TCP connection + service-account bind + search + user bind (no pooling). Under a directory outage or slow DC, session activations stall serially for ~10-20 s each on SDK threads.
|
||||
*Recommendation:* enforce a hard timeout in `LdapOpcUaUserAuthenticator` (`Task.WhenAny` + deny on timeout), surface a `Security:Ldap:TimeoutMs` option, and consider a short negative-cache to shed load during an outage. Fail-closed semantics are otherwise correct end-to-end (deny on error, opaque messages, zero-role fallback on mapper fault).
|
||||
The SDK invokes `ImpersonateUser` synchronously; `OpcUaApplicationHost.HandleImpersonation` block-bridges the authenticator (`OpcUaApplicationHost.cs:271-274`) with `CancellationToken.None`. The doc comment shifts timeout responsibility to the authenticator, but `LdapOpcUaUserAuthenticator` adds none (`Host/OpcUa/LdapOpcUaUserAuthenticator.cs:30-48`), and the shared library's `AuthenticateAsync` is synchronous-in-`Task.FromResult`; its 10 s connect timeout is a library default that `LdapOptions.ToLibraryOptions()` does not project or expose (`Security/Ldap/LdapOptions.cs:94-116`). Every authentication opens a fresh TCP connection + service-account bind + search + user bind. Under a directory outage, session activations stall serially ~10-20 s each on SDK threads.
|
||||
*Recommendation:* enforce a hard timeout in `LdapOpcUaUserAuthenticator` (`Task.WhenAny` + deny on timeout), surface `Security:Ldap:TimeoutMs`, consider a short negative-cache. Fail-closed semantics are otherwise correct end-to-end.
|
||||
|
||||
**S3 — HIGH: HistoryRead block-bridges the gateway per node, sequentially, on SDK request threads.**
|
||||
All four HistoryRead arms call `.GetAwaiter().GetResult()` per node handle with `CancellationToken.None` (`OtOpcUaNodeManager.cs:1902-1911, 2059, 2169-2171, 2201-2203`). This is safe w.r.t. the node-manager `Lock` (the overrides run outside it — correctly documented), but bounded only by the gateway client's per-call `CallTimeout` (30 s default). A single request naming N historized nodes against a slow historian holds one SDK request thread up to N × 30 s; the server is configured `MaxRequestThreadCount = 100`, `MaxQueuedRequestCount = 200` (`OpcUaApplicationHost.cs:329-331`), so a handful of misbehaving history clients can exhaust the request pool and degrade *all* OPC UA services.
|
||||
*Recommendation:* parallelize per-node reads within a batch (bounded), pass a per-request deadline token, and consider a server-side concurrent-HistoryRead limiter.
|
||||
All four HistoryRead arms call `.GetAwaiter().GetResult()` per node handle with `CancellationToken.None` (`OtOpcUaNodeManager.cs:1911, 2059, 2171, 2203`). Safe w.r.t. the node-manager `Lock` (the overrides run outside it — correctly documented), but bounded only by the gateway client's 30 s `CallTimeout`. A single request naming N historized nodes against a slow historian holds one SDK request thread up to N × 30 s against `MaxRequestThreadCount = 100` / `MaxQueuedRequestCount = 200` (`OpcUaApplicationHost.cs:329-331`) — a handful of misbehaving history clients can exhaust the pool and degrade all OPC UA services.
|
||||
*Recommendation:* parallelize per-node reads within a batch (bounded), pass a per-request deadline token, add a server-side concurrent-HistoryRead limiter.
|
||||
|
||||
**S4 — HIGH: Primary-gate "default-allow while role unknown" opens a dual-primary window on the data plane.**
|
||||
`DriverHostActor.HandleRouteNodeWrite` / `HandleRouteNativeAlarmAck` / native-alarm emission all gate on `_localRole is Secondary or Detached` and deliberately default to *allow* until the first redundancy snapshot arrives (`DriverHostActor.cs:1018-1026, 1074-1083, 956-960; OnRedundancyStateChanged :1109-1114`). Combined with DPS delivery being at-most-once and heartbeat-repaired every 10 s, a freshly-booted or snapshot-missed **secondary services device writes and emits alerts as if primary** for up to the heartbeat interval — and indefinitely if the NodeId identity mismatch (S5) bites. The boot-window rationale (single-node deploys) is sound, but the same signal gates *shared-field-device writes*.
|
||||
*Recommendation:* for the write/ack gates specifically, prefer default-*deny* once the cluster has >1 driver member (discoverable from cluster state), or require at least one received snapshot before servicing writes on multi-node clusters.
|
||||
`DriverHostActor` gates native-alarm emission, `HandleRouteNodeWrite`, and `HandleRouteNativeAlarmAck` on `_localRole is Secondary or Detached` and deliberately defaults to *allow* until the first redundancy snapshot arrives (`DriverHostActor.cs:977, 1039, 1095; OnRedundancyStateChanged :1126`). Combined with at-most-once DPS delivery heartbeat-repaired every 10 s, a freshly-booted or snapshot-missed **secondary services device writes and emits alerts as if primary** for up to the heartbeat interval — and indefinitely if the NodeId identity mismatch (S5) bites. The boot-window rationale (single-node deploys) is sound, but the same signal gates shared-field-device writes. The now-proven SBR failover (S1) does not narrow this window — it is a delivery/boot-order gap, not a downing gap.
|
||||
*Recommendation:* default-*deny* for the write/ack gates once the cluster has >1 driver member, or require one received snapshot before servicing writes on multi-node clusters. (Named in STATUS.md's "suggested next" — not started.)
|
||||
|
||||
**S5 — MEDIUM: Redundancy NodeId identity is string-matched across three independently-derived sources with no diagnostic on mismatch.**
|
||||
`OpcUaPublishActor` matches `n.NodeId == _localNode.Value` (`OpcUaPublishActor.cs:512`); the snapshot side derives `host:port` from the gossiped `Member.Address` (`RedundancyStateActor.cs:147-148`) while the local side derives it from configured `PublicHostname:Port` (`ClusterRoleInfo`). If `PublicHostname` ever diverges from the gossiped address (DNS vs IP, container advertised name), the node silently computes ServiceLevel 0 / keeps a stale role — the exact shape of the historical "silently inert delivery" bug, with no log distinguishing it from legitimate Detached.
|
||||
*Recommendation:* log-once (Warning) when the local node is absent from a received snapshot; assert the identity equality in a startup self-check.
|
||||
`OpcUaPublishActor` matches `n.NodeId == _localNode.Value` (`OpcUaPublishActor.cs:512`); the snapshot side derives `host:port` from the gossiped `Member.Address` (`RedundancyStateActor.cs:147-148`), the local side from configured `PublicHostname:Port`. Divergence (DNS vs IP, container advertised name) silently computes ServiceLevel 0 / stale role — the shape of the historical "silently inert delivery" bug, indistinguishable from legitimate Detached.
|
||||
*Recommendation:* log-once (Warning) when the local node is absent from a received snapshot; startup identity self-check.
|
||||
|
||||
**S6 — MEDIUM: A crashed-and-restarted `DriverInstanceActor` loses its message-delivered state (desired subscriptions) until the next deploy.**
|
||||
No actor in Runtime overrides `SupervisorStrategy` (grep confirms), so an exception in a driver child triggers Akka's default one-for-one **unlimited, non-backoff restart**. Restart re-runs `PreStart` (driver re-init from the Props spec) but the desired-subscription set arrives post-spawn via `SetDesiredSubscriptions` messages stored in actor state (`DriverInstanceActor.cs:84, 181, 861`); a restart wipes it and `DriverHostActor` has no restart-detection to re-send (restarts don't fire `Terminated`). A persistently-throwing driver also hot-loops restarts with no backoff.
|
||||
*Recommendation:* wrap driver children in `BackoffSupervisor` (restart with exponential backoff) and have the child request its subscription set from the parent in `PreStart`, or have the parent push it on a `PostRestart` signal. Add a supervision test (none exists — see U6).
|
||||
Still zero `SupervisorStrategy` overrides in Runtime (grep re-confirmed at f6eaa267): a throwing driver child restarts one-for-one, unlimited, no backoff. Restart re-runs `PreStart` but the desired-subscription set arrives post-spawn via `SetDesiredSubscriptions` stored in actor state (`DriverInstanceActor.cs:84, 190-203, 912`); a restart wipes it and the parent has no restart-detection to re-send. A persistently-throwing driver hot-loops. Note the new invoker does *not* change this: retries happen inside a single message handler; a handler that ultimately throws still escalates to the default decider.
|
||||
*Recommendation:* `BackoffSupervisor` for driver children + child-requests-state-on-restart (or parent push on `PostRestart`). Still no supervision test (U6).
|
||||
|
||||
**S7 — MEDIUM: `DriverInstanceActor.PostStop` blocks the actor shutdown path on driver shutdown.**
|
||||
`_driver.ShutdownAsync(CancellationToken.None).GetAwaiter().GetResult()` (`DriverInstanceActor.cs:950`) is the only synchronous block in the Runtime actors; with `CancellationToken.None` a hung protocol stack can stall stop/re-deploy of the whole child set.
|
||||
*Recommendation:* bound it (`WaitAsync(timeout)`) and log-and-abandon on expiry.
|
||||
`_driver.ShutdownAsync(CancellationToken.None).GetAwaiter().GetResult()` (`DriverInstanceActor.cs:1009`) remains the only synchronous block in the Runtime actors; a hung protocol stack stalls stop/re-deploy of the whole child set. Slightly more load-bearing now: #13 makes stop+respawn the path for every ResilienceConfig edit.
|
||||
*Recommendation:* bound it (`WaitAsync(timeout)`), log-and-abandon on expiry.
|
||||
|
||||
**S8 — MEDIUM: Inbound Part 9 commands and native acks are at-most-once; the client sees `Good` even if the command is lost.**
|
||||
The alarm-command router is a fire-and-forget DPS `Publish` and the native-ack router a fire-and-forget `Tell`, both wrapped in catch-log-drop (`OtOpcUaServerHostedService.cs:139-194`); the node manager has already returned `Good` so the SDK applies local condition state regardless (`OtOpcUaNodeManager.cs:769-772`). A mediator hiccup or missing `DriverHostActor` silently strands the engine/upstream state. This is a documented, deliberate trade-off (non-blocking under `Lock`), but it is invisible in operation.
|
||||
*Recommendation:* add a counter/metric for dropped routes (both routers currently only log at Warning) and consider an engine-side reconciliation sweep.
|
||||
Unchanged (`OtOpcUaServerHostedService.cs:139-194`; `OtOpcUaNodeManager.cs:769-772`). Deliberate (non-blocking under `Lock`), but invisible in operation.
|
||||
*Recommendation:* counter/metric for dropped routes; engine-side reconciliation sweep.
|
||||
|
||||
**S9 — MEDIUM: Certificate lifecycle has no renewal or expiry monitoring.**
|
||||
The application certificate is auto-created self-signed with SDK defaults (2048-bit, **12-month lifetime**) and only checked at boot (`OpcUaApplicationHost.cs:305-317`). Nothing monitors expiry; ~12 months after first deploy, Sign/SignAndEncrypt endpoints and UserName-token encryption fail on live servers with no advance warning.
|
||||
*Recommendation:* startup + periodic expiry check with a metric/health-check and a documented rotation runbook (the AdminUI cert-actions work covers client certs; the *server* cert lifecycle is the gap).
|
||||
Unchanged (`OpcUaApplicationHost.cs:305-317`): self-signed, 12-month lifetime, checked only at boot. ~12 months after first deploy, Sign/SignAndEncrypt endpoints and UserName-token encryption fail with no warning.
|
||||
*Recommendation:* startup + periodic expiry check surfaced via health/metric; rotation runbook.
|
||||
|
||||
**S10 — LOW: SDK start failure is swallowed and the node keeps running with Null sinks.**
|
||||
`StartAsync` catches, logs, and returns (`OtOpcUaServerHostedService.cs:110-129`) — a deliberate availability choice (AdminUI stays up), but the node then silently no-ops all OPC UA work and nothing surfaces the condition through `/health` or ServiceLevel (which is itself served by the dead server).
|
||||
*Recommendation:* set a health-check flag consumed by `MapOtOpcUaHealth` so fleet status shows the degraded state.
|
||||
**S10 — LOW: SDK start failure is swallowed and the node keeps running with Null sinks** (`OtOpcUaServerHostedService.cs:110-129`); nothing surfaces it through `/health`.
|
||||
|
||||
**S11 — LOW: `AuditWriterActor` buffer is unbounded between flushes and drops whole batches on DB outage** (`ControlPlane/Audit/AuditWriterActor.cs:35, 83, 113-116`). Explicitly best-effort by contract; flag only if compliance-grade audit is ever required. (Currently moot — see U3: the pipeline has no producers.)
|
||||
**S11 — LOW: `AuditWriterActor` buffer is unbounded between flushes and drops whole batches on DB outage** (`ControlPlane/Audit/AuditWriterActor.cs:35, 83, 113-116`). Explicitly best-effort; currently moot (U3: no producers).
|
||||
|
||||
**Positive stability notes (worth preserving):** the node-manager locking discipline is exemplary — every mutation under `Lock`, every `Server.ReportEvent` deliberately outside it (`OtOpcUaNodeManager.cs:1101-1136, 1503-1504, 1595-1614`), the write-dispatch continuation forced off the SDK thread via `RunContinuationsAsynchronously` (`:884-897`), and the revert guarded by "still holds the optimistic value" (`ShouldRevert`, `:1048`). The Deferred-sink swap-to-Null on stop prevents post-stop writes hitting a disposed manager. `ConfigPublishCoordinator` has real crash-recovery (rebuilds expected-acks from DB on `PreStart`). `HandleServiceLevelChanged`'s first-publish-always correctly defeats the SDK's 255 default. `ActorNodeWriteGateway` bounds its Ask at 10 s with the inner driver Ask at 8 s and the backend call at 5 s — a correctly nested timeout ladder.
|
||||
**S12 — MEDIUM (NEW): The write dispatch site hardcodes `isIdempotent: true`, bypassing the per-tag `WriteIdempotentAttribute` contract.**
|
||||
`DriverInstanceActor.HandleWriteTagAttribute` routes through `_invoker.ExecuteWriteAsync(ResolveHost(msg.TagId), isIdempotent: true, …)` (`DriverInstanceActor.cs:597-601`). The `WriteIdempotentAttribute` (`Core.Abstractions/WriteIdempotentAttribute.cs`) — whose whole purpose is per-tag opt-in to write retries, and which `DriverCapability.Write`'s doc and `ExecuteWriteAsync`'s non-idempotent arm are built around — has **zero readers anywhere in the tree**. Today this is behavior-neutral (the Write tier default is `RetryCount = 0`, and the site comment says so), but the moment an operator sets a Write retry in a per-instance `ResilienceConfig` (#13 made exactly this reachable), retries apply to **every** write on that instance — including command-shaped tags (write-1-to-pulse, PLC auto-resets) where a duplicate write double-triggers. The hardcode silently converts a per-tag safety contract into an instance-wide one.
|
||||
*Recommendation:* either resolve idempotence per tag (plumb the attribute/tag-config through `WriteTagAttribute`) before honoring a configured Write retry, or document loudly on the `ResilienceConfig` authoring surface that Write.RetryCount > 0 retries all writes on the instance. Until then the safest change is `isIdempotent: false` with per-tag opt-in later — that keeps the non-idempotent no-retry arm authoritative.
|
||||
|
||||
**S13 — LOW (NEW): Respawn-on-ResilienceConfig-change has a narrow stale-pipeline re-cache race.**
|
||||
`DriverCapabilityInvokerFactory.Create` invalidates the instance's cached pipelines (`DriverCapabilityInvokerFactory.cs:71`) before the *old* child has terminated (`StopChild` is an async `Context.Stop`, `DriverHostActor.cs:1278-1280, 1658-1664`). An old-child handler already dequeued and mid-await can reach its invoker call *after* the invalidate; `GetOrCreate` then re-caches a pipeline built from the old child's captured options snapshot under the same `(instance, host, capability)` key — and the cache ignores options on a hit (`DriverResiliencePipelineBuilder.cs:75-79`), so the new child silently serves the stale pipeline until the next respawn. Window is milliseconds (the mailbox suspends on Stop; a pipeline is resolved once per `ExecuteAsync` at call start), but the failure is the exact "options-blind cache hit" the invalidate was added to prevent.
|
||||
*Recommendation:* include an options fingerprint in `PipelineKey` (removes the invalidate-ordering dependency entirely), or invalidate from the *new* invoker lazily on first use after the old child's `Terminated`.
|
||||
|
||||
**Positive stability notes (worth preserving):** the node-manager locking discipline remains exemplary (every mutation under `Lock`, every `ReportEvent` outside it, write-dispatch continuation forced off the SDK thread, revert guarded by `ShouldRevert`). The new resilience wiring is carefully done: exception transparency holds (pipeline exceptions — `TimeoutRejectedException`, `BrokenCircuitException` — surface through the same `catch (Exception)` arms the raw calls used, so failure paths like `WriteAttributeResult(false, …)` and Reconnecting transitions are unchanged); retry `ShouldHandle` excludes `OperationCanceledException`, so the actor's outer `cts` deadlines (5 s write/ack/unsub, 10 s subscribe, 30 s discover) still hard-bound mailbox suspension even with retries enabled; the Discover site's ConfigureAwait analysis (invoker-internal `ConfigureAwait(false)` does not leak to the caller's actor-context resume) is correct and covered by a real-`CapabilityInvoker` actor-context test; pipeline order Timeout→Retry→Breaker means the tier timeout bounds the *total* including retries. The `TwoNodeClusterHarness.HardKillNodeAAsync` design (transport shutdown as crash sim, stable-streak takeover polling, diagnostic timeout message naming the inactive-resolver signature) is a model integration test.
|
||||
|
||||
### 2. Performance
|
||||
|
||||
**P1 — HIGH: Any structural deploy change triggers a full address-space teardown/rebuild that severs every client's monitored items server-wide.**
|
||||
`AddressSpaceApplier.Apply` forces `RebuildAddressSpace()` for *any* added/removed equipment, alarm, tag, or virtual tag (`AddressSpaceApplier.cs:134-154`); the rebuild removes every variable/folder/condition under one `Lock` hold and re-materialises from scratch (`OtOpcUaNodeManager.cs:1690-1736`). Recreated nodes are new `NodeState` instances, so existing client subscriptions on the old nodes go dead — the F10b surgical path (`UpdateTagAttributes`/`UpdateFolderDisplayName`) exists precisely because of this, but covers only attribute edits and renames. Adding one tag to one equipment still drops every subscription on a production SCADA server.
|
||||
*Recommendation:* extend the surgical path to pure-add deltas first (`EnsureFolder`/`EnsureVariable` are already idempotent — adds need no teardown; `RaiseNodesAddedModelChange` already exists for announcement), then scope removals per-equipment. This is the highest-leverage performance/stability item in the subsystem.
|
||||
Unchanged (`AddressSpaceApplier.cs:134-154`; `OtOpcUaNodeManager.cs:1690-1736`): any added/removed equipment, alarm, tag, or virtual tag forces `RebuildAddressSpace()`; recreated nodes are new `NodeState` instances, so existing subscriptions go dead. The F10b surgical path covers only attribute edits and renames. Adding one tag still drops every subscription on a production SCADA server. **The declared prerequisite (U2 reflection forwarding guard) is now merged — the surgical pure-adds work is unblocked but not started.**
|
||||
*Recommendation:* extend the surgical path to pure-add deltas first (`EnsureFolder`/`EnsureVariable` are already idempotent; `RaiseNodesAddedModelChange` exists), then scope removals per-equipment. Still the highest-leverage item in the subsystem.
|
||||
|
||||
**P2 — MEDIUM: The value hot path is one global lock acquisition + one actor message per value, with no batching.**
|
||||
Per published value: driver child `Tell` → `DriverHostActor.ForwardToMux` (dictionary lookup + up to two `Tell`s, `DriverHostActor.cs:561-589`) → `OpcUaPublishActor` (pinned thread) → `sink.WriteValue`, which takes the **global node-manager `Lock`** per value (`OtOpcUaNodeManager.cs:266-281`), contending with SDK read/subscription/publish threads. The mux itself is lean (`DependencyMuxActor.cs:95-108` — early-drop for uninterested refs, set-based fan-out). At high tag counts × fast poll intervals this serialises everything through one lock and allocates one record per hop.
|
||||
*Recommendation:* add a batched `WriteValues(IReadOnlyList<…>)` sink call (one lock hold per driver publish cycle) — the seam (`IOpcUaAddressSpaceSink`) makes this a contained change, but remember the Deferred-wrapper forwarding trap (U2).
|
||||
Unchanged: driver child `Tell` → `ForwardToMux` → `OpcUaPublishActor` → `sink.WriteValue` taking the global node-manager `Lock` per value (`OtOpcUaNodeManager.cs:266-281`). Same unblocked-by-U2, not-started status as P1.
|
||||
*Recommendation:* batched `WriteValues(IReadOnlyList<…>)` sink call — the reflection guard now auto-covers the new member's forwarding.
|
||||
|
||||
**P3 — MEDIUM: HistoryRead-Events has no paging and translates `NumValuesPerNode == 0` to `int.MaxValue`.**
|
||||
`EventMaxEvents` deliberately maps "no limit" to unbounded (`OtOpcUaNodeManager.cs:1944-1954`), and the Events arm never issues continuation points ("full window in one shot", `:1914-1921`). A wide time window over a busy alarm source materialises the entire result in memory on both gateway and server.
|
||||
*Recommendation:* impose a server-side max (mirroring `MaxTieClusterOverfetch`'s philosophy) and fail loudly or page beyond it.
|
||||
**P3 — MEDIUM: HistoryRead-Events has no paging and translates `NumValuesPerNode == 0` to `int.MaxValue`** (`OtOpcUaNodeManager.cs:1944-1954`; no continuation points `:1914-1921`). Unchanged.
|
||||
|
||||
**P4 — LOW: Every deploy re-runs the four `Materialise*` passes over the full composition** (`OpcUaPublishActor.cs:338-354`) — acceptable because `EnsureFolder`/`EnsureVariable` early-return on existing ids, but each pass still takes/releases the `Lock` per node; fold into P2's batching if it shows up in profiles. On the positive side, the Raw-paging tie-cluster over-fetch is explicitly bounded (`MaxTieClusterOverfetch`, default 65 536) with a loud-fail backstop, and `HandleDiscoveredNodes` has an unchanged-plan short-circuit preventing ~15× re-subscribe blips during the discovery window (`DriverHostActor.cs:658-673`) — both good designs.
|
||||
**P4 — LOW: Every deploy re-runs the four `Materialise*` passes over the full composition** (`OpcUaPublishActor.cs:338-354`) — acceptable (early-return on existing ids); fold into P2 if it profiles hot. Positive: Raw-paging tie-cluster over-fetch stays bounded; `HandleDiscoveredNodes` unchanged-plan short-circuit preserved. New-code note: the resilience pipeline adds one `GetOrCreate` dictionary hit + options-snapshot closure per capability call on *dispatch* paths only (write/ack/subscribe/discover) — the per-value publish hot path is untouched by #10, which is the right call.
|
||||
|
||||
### 3. Conventions
|
||||
|
||||
**C1 — MEDIUM: The `ServerHistorian` section is bound imperatively in five places with no `IOptions` registration.**
|
||||
`Program.cs:120-122` (own bind + `Validate()` warnings), `AddServerHistorian`, `AddAlarmHistorian`, `AddHistorianProvisioning` (each re-`Get<>`s the section, `Runtime/ServiceCollectionExtensions.cs:86, 132, 168`), and `OtOpcUaServerHostedService`'s constructor (`.cs:88-90`, self-justified as "self-contained"). Warnings can log twice; five bind sites will drift.
|
||||
*Recommendation:* one `AddValidatedOptions<ServerHistorianOptions>` and inject `IOptions<>` everywhere, matching the `OpcUa`/`Ldap` pattern that already exists (`Program.cs:102, 254-255`).
|
||||
Unchanged: `Program.cs:121`, `AddServerHistorian`/`AddAlarmHistorian`/`AddHistorianProvisioning` (`Runtime/ServiceCollectionExtensions.cs:132, 168` et al.), `OtOpcUaServerHostedService.cs:89`.
|
||||
*Recommendation:* one `AddValidatedOptions<ServerHistorianOptions>` + `IOptions<>` everywhere, matching the existing `OpcUa`/`Ldap` pattern.
|
||||
|
||||
**C2 — MEDIUM: Options validation is two-tier and the tiers disagree.**
|
||||
`OpcUa` and `Security:Ldap` get real fail-fast `ValidateOnStart` validators (`OpcUaApplicationHostOptionsValidator` requires ≥1 security profile; `LdapOptionsValidator` refuses plaintext-without-opt-in) — good. `ServerHistorian` / `ContinuousHistorization` / `AlarmHistorian` only get advisory `Validate()` warning lists, and `DevStubMode=true` (accept-any-credentials Administrator grant) is merely log-warned, never blocked, in production (`OtOpcUaLdapAuthService.cs:79-88`).
|
||||
*Recommendation:* promote the historian sections to the validator pattern; add an environment-gated guard (or at minimum a startup validator failure) for `DevStubMode` outside Development.
|
||||
Unchanged: `OpcUa`/`Security:Ldap` get fail-fast `ValidateOnStart` validators; `ServerHistorian`/`ContinuousHistorization`/`AlarmHistorian` get advisory warning lists; `DevStubMode=true` is merely log-warned in production (`OtOpcUaLdapAuthService.cs:79-88`).
|
||||
*Recommendation:* promote the historian sections to the validator pattern; environment-gate `DevStubMode`.
|
||||
|
||||
**C3 — LOW: Library code logs via the static Serilog logger** (`Runtime/ServiceCollectionExtensions.cs:90, 100, 136`) while everything else uses MEL `ILogger` — works only because `Program.cs:305` assigns `Log.Logger`, an ordering-sensitive hidden coupling. The node manager similarly logs through the obsolete `Utils.LogError` static trace with six `#pragma warning disable CS0618` sites (`OtOpcUaNodeManager.cs:449-451` et al.) — the acknowledged follow-up (wire an `ITelemetryContext`) should actually land.
|
||||
**C3 — LOW: Library code logs via the static Serilog logger** (`Runtime/ServiceCollectionExtensions.cs:90, 100, 136`) and the node manager keeps six `#pragma warning disable CS0618` `Utils.LogError` sites. Unchanged — though notably the *new* resilience registrations do it right (`ILoggerFactory`-created categorized loggers, `DriverFactoryBootstrap.cs:58-70`), so the island isn't growing.
|
||||
|
||||
**C4 — LOW: `WithOtOpcUaRuntimeActors` is service-locator style with silent Null fallbacks** (`Runtime/ServiceCollectionExtensions.cs:211-237`) — the accepted Akka.Hosting idiom, and the missing-registration warnings (evaluator, recorder deps) are good; but `resolver.GetService<IHistorianProvisioning>()` (`:298`) has *no* fallback warning parallel to the others, despite being the exact seam that shipped dormant once (PR #423). The `dispatched=N, requested=0` tally in the applier (`AddressSpaceApplier.cs:274-282`) partially compensates.
|
||||
**C4 — LOW: `WithOtOpcUaRuntimeActors` service-locator fallbacks are inconsistently warned.** `resolver.GetService<IHistorianProvisioning>()` (`Runtime/ServiceCollectionExtensions.cs:303`) still has no missing-registration warning despite being the seam that shipped dormant once. The new `IDriverCapabilityInvokerFactory` resolve (`:219-220`) is likewise silent-fallback, but is the better-defended instance of the pattern: the fallback is *deliberate* for admin-only nodes, and `ResilienceInvokerFactoryRegistrationTests` pins the driver-node bind to the real factory — the guard the provisioning seam still lacks.
|
||||
|
||||
**Positive conventions notes:** the layering is clean and deliberate — `OpcUaServer` is Akka-free (delegates/gateway seams), `Runtime` is SDK-free (sink seams), Core holds the pure calculators (`ServiceLevelCalculator`, planners); the Null-object + Deferred late-binding pattern is applied consistently (sink, ServiceLevel, write gateway, historian source, provisioning, history writer); actor `Props` factories + registry marker keys are uniform; scope-correctness around the scoped `IGroupRoleMapper` inside the singleton authenticator is explicitly handled (`LdapOpcUaUserAuthenticator.cs:63-67`); NodeId is canonically `host:port` everywhere except one dormant spot (U4).
|
||||
**Positive conventions notes:** the layering discipline held under pressure — #10's filed plan (thread `CapabilityInvoker` directly into Runtime) would have broken the Polly-free boundary, and the shipped seam (`IDriverCapabilityInvoker`/`IDriverCapabilityInvokerFactory` in Core.Abstractions, mirroring `IDriverFactory`, Null-object defaults, DI-resolved with pass-through fallback) is exactly this codebase's established pattern; the Runtime csproj still references only `Core.Abstractions`. The corrected S1 docs are a conventions win too: `BuildClusterOptions`'s XML comment and `akka.conf` now state precisely *why* the typed option exists, what activates SBR, the keep-oldest-for-2-nodes rationale, and the HOCON/typed-option consistency constraint. Layer inventory unchanged otherwise: Akka-free `OpcUaServer`, SDK-free `Runtime`, pure calculators in Core, uniform Props factories + registry keys, scoped-mapper handling in the singleton authenticator.
|
||||
|
||||
### 4. Underdeveloped areas
|
||||
|
||||
**U1 — HIGH (documentation drift on a load-bearing claim): CLAUDE.md "KNOWN LIMITATION 2" is stale — the ContinuousHistorizationRecorder ref-feed gap is CLOSED in code.**
|
||||
The recorder is still *spawned* with `historizedRefs: Array.Empty<string>()` (`Runtime/ServiceCollectionExtensions.cs:272`), but the applier now feeds it the add/remove delta on every deploy via `ActorHistorizedTagSubscriptionSink` → `UpdateHistorizedRefs` (`Runtime/ServiceCollectionExtensions.cs:287-292`; `AddressSpaceApplier.cs:210-213, 310-353`; `ContinuousHistorizationRecorder.cs:67-78`). Restart convergence works because `OpcUaPublishActor._lastApplied` is in-memory, so the first post-boot rebuild diffs against empty and emits the full set as Added (`OpcUaPublishActor.cs:326-336`). CLAUDE.md still says "the recorder registers interest in nothing and historizes nothing… Until that feed lands, continuous historization records no values."
|
||||
*Recommendation:* update CLAUDE.md; add an explicit restart-convergence test (boot → apply full plan → assert mux interest registered), since the chain (empty seed + delta feed + in-memory `_lastApplied`) is load-bearing and only implicitly covered today.
|
||||
**U3 — MEDIUM: The structured audit pipeline is fully built, tested, and has zero producers.** Unchanged (`ControlPlane/Audit/AuditOutcomeMapper.cs:12-18`; `Security/Audit/AuditActor.cs:18-24`). Wire producers or delete.
|
||||
|
||||
**U2 — MEDIUM: The `DeferredAddressSpaceSink` capability-forwarding trap is currently closed but only half test-guarded.**
|
||||
Audit result: **all** current members forward correctly, including both `ISurgicalAddressSpaceSink` methods with capability-sniffing fallback (`src/Core/…/DeferredAddressSpaceSink.cs:58-70`) — no inert capability today. But tests assert forwarding for only ~5 of the 10 members; `WriteAlarmCondition`, `MaterialiseAlarmCondition`, `EnsureFolder`, `RebuildAddressSpace`, `RaiseNodesAddedModelChange` have no per-member forwarding assertion (`tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests` / `DeferredAddressSpaceSinkTests.cs`). The F10b incident (surgical path dead in prod for exactly this reason) proves the failure mode is real.
|
||||
*Recommendation:* add a reflection-driven exhaustive test: every method of `IOpcUaAddressSpaceSink` + every optional capability interface the wrapper claims must reach a recording inner sink. Same treatment for `DeferredServiceLevelPublisher`.
|
||||
**U4 — MEDIUM: `FleetStatusBroadcaster.DriverHostStatusHeartbeat` is dead code with a latent NodeId-mismatch repeat.** Unchanged (`FleetStatusBroadcaster.cs:38, 110-121`; host-only key `:152-159` vs `host:port` everywhere else).
|
||||
|
||||
**U3 — MEDIUM: The structured audit pipeline is fully built, tested, and has zero producers.** `AuditWriterActor` + `AuditOutcomeMapper` state "no live structured AuditEvent emit sites; all production audit flows through the bespoke stored-procedure path" (`ControlPlane/Audit/AuditOutcomeMapper.cs:12-18`; `Security/Audit/AuditActor.cs:18-24`). Either wire producers or delete — dormant tested code decays.
|
||||
**U5 — MEDIUM: Native Part 9 conditions support Acknowledge-to-driver only.** Unchanged (`OtOpcUaNodeManager.cs:647-658`; Enable/Disable `BadNotSupported` `:698-703`). Shelving a Galaxy alarm from a Part 9 client remains silently ineffective upstream.
|
||||
|
||||
**U4 — MEDIUM: `FleetStatusBroadcaster.DriverHostStatusHeartbeat` is dead code with a latent NodeId-mismatch repeat.** No producer anywhere sends the heartbeat (`FleetStatusBroadcaster.cs:38, 110-121`), so `CurrentRevision` is always null and the freshness branch never fires; and the broadcaster keys nodes by **host only** (`:152-159`) while every other subsystem uses `host:port` — if the heartbeat is ever wired with a `host:port` NodeId it will create phantom node records, re-enacting the historical NodeId bug.
|
||||
*Recommendation:* wire the producer (DriverHostActor already knows its applied revision) or remove; either way canonicalise the key to `host:port`.
|
||||
**U6 — MEDIUM: Test-coverage gaps still concentrate on the fragile seams — though the two worst are now closed.**
|
||||
Closed since 9cad9ed0: **hard-kill failover** (`HardKillFailoverTests`, in-process 2-node, CI-runnable — the single biggest prior blind spot) and **green-because-skipped** (whole-solution CI leg + `assert-not-all-skipped.sh` fail-on-skip gate). Still open, in priority order:
|
||||
- **DPS delivery of redundancy/ServiceLevel state is untested** — `RedundancyStateActorTests` stub the broadcast; `ServiceLevelEndToEndTests` inject state directly; the harness (even hard-kill) asserts cluster membership, not that `RedundancyStateChanged` *arrives* and flips ServiceLevel/gates on the survivor. The known "unit tests can't catch it" blind spot survives the failover test.
|
||||
- **No actor supervision/restart test** anywhere (S6).
|
||||
- **Outbox durability across process restart** and recorder **restart-convergence** (the U1 residual) untested.
|
||||
- `OpcUaServer.IntegrationTests` still contains **one test** (`DualEndpointTests`) — no over-the-wire security-mode matrix, subscription, or HistoryRead round-trip.
|
||||
- **Resilience wiring behavioral coverage is 2 of 6 sites** (`DriverInstanceActorResilienceWiringTests`: Subscribe + Write-with-resolved-host). The analyzer proves the other four *are wrapped* but not that they pass the right `DriverCapability`/host key — a transposed enum member at the ack or alarm-subscribe site would compile, pass the analyzer, and ship.
|
||||
|
||||
**U5 — MEDIUM: Native Part 9 conditions support Acknowledge-to-driver only.** Confirm/AddComment/Shelve on a *native* condition still route to the scripted engine, which does not own them (documented H6c scope, `OtOpcUaNodeManager.cs:647-658`); Enable/Disable short-circuits to `BadNotSupported` (`:698-703`). Operators using a Part 9 client will find shelving a Galaxy alarm silently ineffective upstream.
|
||||
**U7 — LOW:** `IHistoryWriter` permanently Null-wired; H2 HistoryUpdate unimplemented (`OtOpcUaNodeManager.cs:1797-1801`); `LdapAuthFailure` lacks `DirectoryUnavailable`. Tracked backlog, unchanged.
|
||||
|
||||
**U6 — MEDIUM: Test-coverage gaps concentrate exactly where the architecture is most fragile** (from the tests/Server sweep, ~910 tests total across 6 projects):
|
||||
- **DPS delivery of redundancy/ServiceLevel state is untested** — `RedundancyStateActorTests` stub the broadcast, `ServiceLevelEndToEndTests` inject `NodeRedundancyState` directly, and the `TwoNodeClusterHarness` runs in-process with EF InMemory. This is the known "unit tests can't catch it" blind spot, and it isn't even documented in a test comment.
|
||||
- **No actor supervision/restart test** anywhere (relates to S6) — only "init failure keeps Reconnecting state" is covered.
|
||||
- **Outbox durability across process restart** and sustained-outage backpressure untested (a single failure→retry→ack cycle is).
|
||||
- `OpcUaServer.IntegrationTests` contains **one test** (dual-endpoint ServerArray) — no over-the-wire security-mode matrix, subscription, or HistoryRead round-trip.
|
||||
- Docker-fixture-dependent driver-protocol tests self-skip when `10.100.0.35` is unreachable — green local/macOS runs can hide protocol regressions.
|
||||
Strengths worth noting: node-manager HistoryRead/paging (~60 tests), the write-gate/revert path, LDAP fail-closed edges, and Composer/Applier (~120 tests) are genuinely well covered.
|
||||
**U8 — LOW: `BuildSecurityPolicies` doc-comment claims the empty-profile fallback is "logged and very visible" but the method logs nothing** (`OpcUaApplicationHost.cs:376-418`). Unchanged.
|
||||
|
||||
**U7 — LOW: `IHistoryWriter` is permanently Null-wired** (vtag historization, infra-gated on a nonexistent live-write RPC — `Runtime/ServiceCollectionExtensions.cs:56-58`); **H2 HistoryUpdate** remains unimplemented (HistoryRead-only server; `IsReadModified` rejected, `OtOpcUaNodeManager.cs:1797-1801`) — both tracked backlog, listed for completeness. `LdapAuthFailure` lacks a `DirectoryUnavailable` value (directory-down is overloaded onto `ServiceAccountBindFailed`).
|
||||
**U9 — LOW: `EnsureVariable` silently ignores changed historize-intent on an existing node** (`OtOpcUaNodeManager.cs:1345-1349`) — invariant still lives in two files' comments rather than an assert.
|
||||
|
||||
**U8 — LOW: `BuildSecurityPolicies` doc-comment claims the empty-profile fallback is "logged and very visible" but the method logs nothing** (`OpcUaApplicationHost.cs:376-418`). Defused in production by the options validator (`MinCount ≥ 1`, `OpcUaApplicationHostOptionsValidator.cs`), but the silent fail-open-to-None remains for direct embedders/tests; fix the comment or add the log.
|
||||
|
||||
**U9 — LOW: `EnsureVariable` silently ignores changed historize-intent on an existing node** (documented, `OtOpcUaNodeManager.cs:1345-1349`) — correct today because the planner routes such deltas to `UpdateTagAttributes` or a rebuild, but the invariant lives in two files' comments rather than an assert.
|
||||
**U10 — LOW (NEW): The `DriverResilienceStatusTracker` has no reader — pipeline observability is log-only.**
|
||||
The tracker is registered, fed by every retry/breaker event, and consumed by nothing (Admin `/hosts` Stream E.2/E.3 never shipped). The interim mitigation (retry/breaker-open/breaker-close logging in `DriverResiliencePipelineBuilder.cs:124-170`, added with #10) is real and correctly gated, but breaker state — now a production behavior that can reject calls (`BrokenCircuitException`) — is invisible to operators except in logs. Known residual from FOLLOWUP-10; recorded here so the review tracks it.
|
||||
*Recommendation:* ship the `/hosts` resilience column, or at minimum a metric per breaker-open.
|
||||
|
||||
---
|
||||
|
||||
## Maturity ratings
|
||||
|
||||
| Dimension | Rating (1-5) | Justification |
|
||||
|---|---|---|
|
||||
| Stability | **3** | Excellent node-manager locking/self-correction discipline and Null/Deferred boot safety, but the inert SBR (S1) breaks hard-crash failover cluster-wide, and LDAP/HistoryRead thread-blocking (S2/S3) plus the primary-gate boot window (S4) are real production exposures. |
|
||||
| Performance | **3** | Hot paths are lean per-message and history paging is carefully bounded, but the rebuild-on-any-structural-change subscription wipe (P1) and per-value global-lock writes (P2) will not survive large tag counts or frequent deploys gracefully. |
|
||||
| Conventions | **4** | Deliberate, consistent layering (Akka-free OpcUaServer, SDK-free Runtime, Null/Deferred seams, uniform actor patterns); loses a point for the five-way `ServerHistorian` binding, two-tier options validation, and static-logger islands. |
|
||||
| Underdeveloped | **3** | Core capabilities are complete and the once-dormant provisioning/ref-feed seams are now wired, but stale load-bearing docs (U1), a half-guarded forwarding trap (U2), dormant audit/heartbeat code (U3/U4), and test blind spots exactly on the fragile seams (U6) leave meaningful debt. |
|
||||
| Dimension | 2026-07-08 | 2026-07-12 | Justification |
|
||||
|---|---|---|---|
|
||||
| Stability | **3** | **3** | The prior Critical dissolved (S1's premise was wrong — SBR was always active — and hard-kill failover is now integration-proven), which removes the cluster-wide failure mode from the ledger. But the three untouched Highs (S2 LDAP thread-stall, S3 HistoryRead pool exhaustion, S4 dual-primary boot window) are exactly the realistic production exposures the band describes, and the new S12 idempotence bypass adds a loaded footgun. Better-evidenced, not yet safer where it counts. |
|
||||
| Performance | **3** | **3** | Nothing on the hot paths changed. P1 (rebuild severs all subscriptions) and P2 (per-value global lock) remain; their prerequisite guard merged, so the score is now held down purely by unstarted work. The resilience pipeline correctly stayed off the per-value publish path. |
|
||||
| Conventions | **4** | **4** | The new code is the pattern at its best (Polly-free seam mirroring `IDriverFactory`, Null-object + DI-guard, corrected load-bearing docs), and the static-logger island stopped growing — but C1/C2/C3/C4 themselves are all still open, so the score holds rather than rises. |
|
||||
| Underdeveloped | **3** | **4** | The two systemic traps are retired with durable guards (U2 reflection forwarding test closes the F10b/PR#423 class; fail-on-skip CI closes green-because-skipped), U1's doc drift is fixed, the once-dormant CapabilityInvoker is wired end-to-end with analyzer + DI + behavioral guards, and the worst coverage blind spot (hard-kill failover) is tested. Remaining: dormant audit/heartbeat code (U3/U4), the DPS-delivery and supervision test gaps, and the reader-less tracker (U10). |
|
||||
|
||||
---
|
||||
|
||||
## Cross-cutting themes
|
||||
|
||||
1. **Config-in-HOCON is not config-in-effect** — the SBR block shows Akka behavior declared in `akka.conf` can be silently inert without the activating registration; audit other HOCON-declared behaviors (dispatcher blocks, singleton settings) the same way.
|
||||
2. **The Deferred/Null seam pattern is powerful but trap-prone**: every new optional capability must be forwarded through the wrappers *and* exhaustively test-guarded, or it ships inert (proven twice: F10b surgical sink, PR #423 provisioning). A reflection-based forwarding test would retire the trap class.
|
||||
3. **"Default-allow until told otherwise" gates** (primary write gate, peer-probe benefit-of-the-doubt, ServiceLevel legacy seam) consistently trade split-brain safety for boot-window availability; each is individually reasoned but they compound with at-most-once DPS delivery.
|
||||
4. **Unit-test greenness systematically overstates distributed correctness** here: DPS delivery, supervision restarts, hard-kill failover, and Docker-gated protocol paths all sit outside the test net — and the project's own history (redundancy-delivery bug, F10b inertness) shows these are exactly where the bugs live.
|
||||
1. **The review's own premises need negative controls too.** S1 was filed as Critical on a correct-looking static read (no `downing-provider-class` anywhere) that missed a framework default; only the #9 negative control (force `downing-provider-class = ""`) exposed it. The corrected framing is now pinned in code comments, `akka.conf`, and a test that guards the *outcome* regardless of activation path — the right durable form. Treat "config X is inert" findings as unproven until a negative control flips the observed behavior.
|
||||
2. **The Deferred/Null seam pattern's trap class is now guarded, not just known.** The reflection-exhaustive forwarding test (U2) auto-covers future sink/publisher members via `DispatchProxy` plus a guard-of-the-guard on the interface set — new capabilities (P1 surgical adds, P2 `WriteValues`) can no longer ship inert through the wrappers. The same "built-but-never-wired" failure mode got a second guard at the DI layer (`ResilienceInvokerFactoryRegistrationTests`). The remaining unguarded instance of the pattern is the silent `IHistorianProvisioning` fallback (C4).
|
||||
3. **"Default-allow until told otherwise" gates remain the compounding risk.** S4 (primary write gate) + at-most-once DPS delivery + the untested delivery path (U6) still stack; the SBR work hardened membership convergence but none of the delivery-dependent gates. S12 joins this family: `isIdempotent: true` is a default-permissive stance on a safety contract, dormant only because a *different* default (RetryCount=0) currently masks it.
|
||||
4. **Unit-test greenness overstating distributed correctness is being paid down in the right order** — hard-kill failover (done), fail-on-skip (done), DPS delivery / supervision restarts / outbox durability (still open, now the top of the list). The pattern to keep: every guard shipped with a verified negative control.
|
||||
|
||||
Reference in New Issue
Block a user