lmxopcua/archreview/03-server-runtime.md

# Architecture Review — Server & Runtime Subsystem

- **Date:** 2026-07-08
- **Commit:** `9cad9ed0`
- **Scope:** `src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer`, `.Host`, `.Runtime`, `.ControlPlane`, `.Security`, plus the sink/publisher wrappers in `src/Core/ZB.MOM.WW.OtOpcUa.Commons/OpcUa` (they are the load-bearing seam between Host and Runtime) and `tests/Server` for coverage assessment.
- **Dimensions:** Stability · Performance · Conventions · Underdeveloped areas

---

## Architecture overview

### Host composition

`Program.cs` (`src/Server/ZB.MOM.WW.OtOpcUa.Host/Program.cs`) is a single fused binary whose wiring is driven by the `OTOPCUA_ROLES` env var (`admin`, `driver`, or both). Notable mechanics: per-role appsettings overlays with env-vars/args re-appended so deployment overrides keep precedence (`Program.cs:61-73`); CWD anchored to `AppContext.BaseDirectory` for Windows-service path sanity (`:83`); Serilog via the shared `AddZbSerilog` with the static `Log.Logger` assigned post-build so Akka's `SerilogLogger` routes actor logs (`:305`).

On `driver` nodes the host registers the historian stack (three config-gated layers: `AddServerHistorian` read client, `AddAlarmHistorian` SQLite store-and-forward, `AddHistorianProvisioning` EnsureTags, plus the ContinuousHistorization outbox/writer pair), the driver-factory registry (8 drivers, `DriverFactoryBootstrap.cs`), the LDAP data-plane authenticator, and — critically — the **late-binding seams**: `DeferredAddressSpaceSink` and `DeferredServiceLevelPublisher` are DI singletons that actors capture at construction; `OtOpcUaServerHostedService` swaps the real SDK-backed implementations in after `StandardServer` start and swaps Nulls back in on stop (`OtOpcUaServerHostedService.cs:131, 226-231, 241-253`).

### OPC UA server & node manager

`OtOpcUaSdkServer` (a `StandardServer`) creates the single `OtOpcUaNodeManager` (`CustomNodeManager2`, 2 424 lines) that owns the entire writable address space: UNS Area/Line/Equipment folders, typed equipment-tag variables, Part 9 `AlarmConditionState` nodes, historized-node registration, and four HistoryRead override arms (Raw with synthesized continuation-point paging + tie-cluster over-fetch, Processed, AtTime, Events). Reverse paths out of the node manager are deliberately Akka-free delegates/interfaces set by the Host at start: `AlarmCommandRouter` (→ DPS `alarm-commands` topic), `NativeAlarmAckRouter` (→ `DriverHostActor.RouteNativeAlarmAck`), `NodeWriteGateway` (→ `ActorNodeWriteGateway` Ask), `HistorianDataSource` (→ gateway read client). Inbound writes are role-gated (`WriteOperate`), dispatched fire-and-forget under the SDK `Lock`, optimistically applied, and self-corrected (Bad-quality blip + revert + Part 8 audit event) on a failed device outcome.

### Runtime actor topology (per driver node)

`WithOtOpcUaRuntimeActors` (`Runtime/ServiceCollectionExtensions.cs:194-347`) spawns, in order: `DbHealthProbeActor` → `DependencyMuxActor` → (gated) `ContinuousHistorizationRecorder` → `OpcUaPublishActor` (pinned single-thread dispatcher `opcua-synchronized-dispatcher`, owns the `AddressSpaceApplier`) → `PeerProbeSupervisor` → `DriverHostActor` → `HistorianAdapterActor`. `DriverHostActor` spawns one `DriverInstanceActor` per deployed driver plus the `VirtualTagHostActor` and `ScriptedAlarmHostActor` sub-hosts, maintains the `(driver, FullName) ⇄ NodeId` routing maps, and fans every `AttributeValuePublished` to the mux + the publish actor (`ForwardToMux`, `DriverHostActor.cs:561-589`). **No actor in Runtime overrides `SupervisorStrategy`** — everything runs under Akka's default one-for-one unlimited restart decider.

### Redundancy model

Non-transparent warm redundancy. `RedundancyStateActor` (admin-role cluster singleton, `ControlPlane/Redundancy/RedundancyStateActor.cs`) derives Primary/Secondary purely from Akka's `driver` role-leader, debounces 250 ms, publishes `RedundancyStateChanged` on the DPS `redundancy-state` topic, and re-publishes on a 10 s heartbeat (the late-subscriber fix). Consumers: `OpcUaPublishActor` computes ServiceLevel (`ServiceLevelCalculator` + DB-health probe + peer OPC UA probes; 240/200/100/0 bands, first-publish-always so the SDK's 255 default never stands — `OpcUaPublishActor.cs:418-437, 508-542`); `DriverHostActor`, `ScriptedAlarmHostActor`, and `HistorianAdapterActor` use the same snapshot to Primary-gate device writes, alarm emission, and historization. There is no lease or fencing token — correctness rests entirely on Akka's single-role-leader guarantee within a converged cluster.

---

## Findings

Severity scale: **Critical** (correctness/availability broken in a realistic scenario) · **High** (significant risk or debt, plan a fix) · **Medium** (real but bounded) · **Low** (hygiene).

### 1. Stability

**S1 — CRITICAL: The split-brain resolver is configured but never activated; unreachable members are never downed.**
`akka.conf` carries a full `split-brain-resolver { active-strategy = "keep-oldest" … }` block (`src/Core/ZB.MOM.WW.OtOpcUa.Cluster/Resources/akka.conf:40-46`), but there is **no `akka.cluster.downing-provider-class` anywhere in the repo**, and `WithClustering` sets only `SeedNodes`/`Roles` — not `ClusterOptions.SplitBrainResolver` (`src/Core/ZB.MOM.WW.OtOpcUa.Cluster/ServiceCollectionExtensions.cs:80-84`; repo-wide grep for `downing-provider|SplitBrainResolver` returns nothing). In Akka.NET the SBR HOCON block is inert without the provider, so the cluster runs the default **NoDowning** provider. Consequences: (a) a *hard-crashed* node stays "unreachable" forever, so **cluster singletons (`ConfigPublishCoordinator`, `AdminOperationsActor`, `RedundancyStateActor`, `AuditWriterActor`) never fail over** and the `driver` role-leader never moves — i.e. redundancy failover works only for graceful shutdowns; (b) a network partition never resolves, and with `min-nr-of-members = 1` each side elects its own role-leader, so **both sides advertise ServiceLevel 240 indefinitely**. The two-node integration harness only exercises graceful `StopAsync`, which is why this is invisible in tests.
*Recommendation:* set `ClusterOptions.SplitBrainResolver = SplitBrainResolverOption` (or `downing-provider-class = "Akka.Cluster.SBR.SplitBrainResolverProvider"` in `akka.conf`), then add a two-node kill-9 failover test asserting singleton handover + ServiceLevel demotion.

**S2 — HIGH: LDAP authentication blocks the OPC UA session-activation thread with an effectively non-cancellable, non-configurable timeout.**
The SDK invokes `ImpersonateUser` synchronously; `OpcUaApplicationHost.HandleImpersonation` block-bridges the authenticator (`OpcUaApplicationHost.cs:271-273`) with `CancellationToken.None`. The doc comment shifts timeout responsibility to the authenticator ("callers must enforce their own timeouts"), but `LdapOpcUaUserAuthenticator` adds none (`Host/OpcUa/LdapOpcUaUserAuthenticator.cs:30-48`), and the shared library's `AuthenticateAsync` is a **synchronous method wrapped in `Task.FromResult`** that observes the token only at entry; its 10 s connect timeout is a library default that OtOpcUa's `LdapOptions.ToLibraryOptions()` does not project or expose (`Security/Ldap/LdapOptions.cs:94-107`). Every authentication also opens a fresh TCP connection + service-account bind + search + user bind (no pooling). Under a directory outage or slow DC, session activations stall serially for ~10-20 s each on SDK threads.
*Recommendation:* enforce a hard timeout in `LdapOpcUaUserAuthenticator` (`Task.WhenAny` + deny on timeout), surface a `Security:Ldap:TimeoutMs` option, and consider a short negative-cache to shed load during an outage. Fail-closed semantics are otherwise correct end-to-end (deny on error, opaque messages, zero-role fallback on mapper fault).

**S3 — HIGH: HistoryRead block-bridges the gateway per node, sequentially, on SDK request threads.**
All four HistoryRead arms call `.GetAwaiter().GetResult()` per node handle with `CancellationToken.None` (`OtOpcUaNodeManager.cs:1902-1911, 2059, 2169-2171, 2201-2203`). This is safe w.r.t. the node-manager `Lock` (the overrides run outside it — correctly documented), but bounded only by the gateway client's per-call `CallTimeout` (30 s default). A single request naming N historized nodes against a slow historian holds one SDK request thread up to N × 30 s; the server is configured `MaxRequestThreadCount = 100`, `MaxQueuedRequestCount = 200` (`OpcUaApplicationHost.cs:329-331`), so a handful of misbehaving history clients can exhaust the request pool and degrade *all* OPC UA services.
*Recommendation:* parallelize per-node reads within a batch (bounded), pass a per-request deadline token, and consider a server-side concurrent-HistoryRead limiter.

**S4 — HIGH: Primary-gate "default-allow while role unknown" opens a dual-primary window on the data plane.**
`DriverHostActor.HandleRouteNodeWrite` / `HandleRouteNativeAlarmAck` / native-alarm emission all gate on `_localRole is Secondary or Detached` and deliberately default to *allow* until the first redundancy snapshot arrives (`DriverHostActor.cs:1018-1026, 1074-1083, 956-960; OnRedundancyStateChanged :1109-1114`). Combined with DPS delivery being at-most-once and heartbeat-repaired every 10 s, a freshly-booted or snapshot-missed **secondary services device writes and emits alerts as if primary** for up to the heartbeat interval — and indefinitely if the NodeId identity mismatch (S5) bites. The boot-window rationale (single-node deploys) is sound, but the same signal gates *shared-field-device writes*.
*Recommendation:* for the write/ack gates specifically, prefer default-*deny* once the cluster has >1 driver member (discoverable from cluster state), or require at least one received snapshot before servicing writes on multi-node clusters.

**S5 — MEDIUM: Redundancy NodeId identity is string-matched across three independently-derived sources with no diagnostic on mismatch.**
`OpcUaPublishActor` matches `n.NodeId == _localNode.Value` (`OpcUaPublishActor.cs:512`); the snapshot side derives `host:port` from the gossiped `Member.Address` (`RedundancyStateActor.cs:147-148`) while the local side derives it from configured `PublicHostname:Port` (`ClusterRoleInfo`). If `PublicHostname` ever diverges from the gossiped address (DNS vs IP, container advertised name), the node silently computes ServiceLevel 0 / keeps a stale role — the exact shape of the historical "silently inert delivery" bug, with no log distinguishing it from legitimate Detached.
*Recommendation:* log-once (Warning) when the local node is absent from a received snapshot; assert the identity equality in a startup self-check.

**S6 — MEDIUM: A crashed-and-restarted `DriverInstanceActor` loses its message-delivered state (desired subscriptions) until the next deploy.**
No actor in Runtime overrides `SupervisorStrategy` (grep confirms), so an exception in a driver child triggers Akka's default one-for-one **unlimited, non-backoff restart**. Restart re-runs `PreStart` (driver re-init from the Props spec) but the desired-subscription set arrives post-spawn via `SetDesiredSubscriptions` messages stored in actor state (`DriverInstanceActor.cs:84, 181, 861`); a restart wipes it and `DriverHostActor` has no restart-detection to re-send (restarts don't fire `Terminated`). A persistently-throwing driver also hot-loops restarts with no backoff.
*Recommendation:* wrap driver children in `BackoffSupervisor` (restart with exponential backoff) and have the child request its subscription set from the parent in `PreStart`, or have the parent push it on a `PostRestart` signal. Add a supervision test (none exists — see U6).

**S7 — MEDIUM: `DriverInstanceActor.PostStop` blocks the actor shutdown path on driver shutdown.**
`_driver.ShutdownAsync(CancellationToken.None).GetAwaiter().GetResult()` (`DriverInstanceActor.cs:950`) is the only synchronous block in the Runtime actors; with `CancellationToken.None` a hung protocol stack can stall stop/re-deploy of the whole child set.
*Recommendation:* bound it (`WaitAsync(timeout)`) and log-and-abandon on expiry.

**S8 — MEDIUM: Inbound Part 9 commands and native acks are at-most-once; the client sees `Good` even if the command is lost.**
The alarm-command router is a fire-and-forget DPS `Publish` and the native-ack router a fire-and-forget `Tell`, both wrapped in catch-log-drop (`OtOpcUaServerHostedService.cs:139-194`); the node manager has already returned `Good` so the SDK applies local condition state regardless (`OtOpcUaNodeManager.cs:769-772`). A mediator hiccup or missing `DriverHostActor` silently strands the engine/upstream state. This is a documented, deliberate trade-off (non-blocking under `Lock`), but it is invisible in operation.
*Recommendation:* add a counter/metric for dropped routes (both routers currently only log at Warning) and consider an engine-side reconciliation sweep.

**S9 — MEDIUM: Certificate lifecycle has no renewal or expiry monitoring.**
The application certificate is auto-created self-signed with SDK defaults (2048-bit, **12-month lifetime**) and only checked at boot (`OpcUaApplicationHost.cs:305-317`). Nothing monitors expiry; ~12 months after first deploy, Sign/SignAndEncrypt endpoints and UserName-token encryption fail on live servers with no advance warning.
*Recommendation:* startup + periodic expiry check with a metric/health-check and a documented rotation runbook (the AdminUI cert-actions work covers client certs; the *server* cert lifecycle is the gap).

**S10 — LOW: SDK start failure is swallowed and the node keeps running with Null sinks.**
`StartAsync` catches, logs, and returns (`OtOpcUaServerHostedService.cs:110-129`) — a deliberate availability choice (AdminUI stays up), but the node then silently no-ops all OPC UA work and nothing surfaces the condition through `/health` or ServiceLevel (which is itself served by the dead server).
*Recommendation:* set a health-check flag consumed by `MapOtOpcUaHealth` so fleet status shows the degraded state.

**S11 — LOW: `AuditWriterActor` buffer is unbounded between flushes and drops whole batches on DB outage** (`ControlPlane/Audit/AuditWriterActor.cs:35, 83, 113-116`). Explicitly best-effort by contract; flag only if compliance-grade audit is ever required. (Currently moot — see U3: the pipeline has no producers.)

**Positive stability notes (worth preserving):** the node-manager locking discipline is exemplary — every mutation under `Lock`, every `Server.ReportEvent` deliberately outside it (`OtOpcUaNodeManager.cs:1101-1136, 1503-1504, 1595-1614`), the write-dispatch continuation forced off the SDK thread via `RunContinuationsAsynchronously` (`:884-897`), and the revert guarded by "still holds the optimistic value" (`ShouldRevert`, `:1048`). The Deferred-sink swap-to-Null on stop prevents post-stop writes hitting a disposed manager. `ConfigPublishCoordinator` has real crash-recovery (rebuilds expected-acks from DB on `PreStart`). `HandleServiceLevelChanged`'s first-publish-always correctly defeats the SDK's 255 default. `ActorNodeWriteGateway` bounds its Ask at 10 s with the inner driver Ask at 8 s and the backend call at 5 s — a correctly nested timeout ladder.

### 2. Performance

**P1 — HIGH: Any structural deploy change triggers a full address-space teardown/rebuild that severs every client's monitored items server-wide.**
`AddressSpaceApplier.Apply` forces `RebuildAddressSpace()` for *any* added/removed equipment, alarm, tag, or virtual tag (`AddressSpaceApplier.cs:134-154`); the rebuild removes every variable/folder/condition under one `Lock` hold and re-materialises from scratch (`OtOpcUaNodeManager.cs:1690-1736`). Recreated nodes are new `NodeState` instances, so existing client subscriptions on the old nodes go dead — the F10b surgical path (`UpdateTagAttributes`/`UpdateFolderDisplayName`) exists precisely because of this, but covers only attribute edits and renames. Adding one tag to one equipment still drops every subscription on a production SCADA server.
*Recommendation:* extend the surgical path to pure-add deltas first (`EnsureFolder`/`EnsureVariable` are already idempotent — adds need no teardown; `RaiseNodesAddedModelChange` already exists for announcement), then scope removals per-equipment. This is the highest-leverage performance/stability item in the subsystem.

**P2 — MEDIUM: The value hot path is one global lock acquisition + one actor message per value, with no batching.**
Per published value: driver child `Tell` → `DriverHostActor.ForwardToMux` (dictionary lookup + up to two `Tell`s, `DriverHostActor.cs:561-589`) → `OpcUaPublishActor` (pinned thread) → `sink.WriteValue`, which takes the **global node-manager `Lock`** per value (`OtOpcUaNodeManager.cs:266-281`), contending with SDK read/subscription/publish threads. The mux itself is lean (`DependencyMuxActor.cs:95-108` — early-drop for uninterested refs, set-based fan-out). At high tag counts × fast poll intervals this serialises everything through one lock and allocates one record per hop.
*Recommendation:* add a batched `WriteValues(IReadOnlyList<…>)` sink call (one lock hold per driver publish cycle) — the seam (`IOpcUaAddressSpaceSink`) makes this a contained change, but remember the Deferred-wrapper forwarding trap (U2).

**P3 — MEDIUM: HistoryRead-Events has no paging and translates `NumValuesPerNode == 0` to `int.MaxValue`.**
`EventMaxEvents` deliberately maps "no limit" to unbounded (`OtOpcUaNodeManager.cs:1944-1954`), and the Events arm never issues continuation points ("full window in one shot", `:1914-1921`). A wide time window over a busy alarm source materialises the entire result in memory on both gateway and server.
*Recommendation:* impose a server-side max (mirroring `MaxTieClusterOverfetch`'s philosophy) and fail loudly or page beyond it.

**P4 — LOW: Every deploy re-runs the four `Materialise*` passes over the full composition** (`OpcUaPublishActor.cs:338-354`) — acceptable because `EnsureFolder`/`EnsureVariable` early-return on existing ids, but each pass still takes/releases the `Lock` per node; fold into P2's batching if it shows up in profiles. On the positive side, the Raw-paging tie-cluster over-fetch is explicitly bounded (`MaxTieClusterOverfetch`, default 65 536) with a loud-fail backstop, and `HandleDiscoveredNodes` has an unchanged-plan short-circuit preventing ~15× re-subscribe blips during the discovery window (`DriverHostActor.cs:658-673`) — both good designs.

### 3. Conventions

**C1 — MEDIUM: The `ServerHistorian` section is bound imperatively in five places with no `IOptions` registration.**
`Program.cs:120-122` (own bind + `Validate()` warnings), `AddServerHistorian`, `AddAlarmHistorian`, `AddHistorianProvisioning` (each re-`Get<>`s the section, `Runtime/ServiceCollectionExtensions.cs:86, 132, 168`), and `OtOpcUaServerHostedService`'s constructor (`.cs:88-90`, self-justified as "self-contained"). Warnings can log twice; five bind sites will drift.
*Recommendation:* one `AddValidatedOptions<ServerHistorianOptions>` and inject `IOptions<>` everywhere, matching the `OpcUa`/`Ldap` pattern that already exists (`Program.cs:102, 254-255`).

**C2 — MEDIUM: Options validation is two-tier and the tiers disagree.**
`OpcUa` and `Security:Ldap` get real fail-fast `ValidateOnStart` validators (`OpcUaApplicationHostOptionsValidator` requires ≥1 security profile; `LdapOptionsValidator` refuses plaintext-without-opt-in) — good. `ServerHistorian` / `ContinuousHistorization` / `AlarmHistorian` only get advisory `Validate()` warning lists, and `DevStubMode=true` (accept-any-credentials Administrator grant) is merely log-warned, never blocked, in production (`OtOpcUaLdapAuthService.cs:79-88`).
*Recommendation:* promote the historian sections to the validator pattern; add an environment-gated guard (or at minimum a startup validator failure) for `DevStubMode` outside Development.

**C3 — LOW: Library code logs via the static Serilog logger** (`Runtime/ServiceCollectionExtensions.cs:90, 100, 136`) while everything else uses MEL `ILogger` — works only because `Program.cs:305` assigns `Log.Logger`, an ordering-sensitive hidden coupling. The node manager similarly logs through the obsolete `Utils.LogError` static trace with six `#pragma warning disable CS0618` sites (`OtOpcUaNodeManager.cs:449-451` et al.) — the acknowledged follow-up (wire an `ITelemetryContext`) should actually land.

**C4 — LOW: `WithOtOpcUaRuntimeActors` is service-locator style with silent Null fallbacks** (`Runtime/ServiceCollectionExtensions.cs:211-237`) — the accepted Akka.Hosting idiom, and the missing-registration warnings (evaluator, recorder deps) are good; but `resolver.GetService<IHistorianProvisioning>()` (`:298`) has *no* fallback warning parallel to the others, despite being the exact seam that shipped dormant once (PR #423). The `dispatched=N, requested=0` tally in the applier (`AddressSpaceApplier.cs:274-282`) partially compensates.

**Positive conventions notes:** the layering is clean and deliberate — `OpcUaServer` is Akka-free (delegates/gateway seams), `Runtime` is SDK-free (sink seams), Core holds the pure calculators (`ServiceLevelCalculator`, planners); the Null-object + Deferred late-binding pattern is applied consistently (sink, ServiceLevel, write gateway, historian source, provisioning, history writer); actor `Props` factories + registry marker keys are uniform; scope-correctness around the scoped `IGroupRoleMapper` inside the singleton authenticator is explicitly handled (`LdapOpcUaUserAuthenticator.cs:63-67`); NodeId is canonically `host:port` everywhere except one dormant spot (U4).

### 4. Underdeveloped areas

**U1 — HIGH (documentation drift on a load-bearing claim): CLAUDE.md "KNOWN LIMITATION 2" is stale — the ContinuousHistorizationRecorder ref-feed gap is CLOSED in code.**
The recorder is still *spawned* with `historizedRefs: Array.Empty<string>()` (`Runtime/ServiceCollectionExtensions.cs:272`), but the applier now feeds it the add/remove delta on every deploy via `ActorHistorizedTagSubscriptionSink` → `UpdateHistorizedRefs` (`Runtime/ServiceCollectionExtensions.cs:287-292`; `AddressSpaceApplier.cs:210-213, 310-353`; `ContinuousHistorizationRecorder.cs:67-78`). Restart convergence works because `OpcUaPublishActor._lastApplied` is in-memory, so the first post-boot rebuild diffs against empty and emits the full set as Added (`OpcUaPublishActor.cs:326-336`). CLAUDE.md still says "the recorder registers interest in nothing and historizes nothing… Until that feed lands, continuous historization records no values."
*Recommendation:* update CLAUDE.md; add an explicit restart-convergence test (boot → apply full plan → assert mux interest registered), since the chain (empty seed + delta feed + in-memory `_lastApplied`) is load-bearing and only implicitly covered today.

**U2 — MEDIUM: The `DeferredAddressSpaceSink` capability-forwarding trap is currently closed but only half test-guarded.**
Audit result: **all** current members forward correctly, including both `ISurgicalAddressSpaceSink` methods with capability-sniffing fallback (`src/Core/…/DeferredAddressSpaceSink.cs:58-70`) — no inert capability today. But tests assert forwarding for only ~5 of the 10 members; `WriteAlarmCondition`, `MaterialiseAlarmCondition`, `EnsureFolder`, `RebuildAddressSpace`, `RaiseNodesAddedModelChange` have no per-member forwarding assertion (`tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests` / `DeferredAddressSpaceSinkTests.cs`). The F10b incident (surgical path dead in prod for exactly this reason) proves the failure mode is real.
*Recommendation:* add a reflection-driven exhaustive test: every method of `IOpcUaAddressSpaceSink` + every optional capability interface the wrapper claims must reach a recording inner sink. Same treatment for `DeferredServiceLevelPublisher`.

**U3 — MEDIUM: The structured audit pipeline is fully built, tested, and has zero producers.** `AuditWriterActor` + `AuditOutcomeMapper` state "no live structured AuditEvent emit sites; all production audit flows through the bespoke stored-procedure path" (`ControlPlane/Audit/AuditOutcomeMapper.cs:12-18`; `Security/Audit/AuditActor.cs:18-24`). Either wire producers or delete — dormant tested code decays.

**U4 — MEDIUM: `FleetStatusBroadcaster.DriverHostStatusHeartbeat` is dead code with a latent NodeId-mismatch repeat.** No producer anywhere sends the heartbeat (`FleetStatusBroadcaster.cs:38, 110-121`), so `CurrentRevision` is always null and the freshness branch never fires; and the broadcaster keys nodes by **host only** (`:152-159`) while every other subsystem uses `host:port` — if the heartbeat is ever wired with a `host:port` NodeId it will create phantom node records, re-enacting the historical NodeId bug.
*Recommendation:* wire the producer (DriverHostActor already knows its applied revision) or remove; either way canonicalise the key to `host:port`.

**U5 — MEDIUM: Native Part 9 conditions support Acknowledge-to-driver only.** Confirm/AddComment/Shelve on a *native* condition still route to the scripted engine, which does not own them (documented H6c scope, `OtOpcUaNodeManager.cs:647-658`); Enable/Disable short-circuits to `BadNotSupported` (`:698-703`). Operators using a Part 9 client will find shelving a Galaxy alarm silently ineffective upstream.

**U6 — MEDIUM: Test-coverage gaps concentrate exactly where the architecture is most fragile** (from the tests/Server sweep, ~910 tests total across 6 projects):
- **DPS delivery of redundancy/ServiceLevel state is untested** — `RedundancyStateActorTests` stub the broadcast, `ServiceLevelEndToEndTests` inject `NodeRedundancyState` directly, and the `TwoNodeClusterHarness` runs in-process with EF InMemory. This is the known "unit tests can't catch it" blind spot, and it isn't even documented in a test comment.
- **No actor supervision/restart test** anywhere (relates to S6) — only "init failure keeps Reconnecting state" is covered.
- **Outbox durability across process restart** and sustained-outage backpressure untested (a single failure→retry→ack cycle is).
- `OpcUaServer.IntegrationTests` contains **one test** (dual-endpoint ServerArray) — no over-the-wire security-mode matrix, subscription, or HistoryRead round-trip.
- Docker-fixture-dependent driver-protocol tests self-skip when `10.100.0.35` is unreachable — green local/macOS runs can hide protocol regressions.
Strengths worth noting: node-manager HistoryRead/paging (~60 tests), the write-gate/revert path, LDAP fail-closed edges, and Composer/Applier (~120 tests) are genuinely well covered.

**U7 — LOW: `IHistoryWriter` is permanently Null-wired** (vtag historization, infra-gated on a nonexistent live-write RPC — `Runtime/ServiceCollectionExtensions.cs:56-58`); **H2 HistoryUpdate** remains unimplemented (HistoryRead-only server; `IsReadModified` rejected, `OtOpcUaNodeManager.cs:1797-1801`) — both tracked backlog, listed for completeness. `LdapAuthFailure` lacks a `DirectoryUnavailable` value (directory-down is overloaded onto `ServiceAccountBindFailed`).

**U8 — LOW: `BuildSecurityPolicies` doc-comment claims the empty-profile fallback is "logged and very visible" but the method logs nothing** (`OpcUaApplicationHost.cs:376-418`). Defused in production by the options validator (`MinCount ≥ 1`, `OpcUaApplicationHostOptionsValidator.cs`), but the silent fail-open-to-None remains for direct embedders/tests; fix the comment or add the log.

**U9 — LOW: `EnsureVariable` silently ignores changed historize-intent on an existing node** (documented, `OtOpcUaNodeManager.cs:1345-1349`) — correct today because the planner routes such deltas to `UpdateTagAttributes` or a rebuild, but the invariant lives in two files' comments rather than an assert.

---

## Maturity ratings

| Dimension | Rating (1-5) | Justification |
|---|---|---|
| Stability | **3** | Excellent node-manager locking/self-correction discipline and Null/Deferred boot safety, but the inert SBR (S1) breaks hard-crash failover cluster-wide, and LDAP/HistoryRead thread-blocking (S2/S3) plus the primary-gate boot window (S4) are real production exposures. |
| Performance | **3** | Hot paths are lean per-message and history paging is carefully bounded, but the rebuild-on-any-structural-change subscription wipe (P1) and per-value global-lock writes (P2) will not survive large tag counts or frequent deploys gracefully. |
| Conventions | **4** | Deliberate, consistent layering (Akka-free OpcUaServer, SDK-free Runtime, Null/Deferred seams, uniform actor patterns); loses a point for the five-way `ServerHistorian` binding, two-tier options validation, and static-logger islands. |
| Underdeveloped | **3** | Core capabilities are complete and the once-dormant provisioning/ref-feed seams are now wired, but stale load-bearing docs (U1), a half-guarded forwarding trap (U2), dormant audit/heartbeat code (U3/U4), and test blind spots exactly on the fragile seams (U6) leave meaningful debt. |

---

## Cross-cutting themes

1. **Config-in-HOCON is not config-in-effect** — the SBR block shows Akka behavior declared in `akka.conf` can be silently inert without the activating registration; audit other HOCON-declared behaviors (dispatcher blocks, singleton settings) the same way.
2. **The Deferred/Null seam pattern is powerful but trap-prone**: every new optional capability must be forwarded through the wrappers *and* exhaustively test-guarded, or it ships inert (proven twice: F10b surgical sink, PR #423 provisioning). A reflection-based forwarding test would retire the trap class.
3. **"Default-allow until told otherwise" gates** (primary write gate, peer-probe benefit-of-the-doubt, ServiceLevel legacy seam) consistently trade split-brain safety for boot-window availability; each is individually reasoned but they compound with at-most-once DPS delivery.
4. **Unit-test greenness systematically overstates distributed correctness** here: DPS delivery, supervision restarts, hard-kill failover, and Docker-gated protocol paths all sit outside the test net — and the project's own history (redundancy-delivery bug, F10b inertness) shows these are exactly where the bugs live.