lmxopcua/archreview/03-server-runtime.md

Architecture Review — Server & Runtime Subsystem

• Date: 2026-07-08
• Commit: 9cad9ed0
• Scope: src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer, .Host, .Runtime, .ControlPlane, .Security, plus the sink/publisher wrappers in src/Core/ZB.MOM.WW.OtOpcUa.Commons/OpcUa (they are the load-bearing seam between Host and Runtime) and tests/Server for coverage assessment.
• Dimensions: Stability · Performance · Conventions · Underdeveloped areas

---

Architecture overview

Host composition

Program.cs (src/Server/ZB.MOM.WW.OtOpcUa.Host/Program.cs) is a single fused binary whose wiring is driven by the OTOPCUA_ROLES env var (admin, driver, or both). Notable mechanics: per-role appsettings overlays with env-vars/args re-appended so deployment overrides keep precedence (Program.cs:61-73); CWD anchored to AppContext.BaseDirectory for Windows-service path sanity (:83); Serilog via the shared AddZbSerilog with the static Log.Logger assigned post-build so Akka's SerilogLogger routes actor logs (:305).

On driver nodes the host registers the historian stack (three config-gated layers: AddServerHistorian read client, AddAlarmHistorian SQLite store-and-forward, AddHistorianProvisioning EnsureTags, plus the ContinuousHistorization outbox/writer pair), the driver-factory registry (8 drivers, DriverFactoryBootstrap.cs), the LDAP data-plane authenticator, and — critically — the late-binding seams: DeferredAddressSpaceSink and DeferredServiceLevelPublisher are DI singletons that actors capture at construction; OtOpcUaServerHostedService swaps the real SDK-backed implementations in after StandardServer start and swaps Nulls back in on stop (OtOpcUaServerHostedService.cs:131, 226-231, 241-253).

OPC UA server & node manager

OtOpcUaSdkServer (a StandardServer) creates the single OtOpcUaNodeManager (CustomNodeManager2, 2 424 lines) that owns the entire writable address space: UNS Area/Line/Equipment folders, typed equipment-tag variables, Part 9 AlarmConditionState nodes, historized-node registration, and four HistoryRead override arms (Raw with synthesized continuation-point paging + tie-cluster over-fetch, Processed, AtTime, Events). Reverse paths out of the node manager are deliberately Akka-free delegates/interfaces set by the Host at start: AlarmCommandRouter (→ DPS alarm-commands topic), NativeAlarmAckRouter (→ DriverHostActor.RouteNativeAlarmAck), NodeWriteGateway (→ ActorNodeWriteGateway Ask), HistorianDataSource (→ gateway read client). Inbound writes are role-gated (WriteOperate), dispatched fire-and-forget under the SDK Lock, optimistically applied, and self-corrected (Bad-quality blip + revert + Part 8 audit event) on a failed device outcome.

Runtime actor topology (per driver node)

WithOtOpcUaRuntimeActors (Runtime/ServiceCollectionExtensions.cs:194-347) spawns, in order: DbHealthProbeActor → DependencyMuxActor → (gated) ContinuousHistorizationRecorder → OpcUaPublishActor (pinned single-thread dispatcher opcua-synchronized-dispatcher, owns the AddressSpaceApplier) → PeerProbeSupervisor → DriverHostActor → HistorianAdapterActor. DriverHostActor spawns one DriverInstanceActor per deployed driver plus the VirtualTagHostActor and ScriptedAlarmHostActor sub-hosts, maintains the (driver, FullName) ⇄ NodeId routing maps, and fans every AttributeValuePublished to the mux + the publish actor (ForwardToMux, DriverHostActor.cs:561-589). No actor in Runtime overrides SupervisorStrategy — everything runs under Akka's default one-for-one unlimited restart decider.

Redundancy model

Non-transparent warm redundancy. RedundancyStateActor (admin-role cluster singleton, ControlPlane/Redundancy/RedundancyStateActor.cs) derives Primary/Secondary purely from Akka's driver role-leader, debounces 250 ms, publishes RedundancyStateChanged on the DPS redundancy-state topic, and re-publishes on a 10 s heartbeat (the late-subscriber fix). Consumers: OpcUaPublishActor computes ServiceLevel (ServiceLevelCalculator + DB-health probe + peer OPC UA probes; 240/200/100/0 bands, first-publish-always so the SDK's 255 default never stands — OpcUaPublishActor.cs:418-437, 508-542); DriverHostActor, ScriptedAlarmHostActor, and HistorianAdapterActor use the same snapshot to Primary-gate device writes, alarm emission, and historization. There is no lease or fencing token — correctness rests entirely on Akka's single-role-leader guarantee within a converged cluster.

---

Findings

Severity scale: Critical (correctness/availability broken in a realistic scenario) · High (significant risk or debt, plan a fix) · Medium (real but bounded) · Low (hygiene).

1. Stability

S1 — CRITICAL: The split-brain resolver is configured but never activated; unreachable members are never downed. akka.conf carries a full split-brain-resolver { active-strategy = "keep-oldest" … } block (src/Core/ZB.MOM.WW.OtOpcUa.Cluster/Resources/akka.conf:40-46), but there is no akka.cluster.downing-provider-class anywhere in the repo, and WithClustering sets only SeedNodes/Roles — not ClusterOptions.SplitBrainResolver (src/Core/ZB.MOM.WW.OtOpcUa.Cluster/ServiceCollectionExtensions.cs:80-84; repo-wide grep for downing-provider|SplitBrainResolver returns nothing). In Akka.NET the SBR HOCON block is inert without the provider, so the cluster runs the default NoDowning provider. Consequences: (a) a hard-crashed node stays "unreachable" forever, so cluster singletons (ConfigPublishCoordinator, AdminOperationsActor, RedundancyStateActor, AuditWriterActor) never fail over and the driver role-leader never moves — i.e. redundancy failover works only for graceful shutdowns; (b) a network partition never resolves, and with min-nr-of-members = 1 each side elects its own role-leader, so both sides advertise ServiceLevel 240 indefinitely. The two-node integration harness only exercises graceful StopAsync, which is why this is invisible in tests. Recommendation: set ClusterOptions.SplitBrainResolver = SplitBrainResolverOption (or downing-provider-class = "Akka.Cluster.SBR.SplitBrainResolverProvider" in akka.conf), then add a two-node kill-9 failover test asserting singleton handover + ServiceLevel demotion.

S2 — HIGH: LDAP authentication blocks the OPC UA session-activation thread with an effectively non-cancellable, non-configurable timeout. The SDK invokes ImpersonateUser synchronously; OpcUaApplicationHost.HandleImpersonation block-bridges the authenticator (OpcUaApplicationHost.cs:271-273) with CancellationToken.None. The doc comment shifts timeout responsibility to the authenticator ("callers must enforce their own timeouts"), but LdapOpcUaUserAuthenticator adds none (Host/OpcUa/LdapOpcUaUserAuthenticator.cs:30-48), and the shared library's AuthenticateAsync is a synchronous method wrapped in Task.FromResult that observes the token only at entry; its 10 s connect timeout is a library default that OtOpcUa's LdapOptions.ToLibraryOptions() does not project or expose (Security/Ldap/LdapOptions.cs:94-107). Every authentication also opens a fresh TCP connection + service-account bind + search + user bind (no pooling). Under a directory outage or slow DC, session activations stall serially for ~10-20 s each on SDK threads. Recommendation: enforce a hard timeout in LdapOpcUaUserAuthenticator (Task.WhenAny + deny on timeout), surface a Security:Ldap:TimeoutMs option, and consider a short negative-cache to shed load during an outage. Fail-closed semantics are otherwise correct end-to-end (deny on error, opaque messages, zero-role fallback on mapper fault).

S3 — HIGH: HistoryRead block-bridges the gateway per node, sequentially, on SDK request threads. All four HistoryRead arms call .GetAwaiter().GetResult() per node handle with CancellationToken.None (OtOpcUaNodeManager.cs:1902-1911, 2059, 2169-2171, 2201-2203). This is safe w.r.t. the node-manager Lock (the overrides run outside it — correctly documented), but bounded only by the gateway client's per-call CallTimeout (30 s default). A single request naming N historized nodes against a slow historian holds one SDK request thread up to N × 30 s; the server is configured MaxRequestThreadCount = 100, MaxQueuedRequestCount = 200 (OpcUaApplicationHost.cs:329-331), so a handful of misbehaving history clients can exhaust the request pool and degrade all OPC UA services. Recommendation: parallelize per-node reads within a batch (bounded), pass a per-request deadline token, and consider a server-side concurrent-HistoryRead limiter.

S4 — HIGH: Primary-gate "default-allow while role unknown" opens a dual-primary window on the data plane. DriverHostActor.HandleRouteNodeWrite / HandleRouteNativeAlarmAck / native-alarm emission all gate on _localRole is Secondary or Detached and deliberately default to allow until the first redundancy snapshot arrives (DriverHostActor.cs:1018-1026, 1074-1083, 956-960; OnRedundancyStateChanged :1109-1114). Combined with DPS delivery being at-most-once and heartbeat-repaired every 10 s, a freshly-booted or snapshot-missed secondary services device writes and emits alerts as if primary for up to the heartbeat interval — and indefinitely if the NodeId identity mismatch (S5) bites. The boot-window rationale (single-node deploys) is sound, but the same signal gates shared-field-device writes. Recommendation: for the write/ack gates specifically, prefer default-deny once the cluster has >1 driver member (discoverable from cluster state), or require at least one received snapshot before servicing writes on multi-node clusters.

S5 — MEDIUM: Redundancy NodeId identity is string-matched across three independently-derived sources with no diagnostic on mismatch. OpcUaPublishActor matches n.NodeId == _localNode.Value (OpcUaPublishActor.cs:512); the snapshot side derives host:port from the gossiped Member.Address (RedundancyStateActor.cs:147-148) while the local side derives it from configured PublicHostname:Port (ClusterRoleInfo). If PublicHostname ever diverges from the gossiped address (DNS vs IP, container advertised name), the node silently computes ServiceLevel 0 / keeps a stale role — the exact shape of the historical "silently inert delivery" bug, with no log distinguishing it from legitimate Detached. Recommendation: log-once (Warning) when the local node is absent from a received snapshot; assert the identity equality in a startup self-check.

S6 — MEDIUM: A crashed-and-restarted DriverInstanceActor loses its message-delivered state (desired subscriptions) until the next deploy. No actor in Runtime overrides SupervisorStrategy (grep confirms), so an exception in a driver child triggers Akka's default one-for-one unlimited, non-backoff restart. Restart re-runs PreStart (driver re-init from the Props spec) but the desired-subscription set arrives post-spawn via SetDesiredSubscriptions messages stored in actor state (DriverInstanceActor.cs:84, 181, 861); a restart wipes it and DriverHostActor has no restart-detection to re-send (restarts don't fire Terminated). A persistently-throwing driver also hot-loops restarts with no backoff. Recommendation: wrap driver children in BackoffSupervisor (restart with exponential backoff) and have the child request its subscription set from the parent in PreStart, or have the parent push it on a PostRestart signal. Add a supervision test (none exists — see U6).

S7 — MEDIUM: DriverInstanceActor.PostStop blocks the actor shutdown path on driver shutdown. _driver.ShutdownAsync(CancellationToken.None).GetAwaiter().GetResult() (DriverInstanceActor.cs:950) is the only synchronous block in the Runtime actors; with CancellationToken.None a hung protocol stack can stall stop/re-deploy of the whole child set. Recommendation: bound it (WaitAsync(timeout)) and log-and-abandon on expiry.

S8 — MEDIUM: Inbound Part 9 commands and native acks are at-most-once; the client sees Good even if the command is lost. The alarm-command router is a fire-and-forget DPS Publish and the native-ack router a fire-and-forget Tell, both wrapped in catch-log-drop (OtOpcUaServerHostedService.cs:139-194); the node manager has already returned Good so the SDK applies local condition state regardless (OtOpcUaNodeManager.cs:769-772). A mediator hiccup or missing DriverHostActor silently strands the engine/upstream state. This is a documented, deliberate trade-off (non-blocking under Lock), but it is invisible in operation. Recommendation: add a counter/metric for dropped routes (both routers currently only log at Warning) and consider an engine-side reconciliation sweep.

S9 — MEDIUM: Certificate lifecycle has no renewal or expiry monitoring. The application certificate is auto-created self-signed with SDK defaults (2048-bit, 12-month lifetime) and only checked at boot (OpcUaApplicationHost.cs:305-317). Nothing monitors expiry; ~12 months after first deploy, Sign/SignAndEncrypt endpoints and UserName-token encryption fail on live servers with no advance warning. Recommendation: startup + periodic expiry check with a metric/health-check and a documented rotation runbook (the AdminUI cert-actions work covers client certs; the server cert lifecycle is the gap).

S10 — LOW: SDK start failure is swallowed and the node keeps running with Null sinks. StartAsync catches, logs, and returns (OtOpcUaServerHostedService.cs:110-129) — a deliberate availability choice (AdminUI stays up), but the node then silently no-ops all OPC UA work and nothing surfaces the condition through /health or ServiceLevel (which is itself served by the dead server). Recommendation: set a health-check flag consumed by MapOtOpcUaHealth so fleet status shows the degraded state.

S11 — LOW: AuditWriterActor buffer is unbounded between flushes and drops whole batches on DB outage (ControlPlane/Audit/AuditWriterActor.cs:35, 83, 113-116). Explicitly best-effort by contract; flag only if compliance-grade audit is ever required. (Currently moot — see U3: the pipeline has no producers.)

Positive stability notes (worth preserving): the node-manager locking discipline is exemplary — every mutation under Lock, every Server.ReportEvent deliberately outside it (OtOpcUaNodeManager.cs:1101-1136, 1503-1504, 1595-1614), the write-dispatch continuation forced off the SDK thread via RunContinuationsAsynchronously (:884-897), and the revert guarded by "still holds the optimistic value" (ShouldRevert, :1048). The Deferred-sink swap-to-Null on stop prevents post-stop writes hitting a disposed manager. ConfigPublishCoordinator has real crash-recovery (rebuilds expected-acks from DB on PreStart). HandleServiceLevelChanged's first-publish-always correctly defeats the SDK's 255 default. ActorNodeWriteGateway bounds its Ask at 10 s with the inner driver Ask at 8 s and the backend call at 5 s — a correctly nested timeout ladder.

2. Performance

P1 — HIGH: Any structural deploy change triggers a full address-space teardown/rebuild that severs every client's monitored items server-wide. AddressSpaceApplier.Apply forces RebuildAddressSpace() for any added/removed equipment, alarm, tag, or virtual tag (AddressSpaceApplier.cs:134-154); the rebuild removes every variable/folder/condition under one Lock hold and re-materialises from scratch (OtOpcUaNodeManager.cs:1690-1736). Recreated nodes are new NodeState instances, so existing client subscriptions on the old nodes go dead — the F10b surgical path (UpdateTagAttributes/UpdateFolderDisplayName) exists precisely because of this, but covers only attribute edits and renames. Adding one tag to one equipment still drops every subscription on a production SCADA server. Recommendation: extend the surgical path to pure-add deltas first (EnsureFolder/EnsureVariable are already idempotent — adds need no teardown; RaiseNodesAddedModelChange already exists for announcement), then scope removals per-equipment. This is the highest-leverage performance/stability item in the subsystem.

P2 — MEDIUM: The value hot path is one global lock acquisition + one actor message per value, with no batching. Per published value: driver child Tell → DriverHostActor.ForwardToMux (dictionary lookup + up to two Tells, DriverHostActor.cs:561-589) → OpcUaPublishActor (pinned thread) → sink.WriteValue, which takes the global node-manager Lock per value (OtOpcUaNodeManager.cs:266-281), contending with SDK read/subscription/publish threads. The mux itself is lean (DependencyMuxActor.cs:95-108 — early-drop for uninterested refs, set-based fan-out). At high tag counts × fast poll intervals this serialises everything through one lock and allocates one record per hop. Recommendation: add a batched WriteValues(IReadOnlyList<…>) sink call (one lock hold per driver publish cycle) — the seam (IOpcUaAddressSpaceSink) makes this a contained change, but remember the Deferred-wrapper forwarding trap (U2).

P3 — MEDIUM: HistoryRead-Events has no paging and translates NumValuesPerNode == 0 to int.MaxValue. EventMaxEvents deliberately maps "no limit" to unbounded (OtOpcUaNodeManager.cs:1944-1954), and the Events arm never issues continuation points ("full window in one shot", :1914-1921). A wide time window over a busy alarm source materialises the entire result in memory on both gateway and server. Recommendation: impose a server-side max (mirroring MaxTieClusterOverfetch's philosophy) and fail loudly or page beyond it.

P4 — LOW: Every deploy re-runs the four Materialise* passes over the full composition (OpcUaPublishActor.cs:338-354) — acceptable because EnsureFolder/EnsureVariable early-return on existing ids, but each pass still takes/releases the Lock per node; fold into P2's batching if it shows up in profiles. On the positive side, the Raw-paging tie-cluster over-fetch is explicitly bounded (MaxTieClusterOverfetch, default 65 536) with a loud-fail backstop, and HandleDiscoveredNodes has an unchanged-plan short-circuit preventing ~15× re-subscribe blips during the discovery window (DriverHostActor.cs:658-673) — both good designs.

3. Conventions

C1 — MEDIUM: The ServerHistorian section is bound imperatively in five places with no IOptions registration. Program.cs:120-122 (own bind + Validate() warnings), AddServerHistorian, AddAlarmHistorian, AddHistorianProvisioning (each re-Get<>s the section, Runtime/ServiceCollectionExtensions.cs:86, 132, 168), and OtOpcUaServerHostedService's constructor (.cs:88-90, self-justified as "self-contained"). Warnings can log twice; five bind sites will drift. Recommendation: one AddValidatedOptions<ServerHistorianOptions> and inject IOptions<> everywhere, matching the OpcUa/Ldap pattern that already exists (Program.cs:102, 254-255).

C2 — MEDIUM: Options validation is two-tier and the tiers disagree. OpcUa and Security:Ldap get real fail-fast ValidateOnStart validators (OpcUaApplicationHostOptionsValidator requires ≥1 security profile; LdapOptionsValidator refuses plaintext-without-opt-in) — good. ServerHistorian / ContinuousHistorization / AlarmHistorian only get advisory Validate() warning lists, and DevStubMode=true (accept-any-credentials Administrator grant) is merely log-warned, never blocked, in production (OtOpcUaLdapAuthService.cs:79-88). Recommendation: promote the historian sections to the validator pattern; add an environment-gated guard (or at minimum a startup validator failure) for DevStubMode outside Development.

C3 — LOW: Library code logs via the static Serilog logger (Runtime/ServiceCollectionExtensions.cs:90, 100, 136) while everything else uses MEL ILogger — works only because Program.cs:305 assigns Log.Logger, an ordering-sensitive hidden coupling. The node manager similarly logs through the obsolete Utils.LogError static trace with six #pragma warning disable CS0618 sites (OtOpcUaNodeManager.cs:449-451 et al.) — the acknowledged follow-up (wire an ITelemetryContext) should actually land.

C4 — LOW: WithOtOpcUaRuntimeActors is service-locator style with silent Null fallbacks (Runtime/ServiceCollectionExtensions.cs:211-237) — the accepted Akka.Hosting idiom, and the missing-registration warnings (evaluator, recorder deps) are good; but resolver.GetService<IHistorianProvisioning>() (:298) has no fallback warning parallel to the others, despite being the exact seam that shipped dormant once (PR #423). The dispatched=N, requested=0 tally in the applier (AddressSpaceApplier.cs:274-282) partially compensates.

Positive conventions notes: the layering is clean and deliberate — OpcUaServer is Akka-free (delegates/gateway seams), Runtime is SDK-free (sink seams), Core holds the pure calculators (ServiceLevelCalculator, planners); the Null-object + Deferred late-binding pattern is applied consistently (sink, ServiceLevel, write gateway, historian source, provisioning, history writer); actor Props factories + registry marker keys are uniform; scope-correctness around the scoped IGroupRoleMapper inside the singleton authenticator is explicitly handled (LdapOpcUaUserAuthenticator.cs:63-67); NodeId is canonically host:port everywhere except one dormant spot (U4).

4. Underdeveloped areas

U1 — HIGH (documentation drift on a load-bearing claim): CLAUDE.md "KNOWN LIMITATION 2" is stale — the ContinuousHistorizationRecorder ref-feed gap is CLOSED in code. The recorder is still spawned with historizedRefs: Array.Empty<string>() (Runtime/ServiceCollectionExtensions.cs:272), but the applier now feeds it the add/remove delta on every deploy via ActorHistorizedTagSubscriptionSink → UpdateHistorizedRefs (Runtime/ServiceCollectionExtensions.cs:287-292; AddressSpaceApplier.cs:210-213, 310-353; ContinuousHistorizationRecorder.cs:67-78). Restart convergence works because OpcUaPublishActor._lastApplied is in-memory, so the first post-boot rebuild diffs against empty and emits the full set as Added (OpcUaPublishActor.cs:326-336). CLAUDE.md still says "the recorder registers interest in nothing and historizes nothing… Until that feed lands, continuous historization records no values." Recommendation: update CLAUDE.md; add an explicit restart-convergence test (boot → apply full plan → assert mux interest registered), since the chain (empty seed + delta feed + in-memory _lastApplied) is load-bearing and only implicitly covered today.

U2 — MEDIUM: The DeferredAddressSpaceSink capability-forwarding trap is currently closed but only half test-guarded. Audit result: all current members forward correctly, including both ISurgicalAddressSpaceSink methods with capability-sniffing fallback (src/Core/…/DeferredAddressSpaceSink.cs:58-70) — no inert capability today. But tests assert forwarding for only ~5 of the 10 members; WriteAlarmCondition, MaterialiseAlarmCondition, EnsureFolder, RebuildAddressSpace, RaiseNodesAddedModelChange have no per-member forwarding assertion (tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests / DeferredAddressSpaceSinkTests.cs). The F10b incident (surgical path dead in prod for exactly this reason) proves the failure mode is real. Recommendation: add a reflection-driven exhaustive test: every method of IOpcUaAddressSpaceSink + every optional capability interface the wrapper claims must reach a recording inner sink. Same treatment for DeferredServiceLevelPublisher.

U3 — MEDIUM: The structured audit pipeline is fully built, tested, and has zero producers. AuditWriterActor + AuditOutcomeMapper state "no live structured AuditEvent emit sites; all production audit flows through the bespoke stored-procedure path" (ControlPlane/Audit/AuditOutcomeMapper.cs:12-18; Security/Audit/AuditActor.cs:18-24). Either wire producers or delete — dormant tested code decays.

U4 — MEDIUM: FleetStatusBroadcaster.DriverHostStatusHeartbeat is dead code with a latent NodeId-mismatch repeat. No producer anywhere sends the heartbeat (FleetStatusBroadcaster.cs:38, 110-121), so CurrentRevision is always null and the freshness branch never fires; and the broadcaster keys nodes by host only (:152-159) while every other subsystem uses host:port — if the heartbeat is ever wired with a host:port NodeId it will create phantom node records, re-enacting the historical NodeId bug. Recommendation: wire the producer (DriverHostActor already knows its applied revision) or remove; either way canonicalise the key to host:port.

U5 — MEDIUM: Native Part 9 conditions support Acknowledge-to-driver only. Confirm/AddComment/Shelve on a native condition still route to the scripted engine, which does not own them (documented H6c scope, OtOpcUaNodeManager.cs:647-658); Enable/Disable short-circuits to BadNotSupported (:698-703). Operators using a Part 9 client will find shelving a Galaxy alarm silently ineffective upstream.

U6 — MEDIUM: Test-coverage gaps concentrate exactly where the architecture is most fragile (from the tests/Server sweep, ~910 tests total across 6 projects):

• DPS delivery of redundancy/ServiceLevel state is untested — RedundancyStateActorTests stub the broadcast, ServiceLevelEndToEndTests inject NodeRedundancyState directly, and the TwoNodeClusterHarness runs in-process with EF InMemory. This is the known "unit tests can't catch it" blind spot, and it isn't even documented in a test comment.
• No actor supervision/restart test anywhere (relates to S6) — only "init failure keeps Reconnecting state" is covered.
• Outbox durability across process restart and sustained-outage backpressure untested (a single failure→retry→ack cycle is).
• OpcUaServer.IntegrationTests contains one test (dual-endpoint ServerArray) — no over-the-wire security-mode matrix, subscription, or HistoryRead round-trip.
• Docker-fixture-dependent driver-protocol tests self-skip when 10.100.0.35 is unreachable — green local/macOS runs can hide protocol regressions. Strengths worth noting: node-manager HistoryRead/paging (~60 tests), the write-gate/revert path, LDAP fail-closed edges, and Composer/Applier (~120 tests) are genuinely well covered.

U7 — LOW: IHistoryWriter is permanently Null-wired (vtag historization, infra-gated on a nonexistent live-write RPC — Runtime/ServiceCollectionExtensions.cs:56-58); H2 HistoryUpdate remains unimplemented (HistoryRead-only server; IsReadModified rejected, OtOpcUaNodeManager.cs:1797-1801) — both tracked backlog, listed for completeness. LdapAuthFailure lacks a DirectoryUnavailable value (directory-down is overloaded onto ServiceAccountBindFailed).

U8 — LOW: BuildSecurityPolicies doc-comment claims the empty-profile fallback is "logged and very visible" but the method logs nothing (OpcUaApplicationHost.cs:376-418). Defused in production by the options validator (MinCount ≥ 1, OpcUaApplicationHostOptionsValidator.cs), but the silent fail-open-to-None remains for direct embedders/tests; fix the comment or add the log.

U9 — LOW: EnsureVariable silently ignores changed historize-intent on an existing node (documented, OtOpcUaNodeManager.cs:1345-1349) — correct today because the planner routes such deltas to UpdateTagAttributes or a rebuild, but the invariant lives in two files' comments rather than an assert.

---

Maturity ratings

Dimension	Rating (1-5)	Justification
Stability	3	Excellent node-manager locking/self-correction discipline and Null/Deferred boot safety, but the inert SBR (S1) breaks hard-crash failover cluster-wide, and LDAP/HistoryRead thread-blocking (S2/S3) plus the primary-gate boot window (S4) are real production exposures.
Performance	3	Hot paths are lean per-message and history paging is carefully bounded, but the rebuild-on-any-structural-change subscription wipe (P1) and per-value global-lock writes (P2) will not survive large tag counts or frequent deploys gracefully.
Conventions	4	Deliberate, consistent layering (Akka-free OpcUaServer, SDK-free Runtime, Null/Deferred seams, uniform actor patterns); loses a point for the five-way ServerHistorian binding, two-tier options validation, and static-logger islands.
Underdeveloped	3	Core capabilities are complete and the once-dormant provisioning/ref-feed seams are now wired, but stale load-bearing docs (U1), a half-guarded forwarding trap (U2), dormant audit/heartbeat code (U3/U4), and test blind spots exactly on the fragile seams (U6) leave meaningful debt.

---

Cross-cutting themes

1. Config-in-HOCON is not config-in-effect — the SBR block shows Akka behavior declared in akka.conf can be silently inert without the activating registration; audit other HOCON-declared behaviors (dispatcher blocks, singleton settings) the same way.
2. The Deferred/Null seam pattern is powerful but trap-prone: every new optional capability must be forwarded through the wrappers and exhaustively test-guarded, or it ships inert (proven twice: F10b surgical sink, PR #423 provisioning). A reflection-based forwarding test would retire the trap class.
3. "Default-allow until told otherwise" gates (primary write gate, peer-probe benefit-of-the-doubt, ServiceLevel legacy seam) consistently trade split-brain safety for boot-window availability; each is individually reasoned but they compound with at-most-once DPS delivery.
4. Unit-test greenness systematically overstates distributed correctness here: DPS delivery, supervision restarts, hard-kill failover, and Docker-gated protocol paths all sit outside the test net — and the project's own history (redundancy-delivery bug, F10b inertness) shows these are exactly where the bugs live.