Files
lmxopcua/docs/drivers/OpcUaClient.md
Joseph Doherty beb1c583b1 fix(opcuaclient): #676 live gate A1 — a sustained closed connection is a definitive loss
Live gate A1 (docker-dev, opc-plc restart / SIGINT / kill): the fast path never fired —
a real process stop delivers no session status; the keep-alives report
BadSecureChannelClosed (0x80860000) once, then BadConnectionClosed (0x80AE0000)
interleaved with BadNoCommunication for the whole outage.

Reproduced in-process with TcpStallProxy.HardClose (reset every connection, refuse new
ones; server and session alive behind it): the identical sequence, and the SDK 1.5.378
session never opens a new connection on its own (48 s with the proxy accepting again,
zero attempts) — a closed channel cannot recover in place.

The driver now treats BadSecureChannelClosed/BadConnectionClosed as definitive on the
SECOND closed-channel keep-alive failure in one outage (a good keep-alive resets the
count), raising ConnectionLost ~1 keep-alive after the drop. BadNoCommunication /
BadRequestTimeout / BadNotConnected (stall/pause) stay with the watchdog; the stall test
still sees no ConnectionLost. New wire test (hard close + reopen ⇒ raised once, SDK never
reconnects, reinit brings the identical value back Good) and unit tests (sustained raises
once; a single blip that recovers does not) — all red on the previous code. Docs updated.
2026-10-11 12:34:52 -04:00

27 KiB
Raw Permalink Blame History

OPC UA Client (Gateway) Driver

Getting-started guide for the OPC UA Client driver. This is the short path — for the full per-field spec read docs/v2/driver-specs.md §8, and for the test-harness map read OpcUaClient-Test-Fixture.md.

What it talks to

A remote OPC UA server. This driver runs the opposite direction from the usual "server exposes PLC data" flow: it acts as an OPC UA client, opens a Session against an upstream server, and re-exposes that server's address space through the local OtOpcUa server. Browse, read, write, subscribe, alarm, and history calls are passed through to the upstream endpoint.

It is built on the OPC Foundation UA .NET Standard reference SDK and runs in-process in the OtOpcUa server's .NET 10 AnyCPU host — pure managed, no out-of-process isolation.

There is no standalone driver CLI for the OPC UA Client driver. To exercise a remote OPC UA endpoint by hand, point the general-purpose Client CLI at it directly.

Project split

Project Target Role
src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.OpcUaClient/ net10.0 In-process driver — session lifetime, read / write / subscribe / alarm / history passthrough
src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.OpcUaClient.Browser/ net10.0 IDriverBrowser — live address-picker browse used by the AdminUI
src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.OpcUaClient.Contracts/ net10.0 Config records + enums bound from DriverConfig JSON

Minimum deployment

"Drivers": {
  "upstream-1": {
    "Type": "OpcUaClient",
    "Config": {
      "EndpointUrl": "opc.tcp://plc.internal:4840",
      "SecurityPolicy": "None",
      "SecurityMode": "None",
      "AuthType": "Anonymous"
    }
  }
}

EndpointUrls (a list) takes precedence over the single-URL EndpointUrl and provides ordered failover — the driver tries each candidate in turn at every InitializeAsync (including the comms-loss watchdog's re-initialise), and the first to connect wins (e.g. a hot-standby pair on 4840 / 4841). See src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.OpcUaClient.Contracts/OpcUaClientDriverOptions.cs for every field (security policy/mode, auth type, session timeout, keep-alive, browse root, node/depth caps). ReconnectPeriod was deleted (#676 — unused since #670 removed the SDK reconnect handler, and the AdminUI no longer shows it); stored configs that still carry reconnectPeriod keep parsing in the factory, the Test-Connect probe and the browse picker (unknown keys are skipped), and the form drops it on the next save.

Endpoint address: the configured host and port win (#676)

GetEndpoints returns each endpoint with the URL the server believes it lives at. Behind NAT, a port map, a container port publish or a proxy that address is unreachable from the client — or reachable but bypassing the proxy. The driver therefore rewrites the selected endpoint's host and port to the configured URL's whenever the schemes match, keeping the advertised path (the SDK's own CoreClientUtils.SelectEndpoint rule, which the driver's policy-aware endpoint selection used to bypass). A configured URL with no port keeps the advertised port (copying the port-less -1 would have sent the session to 4840). A different scheme is left alone; an unparseable advertised URL falls back to the configured one. OpcUaClientDriver.RewriteEndpointUrl.

Session lifetime

A single Session per driver instance; subscriptions multiplex onto it. Stored NodeIds embed the server-stable namespace URI (not the session-relative ns=N index) so a remote namespace-table reorder across a restart doesn't silently re-point references at the wrong namespace.

Upstream outage and recovery (Gitea #670)

The session keep-alive (KeepAliveInterval, default 5 s) is the liveness signal:

  • Good keep-alive / any data or event notification advances DriverHealth.LastSuccessfulRead — a subscription-only driver never calls ReadAsync, so without this its read stamp froze at connect time.
  • Bad keep-alive ⇒ health Degraded (LastError = "keep-alive: <status>", read stamp preserved), host status Stopped, and — once per outage, not per keep-alive — every subscribed ref is published BadNoCommunication with its last value retained and source + server timestamp = the detection time (live-gate F1: stamped with the last Good's source time it collided with that row's (TagId, SourceTimeUtc) history key and was never stored), plus a quality-only BadNoCommunication annotation (#626 pattern, no alarm-state change, no /alerts row) on every subscribed alarm condition.
  • The session survives (a network blip; the keep-alive turns Good again) ⇒ recovered in place: health Healthy, the refs still Bad are re-read once and published Good (a server whose subscription survived does not resend an unchanged value, and a (value, status) dedup downstream lets the identical value through), alarm quality goes back Good and a ConditionRefresh re-reports anything missed. A notification that lands first wins over the read. For a bounded replay window after such a recovery — max(30 s, 2 x the longest subscription keep-alive window) — each ref is guarded by a high-water mark — its last Good upstream source time, raised by the recovery read and advanced by every notification accepted in the window — and a notification whose source time is at or BELOW it (equal included: OPC UA moves the source timestamp on every value or status change, so an equal one is a replay) is dropped: the SDK was seen delivering a missed one 4–6 s late, which regressed the node (live-gate F2). The guard holds for the WHOLE window — fresh notifications arrive ~1 s after recovery, before the late replay, so a ref is never disarmed early (the F2 re-gate failed on exactly that). A notification with no source time passes without moving the mark. Outside that window every notification is accepted whatever its timestamp, so an upstream clock that steps backwards (NTP, a PLC clock reset) never freezes a node. The outage Bad never moves the comparison; a re-initialise resets it. ⚠️ By design, inside the window a notification whose source time EQUALS the last accepted one is dropped — so a status-only change that keeps the old source timestamp (not what OPC UA servers are supposed to do) is lost during the window only (the trailing re-read below repairs it). When the window closes — or a second recovery re-arms it — the driver logs ONE line, post-recovery replay window closed — dropped N late notifications across M refs after recovery, at Information when N > 0 and Debug otherwise, because the per-drop line is Debug and the stale_notification_dropped counter is OTel-only, which made the guard unobservable on the docker-dev rig (#676). A re-arm starts the count afresh.
  • Trailing re-read after an in-place recovery (#676). Found by the TCP-stall wire test: a change the server published just after the stall ended rode a Publish response that had been in flight across the stall; the SDK timed that request out (BadTimeout), discarded the response and never republished the message (the following keep-alives moved the sequence number past it), so the node stayed Good on the pre-change value indefinitely. Timeout + KeepAliveInterval after the recovery — when every request in flight across the stall has completed or timed out — every subscribed ref is read once more and published when its (value, status) differs from the last value delivered for that ref and is not older than it. Any status counts: a Good→Bad/Uncertain change lost in the pipeline is as stale as a lost value. "Older" compares source timestamps only when both carry one; a read with no source timestamp (a server that does not send them) is the server's current state and counts as newer. An equal source timestamp with different content IS published — it is the server's current state, not a replay, and it repairs the equal-timestamp status-only change the replay guard drops by design. An identical (value, status) is never republished. A ref that received ANY delivery while the read was in flight is skipped (a per-ref delivery counter is captured before the read is issued): without source timestamps the read's sample cannot be ordered against that notification, and publishing it would regress the node. An Information line names how many refs it had to repair. The trailing read and the replay-guard disarm timer run on the driver clock and are cancelled when the session is released (re-initialise / shutdown / dispose).
  • No older value is ever delivered after a newer one. The paths that publish upstream data — live notifications, the recovery read, the trailing read and the comms-loss Bad fan — make their "may I publish this?" decision and the publish inside one driver-wide critical section. A single lock rather than one per ref: it also serialises delivery across subscriptions (the SDK dispatches each subscription's messages on its own worker), which is cheap because each section is a dictionary update plus a subscriber Tell and nothing awaits under it. Subscribers run inside it — the host's handler only Tells an actor — so a subscriber must never block on, or call back into, the driver.
  • The session does not come back (an upstream restart: the server forgot the session) ⇒ the driver stays Degraded with its read stamp frozen, which is the #527 comms-loss watchdog's trip condition; DriverInstanceActor runs ShutdownAsync + InitializeAsync (retried every 10 s while the server is down) and re-subscribes data + alarms; the new subscription's initial values come back Good. Alarm last-told state survives the re-initialise (#619), so the post-subscribe ConditionRefresh raises no un-ack flicker; the host's reconnect annotation restores condition quality.
  • Fast recovery for a DEFINITIVELY lost session (#676). The driver implements IDriverConnectionLossSource and raises ConnectionLost — once per session, re-armed by a re-initialise — when the loss is certain, so the host re-initialises at once instead of after the watchdog's 90 s. Definitive means: a keep-alive status of BadSessionIdInvalid / BadSessionClosed / BadSessionNotActivated (the server forgot, closed or never activated the session) or BadServerHalted, or a subscription the server reported gone (below). BadServerHalted is on the list because it was measured: on the restart wire test a gracefully stopped server reports BadServerHalted (and the keep-alives alternate it with BadNoCommunication), and the SDK never re-opens a channel the server halted — 15 s after the server was back, the same session still failed every request with BadServerHalted; BadSessionIdInvalid never arrived. A CLOSED connection is definitive too, once sustained (live gate A1). Against a real opc-plc on docker-dev, none of docker restart, a graceful SIGINT stop (opc-plc itself logged BadServerHalted) or docker kill delivered a session status to the client: the keep-alives reported BadSecureChannelClosed (0x80860000) once, then BadConnectionClosed (0x80AE0000) interleaved with BadNoCommunication for the whole outage, so the first cut of the fast path never fired and recovery took 91–127 s. Reproduced in-process with TcpStallProxy.HardClose (every connection reset, new ones refused, the server AND its session still alive behind it): the same sequence, and the SDK 1.5.378 session never opens a new connection by itself — 48 s after the proxy accepted connections again, not one attempt. So a closed channel can never recover in place; on the second closed-channel keep-alive failure in one outage (IsClosedChannel; a good keep-alive resets the count, so a single transient does not count) the driver raises ConnectionLost — about one keep-alive interval after the connection dropped. Everything else — BadNoCommunication, BadNotConnected, BadRequestTimeout/BadTimeout (what a stalled or paused link produces) — is ambiguous (the link may resume and keep its session, see the stall test) and stays with the watchdog. Consequence: a restarted, stopped or killed upstream is re-initialised within roughly one keep-alive plus one reconnect interval instead of ~100 s; a pause (docker pause, a congested link) still recovers in place. ⚠️ Known gap: the host acts on the signal only while the driver is Connected, and the driver raises it once per session — so a definitive loss raised while the host is not Connected (e.g. BadServerHalted on the fresh session before the host has processed the successful initialise) is swallowed and that session falls back to the watchdog. Re-raising on later keep-alives would close it but was left out to keep the once-per-session contract simple; the cost is the pre-#676 recovery time, in a narrow window.
  • The subscription dies but the session lives (an outage longer than the subscription lifetime — LifetimeCount x publishing interval, often well under the session timeout — or a server that drops it) ⇒ keep-alives stay Good, so they cannot be the signal. A PublishError of BadNoSubscription (while subscriptions are tracked) or BadSubscriptionIdInvalid naming a tracked subscription, or a tracked subscription whose SDK PublishingStopped persists for one more subscription keep-alive window + 5 s across good keep-alives, declares it lost: Degraded ("subscription lost: …"), the outage published once, and — sticky until the session is released — nothing advances LastSuccessfulRead any more (not good keep-alives, not a sibling subscription's notifications), so the watchdog's re-initialise recreates it. Both detectors are wire-tested independently. A subscription the server reported gone (BadNoSubscription / BadSubscriptionIdInvalid) never comes back on its session, so that also raises ConnectionLost (#676) and the re-initialise happens at once; the PublishingStopped heuristic is an inference from silence and keeps going through the watchdog.
  • An in-place recovery is bound to the session whose keep-alive recovered: one overtaken by a re-initialise neither publishes, nor degrades, nor re-arms the outage latch of the new session.

There is no in-driver reconnect any more. The SDK's SessionReconnectHandler (removed) retried forever in SDK 1.5.378 (its "exhausted" path was unreachable), recreated sessions by cloning subscriptions the driver never re-tracked, and kept health Healthy throughout — so on docker-dev a stopped opc-plc left nodes Good on their last value indefinitely and they never recovered. A teardown against a lost session skips the wire deletes and CloseSession (bounded to 5 s otherwise). SDK warnings (keep-alive / publish / channel errors) now reach the driver's own logger, rate-limited to 20 per minute with a suppressed-count note; the per-driver SDK logger factory is disposed with the driver.

Namespace assignment (retired, Gitea #648)

Earlier versions required a per-instance TargetNamespaceKind (Equipment + a mandatory UnsMappingTable, or SystemPlatform + an empty table). Both options are removed: nothing ever read the mapping table beyond the startup check, and in v3 the raw-to-UNS projection is explicit (UnsTagReference), so the choice carried no behaviour — it only made a config without the key fail Initialize. Remote nodes are authored as /raw tags bound by their upstream nodeId; there is no namespace field on the driver, the AdminUI form or the seed.

Stored DriverConfig JSON that still carries targetNamespaceKind or unsMappingTable keeps working: the keys are unknown and ignored on parse and initialise (the AdminUI form drops them on the next save). Pinned by OpcUaClientReinitConfigAdoptionTests.Legacy_namespace_keys_are_ignored_by_parse_and_initialise.

Capability surface

OpcUaClientDriver : IDriver, ITagDiscovery, IReadable, IWritable, ISubscribable, IHostConnectivityProbe, IAlarmSource, IDriverConnectionLossSource (src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.OpcUaClient/OpcUaClientDriver.cs:31).

Capability Path Notes
ITagDiscovery DiscoverAsync (recursive browse) Mirrors the upstream tree from BrowseRoot (default ObjectsFolder i=85), bounded by MaxDiscoveredNodes / MaxBrowseDepth
IReadable ReadAsync → Session.ReadAsync Upstream StatusCodes pass through verbatim (cascading-quality rule)
IWritable WriteAsync → Session.WriteAsync Passthrough write
ISubscribable native OPC UA subscriptions / monitored items The remote server pushes data changes
IHostConnectivityProbe session keep-alive Host key is the endpoint URL actually connected to after the failover sweep
IAlarmSource SubscribeAlarmsAsync (EventFilter) + AcknowledgeAsync Subscribes to upstream alarm/condition events and forwards acks

Native alarms (#619). An alarm raw tag is TagConfig { "nodeId": "<upstream SourceNode>", "alarm": {...} }. The host subscribes with RawPaths and routes on (driver, RawPath), so the driver matches each event's ConditionId (then its SourceNode) against the tag's authored nodeId and reports the RawPath as ConditionId. Author nodeId as the upstream condition node for a 1:1 mapping (required when one source carries several conditions, e.g. Hi + HiHi); authoring the source node folds every condition of that source into one tag. Acknowledge is called on the condition node of the last event seen for the RawPath with that event's EventId (Part 9 §5.7.3). The event's ActiveState/Id + AckedState/Id are selected and turned into the Raise/Acknowledge/Clear/Retrigger kinds the host projector consumes (without a kind no upstream alarm could ever show active); the kinds are emitted from the state last told to the host per RawPath, so an identical re-delivery emits nothing (no re-raise, which would un-ack). An inactive → active+acked move is ONE ActiveAcknowledged (#656; it used to be Raise + Acknowledge, a client-visible unacked intermediate); a move into inactive+unacked is ONE InactiveUnacknowledged (#657; it used to be Raise/Retrigger + Clear, an Active flicker). The first event per condition after a start or deploy respawn is flagged IsRestatement and states the upstream condition so it lands from whatever the host retained: an inactive+acked one goes out as Clear + Acknowledge (#657; it used to be an Unspecified the host wrote as "Activated", and a retained-Active host condition stayed Active). So the host writes no duplicate row when its condition already shows that state (see docs/AlarmTracking.md). Condition branch events (non-null BranchId) are ignored. ConditionRefresh is called after the event item is created, after a keep-alive outage the session survived, and on a RefreshRequired event, so a condition already active upstream is reported at subscribe; RefreshStart/RefreshEnd markers are dropped. A server without it (BadNotSupported/BadMethodInvalid/…) is logged once and tolerated. Last-told state survives a re-initialise (the host re-subscribes the same set over nodes it did not rebuild) and is forgotten when the subscribed set changes (the host rebuilt, recreating the nodes inactive). Index keys are server-stable (nsu=), so a namespace reorder across a re-initialise keeps matching. docker-dev seeds an opc-plc driver (MAIN-opcua); the alarm tag is created only when @OpcPlcAlarmSourceNodeId is set in docker-dev/seed/seed-clusters.sql.

This driver does not implement IRediscoverable — there is no push-driven rediscovery signal from a remote OPC UA server in this driver.

History: not supported (Gitea #655)

This driver does not proxy the upstream server's HistoryRead. The server serves HistoryRead only through the server-wide IHistorianDataSource (see docs/Historian.md); drivers carry no history capability, and the former IHistoryProvider passthrough (Raw / Processed / AtTime / Events) was never dispatched to and has been deleted. isHistorized on an OpcUaClient tag means OtOpcUa records the value into the historian from now on (numeric only), not that upstream history is forwarded. A tag without isHistorized has no HistoryRead access bit and answers BadHistoryOperationUnsupported. If upstream-history proxying is wanted later, it needs a new intent key, a dispatch path through DriverHostActor, and a continuation-point rework — see #655.

Certificate trust

AutoAcceptCertificates accepts any self-signed / untrusted server certificate. It is dev-only — leave it false in production so a MITM against the opc.tcp channel fails closed. The SDK rejects a revoked certificate (BadCertificateRevoked / BadCertificateIssuerRevoked) before the auto-accept handler runs, and the handler also refuses them explicitly — defence in depth (#277), not a behaviour change.

CertificateValidation (defaults change nothing):

Key Default Meaning
rejectUnknownRevocationStatus false Reject a server certificate whose revocation status cannot be determined (no CRL for its issuer, or a stale one). Leave false for brownfield sites without CRL infrastructure. Only effective with autoAcceptCertificates off — the auto-accept handler accepts every suppressible error, including revocation-unknown.
issuerStorePath null (= %LocalAppData%/OtOpcUa/pki/issuers) Directory used as the trusted-issuer store. The SDK reads issuer CRLs (*.crl) from its crl/ sub-folder, so this is how an operator points the driver at a managed CRL set.

Subscription tuning (#273)

The per-subscription parameters that used to be hardcoded are driver options; the defaults are the old values.

Key Default Notes
subscriptionPublishingInterval null A .NET TimeSpan string ("00:00:02"), not a number. null = the interval the runtime requests per subscribe (floored at 50 ms). When set it overrides that value (floor still applies); monitored-item sampling follows it.
subscriptionKeepAliveCount 10 >= 1.
subscriptionLifetimeCount 1000 Must be >= 3 x keep-alive count (Part 4 §5.13.2).
subscriptionMaxNotificationsPerPublish 0 0 = server decides.
subscriptionPriority 0 0-255.

Out-of-range values (and a non-absolute discoveryUrl) are rejected when the options are loaded — driver creation / InitializeAsync fails once with a clear message rather than the driver faulting and retrying forever. The alarm subscription uses the same counts. Per-tag tuning (deadband / sampling interval / queue size, #274) is not implemented — it needs an ISubscribable signature change on Core.

Server OperationLimits (#275)

Nothing driver-side: the SDK session already honours the remote server's ServerCapabilities/OperationLimits. Opc.Ua.Client.Session derives from Opc.Ua.SessionClientBatched, fetches the limits on open (FetchOperationLimitsAsync) and splits Read / Write / CreateMonitoredItems / Browse / HistoryRead requests that exceed them (a limit of 0 = unlimited). The driver issues one logical call per operation and a unit test pins the Session : SessionClientBatched relationship so an SDK change that dropped it is caught. (A driver-side splitter was tried and removed: it duplicated the SDK and, on a stalled link, held the driver gate for chunks x timeout.)

Discovery URL (#278)

discoveryUrl (default null, e.g. opc.tcp://lds:4840): at InitializeAsync the driver runs FindServers on it and GetEndpoints on every returned server, keeping the endpoints matching securityPolicy + securityMode. Discovered URLs are tried after endpointUrls/endpointUrl (case-insensitive de-dup) — discovery can add fallbacks but never outranks or replaces the configured server. Because an LDS commonly registers several servers (identical line PLCs, a gateway plus a historian) and the first to accept a connection would be bound silently:

  • discoveryServerUri set: only endpoints of the server with that ApplicationUri are used.
  • unset and exactly one distinct server discovered: that server is used.
  • unset and more than one: the connect attempt fails with an error naming the discovered ApplicationUris and telling the operator to set discoveryServerUri.

A discovery network failure (LDS down) is logged and non-fatal — the static list is used (note it is retried on every InitializeAsync, including comms-loss reconnects). The discovery channel itself is always unsecured (spec).

Diagnostics counters (#276)

Session/publish events are published on the shared OTel meter as the counter otopcua.driver.opcuaclient.events, tagged driver_instance (the driver instance id) and event = publish_response, notification (items), data_change, bad_status_value, publish_error, keepalive_miss, keepalive_recovered (a bad keep-alive followed by a good one on the same session), stale_notification_dropped (a late replay dropped inside the post-recovery window, see above). A climbing keepalive_miss / keepalive_recovered rate is the signature of a flapping upstream (reconnect_* were removed with the reconnect handler, #670). They are not part of DriverHealth or the /hosts panel.

Testing

  • Unit tests — tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.OpcUaClient.Tests/ cover the session lifecycle, namespace remapping, alarm/history passthrough, and config binding against a faked SDK session.
  • Integration fixture — exercises the driver against a reference OPC UA server (opc-plc) on the shared docker host; see OpcUaClient-Test-Fixture.md for the coverage map.
  • Wire tests against the in-process server — tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.IntegrationTests/ OpcUaClientDriver*WireTests (no Docker). OpcUaClientDriverStallWireTests (#676) covers the in-place blip that used to be live-gate-only: the driver connects through TcpStallProxy, an in-test TCP forwarder whose pause holds every byte without closing anything (which only works because of the endpoint rewrite above); it asserts one Bad, a frozen read stamp, recovery on the SAME session, the identical value republished Good, a change published right after recovery still arriving, and no source-time regression once the SDK's late backlog flush has arrived. Its second test uses HardClose (reset every connection, refuse new ones) — the shape of a real process stop — and asserts the live-gate status sequence, that the SDK never reconnects on its own once the proxy accepts again, ConnectionLost exactly once, and the re-initialise bringing the identical value back Good. OpcUaClientDriverReconnectWireTests covers the server restart (incl. the BadServerHalted → ConnectionLost signal) and the subscription-lost case.

Further reading