ScadaBridge/docs/requirements/Component-DataConnectionLayer.md

Component: Data Connection Layer

Purpose

The Data Connection Layer provides a uniform interface for reading from and writing to physical machines at site clusters. It abstracts protocol-specific details behind a common interface, manages subscriptions, and delivers live tag value updates to Instance Actors. It is a clean data pipe — it performs no evaluation of triggers, alarm conditions, or business logic.

Location

Site clusters only. Central does not interact with machines directly.

Responsibilities

• Manage data connections defined centrally and deployed to sites as part of artifact deployment (OPC UA servers). Data connection definitions are stored in local SQLite after deployment.
• Establish and maintain connections to data sources based on deployed instance configurations.
• Subscribe to tag paths as requested by Instance Actors (based on attribute data source references in the flattened configuration).
• Deliver tag value updates to the requesting Instance Actors.
• Support writing values to machines (when Instance Actors forward SetAttribute write requests for data-connected attributes).
• Report data connection health status to the Health Monitoring component.

Common Interface

All protocol adapters implement the same interface:

IDataConnection : IAsyncDisposable
├── Connect(connectionDetails) → void
├── Disconnect() → void
├── Subscribe(tagPath, callback) → subscriptionId
├── Unsubscribe(subscriptionId) → void
├── Read(tagPath) → value
├── ReadBatch(tagPaths) → values
├── Write(tagPath, value) → void
├── WriteBatch(values) → void
├── WriteBatchAndWait(values, flagPath, flagValue, responsePath, responseValue, timeout) → bool
├── Status → ConnectionHealth
└── Disconnected → event Action?

The Disconnected event is raised by an adapter when it detects an unexpected connection loss (server offline, network failure, keep-alive timeout). The DataConnectionActor subscribes to this event to trigger the reconnection state machine. Additional protocols can be added by implementing this interface.

Batch Capability Seam

A protocol whose wire form can subscribe MANY tags in one round trip additionally implements the optional capability interface IBatchSubscribableConnection (same pattern as IBrowsableDataConnection / IAlarmSubscribableConnection):

IBatchSubscribableConnection
├── SubscribeBatchAsync(tagPaths, callback) → IReadOnlyList<TagSubscribeResult>
└── UnsubscribeBatchAsync(subscriptionIds) → void

• One shared callback, plain tag list. Every tag in a batch shares one callback; SubscriptionCallback already carries the tag path, so per-tag delegates would carry nothing.
• Partial-failure contract. A per-tag fault (bad node id, unresolvable path) is a result row with Success:false and never aborts the batch — mirroring the gateway's SubscribeResult and OPC UA's per-monitored-item create status. A thrown exception means the whole batch failed; the actor classifies it exactly as the per-tag path does and drives the Reconnecting state machine on a connection-level fault.
• Implementing the interface also asserts that ReadBatchAsync / WriteBatchAsync are true bulk service calls rather than loops over the single-tag methods. Both shipped adapters (OpcUaDataConnection, MxGatewayDataConnection) implement it; an adapter that does not keeps the historical per-tag behaviour, so the capability is purely additive.
• Single-tag SubscribeAsync / ReadAsync / WriteAsync are kept (heartbeat monitor, interactive read/write paths) and delegate to the batch form as a batch of one.

Sizing: at the 37,500-tag site target (docs/deployment/topology-guide.md), the pre-batch path issued one adapter round trip — and, on OPC UA, one ApplyChanges — per tag on every subscribe and every reconnect.

Common Value Type

All protocols produce the same value tuple consumed by Instance Actors. Before the first value update arrives from the DCL, data-sourced attributes are held at uncertain quality by the Instance Actor (see Site Runtime — Initialization):

Concept	ScadaBridge Design
Value container	TagValue(Value, Quality, Timestamp)
Quality	QualityCode enum: Good / Bad / Uncertain
Timestamp	DateTimeOffset (UTC)
Value type	object?

Supported Protocols

OPC UA

• Uses the OPC Foundation .NET Standard Library (OPCFoundation.NetStandard.Opc.Ua.Client).
• Session-based connection with endpoint discovery, certificate handling, and configurable security modes.
• Subscriptions via OPC UA Monitored Items with data change notifications (1000ms sampling, queue size 10, discard-oldest).
• Read/Write via OPC UA Read/Write services with StatusCode-based quality mapping.
• Disconnect detection via Session.KeepAlive event (see Disconnect Detection Pattern below).

MxGateway

• Connects to the MxAccess Gateway (AVEVA/Wonderware MXAccess-backed Galaxy) over gRPC using the ZB.MOM.WW.MxGateway.Client NuGet package (from the Gitea feed); ZB.MOM.WW.MxGateway.Contracts is pulled in transitively.
• Session-based: OpenSession + Register on connect; AddItem + Advise per subscription; value changes arrive on the gateway's server-streaming event feed (StreamEvents), resumable via worker_sequence.
• Read/Write via ReadBulk / WriteBulk; writes carry a configurable WriteUserId. Quality maps the OPC-style quality byte (≥192 Good, ≥64 Uncertain, else Bad), with a failing MXAccess status proxy treated as Bad.
• Galaxy hierarchy browse via the separate GalaxyRepositoryClient — objects are navigable nodes (keyed by Galaxy gobject id), attributes are selectable leaves (keyed by full tag reference). Browse is lazy and attribute-light: navigation uses BrowseChildren with include_attributes=false (child objects only), and an object's own attributes are fetched only when it is expanded, via DiscoverHierarchy(root=<object>, max_depth=0) scoped to that single object. This keeps each browse level's reply small; inlining every child's full attribute set could exceed the Akka remote frame and silently drop the reply.
• Disconnect detection: a fault on the event stream raises IDataConnection.Disconnected, driving the same reconnection state machine as OPC UA. A graceful (OK-status) end of the event stream while the adapter did not cancel it is ALSO a disconnect (fix 2026-08-20): the gateway completes a session's subscriber channels with no error when the session is closed from its dashboard or on a graceful gateway shutdown (CompleteAllSubscribers(error: null)), so before the fix RunEventLoopAsync simply returned, the adapter stayed Connected, subscribed values froze at their last GOOD quality and every write failed with Session … was not found until a process restart (QA incident, vq03 Z28061 MoveIn, one hour of silent loss; the 10-second NotFound heartbeat in the logs was the unresolved-tag resolution probe re-subscribing on the dead session). Same bug class as the gRPC site-stream client's graceful-completion reconnect. Pinned by MxGatewayDataConnectionReconnectTests.EventStreamGracefulCompletion_* (signals once while connected; silent for a superseded loop and after our own DisconnectAsync). Follow-up for the gateway SDK: NotFound maps to a generic MxGatewayException with only the detail text, so a write/subscribe hitting a dead session cannot be classified as a session loss without string matching — a typed session-not-found exception would allow a second, reactive detection layer.
• Implemented as MxGatewayDataConnection over an IMxGatewayClient seam; the seam is decoupled from the generated gRPC types (only RealMxGatewayClient references them), so the adapter is fully unit-testable with a fake.

Endpoint Redundancy

Data connections support an optional backup endpoint for automatic failover when the active endpoint becomes unreachable. Both endpoints use the same protocol.

Entity fields:

Field	Type	Notes
PrimaryConfiguration	string? (max 4000)	Required. Renamed from Configuration
BackupConfiguration	string? (max 4000)	Optional. Null = no backup
FailoverRetryCount	int (default 3)	Retries on active endpoint before switching

Failover state machine:

%%{init: {'theme':'base', 'themeVariables': {'textColor':'#111111','lineColor':'#555555','edgeLabelBackground':'#ffffff','fontSize':'15px'}}}%%
flowchart TD
    connected(["Connected"])
    pushbad["push bad quality"]
    retry["retry active endpoint<br/>(5s)"]
    decide{"N failures<br/>(≥ FailoverRetryCount)?"}
    switch["switch to other endpoint"]
    dispose["dispose adapter,<br/>create fresh adapter<br/>with other config"]
    reconnect["reconnect"]
    resub["ReSubscribeAll"]

    connected -->|disconnect| pushbad
    pushbad --> retry
    retry --> decide
    decide -->|"no (retry again)"| retry
    decide -->|yes| switch
    switch --> dispose
    dispose --> reconnect
    reconnect --> resub
    resub -->|back to Connected| connected

    classDef start fill:#d5e8d4,stroke:#82b366,color:#111111;
    classDef proc fill:#dae8fc,stroke:#6c8ebf,color:#111111;
    classDef dec fill:#fff2cc,stroke:#d6b656,color:#111111;
    classDef warn fill:#ffe6cc,stroke:#d79b00,color:#111111;
    classDef bad fill:#f8cecc,stroke:#b85450,color:#111111;
    class connected start
    class pushbad bad
    class retry,reconnect,resub proc
    class decide dec
    class switch,dispose warn

• Round-robin: primary → backup → primary → backup. No preferred endpoint after first failover — the connection stays on whichever endpoint is working.
• No auto-failback: The connection remains on the active endpoint until it fails.
• Single-endpoint connections (no backup): Retry indefinitely on the same endpoint, preserving existing behavior.
• Adapter lifecycle on failover: The actor disposes the current IDataConnection adapter and creates a fresh one via DataConnectionFactory.Create() with the other endpoint's configuration. Clean slate — no stale state.

Health reporting:

• DataConnectionHealthReport includes ActiveEndpoint: "Primary", "Backup", or "Primary (no backup)".

Site event log entries:

• DataConnectionFailover (Warning) — connection name, from-endpoint, to-endpoint, failure count.
• DataConnectionRestored (Info) — connection name, active endpoint.

See 2026-03-22-primary-backup-data-connections-design.md for the full design.

In-place reconfiguration (register row 97)

A ReconfigureConnectionCommand replaces a live connection's configuration without stopping the connection actor. It is the same adapter swap the failover path above performs — the two share one SwapAdapter helper — so the diagram applies verbatim from dispose adapter onwards:

1. The command lands on the actor and is never stashed, in any of the three states. That is load-bearing, not tidiness: a connection whose configured endpoint is wrong or dead spends its life looping in Connecting or Reconnecting, which is precisely when a corrected configuration is sent. Stashed, it would wait for the old configuration to succeed — i.e. forever.
2. Protocol, primary/backup details and failover retry count are replaced wholesale, and endpoint selection is re-armed from the top: whatever endpoint the previous configuration had failed over to, the new configuration's primary is the operator's intent, so ActiveEndpoint and both failure counters reset.
3. SwapAdapter unhooks and disposes the outgoing adapter (fire-and-forget, never awaited on the actor thread), builds the replacement via IDataConnectionFactory.Create — which is why a protocol change is supported too — and bumps _adapterGeneration last, so every callback and in-flight background task belonging to the outgoing adapter is recognised as stale and dropped rather than delivered as if it came from the new endpoint.
4. The actor re-enters the reconnect cycle immediately (no ReconnectInterval wait — there is nothing to wait for; StartSingleTimer replaces any reconnect already armed, so a reconfigure landing mid-backoff cannot leave two attempts racing). The unstable-disconnect accounting and the failover check are suppressed for this transition: the link was dropped on purpose, and counting it as an unstable cycle could trip a failover onto the new backup before the new primary had been tried once.
5. On connect, ReSubscribeAll / ReSubscribeAllAlarms re-issue everything from _subscriptionsByInstance and _alarmSourceSubscribers, which the swap never touched. Subscribers do nothing and know nothing — the tags go bad quality across the swap, exactly as they do on a failover, and come back Good.

Accepted consequences, all identical to a failover: in-flight writes against the outgoing adapter may fail (returned synchronously to the calling script, as always), and tag quality dips for the duration of the reconnect.

Idempotent. A reconfigure carrying the configuration already in force is a no-op. The sender is hash-guarded and does not send one, but a swap is destructive enough that a duplicate must never cost a reconnect.

Connection Configuration Reference

All settings are parsed from the data connection's configuration JSON dictionaries (PrimaryConfiguration and optional BackupConfiguration, stored as IDictionary<string, string> connection details). Both endpoints use the same protocol-specific keys. Invalid numeric values fall back to defaults silently.

OPC UA Settings

Key	Type	Default	Description
endpoint / EndpointUrl	string	opc.tcp://localhost:4840	OPC UA server endpoint URL
SessionTimeoutMs	int	60000	OPC UA session timeout in milliseconds
OperationTimeoutMs	int	15000	Transport operation timeout in milliseconds
PublishingIntervalMs	int	1000	Subscription publishing interval in milliseconds
KeepAliveCount	int	10	Keep-alive frames before session timeout
LifetimeCount	int	30	Subscription lifetime in publish intervals
MaxNotificationsPerPublish	int	100	Max notifications batched per publish cycle
SamplingIntervalMs	int	1000	Per-item server sampling rate in milliseconds
QueueSize	int	10	Per-item notification buffer size
MaxMonitoredItemsPerSubscription	int	5000	Monitored-item budget per OPC UA subscription. Above it the adapter shards onto an additional subscription on the same session (37,500 tags → 8 shards). Per endpoint, because item-count ceilings are a property of the target server.
SecurityMode	string	None	Preferred endpoint security: None, Sign, or SignAndEncrypt
AutoAcceptUntrustedCerts	bool	true	Accept untrusted server certificates

MxGateway Settings

Key	Type	Default	Description
Endpoint	string	http://localhost:5000	Gateway base URL
ApiKey	string	—	Sent to the gateway as authorization: Bearer <key>
ClientName	string	scadabridge (when blank)	MXAccess client registration name
WriteUserId	int	0	MXAccess user id applied to every write-back (0 = no user context)
UseTls	bool	false	Use TLS to a secured gateway
CaFile	string	—	Path to the CA certificate (TLS only)
ServerName	string	—	TLS server-name override
ReadTimeoutMs	int	5000	ReadBulk per-call timeout in milliseconds

Secret handling for ApiKey follows the same at-rest treatment and log/telemetry redaction as the OPC UA UserIdentity username/password fields.

Shared Settings (appsettings.json)

These are configured via DataConnectionOptions in appsettings.json, not per-connection:

Setting	Default	Description
ReconnectInterval	5s	Fixed interval between reconnection attempts
TagResolutionRetryInterval	10s	Floor interval for the unresolved-tag retry; the retry backs off from here
TagResolutionRetryMaxInterval	5m	Ceiling for the exponential tag-resolution backoff
WriteTimeout	30s	Timeout for write operations
SeedReadTimeout	30s	Per-chunk timeout for seed reads on the initial-subscribe (and reconnect re-seed) path. A hung device read is treated as a failed seed: retried up to SeedReadMaxAttempts, then the tag stays Uncertain until a change notification arrives.
SeedOverallTimeout	120s	Wall-clock deadline for the WHOLE seed (all chunks, all retry rounds). Replaces the old 30s-per-tag serial worst case; remaining tags are logged and stay Uncertain.
SeedReadBatchSize	250	Tags per seed-read chunk
SeedReadMaxParallelism	4	Seed-read chunks in flight concurrently
SubscribeBatchSize	500	Tags per adapter subscribe round trip (initial subscribe, reconnect re-subscribe, resolution probes)
SubscribeBatchDelay	50ms	Delay BETWEEN reconnect re-subscribe chunks. Not applied on the instance-driven subscribe path — the Deployment Manager already staggers instance startup there.
QualityFlushInterval	1s	Coalescing window for pushing tag-quality counters to the health collector
MxSupervisoryAdviseParallelism	16	Max in-flight supervisory advise commands on the MxGateway bulk-subscribe path when the endpoint has no write-user context

Subscription Management

• When an Instance Actor is created (as part of the Site Runtime actor hierarchy), it registers its data source references with the Data Connection Layer.
• The DCL subscribes to the tag paths using the concrete connection details from the flattened configuration.
• Tag value updates are delivered directly to the requesting Instance Actor.
• When an Instance Actor is stopped (due to disable, delete, or redeployment), the DCL cleans up the associated subscriptions.
• When a new Instance Actor is created for a redeployment, subscriptions are established fresh based on the new configuration.

Batching, chunking and pacing

Three subscribe paths, deliberately paced differently:

Path	Shape
Instance-driven subscribe (SubscribeTagsRequest, i.e. site failover / deploy)	One SubscribeBatchAsync per SubscribeBatchSize chunk with no added delay. The Deployment Manager already staggers instance startup (SiteRuntimeOptions.StartupBatchSize/StartupBatchDelayMs), so requests arrive pre-spaced at ~75 tags each — double-staggering would only slow failover.
Reconnect re-subscribe (ReSubscribeAll)	Sequential chunks of SubscribeBatchSize with SubscribeBatchDelay between them, all inside ONE background task; each chunk reports its per-tag results back to the actor as its own message. Replaces a tight loop that fired one fire-and-forget task per tag.
Tag-resolution probe	The whole not-in-flight unresolved set, chunked at SubscribeBatchSize, as one probe round per timer tick.

Seeding (the initial-value read that keeps a STATIC tag from staying Uncertain) issues chunked bulk reads — SeedReadBatchSize per chunk, SeedReadMaxParallelism chunks in flight, SeedReadTimeout per chunk — under a single SeedOverallTimeout deadline covering every chunk and retry round.

OPC UA subscription sharding. RealOpcUaClient places monitored items on the first subscription shard with free capacity (MaxMonitoredItemsPerSubscription), creating a shard when all are full and deleting one once it empties. A batch is N × AddItem plus one ApplyChanges per touched shard. Alarms & Conditions event items are pinned to a dedicated event shard, so ConditionRefresh has a single subscription id to target and QueueSize:1000 event items never consume data-shard capacity. Every structural mutation (AddItem / RemoveItem / ApplyChanges / Create / Delete) and every shard-list mutation serializes behind ONE per-client gate — the SDK Subscription is not safe under concurrent structural mutation, and the subscribe task, resolution probe and alarm subscribe are otherwise unordered.

MxGateway. A batch is one SubscribeBulk gateway command (AddItem + Advise per tag in a single worker pass), replacing the historical 2 RPCs per tag. When the endpoint has no write-user context (WriteUserId == 0) the worker offers no BULK supervisory advise, so the client issues one AddItemBulk and then pipelines the per-item supervisory advises with a bounded window (MxSupervisoryAdviseParallelism). A cross-repo follow-up (an additive supervisory flag on SubscribeBulkCommand in mxaccessgw) would collapse that to one RPC as well.

Write-Back Support

• When a script calls Instance.SetAttribute for an attribute with a data source reference, the Instance Actor sends a write request to the DCL.
• The DCL writes the value to the physical device via the appropriate protocol.
• The existing subscription picks up the confirmed new value from the device and delivers it back to the Instance Actor as a standard value update.
• The Instance Actor's in-memory value is not updated until the device confirms the write.

Browsing the address space

DCL is a clean data pipe on the hot path. Browse is an opt-in capability for protocols that support it, exposed via IBrowsableDataConnection. Only consumed by management/UI (the tag picker on the instance configure page); Instance Actors never call it. The browse path is protocol-agnostic: the same command/service/dialog serve every browsable protocol.

• OpcUaDataConnection and MxGatewayDataConnection both implement IBrowsableDataConnection; other/custom protocols do not (and return a NotBrowsable failure).
• DataConnectionManagerActor handles BrowseNodeCommand (fields: ConnectionName, ParentNodeId, and an optional opaque ContinuationToken for paging) and replies with BrowseNodeResult (children + Truncated + an optional continuation token + structured BrowseFailure?). The Central UI facade is IBrowseService/BrowseService, backing the NodeBrowserDialog tag picker.
• Browse type-info: each child BrowseNode carries optional best-effort type metadata — DataType (friendly name), ValueRank (scalar/array), and Writable. DataType is populated by both OPC UA and MxGateway: RealOpcUaClient batch-reads built-in data-type node ids and maps them to friendly names for Variable nodes; MxGatewayDataConnection surfaces each attribute leaf's GalaxyAttribute.data_type_name (free-form Galaxy type text) as DataType. ValueRank and Writable are OPC-UA-only; MxGateway leaves them unset.
• Node ids are opaque protocol-specific strings: OPC UA uses NodeIds; MxGateway uses Galaxy gobject ids for navigable objects and full tag references for selectable attribute leaves.
• OPC UA namespace binding (Gitea #14): a NodeId's ns=<index> is meaningful only against one server's namespace table, so a server that adds/removes/reorders a namespace silently re-points every stored binding — best case BadNodeIdUnknown, worst case the index names a different namespace holding a colliding identifier and the binding resolves to the wrong node with Good quality. The namespace URI is the stable identity. OpcUaNodeReference is therefore the single translation seam between stored references and the wire:
  • Resolve(reference, session.NamespaceUris) accepts both ns=<index>;s=<id> (existing bindings, unchanged) and the durable nsu=<namespace-uri>;s=<id>, mapping the URI to the live index at use time. It is used at every parse site — subscribe, read, write, alarm-subscribe, browse. A URI absent from the server's NamespaceArray throws with a message naming the URI, rather than binding to whatever sits at that index; a svr= reference to another server is rejected outright.
  • ToDurable(expandedNodeId, session.NamespaceUris) is what the browser emits, so what the picker shows is what gets stored and newly-authored bindings are index-proof from the start. Namespace 0 is exempt (spec-fixed) and keeps its short form. It also closes a round-trip gap: the browse previously emitted ExpandedNodeId.ToString(), which for a URI- or server-index-carrying reference produced a string NodeId.Parse could not read back.
  • Bindings stored before this change keep their ns= form and keep working; they are only as durable as the server's namespace order. Re-authoring against a picker (which now emits nsu=) is what makes them durable — see docs/plans/ and Gitea #14 for the OtOpcUa v3.0 cutover, where v2's sole custom namespace and v3's raw namespace both sit at index 2, so v2-era bindings resolve against v3 without error while meaning something else.
  • OtOpcUa v3.0 dual-namespace cutover: v3.0 splits OtOpcUa's address space into two namespaces, raw (https://zb.com/otopcua/raw) and uns (https://zb.com/otopcua/uns), where v2 exposed one. ScadaBridge is already namespace-URI-durable against this split: OpcUaNodeReference resolves a stored nsu=<uri>;s=... binding against the live session NamespaceArray at use time, so it is index-proof regardless of which namespace a given tag lands in or how many namespaces the server publishes. The node-browser picker already nudges operators toward the durable form, since ToDurable is what it emits for newly-authored bindings. See docs/plans/2026-07-23-otopcua-v3-dual-namespace-cutover-scope.md for the full cutover scope.
• Browse runs against the live session; no caching at DCL.
• Frame-size guard: the reply crosses the site→central Akka frame (default 128 KB) on a temp Ask actor; an oversized reply is silently discarded by remoting, hanging the picker. The child handler caps each BrowseNodeResult to a byte budget (~100 KB) before replying, OR-ing the adapter's own truncation signal into Truncated. This is protocol-agnostic (every adapter's reply funnels through it). Per-protocol upstream caps narrow the window first: OPC UA requests at most 500 references per node (continuation point → Truncated); MxGateway relies on the gateway's BrowseChildren page cap.
• BrowseNext paging (OPC UA): a Truncated level no longer forces manual node-id entry. When the OPC UA browse is truncated, the adapter returns the session's continuation point as the reply's opaque ContinuationToken; a follow-up BrowseNodeCommand carrying that token calls Session.BrowseNext to fetch the next page (the picker exposes this as "Load more"). Continuation points are session-bound and can expire — BadContinuationPointInvalid is caught and the browse restarts from a fresh first page rather than failing. Manual node-id entry remains as a fallback when the site or its session is offline.

Address-space search

A second opt-in capability seam, IAddressSpaceSearchable (in Commons, mirroring the IBrowsableDataConnection / IAlarmSubscribableConnection pattern; implemented by the OPC UA adapter, consumed by management/UI only):

IAddressSpaceSearchable
└── SearchAddressSpaceAsync(query, maxDepth, maxResults, ct) → matches

• DataConnectionManagerActor handles SearchAddressSpaceCommand (ConnectionName, Query, MaxDepth, MaxResults); the OPC UA adapter does a bounded recursive browse (depth + result caps) from the Objects folder, matching a case-insensitive substring against each node's DisplayName and root-relative path, and returns matches as AddressSpaceMatch (the BrowseNode plus its full path). When a cap is hit the result flags it so the UI can prompt "showing first N — refine".
• The Central UI facade is BrowseService.SearchAsync (Design role), surfaced as the search box in NodeBrowserDialog.
• StubOpcUaClient carries a canned browse/search implementation so the picker and its bUnit/unit tests run without a live OPC UA server.

Endpoint verification (OPC UA)

Before saving or deploying an OPC UA data connection, the Central UI can ask the target site to probe the configured endpoint — this exercises connectivity (and TLS trust) against the live config, including edited-but-unsaved values.

• DataConnectionManagerActor handles VerifyEndpointCommand (SiteId, protocol, config JSON). The site spins up a temporary RealOpcUaClient from the submitted config, attempts discovery + a session with a short timeout (a few seconds), then disconnects and disposes the client. The reply is a VerifyEndpointResult (Success, a typed FailureKind, an error message, and an optional captured server cert).
• The probe forces AutoAcceptUntrustedCerts = false and hooks the certificate-validation callback so it can capture an untrusted server certificate (Subject / Issuer / Thumbprint / NotBefore / NotAfter / DER) for the UI to display. Critically, the probe never trusts the cert — the validation callback always rejects (Accept = false); trusting is a separate, explicit, Admin-gated action that writes the cert into the site PKI store (see the cert-trust path in Component-SiteRuntime.md). The probe is read-only with respect to the trust boundary.

Native Alarm Mirroring

Some data sources publish their own alarms — OPC UA Alarms & Conditions servers and the MxAccess Gateway. The DCL can mirror these native alarms into the Site Runtime as a read-only feed: ScadaBridge reflects source alarm state but never acknowledges, confirms, shelves, or otherwise writes back to the source. This complements (does not replace) ScadaBridge's own computed alarms; it feeds the Site Runtime's NativeAlarmActor peer subsystem.

Like browse, this is an opt-in capability for protocols that support it. It does not touch the hot value path — alarm transitions flow over a separate per-connection feed.

Capability Seam

Mirroring is exposed via the optional IAlarmSubscribableConnection capability interface (in Commons), which an IDataConnection implementation may also implement (mirroring the IBrowsableDataConnection pattern; consumed by the DataConnectionActor only):

IAlarmSubscribableConnection
├── SubscribeAlarmsAsync(sourceReference, conditionFilter?, callback, ct) → subscriptionId
└── UnsubscribeAlarmsAsync(subscriptionId, ct) → void

The AlarmTransitionCallback delivers a protocol-neutral NativeAlarmTransition per transition. On every (re)subscribe the adapter replays a snapshot of currently-active conditions (Snapshot… records terminated by a SnapshotComplete sentinel) so consumers can reconcile state after a reconnect.

Protocol Adapters

• OPC UA (OpcUaDataConnection + RealOpcUaClient): one event MonitoredItem per source binding (AttributeId = EventNotifier), sharing an event subscription (the empty binding selects the Server object), with an EventFilter selecting EventType / SourceNode / Severity plus the ConditionType / AcknowledgeableConditionType / AlarmConditionType state fields. ConditionRefresh is invoked on subscribe to replay active conditions as the snapshot. The OPC UA field → NativeAlarmTransition mapping is isolated in the pure helper OpcUaAlarmMapper, unit-testable without a live server.
• MxGateway (MxGatewayDataConnection + RealMxGatewayClient): mirrors over the gateway package's StreamAlarmsAsync — a resumable background stream whose reconnect re-sends a snapshot. The field mapping lives in MxGatewayAlarmMapper.

Other/custom protocols do not implement the capability; a subscribe request against such a connection is replied to with a failure (SubscribeAlarmsResponse.Success = false).

Connection Actor Behavior

The DataConnectionActor opens one alarm feed per source (not per subscriber) and routes incoming transitions to instance subscribers by source-object reference — a prefix match of the transition's SourceObjectReference (falling back to SourceReference) against each subscriber's registered SourceReference. Subscribers (the Site Runtime's NativeAlarmActor instances) are ref-counted per source, so the underlying feed is opened once and torn down only when the last subscriber for that source unsubscribes.

The registered SourceReference is a NodeId for OPC UA (the picker, CSV and config all store NodeIds), so the transition's routing identity must live in the same space or nothing matches. Gitea #17: the OPC UA adapter previously set SourceObjectReference from the event's SourceName (a plain name — the RawPath for OtOpcUa), which never prefix-matches a NodeId binding, so every native OPC UA alarm transition was silently dropped. Because each OPC UA feed is opened for exactly one binding, the adapter now tags every transition on that feed with the binding string verbatim (via the pure OpcUaAlarmMapper.BuildIdentity), making the routing key an exact match independent of whether the binding is stored as ns=<index> or the durable nsu=<uri> form. The Server-object aggregate feed (empty binding) keeps an empty routing identity, so it reaches only "mirror everything" subscribers and never leaks into a specific-node binding. The per-condition SourceReference key stays the human-readable SourceName.ConditionName, so persistence and display are unchanged. MxGateway is unaffected — its bindings are object names and its mapper already emits matching names.

• State gating: SubscribeAlarmsRequest is handled only in the Connected state; requests arriving while Connecting/Reconnecting are coalesced to the latest request per subscriber actor/source and processed on entering Connected. Unsubscribes and subscriber termination are handled in every state.
• Capability check: if _adapter is not IAlarmSubscribableConnection, the actor replies SubscribeAlarmsResponse(Success = false, ...).
• Reconnect handling: on entering Reconnecting, the actor pushes a NativeAlarmSourceUnavailable to every alarm subscriber (consumers mark mirrored alarms uncertain rather than clearing them). On successful reconnection it re-subscribes the feed; the adapter re-emits a snapshot, reconciling state.
• Shared condition filter (last-subscriber-wins): because the feed is opened once per source, it carries a single condition filter. A second subscriber that registers a different filter for the same source overwrites it (last writer wins) and the actor logs a warning — co-subscribers to one source are expected to agree on the filter.
• Routing index: subscribed sources are bucketed by the first path segment of their reference (everything before the first .), and a transition is matched only against the bucket for its own first segment, plus a residue list of sources that carry no separator at all (a prefix shorter than one segment, which can match other buckets). This is sound because a source reference containing a separator can only prefix a reference sharing its entire first segment. The per-source StartsWith test and the condition-type gate inside the bucket are unchanged, so routing decisions are identical to the previous linear scan over every subscribed source; snapshot boundaries bypass the condition filter, but OPC UA boundaries are sent only to the exact source binding. Connection-wide Mx boundaries broadcast to all subscribers while preserving their refresh epoch.
• Gateway union filter (MxGateway): StreamAlarms carries a single alarm_filter_prefix, so the adapter opens the stream on the longest common prefix of the currently subscribed source references and restarts it for each new source, including one covered by the existing prefix, so every newly bound source receives standing state. Each transport refresh has connection-wide start/end markers and a distinct epoch. An unsubscribe never restarts the stream — the prefix it leaves behind is at worst too broad, which may increase bandwidth. Correctness also depends on the epoch-scoped snapshot cache merging concurrent live updates and rejecting stale snapshot ends (DC-5); a broad filter alone provides no such guarantee. The prefix is a bandwidth optimisation only; the actor's per-source + condition-type gate remains authoritative, mirroring the OPC UA server-side WhereClause stance.

Protocol-Neutral Types & Messages

All defined in Commons so the feed is identical across protocols:

Type	Shape
NativeAlarmTransition	SourceReference, SourceObjectReference, AlarmTypeName, Kind, Condition, Category, Description, Message, OperatorUser, OperatorComment, OriginalRaiseTime?, TransitionTime, CurrentValue, LimitValue, AckTime?
AlarmConditionState	Active, Acknowledged, Confirmed? (null when not confirmable), Shelve, Suppressed, Severity (0–1000)
AlarmTransitionKind (enum)	Snapshot, SnapshotComplete, Raise, Acknowledge, Clear, Retrigger, StateChange, SnapshotStart (appended)

OperatorUser / OperatorComment and CurrentValue / LimitValue are display-only mirrors from the source.

Ack Timestamp (AckTime)

AckTime is the instant a condition was acknowledged, mirrored end-to-end (transition → AlarmStateChanged → AlarmStateUpdate field 24 → site native_alarm_state) so MES can report a real AckDT and the Alarms.CurrentAsync() script accessor can surface it. Added additively as a trailing optional parameter — every pre-existing positional construction still compiles and yields null.

One rule decides whether it is set: the condition must be BOTH active and acknowledged. That single predicate gives all three required behaviours — null while unacknowledged, cleared on re-raise (a re-raise transition arrives unacknowledged), and no phantom ack on a return-to-normal (which matters for MxGateway, where INACTIVE maps to Acknowledged = true).

Provenance differs by protocol, and the difference is deliberate — the value is never fabricated:

Source	Ack instant used	Accuracy
OPC UA A&C	AcknowledgeableConditionType/AckedState/TransitionTime — SelectClause index 18, appended after the limit fields so indices 0–17 keep their meaning	The source's own ack instant
OPC UA A&C, server omits the field	The event's Time field	When the DCL observed the acknowledged state
MxAccess Gateway	The transition's own timestamp (TransitionTimestamp, or the DCL's receipt time when the gateway omits it); on a re-subscribe snapshot, LastTransitionTimestamp for an ACTIVE_ACKED entry	When the system saw the ack — the gateway feed carries no dedicated ack timestamp

The decision lives in the pure mappers (OpcUaAlarmMapper.DeriveAckTime, MxGatewayAlarmMapper.DeriveAckTime), so it is unit-tested without a live server or gateway.

Design record: docs/plans/2026-06-30-mes-alarm-status-api.md §6.4.

Messages:

• SubscribeAlarmsRequest / SubscribeAlarmsResponse — instance (via the DCL manager) subscribes a source binding to native alarms; the response carries success + an optional error message.
• UnsubscribeAlarmsRequest — cancels a native alarm subscription for an instance + source.
• NativeAlarmTransitionUpdate(ConnectionName, Transition) — DCL → instance: one routed transition (including snapshot replay).
• NativeAlarmSourceUnavailable(ConnectionName, SourceReference, Timestamp) — DCL → instance: the feed for a source became unavailable (connection lost).

Value Update Message Format

Each value update delivered to an Instance Actor includes:

• Tag path: The relative path of the attribute's data source reference.
• Value: The new value from the device.
• Quality: Data quality indicator (good, bad, uncertain).
• Timestamp: When the value was read from the device.

Connection Actor Model

Where connection actors come from — the Site Runtime is the only creator (register row 85): the DCL manager creates and removes connection actors only when the Deployment Manager tells it to, via CreateConnectionCommand / RemoveConnectionCommand. It never reads storage, and the DCL has no independent notion of "the connections this site ought to have" — the site's data_connection_definitions table is read by the Deployment Manager, never here (see the arbitration note below; it had no reader at all until register rows 89/90). A connection actor's config therefore comes from the Connections map inside a deployed instance's flattened config (pushed as part of starting that instance) or from the system-wide artifact-deploy path. HandleCreateConnection is idempotent on the name — it warns and returns if it already holds that connection — but the Deployment Manager is the side that keeps the traffic quiet, hash-guarding on the last config pushed per name so an unchanged connection sends nothing. Row 85 was the consequence of that ownership split being incompletely honoured on the Site Runtime side: two of the four Instance-Actor start paths omitted the connection push, so instances came up whose every tag subscribe was answered Unknown connection: <name> here, forever. The fix is in the Site Runtime (the ensure moved into its single creation funnel) — not here; giving the DCL manager its own config source would create a second, divergent authority with no arbiter when the two disagree. See Component-SiteRuntime.md and docs/plans/2026-08-15-instance-sweep-self-heal-design.md §7.

The two residual gaps of that family are now closed — but not where the register said they were (rows 89 + 90, 2026-08-17): the peer node's DCL was never "serving the old endpoint". Only the node running the site-{SiteId} Deployment Manager singleton ever receives a CreateConnectionCommand, so the standby's manager is empty. The divergence surfaces when the singleton's next incarnation (failover, or an ordinary restart — no replication required) rebuilds every connection from the Connections snapshots frozen into the instances' flattened configs, because the per-name hash cache that recorded the artifact deploy's change is in-memory and dies with the incarnation. The fix lives entirely in the Site Runtime, as an arbitration rule — the definitions store wins wherever it has an opinion, resolved before the hash is computed so the two sources cannot fight over the same per-name slot — plus in-memory tombstones so a centrally removed connection is not resurrected by a flattened config. Nothing changes here. See Component-SiteRuntime.md → DCL connection arbitration and docs/plans/2026-08-17-dcl-connection-arbitration-decision.md.

A live connection IS reconfigured in place, via a third command (register row 97, 2026-08-17). HandleCreateConnection remains create-only — it warns and returns for a name it already holds — so the Deployment Manager's documented "a changed config re-issues the create so the DCL adopts it" was never true, on any node: a corrected endpoint/credential/backup/failover-count took effect only when the connection was next built from scratch (node restart, singleton failover, remove-then-re-add), while the DM's hash cache recorded it as applied. The Site Runtime now sends a ReconfigureConnectionCommand on the change edge instead. The manager forwards it to the connection actor that owns the name, and falls back to a create for a name it does not hold — so the command is safe whatever the manager's state and a sender never has to know whether the actor exists.

Stop-and-recreate was rejected, and the reason is worth keeping. Subscriptions are held inside DataConnectionActor, and InstanceActor cancels its per-connection subscribe-retry timer on the first success — so a rebuilt connection actor would have no subscribers and nothing would ever re-subscribe. That trades "stale endpoint, data flowing" for "correct endpoint, no data until redeploy". See In-place reconfiguration below for what the actor does instead.

Each data connection is managed by a dedicated connection actor that uses the Akka.NET Become pattern to model its lifecycle as a state machine:

• Connecting: The actor attempts to establish the connection. It retains only the latest tag-subscribe request per instance and alarm-subscribe request per subscriber/source. Single and batch device writes receive an immediate failed response; they are never retained.
• Connected: The actor actively services subscriptions and processes the retained latest subscription requests.
• Reconnecting: The connection was lost. Existing subscribers receive bad quality, the latest subscription requests are coalesced while the actor retries, and single/batch writes fail immediately. A caller timeout never leaves a queued setpoint to replay on recovery.

Instance Actors and NativeAlarmActors send explicit unsubscribe requests from PostStop, with their own actor references as sender. DCL also watches every subscriber. A stopped actor releases only its own registrations; a delayed unsubscribe or termination from an old actor cannot remove the replacement actor with the same instance name. A narrower replacement drops tags it no longer requests. Completion of outstanding device subscribe/resolution I/O adopts handles only for tags still requested and releases orphan handles. Existing identical Mx handles are never released as redundant, and adapter reverse-map removals compare the handle atomically before removing a tag mapping.

OPC UA refreshes carry an exact SnapshotSourceReference (including the empty Server binding) and a SnapshotId shared by SnapshotStart, entries and SnapshotComplete. Each NativeAlarmActor replaces its buffer on a new start and commits only a matching end; duplicate or superseded ends cannot clear a committed snapshot. The adapter tags each monitored item's refresh separately even though ConditionRefresh targets the shared event subscription. Mx snapshots retain their connection-wide framing. An interrupted refresh buffer is discarded on disconnect.

Subscription demand survives connection transitions through coalescing and owner tracking. Device writes during those transitions fail immediately so stale setpoints cannot be replayed.

OPC UA-specific notes: The RealOpcUaClient uses the OPC Foundation SDK's Session.KeepAlive event for proactive disconnect detection. The SDK sends keep-alive requests at the subscription's KeepAliveCount × PublishingInterval (default: 10s). When keep-alive fails, the ConnectionLost event fires, triggering the same reconnection flow. On reconnection, the DCL re-creates the OPC UA session and subscription, then re-adds all monitored items.

Connection Lifecycle & Reconnection

The DCL manages connection lifecycle automatically:

1. Connection drop detection: When a connection to a data source is lost, the DCL immediately pushes a value update with quality bad for every tag subscribed on that connection. Instance Actors see the staleness immediately and publish it to the site stream / debug view; script and alarm actors are deliberately NOT re-notified on a quality-only change (no value changed — re-evaluating triggers would cause spurious firings). Scripts observe quality on their next attribute read; a computed alarm holds its last-known state until fresh values arrive.
2. Auto-reconnect with fixed interval: The DCL retries the connection at a configurable fixed interval (e.g., every 5 seconds). The retry interval is a shared setting for all data connections (ReconnectInterval in the Shared Settings table). This is consistent with the fixed-interval retry philosophy used throughout the system. Individual gRPC/OPC UA operations (reads, writes) fail immediately to the caller on error; there is no operation-level retry within the adapter.
3. Connection state transitions: The DCL tracks each connection's state as connected, disconnected, or reconnecting. All transitions are logged to Site Event Logging.
4. Transparent re-subscribe: On successful reconnection, the DCL automatically re-establishes all previously active subscriptions for that connection. Instance Actors require no action — they simply see quality return to good as fresh values arrive from restored subscriptions.
5. Clean client replacement on (re)connect: On every (re)connect the adapter detaches and disposes the previous protocol client (session, keep-alive timers, event loop) before creating a new one — a stale client can neither leak nor signal a disconnect against the new session.

Disconnect Detection Pattern

Each adapter implements the IDataConnection.Disconnected event to proactively signal connection loss to the DataConnectionActor. Detection uses two complementary paths:

Proactive detection (server goes offline between operations):

• OPC UA: The OPC Foundation SDK fires Session.KeepAlive events at regular intervals. RealOpcUaClient hooks this event; when ServiceResult.IsBad(e.Status) (server unreachable, keep-alive timeout), it fires ConnectionLost. The OpcUaDataConnection adapter translates this into IDataConnection.Disconnected.

Reactive detection (failure discovered during an operation):

• Both adapters wrap ReadAsync (and by extension ReadBatchAsync) with exception handling. If a read throws a non-cancellation exception, the adapter calls RaiseDisconnected() and re-throws. The DataConnectionActor's existing error handling catches the exception while the disconnect event triggers the reconnection state machine.

Event marshalling: The DataConnectionActor subscribes to _adapter.Disconnected in PreStart(). Since Disconnected may fire from a background thread (gRPC stream task, OPC UA keep-alive timer), the handler sends an AdapterDisconnected message to Self, marshalling the notification onto the actor's message loop. This triggers BecomeReconnecting() → bad quality push → retry timer.

Once-only guard: OpcUaDataConnection uses an int _disconnectFired guard with Interlocked.Exchange(ref _disconnectFired, 1) to ensure RaiseDisconnected() fires exactly once per connection session. The guard resets atomically with Interlocked.Exchange(ref _disconnectFired, 0) on successful reconnection (ConnectAsync).

Write Failure Handling

Writes to physical devices are synchronous from the script's perspective:

• If the write fails (connection down, device rejection, timeout), the error is returned to the calling script. Script authors can catch and handle write errors (log, notify, retry, etc.).
• Write failures are also logged to Site Event Logging.
• There is no store-and-forward for device writes — these are real-time control operations. Buffering stale setpoints for later application would be dangerous in an industrial context.

Tag Path Resolution

When the DCL subscribes to a tag path from the flattened configuration but the path does not exist on the physical device (e.g., typo in the template, device firmware changed, device still booting):

1. The failure is logged to Site Event Logging.
2. The attribute is marked with quality bad.
3. The DCL retries resolution with exponential backoff, accommodating devices that come online in stages or load modules after startup: the interval starts at TagResolutionRetryInterval (10s), doubles after a round in which nothing resolved, and is capped at TagResolutionRetryMaxInterval (5m). It resets to the floor as soon as any tag resolves or the connection reconnects, so a device that is merely slow to boot is still picked up quickly while a dead one is not probed at full width forever. Each round probes the whole unresolved set in SubscribeBatchSize chunks, not one call per tag.
4. The retry timer is single-shot, rescheduled on probe completion — a periodic timer cannot back off, and rescheduling only on completion is what keeps a fan-out of failures from resetting the clock (the anti-starvation property previously defended by an IsTimerActive gate, which is retained).
5. On successful resolution, the subscription activates normally and quality reflects the live value from the device.

Note: Pre-deployment validation at central does not verify that tag paths resolve to real tags on physical devices — that is a runtime concern handled here.

Health Reporting

The DCL reports the following metrics to the Health Monitoring component via the existing periodic heartbeat:

• Connection status: connected, disconnected, or reconnecting per data connection.

• Tag resolution counts: Per connection, the number of total subscribed tags vs. successfully resolved tags. This gives operators visibility into misconfigured templates without needing to open the debug view for individual instances. Both numbers are derived from the connection actor's authoritative per-tag state, never accumulated in counters: the total is the number of distinct tag paths at least one instance currently subscribes to (the per-tag counted set that also drives value fan-out and the last-subscriber release decision), and the resolved count is the number of tags for which the adapter currently holds a subscription handle. A tag counts toward the total from the moment an instance registers it, whatever its resolution outcome — resolved, awaiting a resolution retry, or failed at connection level — so resolved can never exceed total. Deriving rather than accumulating is deliberate: increment/decrement counters drifted whenever an unsubscribe landed inside a reconnect window (the reconnect clears the per-tag maps the decrements keyed off), leaking a phantom tag into the reported total on every churn cycle.

  The pair is pushed to the health collector, not pulled from it, so it is a cached value that is only as fresh as its last push — and the collector holds it until the next one. Every site that mutates either underlying set must therefore push: entering the Connected state (a legitimate 0/0 — no instance has subscribed yet), the first-subscribe apply, the batch apply (reconnect re-subscribe and resolution probe), and unsubscribe. Missing the first-subscribe push is what made a perfectly healthy connection report 0/0 indefinitely — the connect-time 0/0 was never superseded, because a connection whose tags all resolve on the first attempt never reaches the batch path and has its resolution-retry timer cancelled outright for having nothing unresolved. That is the reverse of the usual staleness risk, and it blinded the diagnostic exactly when it should have read clean (deferred-work register row 55; the actor's own GetHealthReport reply, which really is derived on demand, was correct throughout).

• Tag quality counters are pushed on a genuine quality transition only, coalesced onto a QualityFlushInterval (1s) single-shot timer. Counter arithmetic still runs per message; only the collector push is deferred, and a value whose quality is unchanged moves no counter at all. Health reports poll at 30s, so the coalescing loses nothing. Three paths flush synchronously because their correctness depends on it: the bad-quality push on disconnect, unsubscribe, and the reconnect counter reset.

Dependencies

• Site Runtime (Instance Actors): Receives subscription registrations and delivers value updates. Receives write requests.
• Site Runtime (NativeAlarmActor): For alarm-subscribable connections, receives SubscribeAlarmsRequest/UnsubscribeAlarmsRequest and delivers NativeAlarmTransitionUpdate / NativeAlarmSourceUnavailable (read-only native alarm mirroring).
• Health Monitoring: Reports connection status.
• Site Event Logging: Logs connection status changes.

Interactions

• Site Runtime (Deployment Manager): Receives CreateConnectionCommand / ReconfigureConnectionCommand / RemoveConnectionCommand. It is the sole creator, reconfigurer and remover of connection actors; the DCL never reads storage.
• Site Runtime (Instance Actors): Bidirectional — delivers value updates, receives subscription registrations and write-back commands.
• Site Runtime (NativeAlarmActor): Bidirectional — receives alarm subscribe/unsubscribe requests, delivers native alarm transitions and source-unavailable notifications (read-only; no ack-back to the source).
• Health Monitoring: Reports connection health periodically.
• Site Event Logging: Logs connection/disconnection events.

Medium architecture review connection guarantees (2026-09-09)

OPC UA connection and endpoint verification select exactly the configured security mode. If a protected mode is unavailable, or discovery fails while protection was requested, the operation fails with the requested mode in its diagnostic; it never falls back to an unsecured session.

DCL retains each subscribed alarm source's current standing state and its in-progress snapshot. A new subscriber sharing a feed receives a scoped replay, or the already-received prefix of an in-progress frame followed by the remaining frame. Cache state is discarded on reconnect and last-owner unsubscribe. A new Mx source reopens the shared stream even when the existing prefix covers it, obtaining a fresh standing-state snapshot; callbacks from the canceled stream are ignored. Mx refresh boundaries remain connection-wide but share a distinct per-stream epoch. Both replay cache and runtime mirror ignore stale-epoch rows and apply newer live raises/clears over a concurrent refresh before committing it.