# Component: Data Connection Layer

## Purpose

The Data Connection Layer provides a uniform interface for reading from and writing to physical machines at site clusters. It abstracts protocol-specific details behind a common interface, manages subscriptions, and delivers live tag value updates to Instance Actors. It is a **clean data pipe** — it performs no evaluation of triggers, alarm conditions, or business logic.

## Location

Site clusters only. Central does not interact with machines directly.

## Responsibilities

- Manage data connections defined centrally and deployed to sites as part of artifact deployment (OPC UA servers). Data connection definitions are stored in local SQLite after deployment.
- Establish and maintain connections to data sources based on deployed instance configurations.
- Subscribe to tag paths as requested by Instance Actors (based on attribute data source references in the flattened configuration).
- Deliver tag value updates to the requesting Instance Actors.
- Support writing values to machines (when Instance Actors forward `SetAttribute` write requests for data-connected attributes).
- Report data connection health status to the Health Monitoring component.

## Common Interface

All protocol adapters implement the same interface:

```
IDataConnection : IAsyncDisposable
├── Connect(connectionDetails) → void
├── Disconnect() → void
├── Subscribe(tagPath, callback) → subscriptionId
├── Unsubscribe(subscriptionId) → void
├── Read(tagPath) → value
├── ReadBatch(tagPaths) → values
├── Write(tagPath, value) → void
├── WriteBatch(values) → void
├── WriteBatchAndWait(values, flagPath, flagValue, responsePath, responseValue, timeout) → bool
├── Status → ConnectionHealth
└── Disconnected → event Action?
```

The `Disconnected` event is raised by an adapter when it detects an unexpected connection loss (server offline, network failure, keep-alive timeout). The `DataConnectionActor` subscribes to this event to trigger the reconnection state machine. Additional protocols can be added by implementing this interface.

### Batch Capability Seam

A protocol whose wire form can subscribe MANY tags in one round trip additionally implements the optional capability interface `IBatchSubscribableConnection` (same pattern as `IBrowsableDataConnection` / `IAlarmSubscribableConnection`):

```
IBatchSubscribableConnection
├── SubscribeBatchAsync(tagPaths, callback) → IReadOnlyList<TagSubscribeResult>
└── UnsubscribeBatchAsync(subscriptionIds) → void
```

- **One shared callback, plain tag list.** Every tag in a batch shares one callback; `SubscriptionCallback` already carries the tag path, so per-tag delegates would carry nothing.
- **Partial-failure contract.** A per-tag fault (bad node id, unresolvable path) is a result row with `Success:false` and never aborts the batch — mirroring the gateway's `SubscribeResult` and OPC UA's per-monitored-item create status. A **thrown** exception means the whole batch failed; the actor classifies it exactly as the per-tag path does and drives the Reconnecting state machine on a connection-level fault.
- Implementing the interface also asserts that `ReadBatchAsync` / `WriteBatchAsync` are **true bulk** service calls rather than loops over the single-tag methods. Both shipped adapters (`OpcUaDataConnection`, `MxGatewayDataConnection`) implement it; an adapter that does not keeps the historical per-tag behaviour, so the capability is purely additive.
- Single-tag `SubscribeAsync` / `ReadAsync` / `WriteAsync` are **kept** (heartbeat monitor, interactive read/write paths) and delegate to the batch form as a batch of one.

Sizing: at the 37,500-tag site target (`docs/deployment/topology-guide.md`), the pre-batch path issued one adapter round trip — and, on OPC UA, one `ApplyChanges` — per tag on every subscribe and every reconnect.

### Common Value Type

All protocols produce the same value tuple consumed by Instance Actors. Before the first value update arrives from the DCL, data-sourced attributes are held at **uncertain** quality by the Instance Actor (see Site Runtime — Initialization):

| Concept | ScadaBridge Design |
|---|---|
| Value container | `TagValue(Value, Quality, Timestamp)` |
| Quality | `QualityCode` enum: Good / Bad / Uncertain |
| Timestamp | `DateTimeOffset` (UTC) |
| Value type | `object?` |

## Supported Protocols

### OPC UA
- Uses the **OPC Foundation .NET Standard Library** (`OPCFoundation.NetStandard.Opc.Ua.Client`).
- Session-based connection with endpoint discovery, certificate handling, and configurable security modes.
- Subscriptions via OPC UA Monitored Items with data change notifications (1000ms sampling, queue size 10, discard-oldest).
- Read/Write via OPC UA Read/Write services with StatusCode-based quality mapping.
- Disconnect detection via `Session.KeepAlive` event (see Disconnect Detection Pattern below).

### MxGateway
- Connects to the **MxAccess Gateway** (AVEVA/Wonderware MXAccess-backed Galaxy) over gRPC using the `ZB.MOM.WW.MxGateway.Client` NuGet package (from the Gitea feed); `ZB.MOM.WW.MxGateway.Contracts` is pulled in transitively.
- Session-based: `OpenSession` + `Register` on connect; `AddItem` + `Advise` per subscription; value changes arrive on the gateway's server-streaming event feed (`StreamEvents`), resumable via `worker_sequence`.
- Read/Write via `ReadBulk` / `WriteBulk`; writes carry a configurable `WriteUserId`. Quality maps the OPC-style quality byte (≥192 Good, ≥64 Uncertain, else Bad), with a failing MXAccess status proxy treated as Bad.
- Galaxy hierarchy browse via the separate `GalaxyRepositoryClient` — objects are navigable nodes (keyed by Galaxy gobject id), attributes are selectable leaves (keyed by full tag reference). Browse is **lazy and attribute-light**: navigation uses `BrowseChildren` with `include_attributes=false` (child objects only), and an object's own attributes are fetched only when it is expanded, via `DiscoverHierarchy(root=<object>, max_depth=0)` scoped to that single object. This keeps each browse level's reply small; inlining every child's full attribute set could exceed the Akka remote frame and silently drop the reply.
- Disconnect detection: a fault on the event stream raises `IDataConnection.Disconnected`, driving the same reconnection state machine as OPC UA.
- Implemented as `MxGatewayDataConnection` over an `IMxGatewayClient` seam; the seam is decoupled from the generated gRPC types (only `RealMxGatewayClient` references them), so the adapter is fully unit-testable with a fake.

## Endpoint Redundancy

Data connections support an optional backup endpoint for automatic failover when the active endpoint becomes unreachable. Both endpoints use the same protocol.

**Entity fields:**

| Field | Type | Notes |
|-------|------|-------|
| `PrimaryConfiguration` | string? (max 4000) | Required. Renamed from `Configuration` |
| `BackupConfiguration` | string? (max 4000) | Optional. Null = no backup |
| `FailoverRetryCount` | int (default 3) | Retries on active endpoint before switching |

**Failover state machine:**

```mermaid
%%{init: {'theme':'base', 'themeVariables': {'textColor':'#111111','lineColor':'#555555','edgeLabelBackground':'#ffffff','fontSize':'15px'}}}%%
flowchart TD
    connected(["Connected"])
    pushbad["push bad quality"]
    retry["retry active endpoint<br/>(5s)"]
    decide{"N failures<br/>(≥ FailoverRetryCount)?"}
    switch["switch to other endpoint"]
    dispose["dispose adapter,<br/>create fresh adapter<br/>with other config"]
    reconnect["reconnect"]
    resub["ReSubscribeAll"]

    connected -->|disconnect| pushbad
    pushbad --> retry
    retry --> decide
    decide -->|"no (retry again)"| retry
    decide -->|yes| switch
    switch --> dispose
    dispose --> reconnect
    reconnect --> resub
    resub -->|back to Connected| connected

    classDef start fill:#d5e8d4,stroke:#82b366,color:#111111;
    classDef proc fill:#dae8fc,stroke:#6c8ebf,color:#111111;
    classDef dec fill:#fff2cc,stroke:#d6b656,color:#111111;
    classDef warn fill:#ffe6cc,stroke:#d79b00,color:#111111;
    classDef bad fill:#f8cecc,stroke:#b85450,color:#111111;
    class connected start
    class pushbad bad
    class retry,reconnect,resub proc
    class decide dec
    class switch,dispose warn
```

- **Round-robin**: primary → backup → primary → backup. No preferred endpoint after first failover — the connection stays on whichever endpoint is working.
- **No auto-failback**: The connection remains on the active endpoint until it fails.
- **Single-endpoint connections** (no backup): Retry indefinitely on the same endpoint, preserving existing behavior.
- **Adapter lifecycle on failover**: The actor disposes the current `IDataConnection` adapter and creates a fresh one via `DataConnectionFactory.Create()` with the other endpoint's configuration. Clean slate — no stale state.

**Health reporting:**

- `DataConnectionHealthReport` includes `ActiveEndpoint`: `"Primary"`, `"Backup"`, or `"Primary (no backup)"`.

**Site event log entries:**

- `DataConnectionFailover` (Warning) — connection name, from-endpoint, to-endpoint, failure count.
- `DataConnectionRestored` (Info) — connection name, active endpoint.

See [`2026-03-22-primary-backup-data-connections-design.md`](../plans/2026-03-22-primary-backup-data-connections-design.md) for the full design.

## Connection Configuration Reference

All settings are parsed from the data connection's configuration JSON dictionaries (`PrimaryConfiguration` and optional `BackupConfiguration`, stored as `IDictionary<string, string>` connection details). Both endpoints use the same protocol-specific keys. Invalid numeric values fall back to defaults silently.

### OPC UA Settings

| Key | Type | Default | Description |
|-----|------|---------|-------------|
| `endpoint` / `EndpointUrl` | string | `opc.tcp://localhost:4840` | OPC UA server endpoint URL |
| `SessionTimeoutMs` | int | `60000` | OPC UA session timeout in milliseconds |
| `OperationTimeoutMs` | int | `15000` | Transport operation timeout in milliseconds |
| `PublishingIntervalMs` | int | `1000` | Subscription publishing interval in milliseconds |
| `KeepAliveCount` | int | `10` | Keep-alive frames before session timeout |
| `LifetimeCount` | int | `30` | Subscription lifetime in publish intervals |
| `MaxNotificationsPerPublish` | int | `100` | Max notifications batched per publish cycle |
| `SamplingIntervalMs` | int | `1000` | Per-item server sampling rate in milliseconds |
| `QueueSize` | int | `10` | Per-item notification buffer size |
| `MaxMonitoredItemsPerSubscription` | int | `5000` | Monitored-item budget per OPC UA subscription. Above it the adapter **shards** onto an additional subscription on the same session (37,500 tags → 8 shards). Per endpoint, because item-count ceilings are a property of the target server. |
| `SecurityMode` | string | `None` | Preferred endpoint security: `None`, `Sign`, or `SignAndEncrypt` |
| `AutoAcceptUntrustedCerts` | bool | `true` | Accept untrusted server certificates |

### MxGateway Settings

| Key | Type | Default | Description |
|-----|------|---------|-------------|
| `Endpoint` | string | `http://localhost:5000` | Gateway base URL |
| `ApiKey` | string | — | Sent to the gateway as `authorization: Bearer <key>` |
| `ClientName` | string | `scadabridge` (when blank) | MXAccess client registration name |
| `WriteUserId` | int | `0` | MXAccess user id applied to every write-back (0 = no user context) |
| `UseTls` | bool | `false` | Use TLS to a secured gateway |
| `CaFile` | string | — | Path to the CA certificate (TLS only) |
| `ServerName` | string | — | TLS server-name override |
| `ReadTimeoutMs` | int | `5000` | `ReadBulk` per-call timeout in milliseconds |

Secret handling for `ApiKey` follows the same at-rest treatment and log/telemetry redaction as the OPC UA `UserIdentity` username/password fields.

### Shared Settings (appsettings.json)

These are configured via `DataConnectionOptions` in `appsettings.json`, not per-connection:

| Setting | Default | Description |
|---------|---------|-------------|
| `ReconnectInterval` | 5s | Fixed interval between reconnection attempts |
| `TagResolutionRetryInterval` | 10s | **Floor** interval for the unresolved-tag retry; the retry backs off from here |
| `TagResolutionRetryMaxInterval` | 5m | Ceiling for the exponential tag-resolution backoff |
| `WriteTimeout` | 30s | Timeout for write operations |
| `SeedReadTimeout` | 30s | **Per-chunk** timeout for seed reads on the initial-subscribe (and reconnect re-seed) path. A hung device read is treated as a failed seed: retried up to `SeedReadMaxAttempts`, then the tag stays Uncertain until a change notification arrives. |
| `SeedOverallTimeout` | 120s | Wall-clock deadline for the WHOLE seed (all chunks, all retry rounds). Replaces the old 30s-per-tag serial worst case; remaining tags are logged and stay Uncertain. |
| `SeedReadBatchSize` | 250 | Tags per seed-read chunk |
| `SeedReadMaxParallelism` | 4 | Seed-read chunks in flight concurrently |
| `SubscribeBatchSize` | 500 | Tags per adapter subscribe round trip (initial subscribe, reconnect re-subscribe, resolution probes) |
| `SubscribeBatchDelay` | 50ms | Delay BETWEEN reconnect re-subscribe chunks. Not applied on the instance-driven subscribe path — the Deployment Manager already staggers instance startup there. |
| `QualityFlushInterval` | 1s | Coalescing window for pushing tag-quality counters to the health collector |
| `MxSupervisoryAdviseParallelism` | 16 | Max in-flight supervisory advise commands on the MxGateway bulk-subscribe path when the endpoint has no write-user context |

## Subscription Management

- When an Instance Actor is created (as part of the Site Runtime actor hierarchy), it registers its data source references with the Data Connection Layer.
- The DCL subscribes to the tag paths using the concrete connection details from the flattened configuration.
- Tag value updates are delivered directly to the requesting Instance Actor.
- When an Instance Actor is stopped (due to disable, delete, or redeployment), the DCL cleans up the associated subscriptions.
- When a new Instance Actor is created for a redeployment, subscriptions are established fresh based on the new configuration.

### Batching, chunking and pacing

Three subscribe paths, deliberately paced differently:

| Path | Shape |
|---|---|
| **Instance-driven subscribe** (`SubscribeTagsRequest`, i.e. site failover / deploy) | One `SubscribeBatchAsync` per `SubscribeBatchSize` chunk with **no** added delay. The Deployment Manager already staggers instance startup (`SiteRuntimeOptions.StartupBatchSize`/`StartupBatchDelayMs`), so requests arrive pre-spaced at ~75 tags each — double-staggering would only slow failover. |
| **Reconnect re-subscribe** (`ReSubscribeAll`) | **Sequential** chunks of `SubscribeBatchSize` with `SubscribeBatchDelay` between them, all inside ONE background task; each chunk reports its per-tag results back to the actor as its own message. Replaces a tight loop that fired one fire-and-forget task per tag. |
| **Tag-resolution probe** | The whole not-in-flight unresolved set, chunked at `SubscribeBatchSize`, as one probe round per timer tick. |

Seeding (the initial-value read that keeps a STATIC tag from staying Uncertain) issues chunked bulk reads — `SeedReadBatchSize` per chunk, `SeedReadMaxParallelism` chunks in flight, `SeedReadTimeout` per chunk — under a single `SeedOverallTimeout` deadline covering every chunk and retry round.

**OPC UA subscription sharding.** `RealOpcUaClient` places monitored items on the first subscription shard with free capacity (`MaxMonitoredItemsPerSubscription`), creating a shard when all are full and deleting one once it empties. A batch is N × `AddItem` plus **one** `ApplyChanges` per touched shard. Alarms & Conditions event items are pinned to a **dedicated event shard**, so `ConditionRefresh` has a single subscription id to target and `QueueSize:1000` event items never consume data-shard capacity. Every structural mutation (AddItem / RemoveItem / ApplyChanges / Create / Delete) and every shard-list mutation serializes behind ONE per-client gate — the SDK `Subscription` is not safe under concurrent structural mutation, and the subscribe task, resolution probe and alarm subscribe are otherwise unordered.

**MxGateway.** A batch is one `SubscribeBulk` gateway command (AddItem + Advise per tag in a single worker pass), replacing the historical 2 RPCs per tag. When the endpoint has no write-user context (`WriteUserId == 0`) the worker offers no BULK supervisory advise, so the client issues one `AddItemBulk` and then pipelines the per-item supervisory advises with a bounded window (`MxSupervisoryAdviseParallelism`). A cross-repo follow-up (an additive `supervisory` flag on `SubscribeBulkCommand` in mxaccessgw) would collapse that to one RPC as well.

## Write-Back Support

- When a script calls `Instance.SetAttribute` for an attribute with a data source reference, the Instance Actor sends a write request to the DCL.
- The DCL writes the value to the physical device via the appropriate protocol.
- The existing subscription picks up the confirmed new value from the device and delivers it back to the Instance Actor as a standard value update.
- The Instance Actor's in-memory value is **not** updated until the device confirms the write.

## Browsing the address space

DCL is a clean data pipe on the hot path. Browse is an **opt-in capability** for protocols that support it, exposed via `IBrowsableDataConnection`. Only consumed by management/UI (the tag picker on the instance configure page); Instance Actors never call it. The browse path is **protocol-agnostic**: the same command/service/dialog serve every browsable protocol.

- `OpcUaDataConnection` and `MxGatewayDataConnection` both implement `IBrowsableDataConnection`; other/custom protocols do not (and return a `NotBrowsable` failure).
- `DataConnectionManagerActor` handles `BrowseNodeCommand` (fields: `ConnectionName`, `ParentNodeId`, and an optional opaque `ContinuationToken` for paging) and replies with `BrowseNodeResult` (children + `Truncated` + an optional continuation token + structured `BrowseFailure?`). The Central UI facade is `IBrowseService`/`BrowseService`, backing the `NodeBrowserDialog` tag picker.
- **Browse type-info**: each child `BrowseNode` carries optional best-effort type metadata — `DataType` (friendly name), `ValueRank` (scalar/array), and `Writable`. `DataType` is populated by **both** OPC UA and MxGateway: `RealOpcUaClient` batch-reads built-in data-type node ids and maps them to friendly names for **Variable** nodes; `MxGatewayDataConnection` surfaces each attribute leaf's `GalaxyAttribute.data_type_name` (free-form Galaxy type text) as `DataType`. `ValueRank` and `Writable` are OPC-UA-only; MxGateway leaves them unset.
- Node ids are opaque protocol-specific strings: OPC UA uses NodeIds; MxGateway uses Galaxy gobject ids for navigable objects and full tag references for selectable attribute leaves.
- **OPC UA namespace binding (Gitea #14)**: a NodeId's `ns=<index>` is meaningful only against one server's namespace table, so a server that adds/removes/reorders a namespace silently re-points every stored binding — best case `BadNodeIdUnknown`, worst case the index names a different namespace holding a colliding identifier and the binding resolves to the **wrong node with Good quality**. The namespace **URI** is the stable identity. `OpcUaNodeReference` is therefore the single translation seam between stored references and the wire:
  - `Resolve(reference, session.NamespaceUris)` accepts **both** `ns=<index>;s=<id>` (existing bindings, unchanged) and the durable `nsu=<namespace-uri>;s=<id>`, mapping the URI to the live index at use time. It is used at every parse site — subscribe, read, write, alarm-subscribe, browse. A URI absent from the server's `NamespaceArray` throws with a message naming the URI, rather than binding to whatever sits at that index; a `svr=` reference to another server is rejected outright.
  - `ToDurable(expandedNodeId, session.NamespaceUris)` is what the browser emits, so **what the picker shows is what gets stored** and newly-authored bindings are index-proof from the start. Namespace 0 is exempt (spec-fixed) and keeps its short form. It also closes a round-trip gap: the browse previously emitted `ExpandedNodeId.ToString()`, which for a URI- or server-index-carrying reference produced a string `NodeId.Parse` could not read back.
  - Bindings stored before this change keep their `ns=` form and keep working; they are only as durable as the server's namespace order. Re-authoring against a picker (which now emits `nsu=`) is what makes them durable — see `docs/plans/` and Gitea #14 for the OtOpcUa v3.0 cutover, where v2's sole custom namespace and v3's `raw` namespace both sit at index 2, so v2-era bindings resolve against v3 without error while meaning something else.
  - **OtOpcUa v3.0 dual-namespace cutover**: v3.0 splits OtOpcUa's address space into two namespaces, `raw` (`https://zb.com/otopcua/raw`) and `uns` (`https://zb.com/otopcua/uns`), where v2 exposed one. ScadaBridge is already namespace-URI-durable against this split: `OpcUaNodeReference` resolves a stored `nsu=<uri>;s=...` binding against the live session `NamespaceArray` at use time, so it is index-proof regardless of which namespace a given tag lands in or how many namespaces the server publishes. The node-browser picker already nudges operators toward the durable form, since `ToDurable` is what it emits for newly-authored bindings. See `docs/plans/2026-07-23-otopcua-v3-dual-namespace-cutover-scope.md` for the full cutover scope.
- Browse runs against the live session; no caching at DCL.
- **Frame-size guard**: the reply crosses the site→central Akka frame (default 128 KB) on a temp Ask actor; an oversized reply is silently discarded by remoting, hanging the picker. The child handler caps each `BrowseNodeResult` to a byte budget (~100 KB) before replying, OR-ing the adapter's own truncation signal into `Truncated`. This is protocol-agnostic (every adapter's reply funnels through it). Per-protocol upstream caps narrow the window first: OPC UA requests at most 500 references per node (continuation point → `Truncated`); MxGateway relies on the gateway's `BrowseChildren` page cap.
- **`BrowseNext` paging** (OPC UA): a `Truncated` level no longer forces manual node-id entry. When the OPC UA browse is truncated, the adapter returns the session's continuation point as the reply's opaque `ContinuationToken`; a follow-up `BrowseNodeCommand` carrying that token calls `Session.BrowseNext` to fetch the next page (the picker exposes this as **"Load more"**). Continuation points are session-bound and can expire — `BadContinuationPointInvalid` is caught and the browse restarts from a fresh first page rather than failing. Manual node-id entry remains as a fallback when the site or its session is offline.

### Address-space search

A second opt-in capability seam, `IAddressSpaceSearchable` (in Commons, mirroring the `IBrowsableDataConnection` / `IAlarmSubscribableConnection` pattern; implemented by the OPC UA adapter, consumed by management/UI only):

```
IAddressSpaceSearchable
└── SearchAddressSpaceAsync(query, maxDepth, maxResults, ct) → matches
```

- `DataConnectionManagerActor` handles `SearchAddressSpaceCommand` (`ConnectionName`, `Query`, `MaxDepth`, `MaxResults`); the OPC UA adapter does a **bounded recursive browse** (depth + result caps) from the Objects folder, matching a **case-insensitive substring** against each node's DisplayName and root-relative path, and returns matches as `AddressSpaceMatch` (the `BrowseNode` plus its full path). When a cap is hit the result flags it so the UI can prompt "showing first N — refine".
- The Central UI facade is `BrowseService.SearchAsync` (Design role), surfaced as the search box in `NodeBrowserDialog`.
- `StubOpcUaClient` carries a canned browse/search implementation so the picker and its bUnit/unit tests run without a live OPC UA server.

## Endpoint verification (OPC UA)

Before saving or deploying an OPC UA data connection, the Central UI can ask the target site to **probe** the configured endpoint — this exercises connectivity (and TLS trust) against the live config, including edited-but-unsaved values.

- `DataConnectionManagerActor` handles `VerifyEndpointCommand` (`SiteId`, protocol, config JSON). The site spins up a **temporary** `RealOpcUaClient` from the submitted config, attempts discovery + a session with a short timeout (a few seconds), then disconnects and disposes the client. The reply is a `VerifyEndpointResult` (`Success`, a typed `FailureKind`, an error message, and an optional captured server cert).
- The probe forces **`AutoAcceptUntrustedCerts = false`** and hooks the certificate-validation callback so it can **capture** an untrusted server certificate (Subject / Issuer / Thumbprint / NotBefore / NotAfter / DER) for the UI to display. Critically, the probe **never trusts** the cert — the validation callback always rejects (`Accept = false`); trusting is a separate, explicit, Admin-gated action that writes the cert into the site PKI store (see the cert-trust path in Component-SiteRuntime.md). The probe is read-only with respect to the trust boundary.

## Native Alarm Mirroring

Some data sources publish their own alarms — OPC UA **Alarms & Conditions** servers and the **MxAccess Gateway**. The DCL can mirror these native alarms into the Site Runtime as a **read-only** feed: ScadaBridge reflects source alarm state but never acknowledges, confirms, shelves, or otherwise writes back to the source. This complements (does not replace) ScadaBridge's own computed alarms; it feeds the Site Runtime's `NativeAlarmActor` peer subsystem.

Like browse, this is an **opt-in capability** for protocols that support it. It does not touch the hot value path — alarm transitions flow over a separate per-connection feed.

### Capability Seam

Mirroring is exposed via the optional `IAlarmSubscribableConnection` capability interface (in Commons), which an `IDataConnection` implementation **may also** implement (mirroring the `IBrowsableDataConnection` pattern; consumed by the `DataConnectionActor` only):

```
IAlarmSubscribableConnection
├── SubscribeAlarmsAsync(sourceReference, conditionFilter?, callback, ct) → subscriptionId
└── UnsubscribeAlarmsAsync(subscriptionId, ct) → void
```

The `AlarmTransitionCallback` delivers a protocol-neutral `NativeAlarmTransition` per transition. On every (re)subscribe the adapter replays a **snapshot** of currently-active conditions (`Snapshot…` records terminated by a `SnapshotComplete` sentinel) so consumers can reconcile state after a reconnect.

### Protocol Adapters

- **OPC UA** (`OpcUaDataConnection` + `RealOpcUaClient`): a single **event MonitoredItem** (`AttributeId = EventNotifier`) on the Server object, with an `EventFilter` selecting `EventType` / `SourceNode` / `Severity` plus the `ConditionType` / `AcknowledgeableConditionType` / `AlarmConditionType` state fields. `ConditionRefresh` is invoked on subscribe to replay active conditions as the snapshot. The OPC UA field → `NativeAlarmTransition` mapping is isolated in the pure helper `OpcUaAlarmMapper`, unit-testable without a live server.
- **MxGateway** (`MxGatewayDataConnection` + `RealMxGatewayClient`): mirrors over the gateway package's `StreamAlarmsAsync` — a resumable background stream whose reconnect re-sends a snapshot. The field mapping lives in `MxGatewayAlarmMapper`.

Other/custom protocols do not implement the capability; a subscribe request against such a connection is replied to with a failure (`SubscribeAlarmsResponse.Success = false`).

### Connection Actor Behavior

The `DataConnectionActor` opens **one alarm feed per source** (not per subscriber) and routes incoming transitions to instance subscribers by **source-object reference** — a prefix match of the transition's `SourceObjectReference` (falling back to `SourceReference`) against each subscriber's registered `SourceReference`. Subscribers (the Site Runtime's `NativeAlarmActor` instances) are **ref-counted per source**, so the underlying feed is opened once and torn down only when the last subscriber for that source unsubscribes.

The registered `SourceReference` is a **NodeId** for OPC UA (the picker, CSV and config all store NodeIds), so the transition's routing identity must live in the same space or nothing matches. **Gitea #17**: the OPC UA adapter previously set `SourceObjectReference` from the event's `SourceName` (a plain name — the RawPath for OtOpcUa), which never prefix-matches a NodeId binding, so every native OPC UA alarm transition was silently dropped. Because each OPC UA feed is opened for exactly one binding, the adapter now tags every transition on that feed with the **binding string verbatim** (via the pure `OpcUaAlarmMapper.BuildIdentity`), making the routing key an exact match independent of whether the binding is stored as `ns=<index>` or the durable `nsu=<uri>` form. The Server-object aggregate feed (empty binding) keeps an empty routing identity, so it reaches only "mirror everything" subscribers and never leaks into a specific-node binding. The per-condition `SourceReference` key stays the human-readable `SourceName.ConditionName`, so persistence and display are unchanged. MxGateway is unaffected — its bindings are object names and its mapper already emits matching names.

- **State gating**: `SubscribeAlarmsRequest` is handled only in the **Connected** state; requests arriving while **Connecting**/**Reconnecting** are stashed (standard Become/Stash) and processed on entering Connected.
- **Capability check**: if `_adapter is not IAlarmSubscribableConnection`, the actor replies `SubscribeAlarmsResponse(Success = false, ...)`.
- **Reconnect handling**: on entering **Reconnecting**, the actor pushes a `NativeAlarmSourceUnavailable` to every alarm subscriber (consumers mark mirrored alarms uncertain rather than clearing them). On successful reconnection it re-subscribes the feed; the adapter re-emits a snapshot, reconciling state.
- **Shared condition filter (last-subscriber-wins)**: because the feed is opened once per source, it carries a single condition filter. A second subscriber that registers a *different* filter for the same source overwrites it (last writer wins) and the actor logs a warning — co-subscribers to one source are expected to agree on the filter.
- **Routing index**: subscribed sources are bucketed by the **first path segment** of their reference (everything before the first `.`), and a transition is matched only against the bucket for its own first segment, plus a residue list of sources that carry no separator at all (a prefix shorter than one segment, which can match other buckets). This is sound because a source reference containing a separator can only prefix a reference sharing its entire first segment. The per-source `StartsWith` test and the condition-type gate inside the bucket are unchanged, so routing decisions are identical to the previous linear scan over every subscribed source; the `SnapshotComplete` sentinel still bypasses the index entirely and is broadcast to every subscriber.
- **Gateway union filter (MxGateway)**: `StreamAlarms` carries a single `alarm_filter_prefix`, so the adapter opens the stream on the **longest common prefix** of the currently subscribed source references and restarts it only when a NEW source falls outside that prefix (the source then replays a fresh snapshot, which `NativeAlarmActor` already handles). An unsubscribe never restarts the stream — the prefix it leaves behind is at worst too broad, which costs bandwidth, not correctness. The prefix is a bandwidth optimisation only; the actor's per-source + condition-type gate remains authoritative, mirroring the OPC UA server-side WhereClause stance.

### Protocol-Neutral Types & Messages

All defined in Commons so the feed is identical across protocols:

| Type | Shape |
|------|-------|
| `NativeAlarmTransition` | `SourceReference`, `SourceObjectReference`, `AlarmTypeName`, `Kind`, `Condition`, `Category`, `Description`, `Message`, `OperatorUser`, `OperatorComment`, `OriginalRaiseTime?`, `TransitionTime`, `CurrentValue`, `LimitValue`, `AckTime?` |
| `AlarmConditionState` | `Active`, `Acknowledged`, `Confirmed?` (null when not confirmable), `Shelve`, `Suppressed`, `Severity` (0–1000) |
| `AlarmTransitionKind` (enum) | `Snapshot`, `SnapshotComplete`, `Raise`, `Acknowledge`, `Clear`, `Retrigger`, `StateChange` |

`OperatorUser` / `OperatorComment` and `CurrentValue` / `LimitValue` are display-only mirrors from the source.

### Ack Timestamp (`AckTime`)

`AckTime` is the instant a condition was acknowledged, mirrored end-to-end (transition → `AlarmStateChanged` → `AlarmStateUpdate` field 24 → site `native_alarm_state`) so MES can report a real `AckDT` and the `Alarms.CurrentAsync()` script accessor can surface it. Added additively as a trailing optional parameter — every pre-existing positional construction still compiles and yields `null`.

**One rule decides whether it is set: the condition must be BOTH active and acknowledged.** That single predicate gives all three required behaviours — null while unacknowledged, cleared on re-raise (a re-raise transition arrives unacknowledged), and no phantom ack on a return-to-normal (which matters for MxGateway, where `INACTIVE` maps to `Acknowledged = true`).

**Provenance differs by protocol, and the difference is deliberate — the value is never fabricated:**

| Source | Ack instant used | Accuracy |
|---|---|---|
| OPC UA A&C | `AcknowledgeableConditionType/AckedState/TransitionTime` — **SelectClause index 18**, appended after the limit fields so indices 0–17 keep their meaning | The source's own ack instant |
| OPC UA A&C, server omits the field | The event's `Time` field | When the DCL observed the acknowledged state |
| MxAccess Gateway | The transition's own timestamp (`TransitionTimestamp`, or the DCL's receipt time when the gateway omits it); on a re-subscribe snapshot, `LastTransitionTimestamp` for an `ACTIVE_ACKED` entry | When the system saw the ack — the gateway feed carries no dedicated ack timestamp |

The decision lives in the pure mappers (`OpcUaAlarmMapper.DeriveAckTime`, `MxGatewayAlarmMapper.DeriveAckTime`), so it is unit-tested without a live server or gateway.

Design record: `docs/plans/2026-06-30-mes-alarm-status-api.md` §6.4.

**Messages:**

- `SubscribeAlarmsRequest` / `SubscribeAlarmsResponse` — instance (via the DCL manager) subscribes a source binding to native alarms; the response carries success + an optional error message.
- `UnsubscribeAlarmsRequest` — cancels a native alarm subscription for an instance + source.
- `NativeAlarmTransitionUpdate(ConnectionName, Transition)` — DCL → instance: one routed transition (including snapshot replay).
- `NativeAlarmSourceUnavailable(ConnectionName, SourceReference, Timestamp)` — DCL → instance: the feed for a source became unavailable (connection lost).

## Value Update Message Format

Each value update delivered to an Instance Actor includes:
- **Tag path**: The relative path of the attribute's data source reference.
- **Value**: The new value from the device.
- **Quality**: Data quality indicator (good, bad, uncertain).
- **Timestamp**: When the value was read from the device.

## Connection Actor Model

Each data connection is managed by a dedicated connection actor that uses the Akka.NET **Become/Stash** pattern to model its lifecycle as a state machine:

- **Connecting**: The actor attempts to establish the connection. Subscription requests and write commands received during this phase are **stashed** (buffered in the actor's stash).
- **Connected**: The actor is actively servicing subscriptions. On entering this state, all stashed messages are unstashed and processed.
- **Reconnecting**: The connection was lost. The actor transitions back to a connecting-like state, stashing new requests while it retries.

This pattern ensures no messages are lost during connection transitions and is the standard Akka.NET approach for actors with I/O lifecycle dependencies.

**OPC UA-specific notes**: The `RealOpcUaClient` uses the OPC Foundation SDK's `Session.KeepAlive` event for proactive disconnect detection. The SDK sends keep-alive requests at the subscription's `KeepAliveCount × PublishingInterval` (default: 10s). When keep-alive fails, the `ConnectionLost` event fires, triggering the same reconnection flow. On reconnection, the DCL re-creates the OPC UA session and subscription, then re-adds all monitored items.

## Connection Lifecycle & Reconnection

The DCL manages connection lifecycle automatically:

1. **Connection drop detection**: When a connection to a data source is lost, the DCL immediately pushes a value update with quality `bad` for **every tag subscribed on that connection**. Instance Actors see the staleness immediately and publish it to the site stream / debug view; **script and alarm actors are deliberately NOT re-notified on a quality-only change** (no value changed — re-evaluating triggers would cause spurious firings). Scripts observe quality on their next attribute read; a computed alarm holds its last-known state until fresh values arrive.
2. **Auto-reconnect with fixed interval**: The DCL retries the connection at a configurable fixed interval (e.g., every 5 seconds). The retry interval is a **shared setting for all data connections** (`ReconnectInterval` in the Shared Settings table). This is consistent with the fixed-interval retry philosophy used throughout the system. Individual gRPC/OPC UA operations (reads, writes) fail immediately to the caller on error; there is no operation-level retry within the adapter.
3. **Connection state transitions**: The DCL tracks each connection's state as `connected`, `disconnected`, or `reconnecting`. All transitions are logged to Site Event Logging.
4. **Transparent re-subscribe**: On successful reconnection, the DCL automatically re-establishes all previously active subscriptions for that connection. Instance Actors require no action — they simply see quality return to `good` as fresh values arrive from restored subscriptions.
5. **Clean client replacement on (re)connect**: On every (re)connect the adapter detaches and disposes the previous protocol client (session, keep-alive timers, event loop) before creating a new one — a stale client can neither leak nor signal a disconnect against the new session.

### Disconnect Detection Pattern

Each adapter implements the `IDataConnection.Disconnected` event to proactively signal connection loss to the `DataConnectionActor`. Detection uses two complementary paths:

**Proactive detection** (server goes offline between operations):
- **OPC UA**: The OPC Foundation SDK fires `Session.KeepAlive` events at regular intervals. `RealOpcUaClient` hooks this event; when `ServiceResult.IsBad(e.Status)` (server unreachable, keep-alive timeout), it fires `ConnectionLost`. The `OpcUaDataConnection` adapter translates this into `IDataConnection.Disconnected`.

**Reactive detection** (failure discovered during an operation):
- Both adapters wrap `ReadAsync` (and by extension `ReadBatchAsync`) with exception handling. If a read throws a non-cancellation exception, the adapter calls `RaiseDisconnected()` and re-throws. The `DataConnectionActor`'s existing error handling catches the exception while the disconnect event triggers the reconnection state machine.

**Event marshalling**: The `DataConnectionActor` subscribes to `_adapter.Disconnected` in `PreStart()`. Since `Disconnected` may fire from a background thread (gRPC stream task, OPC UA keep-alive timer), the handler sends an `AdapterDisconnected` message to `Self`, marshalling the notification onto the actor's message loop. This triggers `BecomeReconnecting()` → bad quality push → retry timer.

**Once-only guard**: `OpcUaDataConnection` uses a `volatile bool _disconnectFired` flag to ensure `RaiseDisconnected()` fires exactly once per connection session. The flag resets on successful reconnection (`ConnectAsync`).

## Write Failure Handling

Writes to physical devices are **synchronous** from the script's perspective:

- If the write fails (connection down, device rejection, timeout), the error is **returned to the calling script**. Script authors can catch and handle write errors (log, notify, retry, etc.).
- Write failures are also logged to Site Event Logging.
- There is **no store-and-forward for device writes** — these are real-time control operations. Buffering stale setpoints for later application would be dangerous in an industrial context.

## Tag Path Resolution

When the DCL subscribes to a tag path from the flattened configuration but the path does not exist on the physical device (e.g., typo in the template, device firmware changed, device still booting):

1. The failure is **logged to Site Event Logging**.
2. The attribute is marked with quality `bad`.
3. The DCL **retries resolution with exponential backoff**, accommodating devices that come online in stages or load modules after startup: the interval starts at `TagResolutionRetryInterval` (10s), doubles after a round in which nothing resolved, and is capped at `TagResolutionRetryMaxInterval` (5m). It resets to the floor as soon as **any** tag resolves or the connection reconnects, so a device that is merely slow to boot is still picked up quickly while a dead one is not probed at full width forever. Each round probes the whole unresolved set in `SubscribeBatchSize` chunks, not one call per tag.
4. The retry timer is **single-shot, rescheduled on probe completion** — a periodic timer cannot back off, and rescheduling only on completion is what keeps a fan-out of failures from resetting the clock (the anti-starvation property previously defended by an `IsTimerActive` gate, which is retained).
5. On successful resolution, the subscription activates normally and quality reflects the live value from the device.

Note: Pre-deployment validation at central does **not** verify that tag paths resolve to real tags on physical devices — that is a runtime concern handled here.

## Health Reporting

The DCL reports the following metrics to the Health Monitoring component via the existing periodic heartbeat:

- **Connection status**: `connected`, `disconnected`, or `reconnecting` per data connection.
- **Tag resolution counts**: Per connection, the number of total subscribed tags vs. successfully resolved tags. This gives operators visibility into misconfigured templates without needing to open the debug view for individual instances. Both numbers are **derived at report time from the connection actor's authoritative per-tag state**, never accumulated in counters: the total is the number of distinct tag paths at least one instance currently subscribes to (the per-tag counted set that also drives value fan-out and the last-subscriber release decision), and the resolved count is the number of tags for which the adapter currently holds a subscription handle. A tag counts toward the total from the moment an instance registers it, whatever its resolution outcome — resolved, awaiting a resolution retry, or failed at connection level — so resolved can never exceed total. Deriving rather than accumulating is deliberate: increment/decrement counters drifted whenever an unsubscribe landed inside a reconnect window (the reconnect clears the per-tag maps the decrements keyed off), leaking a phantom tag into the reported total on every churn cycle.
- **Tag quality counters** are pushed on a genuine quality **transition** only, coalesced onto a `QualityFlushInterval` (1s) single-shot timer. Counter arithmetic still runs per message; only the collector push is deferred, and a value whose quality is unchanged moves no counter at all. Health reports poll at 30s, so the coalescing loses nothing. Three paths flush **synchronously** because their correctness depends on it: the bad-quality push on disconnect, unsubscribe, and the reconnect counter reset.

## Dependencies

- **Site Runtime (Instance Actors)**: Receives subscription registrations and delivers value updates. Receives write requests.
- **Site Runtime (NativeAlarmActor)**: For alarm-subscribable connections, receives `SubscribeAlarmsRequest`/`UnsubscribeAlarmsRequest` and delivers `NativeAlarmTransitionUpdate` / `NativeAlarmSourceUnavailable` (read-only native alarm mirroring).
- **Health Monitoring**: Reports connection status.
- **Site Event Logging**: Logs connection status changes.

## Interactions

- **Site Runtime (Instance Actors)**: Bidirectional — delivers value updates, receives subscription registrations and write-back commands.
- **Site Runtime (NativeAlarmActor)**: Bidirectional — receives alarm subscribe/unsubscribe requests, delivers native alarm transitions and source-unavailable notifications (read-only; no ack-back to the source).
- **Health Monitoring**: Reports connection health periodically.
- **Site Event Logging**: Logs connection/disconnection events.
