lmxopcua/docs/Historian.md

# Historian — server-side OPC UA HistoryRead (Phase C)

Phase C wires server-side OPC UA **HistoryRead** for authored equipment tags flagged
historized. The feature is driver-agnostic: any equipment tag (Galaxy, Modbus, OpcUaClient,
or any other driver) can be marked historized; the server dispatches all history reads to the
registered `IHistorianDataSource` — today, the Wonderware sidecar client
(`WonderwareHistorianClient`). No EF migration is required; the historian flag rides in the
existing schemaless `TagConfig` JSON blob alongside the Phase B `alarm` object.

Design reference: [docs/plans/2026-06-14-galaxy-phase-c-historian-design.md](plans/2026-06-14-galaxy-phase-c-historian-design.md).

---

## Historized TagConfig schema

A tag is historized by setting the **Historize this tag** checkbox and optional **Historian
tagname (override)** textbox in the Tag modal on the `/uns` equipment page Tags tab. These
controls work for **all drivers** — typed editors (Modbus, S7, OpcUaClient, etc.) and the
raw-JSON textarea (Galaxy) alike. The controls merge `isHistorized` / `historianTagname`
into the existing `TagConfig` JSON blob via the `TagHistorizeConfig` helper, preserving all
other keys byte-stable. Drivers that still use the raw-JSON editor (Galaxy) can also add the
fields directly in the textarea.

### Fields

| Field | Type | Required | Description |
|---|---|---|---|
| `isHistorized` | bool | yes | Marks the tag historized. Materialises the OPC UA node with `Historizing=true` and the `HistoryRead` AccessLevel bit. |
| `historianTagname` | string | no | Explicit historian tagname to query. When absent or empty, defaults to the tag's driver `FullName` (the `TagConfig.FullName`). |

### Examples

Default historian tagname (uses `FullName` = `TestMachine_002.TestFloat`):

```json
{"FullName":"TestMachine_002.TestFloat","isHistorized":true}
```

Explicit historian tagname override:

```json
{"FullName":"TestMachine_002.TestFloat","isHistorized":true,"historianTagname":"Plant.Line1.Flow"}
```

A tag that is both historized and a native alarm:

```json
{"FullName":"TestMachine_002.HiAlarm","isHistorized":true,"alarm":{"alarmType":"OffNormalAlarm","severity":700}}
```

---

## ServerHistorian configuration

The `ServerHistorian` section in `appsettings.json` controls the historian read path.
`Enabled` defaults to `false`; when disabled, the server registers `NullHistorianDataSource`
and all HistoryRead calls on historized nodes return `GoodNoData` (empty, not an error).

```json
{
  "ServerHistorian": {
    "Enabled": false,
    "Host": "localhost",
    "Port": 32569,
    "UseTls": false,
    "ServerCertThumbprint": "",
    "SharedSecret": ""
  }
}
```

| Key | Type | Default | Description |
|---|---|---|---|
| `Enabled` | bool | `false` | Enable the live `WonderwareHistorianClient`. `false` → `NullHistorianDataSource` (empty reads). |
| `Host` | string | `localhost` | DNS name or IP of the machine running the historian sidecar. |
| `Port` | int | `32569` | TCP port the sidecar listens on (`OTOPCUA_HISTORIAN_TCP_PORT`). |
| `UseTls` | bool | `false` | Wrap the TCP connection in TLS. |
| `ServerCertThumbprint` | string | — | Optional SHA-1 thumbprint to pin the sidecar's TLS certificate. Leave empty for CA-chain validation. |
| `SharedSecret` | string | — | Shared secret token the sidecar expects on every connection. Required when `Enabled`. |

> **Do not commit `SharedSecret` to `appsettings.json`.** Set it via an environment variable,
> a secrets store, or a deployment-time overlay. The checked-in default is always empty.

The `ServerHistorian` section is independent of the `AlarmHistorian` section (the alarm
write path). They share the same Wonderware sidecar process but hold separate client
instances and separate `SharedSecret` values.

---

## HistoryRead behavior

### Read variants

The server supports all four OPC UA HistoryRead variants:

| Variant | Node type | CLI `--aggregate` |
|---|---|---|
| **Raw** | Historized variable | (omit `--aggregate`) |
| **Processed** | Historized variable | `Average`, `Minimum`, `Maximum`, `Total`, `Count` |
| **AtTime** | Historized variable | n/a (client supplies exact timestamps) |
| **Events** | Equipment folder (event notifier) | n/a |

**Variable nodes** (historized tags) serve Raw, Processed, and AtTime history.
`Historizing=true` and `AccessLevels.HistoryRead` are set at materialization so any compliant
OPC UA client can discover historized capability from the node's attributes.

**Equipment-folder event-notifier nodes** serve Event history. Every equipment folder that
owns at least one alarm condition is already an event notifier; the server registers a
`sourceName` (the equipment id) for each such folder and maps event history reads to the
Wonderware historian using that source. Event-field projection supports the standard
`BaseEventType` select clauses — `EventId`, `SourceName`, `Time`, `ReceiveTime`, `Message`,
and `Severity`; an unsupported select operand returns a null field (spec-conformant).

### Graceful degradation

| Situation | HistoryRead status |
|---|---|
| Historized node, historian configured and reachable, results found | `Good` |
| Historized node, historian configured, time range empty | `GoodNoData` |
| Historized node, historian NOT configured (`Enabled=false` / Null source) | `GoodNoData` (empty) |
| Non-historized node | `BadHistoryOperationUnsupported` |
| Backend timeout or exception | `BadHistoryOperationUnsupported` per node; other nodes in the same batch are unaffected |

A historized node with no historian configured never returns an error status — it returns
empty. This means a deployment can author and publish historized tags before the historian
sidecar is provisioned, without producing error spikes in connected clients.

### Continuation-point paging (Raw)

`HistoryRead-Raw` is paged server-side. The historian backend is single-shot (it returns up to
`NumValuesPerNode` samples with no continuation point of its own), so the server synthesises
paging time-based:

- A page that returns **exactly** `NumValuesPerNode` samples (with `NumValuesPerNode > 0`) MAY
  have more behind it, so the server stores a resume cursor and returns an opaque continuation
  point (16 bytes). The client hands it back to fetch the next page.
- A short page (fewer than the cap) is the last page — no continuation point.
- `NumValuesPerNode == 0` ("all values, no limit") is never paged; the whole window returns in one
  shot.
- The resume cursor is **tie-safe**: the next page resumes from the last returned sample's
  SourceTimestamp *inclusive* and drops the boundary samples already emitted, so samples sharing
  the boundary timestamp are neither duplicated nor skipped.

> **Paging limitation — oversized tie clusters.** The tie-safe cursor is a `(timestamp, skip)`
> pair, and the single-shot backend only accepts `(start, end, cap)` — it cannot skip. So if **more
> samples share one `SourceTimestamp` than `NumValuesPerNode`** (a tie cluster larger than the page
> cap), the cursor cannot advance past that timestamp: every resume re-reads the same first `cap`
> ties. Rather than silently truncate the read to `GoodNoData` (which would permanently drop the
> un-emitted ties), the resume read fails that node **loudly** with
> `BadHistoryOperationUnsupported` and logs the tag + timestamp + cap. The operator's remedy is to
> re-issue the read with a larger `NumValuesPerNode`. For a single tag's raw history this is a data
> anomaly (raw samples normally carry strictly increasing distinct timestamps); a fully cursor-based
> fix that pages *within* a single timestamp is a possible follow-up.

Continuation points are bound to the OPC UA session (the SDK's
`ServerConfiguration.MaxHistoryContinuationPoints` cap, default 100, with oldest-eviction; points
are disposed when the session closes). Resuming an unknown / evicted / released point returns
`BadContinuationPointInvalid`. `releaseContinuationPoints` drops the stored cursors without reading
data.

### Total aggregate derivation

The OPC UA `Total` aggregate is **supported** over the Wonderware backend. Because the
Wonderware `AnalogSummary` query exposes no `Total` column, the value is derived client-side
using the time-integral identity:

> **Total = time-weighted Average × interval-seconds**

The wire request is issued with the `Average` column; each returned bucket's value is
multiplied by `interval.TotalSeconds` before the result is returned to the OPC UA client.
Bucket status codes and timestamps are preserved unchanged. Null (unavailable) Average
buckets produce a null Total (`BadNoData` downstream) — the scaling is not applied.

This derivation is exact for piecewise-constant (step) signals. For continuously varying
signals it is an approximation identical to the one Wonderware would apply internally, so
the result is consistent with what AVEVA Historian reports for the same window.

### Known limitations

- **Processed and AtTime are single-shot** (no continuation points). Unlike Raw, neither
  `ReadProcessedDetails` nor `ReadAtTimeDetails` carries a client count cap (`NumValuesPerNode`):
  the Processed bucket count is deterministic (window / interval) and AtTime returns exactly one
  sample per requested timestamp, so the single-shot backend returns the complete result in one
  read and there is no "full page ⇒ maybe more" signal to page on. Returning the full result with
  no continuation point is spec-conformant.
- **No modified-value history** (`HistoryReadModified`). Requests for modified values return
  `BadHistoryOperationUnsupported`. This is **infra-gated, not a server-code gap**: the AVEVA
  Wonderware historian backend (`IHistorianDataSource`, the TCP sidecar client) exposes only a
  current-value read path — there is no modified/edited-history surface to source the data from. The
  server-side override is in place (it cleanly rejects modified reads per node) and `IsReadModified`
  is honoured; serving real modified-value history is unblocked only once the historian client/sidecar
  grows a modified-read RPC. Until then, rejecting is the correct, spec-conformant behaviour.

### Redundancy and authorization

History reads are served from any node — there is no Primary gate. Authorization is the
standard OPC UA `HistoryRead` permission enforced by the SDK through the `AccessLevels.HistoryRead`
bit set at materialization. A session without sufficient permissions receives
`BadUserAccessDenied` from the SDK before the dispatch reaches the historian.

---

## Authoring workflow

1. Open the equipment's **Tags** tab on `/uns/equipment/{id}`.
2. Create or edit the tag.
   - For **typed-editor drivers** (Modbus, S7, OpcUaClient, etc.): check the
     **Historize this tag** checkbox and, if needed, fill in the **Historian tagname
     (override)** textbox.
   - For **raw-JSON editors** (Galaxy): you can check the same first-class checkboxes
     (they appear below the JSON textarea), or add `"isHistorized":true` (and optionally
     `"historianTagname":"..."`) directly in the textarea.
3. Save and publish. The server rebuilds its address space; the node materialises with
   `Historizing=true` and the `HistoryRead` AccessLevel bit.
4. Confirm with Client.CLI `read` that the node's `Status` is `Good` and that the value
   is updating. Then issue a `historyread` to verify the historian connection returns data.

### Native-alarm historian opt-out (`alarm.historizeToAveva`)

A tag carrying a native `"alarm"` object has a **separate** historian opt-out for its
**alarm transitions** (distinct from tag-value historization). On the Tag modal, check or
uncheck **Historize to AVEVA** in the alarm section. This maps to `alarm.historizeToAveva`
(`bool?`) in the `TagConfig` JSON:

- **Absent or `true`** (default) — the alarm's transitions are written to AVEVA Historian
  via `HistorianAdapterActor`.
- **`false`** — the durable AVEVA write is suppressed for this alarm's transitions. The
  live `/alerts` feed and OPC UA condition events are unaffected.

The gate is applied in `HistorianAdapterActor` using `is not false` semantics, matching the
scripted-alarm `HistorizeToAveva` posture. See
[ScriptedAlarms.md §Native driver alarms](ScriptedAlarms.md#native-driver-alarms-equipment-tag-path)
and [AlarmTracking.md §Historian write-back](AlarmTracking.md#historian-write-back-non-galaxy-alarms)
for the full alarm-historian routing.

---

## Client.CLI historyread examples

The `historyread` command reads historical data from any node. Supply start and end times in
ISO 8601 UTC form. See [docs/Client.CLI.md](Client.CLI.md) for the full flag reference.

```bash
# Raw history for a historized Galaxy tag (last 24 hours by default)
otopcua-cli historyread \
  -u opc.tcp://localhost:4840/OtOpcUa \
  -n "ns=2;s=EQ-55297329838d/GalaxyTestTag" \
  --start "2026-06-13T00:00:00Z" --end "2026-06-14T00:00:00Z"

# Limit to 100 values
otopcua-cli historyread \
  -u opc.tcp://localhost:4840/OtOpcUa \
  -n "ns=2;s=EQ-55297329838d/GalaxyTestTag" \
  --start "2026-06-13T00:00:00Z" --end "2026-06-14T00:00:00Z" \
  --max 100

# 1-hour average aggregate
otopcua-cli historyread \
  -u opc.tcp://localhost:4840/OtOpcUa \
  -n "ns=2;s=EQ-55297329838d/GalaxyTestTag" \
  --start "2026-06-13T00:00:00Z" --end "2026-06-14T00:00:00Z" \
  --aggregate Average --interval 3600000

# Authenticated read (ReadOnly role or higher required)
otopcua-cli historyread \
  -u opc.tcp://localhost:4840/OtOpcUa \
  -n "ns=2;s=EQ-55297329838d/GalaxyTestTag" \
  --start "2026-06-13T00:00:00Z" --end "2026-06-14T00:00:00Z" \
  -U reader -P password
```

Supported `--aggregate` values: `Average`, `Minimum`, `Maximum`, `Count`, `Start`, `End`,
`StandardDeviation` (aliases: `avg`, `min`, `max`, `stddev`/`stdev`, `first`, `last`).
`--interval` is the processing interval in milliseconds (default 3600000 = 1 hour).
The `Total` aggregate is served by the server's OPC UA HistoryRead endpoint for any conformant
client (derived as time-weighted Average × interval-seconds — see "Total aggregate derivation"
above), but is not exposed by this bundled CLI.

---

## Live /run gate

The live read gate requires the Wonderware historian sidecar running on the WW Historian VM
(`10.100.0.48`) and AVEVA Historian healthy. Set `ServerHistorian:Enabled=true` with the
correct `Host`, `Port`, and `SharedSecret` in `appsettings.json` (or via environment
variables), then deploy and publish at least one historized Galaxy tag. The gate is
operator-driven — it is not part of the local docker-dev rig.

See [AlarmHistorian.md](AlarmHistorian.md) for the historian sidecar setup and
[ServiceHosting.md](ServiceHosting.md) for the sidecar service configuration.

---

---

## Cross-driver TagConfig key reference

The `TagConfig` JSON blob is the single extension surface for per-tag server-side
behaviours across all drivers.  Keys from multiple phases coexist in the same blob;
unknown keys are preserved byte-stable on round-trip by all typed editors.

### Historian + alarm keys (Phase B / Phase C)

| Key | Type | Phase | Description |
|---|---|---|---|
| `isHistorized` | bool | C | Marks the tag historized (HistoryRead + Historizing=true). |
| `historianTagname` | string | C | Explicit historian tagname override (defaults to `FullName`). |
| `alarm` | object | B | Native alarm definition (`alarmType`, `severity`, `historizeToAveva`, …). |

### Array keys (Phase 4c)

| Key | Type | Phase | Description |
|---|---|---|---|
| `isArray` | bool | 4c | When `true` and `arrayLength >= 1`, materialises the node as a 1-D array (`ValueRank=OneDimension`). Absent or `false` → scalar. |
| `arrayLength` | uint (≥1) | 4c | Element count; sets `ArrayDimensions`. Required when `isArray: true`. |

### Cross-driver array coverage matrix

| Driver | Read mechanism | Live-verify status |
|---|---|---|
| **Modbus** | Contiguous FC03/FC04 block; String/BitInRegister modes supported | Mac-verifiable (sim `10.100.0.35:5020`) |
| **S7** | `ReadBytesAsync` block + per-element decode loop | Unit-proven (fixture down) |
| **AB CIP** | libplctag native array read (atomic + UDT member arrays) | Unit-proven (fixture down) |
| **AB Legacy** | PCCC multi-element file read via libplctag (cap 256 elements) | Unit-proven (fixture down) |
| **TwinCAT** | ADS native array symbol read | Unit-proven (fixture down) |

Array writes, multi-dimensional arrays (ValueRank>1), and per-element historization are out of scope — see [Uns.md §Array tags](Uns.md#array-tags-1-d).

---

## Closed stillpending.md §2 items (Phase 4 / Phase 4b)

The following items from the `stillpending.md` backlog §2 were closed by earlier
phases and are recorded here so future audits don't re-flag them.

| Item | Closed by | Commit |
|---|---|---|
| Modbus Int64 / UInt64 OPC UA node `DataType` correction | Phase 4 | `bd8fee61` |
| `HistoryAggregateType.Total` — `Total` aggregate | Phase 4 (derived client-side as time-weighted Average × interval-seconds) | `5e27b5f7` |
| Historian poison alarm-event indefinite retry — dead-letter cap (task #437) | Phase 4 | `fcb38014` |

---

## See also

- [docs/plans/2026-06-14-galaxy-phase-c-historian-design.md](plans/2026-06-14-galaxy-phase-c-historian-design.md) — full design and implementation notes
- [AlarmHistorian.md](AlarmHistorian.md) — alarm write path; shares the same Wonderware sidecar
- [AlarmTracking.md](AlarmTracking.md) — OPC UA Part 9 alarm surface (event history source)
- [Client.CLI.md](Client.CLI.md) — full `historyread` flag reference
- [ScriptedAlarms.md](ScriptedAlarms.md) §"Native driver alarms" — the Phase B `alarm` object in `TagConfig` (parallel carrier)
- [Uns.md §Array tags](Uns.md#array-tags-1-d) — `isArray`/`arrayLength` keys, cross-driver coverage, and deferrals