lmxopcua/historian_plan.md

# OPC UA Historical Data Access Plan

## Context

Galaxy attributes with `HistoryExtension` primitives are historized by the Wonderware Historian. The Historian exposes its data via SQL queries against the `Runtime` database. This plan documents how to implement OPC UA Historical Data Access (HDA) so OPC UA clients can read historical values through the server.

## 1. Wonderware Historian Data Source

### Connection

- **Database**: `Runtime` on `localhost` (Windows Auth)
- **Constraint**: History views require a `WHERE TagName='...'` clause — queries without a tag filter will fail.

### History View Schema (31 columns)

Key columns for OPC UA HDA:

| Column | Type | Description |
|---|---|---|
| `DateTime` | datetime2 | Timestamp of the value |
| `TagName` | nvarchar(256) | Galaxy tag reference (e.g., `TestMachine_001.TestHistoryValue`) |
| `Value` | float | Numeric value |
| `vValue` | nvarchar(4000) | String representation of value |
| `Quality` | tinyint | Quality code (0=Good, 1=Bad, 133=Uncertain) |
| `QualityDetail` | int | Detailed quality (192=Good) |
| `OPCQuality` | int | OPC-style quality code |

### Raw Data Query

```sql
SELECT DateTime, Value, vValue, Quality, QualityDetail
FROM Runtime.dbo.History
WHERE TagName = 'TestMachine_001.TestHistoryValue'
  AND DateTime BETWEEN @StartTime AND @EndTime
ORDER BY DateTime
```

### Aggregate Data (AnalogSummaryHistory)

The Historian provides pre-calculated aggregates via the `AnalogSummaryHistory` view:

| Column | Description |
|---|---|
| `StartDateTime` | Start of aggregate interval |
| `EndDateTime` | End of aggregate interval |
| `First` | First value in interval |
| `Last` | Last value in interval |
| `Minimum` | Minimum value |
| `Maximum` | Maximum value |
| `Average` | Average value |
| `StdDev` | Standard deviation |
| `Integral` | Time-weighted integral |
| `ValueCount` | Number of values |

```sql
SELECT StartDateTime, EndDateTime, Average, Minimum, Maximum, ValueCount
FROM Runtime.dbo.AnalogSummaryHistory
WHERE TagName = 'TestMachine_001.TestHistoryValue'
  AND StartDateTime BETWEEN @StartTime AND @EndTime
  AND wwResolution = @IntervalMs
```

### Retrieval Modes

| Mode | Description |
|---|---|
| `DELTA` | Change-based retrieval (default) — returns values when they changed |
| `CYCLIC` | Periodic sampling — returns interpolated values at fixed intervals |

### Quality Mapping

| Historian Quality | OPC UA StatusCode |
|---|---|
| 0 (Good) | `Good` (0x00000000) |
| 1 (Bad) | `Bad` (0x80000000) |
| 133 (Uncertain) | `Uncertain` (0x40000000) |

### Test Data

Tag: `TestMachine_001.TestHistoryValue` (Analog, Integer)
- 4 records from 2026-03-26 00:44 to 01:09
- Values: 0, 3, 4, 7, 9
- InterpolationType: STAIRSTEP

## 2. OPC UA HDA Implementation

### Marking Variables as Historized

For attributes where `is_historized = 1` from the Galaxy query:

```csharp
variable.Historizing = true;
variable.AccessLevel |= AccessLevels.HistoryRead;
variable.UserAccessLevel |= AccessLevels.HistoryRead;
```

This tells OPC UA clients the variable supports `HistoryRead` requests.

### Server-Side Handler

Override `HistoryRead` on `LmxNodeManager` (inherits from `CustomNodeManager2`):

```csharp
public override void HistoryRead(
    OperationContext context,
    HistoryReadDetails details,
    TimestampsToReturn timestampsToReturn,
    bool releaseContinuationPoints,
    IList<HistoryReadValueId> nodesToRead,
    IList<HistoryReadResult> results,
    IList<ServiceResult> errors)
```

Dispatch based on `details` type:
- `ReadRawModifiedDetails` → `HistoryReadRaw` → query `Runtime.dbo.History`
- `ReadProcessedDetails` → `HistoryReadProcessed` → query `Runtime.dbo.AnalogSummaryHistory`
- `ReadAtTimeDetails` → `HistoryReadAtTime` → query with `wwRetrievalMode = 'Cyclic'`

### ReadRaw Implementation

Map `HistoryReadRawModifiedDetails` to a Historian SQL query:

| OPC UA Parameter | SQL Mapping |
|---|---|
| `StartTime` | `DateTime >= @StartTime` |
| `EndTime` | `DateTime <= @EndTime` |
| `NumValuesPerNode` | `TOP @NumValues` |
| `ReturnBounds` | Include one value before StartTime and one after EndTime |

Result: populate `HistoryData` with `DataValue` list:
```csharp
new DataValue
{
    Value = row.Value,
    SourceTimestamp = row.DateTime,
    StatusCode = MapQuality(row.Quality)
}
```

### ReadProcessed Implementation

Map `HistoryReadProcessedDetails` to `AnalogSummaryHistory`:

| OPC UA Aggregate | Historian Column |
|---|---|
| `Average` | `Average` |
| `Minimum` | `Minimum` |
| `Maximum` | `Maximum` |
| `Count` | `ValueCount` |
| `Start` | `First` |
| `End` | `Last` |
| `StandardDeviationPopulation` | `StdDev` |

`ProcessingInterval` maps to `wwResolution` (milliseconds).

### Continuation Points for Paging

When `NumValuesPerNode` limits the result:

1. Query `NumValuesPerNode + 1` rows
2. If more exist, save a continuation point (store last timestamp + query params)
3. Return `StatusCodes.GoodMoreData` with the continuation point
4. On next request, restore the continuation point and resume from last timestamp

Use `Session.SaveHistoryContinuationPoint()` / `RestoreHistoryContinuationPoint()` to manage state.

### Tag Name Resolution

The `FullTagReference` stored on each variable node (e.g., `TestMachine_001.TestHistoryValue`) is exactly the `TagName` used in the Historian query — no translation needed.

## 3. Galaxy Repository Detection

Already implemented: `is_historized` column in the attributes queries detects `HistoryExtension` primitives in the deployed package chain.

## 4. Implementation Steps

### Phase 1: Mark historized nodes
- Read `is_historized` from query results into `GalaxyAttributeInfo`
- In `LmxNodeManager.CreateAttributeVariable`, set `Historizing = true` and add `HistoryRead` to `AccessLevel`

### Phase 2: Historian data source
- New class: `HistorianDataSource` — executes SQL queries against `Runtime.dbo.History` and `AnalogSummaryHistory`
- Connection string configurable in `appsettings.json`
- Parameterized queries only (no dynamic SQL)

### Phase 3: HistoryRead handler
- Override `HistoryRead` on `LmxNodeManager`
- Implement `HistoryReadRaw` — query `History` view, map results to `HistoryData`
- Implement `HistoryReadProcessed` — query `AnalogSummaryHistory`, map aggregates
- Implement continuation points for large result sets

### Phase 4: Testing
- Unit tests for quality mapping, tag name resolution, SQL parameter building
- Integration test: create a historized variable, verify `Historizing = true` and `HistoryRead` access level
- Manual test: use OPC UA client to read historical data from deployed server

## 5. OPC UA CLI Tool — History Command

Add a `historyread` command to `tools/opcuacli-dotnet/` for manual testing of HDA.

### Usage

```bash
# Read raw history (last 24 hours)
dotnet run -- historyread -u opc.tcp://localhost:4840/LmxOpcUa -n "ns=1;s=TestMachine_001.TestHistoryValue"

# Read raw history with explicit time range
dotnet run -- historyread -u opc.tcp://localhost:4840/LmxOpcUa -n "ns=1;s=TestMachine_001.TestHistoryValue" --start "2026-03-25" --end "2026-03-30"

# Read with max values limit
dotnet run -- historyread -u opc.tcp://localhost:4840/LmxOpcUa -n "ns=1;s=TestMachine_001.TestHistoryValue" --start "2026-03-25" --end "2026-03-30" --max 100

# Read processed/aggregate history (1-hour intervals, Average)
dotnet run -- historyread -u opc.tcp://localhost:4840/LmxOpcUa -n "ns=1;s=TestMachine_001.TestHistoryValue" --start "2026-03-25" --end "2026-03-30" --aggregate Average --interval 3600000
```

### Command Options

| Flag | Description |
|------|-------------|
| `-u, --url` | OPC UA server endpoint URL (required) |
| `-n, --node` | Node ID to read history for (required) |
| `--start` | Start time, ISO 8601 or date string (default: 24 hours ago) |
| `--end` | End time, ISO 8601 or date string (default: now) |
| `--max` | Maximum number of values to return (default: 1000) |
| `--aggregate` | Aggregate function: Average, Minimum, Maximum, Count (default: none = raw) |
| `--interval` | Processing interval in milliseconds for aggregates (default: 3600000 = 1 hour) |

### Output Format

**Raw history:**
```
History for ns=1;s=TestMachine_001.TestHistoryValue (2026-03-25 → 2026-03-30)

Timestamp                      Value       Status
2026-03-26T00:44:03.000Z       0           Good
2026-03-26T00:52:17.000Z       3           Good
2026-03-26T01:01:44.000Z       7           Good
2026-03-26T01:09:00.000Z       9           Good

4 values returned.
```

**Aggregate history:**
```
History for ns=1;s=TestMachine_001.TestHistoryValue (Average, interval=3600000ms)

Timestamp                      Value       Status
2026-03-26T00:00:00.000Z       4.75        Good

1 values returned.
```

### Implementation

New file: `tools/opcuacli-dotnet/Commands/HistoryReadCommand.cs`

Uses the OPC UA client SDK's `Session.ReadRawHistory` and `Session.ReadProcessedHistory` methods (or `HistoryReadAsync` with appropriate `HistoryReadDetails`):

```csharp
// Raw read
var details = new ReadRawModifiedDetails
{
    StartTime = startTime,
    EndTime = endTime,
    NumValuesPerNode = (uint)maxValues,
    IsReadModified = false,
    ReturnBounds = false
};

// Processed read
var details = new ReadProcessedDetails
{
    StartTime = startTime,
    EndTime = endTime,
    ProcessingInterval = intervalMs,
    AggregateType = new NodeIdCollection { aggregateNodeId }
};
```

Follow the same pattern as existing commands: use `OpcUaHelper.ConnectAsync()`, parse NodeId, call history read, print results.

### Continuation Point Handling

If the server returns `GoodMoreData` with a continuation point, automatically follow up with subsequent requests until all data is retrieved or `--max` is reached.

### README Update

Add `historyread` section to `tools/opcuacli-dotnet/README.md` documenting the new command.

## 6. Files to Modify/Create

| File | Change |
|---|---|
| `src/.../Domain/GalaxyAttributeInfo.cs` | Add `IsHistorized` property |
| `src/.../GalaxyRepository/GalaxyRepositoryService.cs` | Read `is_historized` column |
| `src/.../OpcUa/LmxNodeManager.cs` | Set `Historizing`/`AccessLevel` for historized nodes; override `HistoryRead` |
| `src/.../Configuration/HistorianConfiguration.cs` | NEW — connection string, query timeout |
| `src/.../Historian/HistorianDataSource.cs` | NEW — SQL queries against Runtime DB |
| `appsettings.json` | Add `Historian` section with connection string |
| `tools/opcuacli-dotnet/Commands/HistoryReadCommand.cs` | NEW — `historyread` CLI command |
| `tools/opcuacli-dotnet/README.md` | Add `historyread` command documentation |

## 6. Configuration

```json
{
  "Historian": {
    "ConnectionString": "Server=localhost;Database=Runtime;Integrated Security=true;",
    "CommandTimeoutSeconds": 30,
    "MaxValuesPerRead": 10000
  }
}
```