histsdk/docs/reverse-engineering/wcf-exec-sql.md

# SQL command execution over 2020 WCF — ExeC + GetR (HCAL R1.1)

**Status: ✅ DONE + live-verified (2026-06-20).** `HistorianClient.ExecuteSqlCommandAsync(sql)` runs a
SQL command against the Historian over the 2020 WCF ops `aa/Retr/ExeC` (ExecuteSqlCommand) +
`aa/Retr/GetR` (GetRecordSetByteStream) and returns the record set as a `HistorianSqlResult` (the
managed equivalent of the native `DataTable`). Live-verified end-to-end from the pure-managed .NET 10
client against the local 2020 Historian (single-cell, multi-column/multi-row, and NULL cases).

## The ops

Both are on the Retrieval service (`IRetrievalServiceContract3`), **string-handle** ops reached with
the Open2 storage-session GUID formatted `storageSessionId.ToString("D").ToUpperInvariant()` (the same
uppercase handle that unlocked GETRP/GETHI; see `wcf-string-handle-wall.md`). `Retr.GetV` is primed
first.

```
bool ExecuteSqlCommand(string handle, string command, uint option,
                       ref uint queryHandle, out int retValue, out uint errorSize, out byte[] errorBuffer)

bool GetRecordSetByteStream(string handle, uint queryHandle, ref uint sequence,
                            out uint resultSize, out byte[] pResultBuff, out uint errorSize, out byte[] errorBuffer)
```

- **`command`** is sent as a plain string (MDAS-encoded ASCII), e.g. `SELECT 1 AS ProbeValue`.
- **`option`** = `HistorianSqlExecuteOption` (`ExecuteRecord=0` is the captured/proven record-set path).
- **`ExeC`** returns the assigned `queryHandle` (and `retValue`); pass it to `GetR`.
- **`GetR` returns `false` even on success** — the byte stream is in `pResultBuff` regardless; a
  `false` result just signals the final page. So the orchestrator always consumes `pResultBuff`, then
  stops on a `false` result or an empty page. (`sequence` is the paging cursor; small record sets
  return everything in one call.)

## The result stream — NRBF-wrapped DataTable (no BinaryFormatter)

`GetR`'s `pResultBuff` is a **.NET Remoting Binary Format (NRBF)** stream wrapping a
`System.Data.DataTable` serialized with `SerializationFormat.Xml`. Stream shape (captured):

```
SerializationHeader (00 01 00 00 00 FF FF FF FF 01 00 00 00 00 00 00 00)
BinaryLibrary (0C): "System.Data, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089"
ClassWithMembersAndTypes (05): System.Data.DataTable, members:
    DataTable.RemotingVersion  -> System.Version object
    XmlSchema                  -> BinaryObjectString (XSD: column names + xs types)
    XmlDiffGram                -> BinaryObjectString (ADO.NET diffgram: the rows)
System.Version object (_Major/_Minor/_Build/_Revision)
MessageEnd (0B)
```

**BinaryFormatter is removed from .NET 10**, so the stream is decoded read-only with
`System.Formats.Nrbf.NrbfDecoder` (the sanctioned successor — it parses records without instantiating
or executing any payload type; added as a managed `PackageReference`). The two embedded XML strings are
then parsed with `XDocument`:

- **XmlSchema** → columns (name + XSD type) under the `msdata:MainDataTable` element's sequence.
- **XmlDiffGram** → rows (each row element under the dataset; cells are child elements or attributes
  matching column names). NULL cells are simply absent → parsed as `null`.

Values are typed per the XSD type (`xs:int`→int, `xs:string`→string, `xs:dateTime`→DateTime, …),
falling back to the raw string for unrecognized types. Only the `SerializationFormat.Xml` DataTable
shape is evidence-backed; a stream whose root is not a DataTable class record, or that lacks the two
XML members, throws `ProtocolEvidenceMissingException`.

## Capture / decode tooling

`scripts/Capture-ExecSql.ps1` (NativeTraceHarness `exec-sql` scenario + instrument-wcf-{write,read}message)
captures the ExeC/GetR exchange. ⚠️ A **raw** instrument-wcf capture interleaves MDAS transport chunk
markers (`0x9F`/`0x9E`) into a large `pResultBuff`, so raw byte-slicing yields a corrupted NRBF stream.
The **clean** contract-level byte[] (what the WCF channel reassembles) is dumped via the
`AVEVA_HISTORIAN_SQL_DUMP` env var on `HistorianWcfSqlClient` — that is the golden fixture in
`WcfSqlResultProtocolTests` (the benign `SELECT 1 AS ProbeValue`, no sensitive data).

## Shipped surface

- `HistorianClient.ExecuteSqlCommandAsync(command, option = ExecuteRecord)` → `HistorianSqlResult`
  (`Columns` name+type, `Rows` typed values, `ReturnValue`).
- Models `HistorianSqlResult` / `HistorianSqlColumn` / `HistorianSqlExecuteOption`;
  `HistorianSqlResultProtocol` (NRBF + diffgram parser); `HistorianWcfSqlClient` (ExeC/GetR
  orchestration); golden `WcfSqlResultProtocolTests`; gated live tests
  (`ExecuteSqlCommandAsync_AgainstLocalHistorian_ReturnsRecordSet` and `_MultiColumnMultiRow`).

## Scope notes

- `ExecuteRecord` (record set) is the evidence-backed path. `ExecuteNonQuery`/`ExecuteScalar`/
  `ExecuteRecordDirect` are accepted via the option enum but their non-record-set return shapes are not
  separately captured — a non-record result yields empty `Columns`/`Rows` with the `ReturnValue` set.
- The command is whatever the Historian's SQL surface accepts (it routes to the Runtime DB). No
  client-side SQL validation is performed.