R1.1 ExecuteSqlCommandAsync (ExeC + GetR, NRBF DataTable, no BinaryFormatter)

Ship SQL command execution over the 2020 WCF aa/Retr/ExeC + aa/Retr/GetR ops:
HistorianClient.ExecuteSqlCommandAsync(sql) -> HistorianSqlResult (columns +
typed rows). String-handle ops reached with the Open2 storage-session GUID
formatted uppercase (the handle format that unlocked GETRP/GETHI).

Chain: Retr.GetV prime -> ExeC(handle, sql, option=0, ref queryHandle) ->
GetR loop. Key gotcha captured: GetR returns FALSE even on success -- the byte
stream is in pResultBuff regardless; false just signals the final page. So the
orchestrator consumes the buffer first, then stops on a false result / empty page.

GetR's pResultBuff is an NRBF-serialized System.Data.DataTable
(SerializationFormat.Xml: members XmlSchema (XSD) + XmlDiffGram (rows)).
BinaryFormatter is removed from .NET 10, so the stream is decoded read-only with
the System.Formats.Nrbf package (NrbfDecoder) + XDocument -- no BinaryFormatter,
no code execution. Values are typed per the XSD type, falling back to string.

Adds: HistorianSqlResult / HistorianSqlColumn / HistorianSqlExecuteOption models,
HistorianSqlResultProtocol (NRBF + diffgram parser), HistorianWcfSqlClient
(ExeC/GetR orchestration with an AVEVA_HISTORIAN_SQL_DUMP diagnostic), dialect +
public API. Golden WcfSqlResultProtocolTests pinned to the real clean GetR stream
for the benign "SELECT 1 AS ProbeValue" (no sensitive data); gated live tests
(single cell + multi-column/multi-row/NULL). Doc: wcf-exec-sql.md; roadmap R1.1
DONE; wall doc + memory updated (incl. the QTB-server-side nuance). 229 tests green.

Note: a raw instrument-wcf capture corrupts a large pResultBuff with MDAS
transport chunk markers (0x9F); the clean contract-level byte[] is dumped via the
AVEVA_HISTORIAN_SQL_DUMP env var for the golden fixture.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B6mcaT2PjRFKcogzp9UkfC
This commit is contained in:
Joseph Doherty
2026-06-20 23:16:06 -04:00
parent 4da5287d01
commit 1a539882d0
12 changed files with 676 additions and 9 deletions
+1 -1
View File
@@ -90,7 +90,7 @@ read/browse/status surface is Windows-free and the gRPC stack is the default pat
### 1a. Trivial (XSS each, no new payload format)
| ID | Capability | gRPC op | Notes |
|---|---|---|---|
| R1.1 | `ExecuteSqlCommandAsync` | `Retrieval.ExecuteSqlCommand` (`ExeC`+`GetR`) | ✅ **REACHABLE (2026-06-20, live-probed).** The earlier "code 51 blocked" verdict was a handle-**format** bug — `ExeC` succeeds with the Open2 storage GUID sent **uppercase** (`ToString("D").ToUpperInvariant()`). Chain: `Retr.GetV` prime → `ExeC(handle, sqlString, option=0, ref queryHandle)``GetR(handle, queryHandle, ref sequence)` returns the result as a **BinaryFormatter-serialized .NET DataTable**. Proven by `StringHandleProbeDiagnosticTests` + `scripts/Capture-ExecSql.ps1`. **Public API not yet shipped** — needs a `GetR` continuation loop + a custom BinaryFormatter-stream parser (BinaryFormatter is removed from .NET 10, so a DataTable can't just be deserialized). |
| ~~R1.1~~ | ~~`ExecuteSqlCommandAsync`~~ | `Retrieval.ExecuteSqlCommand` (`ExeC`+`GetR`) | ✅ **DONE (2026-06-20), live-verified.** `ExecuteSqlCommandAsync(sql)``HistorianSqlResult` (columns + typed rows). String-handle op via the uppercase storage GUID. Chain: `Retr.GetV` prime → `ExeC(handle, sql, option=0, ref queryHandle)``GetR` loop (note: `GetR` returns **false even on success** — the stream is in `pResultBuff` regardless; false = final page). `GetR`'s `pResultBuff` is an **NRBF-serialized `DataTable`** (`SerializationFormat.Xml`: members `XmlSchema` + `XmlDiffGram`). BinaryFormatter is gone from .NET 10, so it's decoded read-only with `System.Formats.Nrbf` + `XDocument` (no BinaryFormatter). Shipped: `HistorianSqlResult`/`HistorianSqlColumn`/`HistorianSqlExecuteOption`, `HistorianSqlResultProtocol`, `HistorianWcfSqlClient`, golden `WcfSqlResultProtocolTests`, gated live tests. See `docs/reverse-engineering/wcf-exec-sql.md`. |
| ~~R1.2~~ | ~~`GetRuntimeParameterAsync`~~ | `Status.GetRuntimeParameter` (`aa/Stat/GETRP`) | ✅ **DONE (2026-06-20), live-verified.** Captured (`scripts/Capture-RuntimeParam.ps1`): GETRP is a **`string`-handle** op (GETHI's shape), but reachable from the managed client using the Open2 storage-session GUID as an **uppercase** string handle (`ToString("D").ToUpperInvariant()`). Returns `HistorianVersion` = `20,0,000,000` live. pRequestBuff = `54 67 01 00` + uint nameCount + per-name(uint charCount + UTF-16); pResponseBuff = version + uint resultCount + CRetVariant(`0x43` VT_BSTR + uint16 len + uint16 charCount + UTF-16). Single string-valued param only (multi-name framing inferred, not captured). Shipped: `HistorianClient.GetRuntimeParameterAsync(name)`; golden `WcfRuntimeParameterProtocolTests`. **Note:** GETRP punching through the string-handle wall with the uppercase storage GUID is a strong lead that GETHI/ExeC may be a handle-*format* issue — see `wcf-string-handle-wall.md` §Update. |
| ~~R1.3~~ | ~~`GetServerTimeZoneAsync`~~ | `Status.GetSystemTimeZoneName` | ⚠ **gRPC/2023R2-only.** Verified 2026-06-20: over **2020 WCF** this op is a stub (rc=0, empty value) in the `GetServerTime` family — not shippable here. Build+verify only against a live 2023 R2 server. See `docs/reverse-engineering/wcf-status-localhost.md`. |
+86
View File
@@ -0,0 +1,86 @@
# SQL command execution over 2020 WCF — ExeC + GetR (HCAL R1.1)
**Status: ✅ DONE + live-verified (2026-06-20).** `HistorianClient.ExecuteSqlCommandAsync(sql)` runs a
SQL command against the Historian over the 2020 WCF ops `aa/Retr/ExeC` (ExecuteSqlCommand) +
`aa/Retr/GetR` (GetRecordSetByteStream) and returns the record set as a `HistorianSqlResult` (the
managed equivalent of the native `DataTable`). Live-verified end-to-end from the pure-managed .NET 10
client against the local 2020 Historian (single-cell, multi-column/multi-row, and NULL cases).
## The ops
Both are on the Retrieval service (`IRetrievalServiceContract3`), **string-handle** ops reached with
the Open2 storage-session GUID formatted `storageSessionId.ToString("D").ToUpperInvariant()` (the same
uppercase handle that unlocked GETRP/GETHI; see `wcf-string-handle-wall.md`). `Retr.GetV` is primed
first.
```
bool ExecuteSqlCommand(string handle, string command, uint option,
ref uint queryHandle, out int retValue, out uint errorSize, out byte[] errorBuffer)
bool GetRecordSetByteStream(string handle, uint queryHandle, ref uint sequence,
out uint resultSize, out byte[] pResultBuff, out uint errorSize, out byte[] errorBuffer)
```
- **`command`** is sent as a plain string (MDAS-encoded ASCII), e.g. `SELECT 1 AS ProbeValue`.
- **`option`** = `HistorianSqlExecuteOption` (`ExecuteRecord=0` is the captured/proven record-set path).
- **`ExeC`** returns the assigned `queryHandle` (and `retValue`); pass it to `GetR`.
- **`GetR` returns `false` even on success** — the byte stream is in `pResultBuff` regardless; a
`false` result just signals the final page. So the orchestrator always consumes `pResultBuff`, then
stops on a `false` result or an empty page. (`sequence` is the paging cursor; small record sets
return everything in one call.)
## The result stream — NRBF-wrapped DataTable (no BinaryFormatter)
`GetR`'s `pResultBuff` is a **.NET Remoting Binary Format (NRBF)** stream wrapping a
`System.Data.DataTable` serialized with `SerializationFormat.Xml`. Stream shape (captured):
```
SerializationHeader (00 01 00 00 00 FF FF FF FF 01 00 00 00 00 00 00 00)
BinaryLibrary (0C): "System.Data, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089"
ClassWithMembersAndTypes (05): System.Data.DataTable, members:
DataTable.RemotingVersion -> System.Version object
XmlSchema -> BinaryObjectString (XSD: column names + xs types)
XmlDiffGram -> BinaryObjectString (ADO.NET diffgram: the rows)
System.Version object (_Major/_Minor/_Build/_Revision)
MessageEnd (0B)
```
**BinaryFormatter is removed from .NET 10**, so the stream is decoded read-only with
`System.Formats.Nrbf.NrbfDecoder` (the sanctioned successor — it parses records without instantiating
or executing any payload type; added as a managed `PackageReference`). The two embedded XML strings are
then parsed with `XDocument`:
- **XmlSchema** → columns (name + XSD type) under the `msdata:MainDataTable` element's sequence.
- **XmlDiffGram** → rows (each row element under the dataset; cells are child elements or attributes
matching column names). NULL cells are simply absent → parsed as `null`.
Values are typed per the XSD type (`xs:int`→int, `xs:string`→string, `xs:dateTime`→DateTime, …),
falling back to the raw string for unrecognized types. Only the `SerializationFormat.Xml` DataTable
shape is evidence-backed; a stream whose root is not a DataTable class record, or that lacks the two
XML members, throws `ProtocolEvidenceMissingException`.
## Capture / decode tooling
`scripts/Capture-ExecSql.ps1` (NativeTraceHarness `exec-sql` scenario + instrument-wcf-{write,read}message)
captures the ExeC/GetR exchange. ⚠️ A **raw** instrument-wcf capture interleaves MDAS transport chunk
markers (`0x9F`/`0x9E`) into a large `pResultBuff`, so raw byte-slicing yields a corrupted NRBF stream.
The **clean** contract-level byte[] (what the WCF channel reassembles) is dumped via the
`AVEVA_HISTORIAN_SQL_DUMP` env var on `HistorianWcfSqlClient` — that is the golden fixture in
`WcfSqlResultProtocolTests` (the benign `SELECT 1 AS ProbeValue`, no sensitive data).
## Shipped surface
- `HistorianClient.ExecuteSqlCommandAsync(command, option = ExecuteRecord)``HistorianSqlResult`
(`Columns` name+type, `Rows` typed values, `ReturnValue`).
- Models `HistorianSqlResult` / `HistorianSqlColumn` / `HistorianSqlExecuteOption`;
`HistorianSqlResultProtocol` (NRBF + diffgram parser); `HistorianWcfSqlClient` (ExeC/GetR
orchestration); golden `WcfSqlResultProtocolTests`; gated live tests
(`ExecuteSqlCommandAsync_AgainstLocalHistorian_ReturnsRecordSet` and `_MultiColumnMultiRow`).
## Scope notes
- `ExecuteRecord` (record set) is the evidence-backed path. `ExecuteNonQuery`/`ExecuteScalar`/
`ExecuteRecordDirect` are accepted via the option enum but their non-record-set return shapes are not
separately captured — a non-record result yields empty `Columns`/`Rows` with the `ReturnValue` set.
- The command is whatever the Historian's SQL surface accepts (it routes to the Runtime DB). No
client-side SQL validation is performed.
@@ -100,14 +100,24 @@ GETHI probe is the **handle string format**: the native client sends the GUID
lowercase, so a probe that passed the GUID without upcasing would not byte-match what
the server's session table is keyed on.
**Implication / open lead (not yet retested):** the GETHI/ExeC/QTB/QTG family failures
may be (at least partly) a **handle-format** issue, not (only) a missing native
registration step. The highest-value cheap follow-up is to **re-probe GETHI and ExeC
with the uppercased storage-session GUID** before assuming the registration wall. If
they also return data, the "wall" collapses to a formatting bug and R1.4/R1.5/R1.6/R1.1
may be reachable without any new RE. This has **not** been done yet — do not reclassify
those items until it is. GETRP is shipped because it was directly captured + live-verified
end-to-end; the rest remain `ProtocolEvidenceMissingException`/unprobed until tested.
**Implication — CONFIRMED, the wall is largely a handle-format bug.** The follow-up was done:
GETHI and **ExeC both return data with the uppercased storage-session GUID**.
- **R1.1 `ExecuteSqlCommandAsync` (ExeC + GetR) — SHIPPED + live-verified (2026-06-20).**
`ExecuteSqlCommandAsync(sql)``HistorianSqlResult`. `Retr.GetV` prime → `ExeC(handle,
sql, option=0, ref queryHandle)``GetR` loop. Note: **`GetR` returns false even on
success** (the byte stream is in `pResultBuff` regardless; false = final page). `pResultBuff`
is an **NRBF `DataTable`** (`SerializationFormat.Xml`: `XmlSchema` + `XmlDiffGram`), decoded
read-only with `System.Formats.Nrbf` + `XDocument` (BinaryFormatter is gone from .NET 10).
Shipped: `HistorianSqlResultProtocol`, `HistorianWcfSqlClient`, golden `WcfSqlResultProtocolTests`,
gated live tests. See `docs/reverse-engineering/wcf-exec-sql.md`.
- **GETHI (R1.4)** also returns data with the uppercase handle (probe; public API not yet shipped).
So the "wall" collapses to the handle **format** for the Retrieval/Status string-handle ops.
**Exception — QTB/QTG:** `StartTagQuery` does *not* punch through; captured with a correctly
uppercase handle it still fails `error 1` **server-side** (`CMdServer::StartActiveTagnamesQuery`
over `\\.\pipe\aahMetadataServer\console`) — a metadata-server blocker, independent of handle
format. Name-based ops route around it.
See `HistorianRuntimeParameterProtocol`, `IStatusServiceContract2.GetRuntimeParameter`,
golden `WcfRuntimeParameterProtocolTests`, and capture tooling
@@ -7,6 +7,7 @@
</PropertyGroup>
<ItemGroup>
<PackageReference Include="System.Formats.Nrbf" Version="10.0.7" />
<PackageReference Include="System.Security.Cryptography.Xml" Version="10.0.7" />
<PackageReference Include="System.ServiceModel.NetNamedPipe" Version="10.0.652802" />
<PackageReference Include="System.ServiceModel.NetTcp" Version="10.0.652802" />
@@ -166,6 +166,22 @@ public sealed class HistorianClient : IAsyncDisposable
return _protocol.GetRuntimeParameterAsync(name, cancellationToken);
}
/// <summary>
/// Executes a SQL command against the Historian over the WCF <c>ExeC</c>/<c>GetR</c> ops and
/// returns the record set as a <see cref="HistorianSqlResult"/> (the managed equivalent of the
/// native <c>DataTable</c>). The record-set path (<see cref="HistorianSqlExecuteOption.ExecuteRecord"/>,
/// the default) is the evidence-backed surface; the result is decoded from the server's
/// NRBF-serialized DataTable without BinaryFormatter. See <c>HistorianSqlResultProtocol</c>.
/// </summary>
public Task<HistorianSqlResult> ExecuteSqlCommandAsync(
string command,
HistorianSqlExecuteOption option = HistorianSqlExecuteOption.ExecuteRecord,
CancellationToken cancellationToken = default)
{
ArgumentException.ThrowIfNullOrWhiteSpace(command);
return _protocol.ExecuteSqlCommandAsync(command, option, cancellationToken);
}
/// <summary>
/// Creates or updates the named tag in the Historian Runtime database via
/// <c>EnsureTags2</c>. Currently only <see cref="HistorianDataType.Float"/> is
@@ -0,0 +1,21 @@
namespace AVEVA.Historian.Client.Models;
/// <summary>
/// Execution mode for <c>ExecuteSqlCommandAsync</c>, mirroring the native
/// <c>HistorianSqlExecuteOption</c> enum passed to the <c>ExeC</c> op. Only
/// <see cref="ExecuteRecord"/> (the record-set path) is evidence-backed end-to-end.
/// </summary>
public enum HistorianSqlExecuteOption
{
/// <summary>Execute and return a record set (the captured/proven path).</summary>
ExecuteRecord = 0,
/// <summary>Execute without returning a record set; the result carries only the return value.</summary>
ExecuteNonQuery = 1,
/// <summary>Execute and return a single scalar value.</summary>
ExecuteScalar = 2,
/// <summary>Execute a record set directly (server-side direct path).</summary>
ExecuteRecordDirect = 3,
}
@@ -0,0 +1,35 @@
namespace AVEVA.Historian.Client.Models;
/// <summary>
/// A column in a <see cref="HistorianSqlResult"/>: its name and the XSD type from the result-set
/// schema (e.g. <c>xs:int</c>, <c>xs:string</c>, <c>xs:dateTime</c>).
/// </summary>
public sealed record HistorianSqlColumn(string Name, string SchemaType);
/// <summary>
/// The result of <c>ExecuteSqlCommandAsync</c> — the managed equivalent of the <c>DataTable</c>
/// the native client returns. Columns carry name + XSD type; each row is a list of values aligned
/// to <see cref="Columns"/> (typed per the schema where the XSD type is recognized, otherwise the
/// raw string; <c>null</c> for a SQL NULL / absent cell).
/// </summary>
public sealed class HistorianSqlResult
{
public HistorianSqlResult(
IReadOnlyList<HistorianSqlColumn> columns,
IReadOnlyList<IReadOnlyList<object?>> rows,
int returnValue)
{
Columns = columns;
Rows = rows;
ReturnValue = returnValue;
}
/// <summary>The result-set columns, in order.</summary>
public IReadOnlyList<HistorianSqlColumn> Columns { get; }
/// <summary>The rows; each row's values align positionally to <see cref="Columns"/>.</summary>
public IReadOnlyList<IReadOnlyList<object?>> Rows { get; }
/// <summary>The native <c>retValue</c> from <c>ExeC</c> (e.g. rows affected for non-queries).</summary>
public int ReturnValue { get; }
}
@@ -74,6 +74,13 @@ internal sealed class Historian2020ProtocolDialect
return Wcf.HistorianWcfStatusClient.GetRuntimeParameterAsync(_options, name, cancellationToken);
}
public Task<HistorianSqlResult> ExecuteSqlCommandAsync(string command, HistorianSqlExecuteOption option, CancellationToken cancellationToken)
{
cancellationToken.ThrowIfCancellationRequested();
ArgumentException.ThrowIfNullOrWhiteSpace(command);
return Wcf.HistorianWcfSqlClient.ExecuteSqlCommandAsync(_options, command, option, cancellationToken);
}
private static async IAsyncEnumerable<T> Missing<T>(
string operation,
[System.Runtime.CompilerServices.EnumeratorCancellation] CancellationToken cancellationToken)
@@ -0,0 +1,221 @@
using System.Formats.Nrbf;
using System.Globalization;
using System.Xml.Linq;
using AVEVA.Historian.Client.Models;
using AVEVA.Historian.Client.Protocol;
namespace AVEVA.Historian.Client.Wcf;
/// <summary>
/// Parses the <c>GetR</c> (GetRecordSetByteStream) result buffer returned by the 2020 WCF
/// <c>ExeC</c>/<c>GetR</c> SQL surface into a <see cref="HistorianSqlResult"/>.
/// </summary>
/// <remarks>
/// Captured from the native client (<c>scripts/Capture-ExecSql.ps1</c> +
/// instrument-wcf-{write,read}message; golden-pinned in <c>WcfSqlResultProtocolTests</c>). The
/// <c>GetR</c> <c>pResultBuff</c> is a <b>.NET Remoting Binary Format (NRBF)</b> stream wrapping a
/// <c>System.Data.DataTable</c> serialized with <c>SerializationFormat.Xml</c>: its data lives in
/// two string members — <c>XmlSchema</c> (an XSD describing the columns + types) and
/// <c>XmlDiffGram</c> (an ADO.NET diffgram carrying the rows).
///
/// <para>BinaryFormatter is removed from .NET 10, so the stream is decoded read-only with
/// <see cref="NrbfDecoder"/> (the sanctioned successor — it parses records without instantiating
/// or executing any payload type). The two embedded XML strings are then parsed with
/// <see cref="XDocument"/>.</para>
///
/// <para>Only the <c>SerializationFormat.Xml</c> DataTable shape is evidence-backed; a stream whose
/// root is not a <c>DataTable</c> class record, or that lacks the two XML members, throws
/// <see cref="ProtocolEvidenceMissingException"/>.</para>
/// </remarks>
internal static class HistorianSqlResultProtocol
{
private const string DataTableTypeName = "System.Data.DataTable";
private const string XmlSchemaMember = "XmlSchema";
private const string XmlDiffGramMember = "XmlDiffGram";
private static readonly XNamespace Xsd = "http://www.w3.org/2001/XMLSchema";
private static readonly XNamespace MsData = "urn:schemas-microsoft-com:xml-msdata";
private static readonly XNamespace DiffGr = "urn:schemas-microsoft-com:xml-diffgram-v1";
public static HistorianSqlResult Parse(ReadOnlyMemory<byte> resultBuffer, int returnValue)
{
(string? schemaXml, string? diffGramXml) = ExtractXmlMembers(resultBuffer);
if (string.IsNullOrEmpty(schemaXml))
{
// No record set (e.g. a non-query, or an empty stream): return columns/rows empty.
return new HistorianSqlResult([], [], returnValue);
}
(string? tableName, List<HistorianSqlColumn> columns) = ParseSchema(schemaXml);
IReadOnlyList<IReadOnlyList<object?>> rows = ParseDiffGram(diffGramXml, tableName, columns);
return new HistorianSqlResult(columns, rows, returnValue);
}
private static (string? schemaXml, string? diffGramXml) ExtractXmlMembers(ReadOnlyMemory<byte> resultBuffer)
{
if (resultBuffer.Length == 0)
{
return (null, null);
}
SerializationRecord root;
try
{
using MemoryStream stream = new(resultBuffer.ToArray(), writable: false);
root = NrbfDecoder.Decode(stream);
}
catch (Exception ex) when (ex is not ProtocolEvidenceMissingException)
{
throw new ProtocolEvidenceMissingException(
$"GetR result buffer is not a decodable NRBF stream: {ex.Message}");
}
if (root is not ClassRecord classRecord || !classRecord.TypeNameMatches(typeof(System.Data.DataTable)))
{
// TypeNameMatches needs the real type; fall back to the raw type name when the assembly
// identity differs from the local one.
if (root is not ClassRecord cr || !string.Equals(cr.TypeName.FullName, DataTableTypeName, StringComparison.Ordinal))
{
throw new ProtocolEvidenceMissingException(
$"GetR result root record is not a {DataTableTypeName} (got '{(root as ClassRecord)?.TypeName.FullName ?? root.GetType().Name}').");
}
classRecord = cr;
}
if (!classRecord.HasMember(XmlSchemaMember) || !classRecord.HasMember(XmlDiffGramMember))
{
throw new ProtocolEvidenceMissingException(
$"GetR DataTable record is missing the {XmlSchemaMember}/{XmlDiffGramMember} members (non-Xml SerializationFormat not supported).");
}
return (classRecord.GetString(XmlSchemaMember), classRecord.GetString(XmlDiffGramMember));
}
private static (string? tableName, List<HistorianSqlColumn> columns) ParseSchema(string schemaXml)
{
XDocument doc = XDocument.Parse(schemaXml);
XElement? schema = doc.Root;
List<HistorianSqlColumn> columns = [];
if (schema is null)
{
return (null, columns);
}
// The dataset element advertises the main data table via msdata:MainDataTable.
XElement? dataSet = schema.Elements(Xsd + "element")
.FirstOrDefault(e => (string?)e.Attribute(MsData + "IsDataSet") == "true");
string? mainTable = (string?)dataSet?.Attribute(MsData + "MainDataTable");
// The table element (named by MainDataTable, else the first non-dataset top-level element).
XElement? tableElement = schema.Elements(Xsd + "element")
.FirstOrDefault(e => mainTable is not null && (string?)e.Attribute("name") == mainTable)
?? schema.Elements(Xsd + "element").FirstOrDefault(e => (string?)e.Attribute(MsData + "IsDataSet") != "true");
string? tableName = (string?)tableElement?.Attribute("name");
// Columns are the xs:element children of the table's complexType/sequence.
IEnumerable<XElement> columnElements = tableElement
?.Descendants(Xsd + "element")
.Where(e => e.Attribute("name") is not null)
?? [];
foreach (XElement column in columnElements)
{
string name = (string)column.Attribute("name")!;
string type = (string?)column.Attribute("type") ?? InlineSimpleType(column) ?? "xs:string";
columns.Add(new HistorianSqlColumn(name, type));
}
return (tableName, columns);
}
private static string? InlineSimpleType(XElement column)
{
XElement? restriction = column.Element(Xsd + "simpleType")?.Element(Xsd + "restriction");
return (string?)restriction?.Attribute("base");
}
private static IReadOnlyList<IReadOnlyList<object?>> ParseDiffGram(
string? diffGramXml,
string? tableName,
List<HistorianSqlColumn> columns)
{
if (string.IsNullOrEmpty(diffGramXml) || columns.Count == 0)
{
return [];
}
XDocument doc = XDocument.Parse(diffGramXml);
XElement? diffgram = doc.Root;
if (diffgram is null)
{
return [];
}
// The dataset element is the diffgram's child that is not in the diffgr: namespace
// (diffgr:before / diffgr:errors are change-tracking sections we don't need for a SELECT).
XElement? dataSet = diffgram.Elements().FirstOrDefault(e => e.Name.Namespace != DiffGr);
if (dataSet is null)
{
return [];
}
// Row elements are the dataset's children whose local name matches the table (or any child
// when the table name is unknown).
IEnumerable<XElement> rowElements = dataSet.Elements()
.Where(e => tableName is null || e.Name.LocalName == tableName);
List<IReadOnlyList<object?>> rows = [];
foreach (XElement rowElement in rowElements)
{
object?[] values = new object?[columns.Count];
for (int i = 0; i < columns.Count; i++)
{
HistorianSqlColumn column = columns[i];
// ADO.NET maps a column either as a child element or as an attribute on the row.
XElement? cell = rowElement.Elements().FirstOrDefault(e => e.Name.LocalName == column.Name);
string? raw = cell?.Value
?? rowElement.Attributes().FirstOrDefault(a => a.Name.LocalName == column.Name)?.Value;
values[i] = raw is null ? null : ConvertValue(raw, column.SchemaType);
}
rows.Add(values);
}
return rows;
}
private static object ConvertValue(string raw, string schemaType)
{
string type = schemaType.Contains(':') ? schemaType[(schemaType.IndexOf(':') + 1)..] : schemaType;
try
{
return type switch
{
"int" or "integer" => int.Parse(raw, CultureInfo.InvariantCulture),
"long" => long.Parse(raw, CultureInfo.InvariantCulture),
"short" => short.Parse(raw, CultureInfo.InvariantCulture),
"byte" => sbyte.Parse(raw, CultureInfo.InvariantCulture),
"unsignedByte" => byte.Parse(raw, CultureInfo.InvariantCulture),
"unsignedShort" => ushort.Parse(raw, CultureInfo.InvariantCulture),
"unsignedInt" => uint.Parse(raw, CultureInfo.InvariantCulture),
"unsignedLong" => ulong.Parse(raw, CultureInfo.InvariantCulture),
"double" => double.Parse(raw, CultureInfo.InvariantCulture),
"float" => float.Parse(raw, CultureInfo.InvariantCulture),
"decimal" => decimal.Parse(raw, CultureInfo.InvariantCulture),
"boolean" => ParseXsdBoolean(raw),
"dateTime" => DateTime.Parse(raw, CultureInfo.InvariantCulture, DateTimeStyles.RoundtripKind),
"base64Binary" => Convert.FromBase64String(raw),
_ => raw,
};
}
catch (Exception ex) when (ex is FormatException or OverflowException)
{
// Keep the raw string rather than throwing — the column type is informational and the
// server's textual rendering is always available.
return raw;
}
}
private static bool ParseXsdBoolean(string raw) =>
raw is "true" or "1" || (raw is not ("false" or "0") && bool.Parse(raw));
}
@@ -0,0 +1,132 @@
using System.ServiceModel;
using System.ServiceModel.Channels;
using AVEVA.Historian.Client.Models;
using AVEVA.Historian.Client.Wcf.Contracts;
namespace AVEVA.Historian.Client.Wcf;
/// <summary>
/// Executes SQL commands (HCAL R1.1) over the 2020 WCF <c>aa/Retr/ExeC</c> + <c>aa/Retr/GetR</c>
/// ops. Both are string-handle ops reached with the Open2 storage-session GUID formatted UPPERCASE
/// (the same handle format that unlocked GETRP/GETHI; see <c>wcf-string-handle-wall.md</c>). The
/// record set comes back from <c>GetR</c> as an NRBF-serialized <c>DataTable</c>, parsed by
/// <see cref="HistorianSqlResultProtocol"/>.
/// </summary>
internal static class HistorianWcfSqlClient
{
// GetR is byte-stream-paged. A small record set returns the whole stream in the first call;
// larger ones chunk across calls accumulated here. The cap is a runaway guard.
private const int MaxPages = 4096;
/// <summary>Diagnostic: the ExeC/GetR error from the last call (set only on server rejection).</summary>
public static string? LastError { get; private set; }
public static Task<HistorianSqlResult> ExecuteSqlCommandAsync(
HistorianClientOptions options,
string command,
HistorianSqlExecuteOption option,
CancellationToken cancellationToken)
{
ArgumentException.ThrowIfNullOrWhiteSpace(command);
return Task.Run(() => ExecuteSqlCommand(options, command, option), cancellationToken);
}
private static HistorianSqlResult ExecuteSqlCommand(
HistorianClientOptions options,
string command,
HistorianSqlExecuteOption option)
{
Guid contextKey = Guid.NewGuid();
var (histBinding, histEndpoint, retrBinding, retrEndpoint) = HistorianWcfBindingFactory.CreateBindingPair(options);
LastError = null;
HistorianSqlResult? result = null;
HistorianWcfAuthChainHelper.OpenAuthenticatedConnection(
options, histBinding, histEndpoint, contextKey, CancellationToken.None,
additionalSetup: (_, context) =>
result = RunSql(retrBinding, retrEndpoint, options, context.StorageSessionId, command, option));
return result ?? new HistorianSqlResult([], [], 0);
}
private static HistorianSqlResult RunSql(
Binding retrBinding,
EndpointAddress retrEndpoint,
HistorianClientOptions options,
Guid storageSessionId,
string command,
HistorianSqlExecuteOption option)
{
// ExeC/GetR take the storage-session GUID as a string handle, uppercase dash-no-braces.
string handle = storageSessionId.ToString("D").ToUpperInvariant();
ChannelFactory<IRetrievalServiceContract3> factory = new(retrBinding, retrEndpoint);
HistorianWcfClientCredentialsHelper.Configure(factory, options);
IRetrievalServiceContract3 channel = factory.CreateChannel();
ICommunicationObject co = (ICommunicationObject)channel;
try
{
// Prime the Retrieval service version handshake (Retr.GetV).
channel.GetInterfaceVersion(out _);
uint queryHandle = 0;
bool execOk = channel.ExecuteSqlCommand(
handle, command, (uint)option, ref queryHandle,
out int returnValue, out uint errSize, out byte[] errBuf);
if (!execOk)
{
LastError = $"ExeC returned false (retValue={returnValue}, errSize={errSize}, errLen={errBuf?.Length ?? 0}).";
return new HistorianSqlResult([], [], returnValue);
}
using MemoryStream accumulated = new();
uint sequence = 0;
for (int page = 0; page < MaxPages; page++)
{
bool getrOk = channel.GetRecordSetByteStream(
handle, queryHandle, ref sequence,
out uint resultSize, out byte[] resultBuffer, out uint gErrSize, out byte[] gErrBuf);
// GetRResult is false even on a successful single-page read — the byte stream is in
// resultBuffer regardless, and false signals "this is the final page". So always
// consume the buffer first, then stop on a false result or an empty page.
if (resultBuffer is { Length: > 0 })
{
accumulated.Write(resultBuffer, 0, resultBuffer.Length);
}
if (!getrOk || resultBuffer is null || resultBuffer.Length == 0)
{
if (page == 0 && accumulated.Length == 0 && gErrBuf is { Length: > 0 })
{
LastError = $"GetR error (resultSize={resultSize}, errSize={gErrSize}, errLen={gErrBuf.Length}).";
}
break;
}
}
byte[] stream = accumulated.ToArray();
DumpStreamIfRequested(stream);
return HistorianSqlResultProtocol.Parse(stream, returnValue);
}
finally
{
try { if (co.State == CommunicationState.Faulted) co.Abort(); else co.Close(); } catch { try { co.Abort(); } catch { } }
try { if (factory.State == CommunicationState.Faulted) factory.Abort(); else factory.Close(); } catch { try { factory.Abort(); } catch { } }
}
}
/// <summary>
/// Reverse-engineering aid: when <c>AVEVA_HISTORIAN_SQL_DUMP</c> is set to a path, writes the
/// base64 of the reassembled GetR NRBF stream there (the clean contract-level byte[], free of
/// the MDAS transport chunk markers that corrupt a raw instrument-wcf capture). Used once to
/// produce the golden fixture for <c>WcfSqlResultProtocolTests</c>.
/// </summary>
private static void DumpStreamIfRequested(byte[] stream)
{
string? path = Environment.GetEnvironmentVariable("AVEVA_HISTORIAN_SQL_DUMP");
if (!string.IsNullOrWhiteSpace(path) && stream.Length > 0)
{
try { File.WriteAllText(path, Convert.ToBase64String(stream)); } catch { /* diagnostic only */ }
}
}
}
@@ -296,6 +296,65 @@ public sealed class HistorianClientIntegrationTests
Assert.False(string.IsNullOrWhiteSpace(value));
}
[Fact]
public async Task ExecuteSqlCommandAsync_AgainstLocalHistorian_ReturnsRecordSet()
{
string? host = Environment.GetEnvironmentVariable("HISTORIAN_HOST");
if (string.IsNullOrWhiteSpace(host) || !string.Equals(host, "localhost", StringComparison.OrdinalIgnoreCase) || !OperatingSystem.IsWindows())
{
return;
}
HistorianClient client = new(new HistorianClientOptions
{
Host = host,
IntegratedSecurity = true,
Transport = HistorianTransport.LocalPipe
});
// ExeC/GetR ride the storage-session GUID as an uppercase string handle. A constant SELECT
// returns a single int column; the DataTable is decoded from the NRBF stream (no BinaryFormatter).
AVEVA.Historian.Client.Models.HistorianSqlResult result =
await client.ExecuteSqlCommandAsync("SELECT 1 AS ProbeValue", cancellationToken: CancellationToken.None);
AVEVA.Historian.Client.Models.HistorianSqlColumn column = Assert.Single(result.Columns);
Assert.Equal("ProbeValue", column.Name);
IReadOnlyList<object?> row = Assert.Single(result.Rows);
Assert.Equal(1, Assert.Single(row));
}
[Fact]
public async Task ExecuteSqlCommandAsync_AgainstLocalHistorian_MultiColumnMultiRow()
{
string? host = Environment.GetEnvironmentVariable("HISTORIAN_HOST");
if (string.IsNullOrWhiteSpace(host) || !string.Equals(host, "localhost", StringComparison.OrdinalIgnoreCase) || !OperatingSystem.IsWindows())
{
return;
}
HistorianClient client = new(new HistorianClientOptions
{
Host = host,
IntegratedSecurity = true,
Transport = HistorianTransport.LocalPipe
});
// Fully synthetic query (no server data): two columns (int + string), two rows, one NULL —
// exercises the schema/diffgram parser beyond the single-cell case.
AVEVA.Historian.Client.Models.HistorianSqlResult result = await client.ExecuteSqlCommandAsync(
"SELECT 10 AS Num, 'alpha' AS Word UNION ALL SELECT 20, NULL",
cancellationToken: CancellationToken.None);
Assert.Equal(2, result.Columns.Count);
Assert.Equal("Num", result.Columns[0].Name);
Assert.Equal("Word", result.Columns[1].Name);
Assert.Equal(2, result.Rows.Count);
Assert.Equal(10, result.Rows[0][0]);
Assert.Equal("alpha", result.Rows[0][1]);
Assert.Equal(20, result.Rows[1][0]);
Assert.Null(result.Rows[1][1]);
}
[Fact]
public async Task GetConnectionStatusAsync_AgainstLocalHistorian_ReportsConnectedToServer()
{
@@ -0,0 +1,79 @@
using AVEVA.Historian.Client.Models;
using AVEVA.Historian.Client.Protocol;
using AVEVA.Historian.Client.Wcf;
namespace AVEVA.Historian.Client.Tests;
/// <summary>
/// Golden tests for the ExeC/GetR SQL result codec. The fixture is the real, clean GetR
/// <c>pResultBuff</c> captured from the live server for the benign query
/// <c>SELECT 1 AS ProbeValue</c> (no sensitive data) — an NRBF stream wrapping a
/// <c>System.Data.DataTable</c> serialized with <c>SerializationFormat.Xml</c>. It is the
/// contract-level byte[] the WCF channel reassembles (free of MDAS transport chunk markers).
/// </summary>
public sealed class WcfSqlResultProtocolTests
{
// GetR pResultBuff for "SELECT 1 AS ProbeValue" (1232 bytes), base64. Captured via the
// AVEVA_HISTORIAN_SQL_DUMP diagnostic in HistorianWcfSqlClient.
private const string GetRStreamBase64 =
"AAEAAAD/////AQAAAAAAAAAMAgAAAE5TeXN0ZW0uRGF0YSwgVmVyc2lvbj00LjAuMC4wLCBDdWx0dXJlPW5ldXRyYWwsIFB1Ymxp" +
"Y0tleVRva2VuPWI3N2E1YzU2MTkzNGUwODkFAQAAABVTeXN0ZW0uRGF0YS5EYXRhVGFibGUDAAAAGURhdGFUYWJsZS5SZW1vdGlu" +
"Z1ZlcnNpb24JWG1sU2NoZW1hC1htbERpZmZHcmFtAwEBDlN5c3RlbS5WZXJzaW9uAgAAAAkDAAAABgQAAACJBTw/eG1sIHZlcnNp" +
"b249IjEuMCIgZW5jb2Rpbmc9InV0Zi0xNiI/Pg0KPHhzOnNjaGVtYSB4bWxucz0iIiB4bWxuczp4cz0iaHR0cDovL3d3dy53My5v" +
"cmcvMjAwMS9YTUxTY2hlbWEiIHhtbG5zOm1zZGF0YT0idXJuOnNjaGVtYXMtbWljcm9zb2Z0LWNvbTp4bWwtbXNkYXRhIj4NCiAg" +
"PHhzOmVsZW1lbnQgbmFtZT0iVGFibGUxIj4NCiAgICA8eHM6Y29tcGxleFR5cGU+DQogICAgICA8eHM6c2VxdWVuY2U+DQogICAg" +
"ICAgIDx4czplbGVtZW50IG5hbWU9IlByb2JlVmFsdWUiIG1zZGF0YTpSZWFkT25seT0idHJ1ZSIgdHlwZT0ieHM6aW50IiBtc2Rh" +
"dGE6dGFyZ2V0TmFtZXNwYWNlPSIiIC8+DQogICAgICA8L3hzOnNlcXVlbmNlPg0KICAgIDwveHM6Y29tcGxleFR5cGU+DQogIDwv" +
"eHM6ZWxlbWVudD4NCiAgPHhzOmVsZW1lbnQgbmFtZT0idG1wRGF0YVNldCIgbXNkYXRhOklzRGF0YVNldD0idHJ1ZSIgbXNkYXRh" +
"Ok1haW5EYXRhVGFibGU9IlRhYmxlMSIgbXNkYXRhOlVzZUN1cnJlbnRMb2NhbGU9InRydWUiPg0KICAgIDx4czpjb21wbGV4VHlw" +
"ZT4NCiAgICAgIDx4czpjaG9pY2UgbWluT2NjdXJzPSIwIiBtYXhPY2N1cnM9InVuYm91bmRlZCIgLz4NCiAgICA8L3hzOmNvbXBs" +
"ZXhUeXBlPg0KICA8L3hzOmVsZW1lbnQ+DQo8L3hzOnNjaGVtYT4GBQAAAJoCPGRpZmZncjpkaWZmZ3JhbSB4bWxuczptc2RhdGE9" +
"InVybjpzY2hlbWFzLW1pY3Jvc29mdC1jb206eG1sLW1zZGF0YSIgeG1sbnM6ZGlmZmdyPSJ1cm46c2NoZW1hcy1taWNyb3NvZnQt" +
"Y29tOnhtbC1kaWZmZ3JhbS12MSI+DQogIDx0bXBEYXRhU2V0Pg0KICAgIDxUYWJsZTEgZGlmZmdyOmlkPSJUYWJsZTExIiBtc2Rh" +
"dGE6cm93T3JkZXI9IjAiPg0KICAgICAgPFByb2JlVmFsdWU+MTwvUHJvYmVWYWx1ZT4NCiAgICA8L1RhYmxlMT4NCiAgPC90bXBE" +
"YXRhU2V0Pg0KPC9kaWZmZ3I6ZGlmZmdyYW0+BAMAAAAOU3lzdGVtLlZlcnNpb24EAAAABl9NYWpvcgZfTWlub3IGX0J1aWxkCV9S" +
"ZXZpc2lvbgAAAAAICAgIAgAAAAAAAAD//////////ws=";
[Fact]
public void Parse_DataTableStream_ExtractsColumnAndTypedRow()
{
byte[] stream = Convert.FromBase64String(GetRStreamBase64);
HistorianSqlResult result = HistorianSqlResultProtocol.Parse(stream, returnValue: 0);
HistorianSqlColumn column = Assert.Single(result.Columns);
Assert.Equal("ProbeValue", column.Name);
Assert.Equal("xs:int", column.SchemaType);
IReadOnlyList<object?> row = Assert.Single(result.Rows);
object? value = Assert.Single(row);
Assert.Equal(1, value); // typed to int per the xs:int schema
Assert.IsType<int>(value);
Assert.Equal(0, result.ReturnValue);
}
[Fact]
public void Parse_PreservesReturnValue()
{
byte[] stream = Convert.FromBase64String(GetRStreamBase64);
HistorianSqlResult result = HistorianSqlResultProtocol.Parse(stream, returnValue: 7);
Assert.Equal(7, result.ReturnValue);
}
[Fact]
public void Parse_EmptyBuffer_ReturnsEmptyResult()
{
HistorianSqlResult result = HistorianSqlResultProtocol.Parse(ReadOnlyMemory<byte>.Empty, returnValue: 3);
Assert.Empty(result.Columns);
Assert.Empty(result.Rows);
Assert.Equal(3, result.ReturnValue);
}
[Fact]
public void Parse_NonNrbfBuffer_Throws()
{
byte[] garbage = [0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, 0x08];
Assert.Throws<ProtocolEvidenceMissingException>(
() => HistorianSqlResultProtocol.Parse(garbage, returnValue: 0));
}
}