diff --git a/docs/plans/hcal-roadmap.md b/docs/plans/hcal-roadmap.md index 9c8558c..57f3e84 100644 --- a/docs/plans/hcal-roadmap.md +++ b/docs/plans/hcal-roadmap.md @@ -90,7 +90,7 @@ read/browse/status surface is Windows-free and the gRPC stack is the default pat ### 1a. Trivial (XS–S each, no new payload format) | ID | Capability | gRPC op | Notes | |---|---|---|---| -| R1.1 | `ExecuteSqlCommandAsync` | `Retrieval.ExecuteSqlCommand` (`ExeC`+`GetR`) | ✅ **REACHABLE (2026-06-20, live-probed).** The earlier "code 51 blocked" verdict was a handle-**format** bug — `ExeC` succeeds with the Open2 storage GUID sent **uppercase** (`ToString("D").ToUpperInvariant()`). Chain: `Retr.GetV` prime → `ExeC(handle, sqlString, option=0, ref queryHandle)` → `GetR(handle, queryHandle, ref sequence)` returns the result as a **BinaryFormatter-serialized .NET DataTable**. Proven by `StringHandleProbeDiagnosticTests` + `scripts/Capture-ExecSql.ps1`. **Public API not yet shipped** — needs a `GetR` continuation loop + a custom BinaryFormatter-stream parser (BinaryFormatter is removed from .NET 10, so a DataTable can't just be deserialized). | +| ~~R1.1~~ | ~~`ExecuteSqlCommandAsync`~~ | `Retrieval.ExecuteSqlCommand` (`ExeC`+`GetR`) | ✅ **DONE (2026-06-20), live-verified.** `ExecuteSqlCommandAsync(sql)` → `HistorianSqlResult` (columns + typed rows). String-handle op via the uppercase storage GUID. Chain: `Retr.GetV` prime → `ExeC(handle, sql, option=0, ref queryHandle)` → `GetR` loop (note: `GetR` returns **false even on success** — the stream is in `pResultBuff` regardless; false = final page). `GetR`'s `pResultBuff` is an **NRBF-serialized `DataTable`** (`SerializationFormat.Xml`: members `XmlSchema` + `XmlDiffGram`). BinaryFormatter is gone from .NET 10, so it's decoded read-only with `System.Formats.Nrbf` + `XDocument` (no BinaryFormatter). Shipped: `HistorianSqlResult`/`HistorianSqlColumn`/`HistorianSqlExecuteOption`, `HistorianSqlResultProtocol`, `HistorianWcfSqlClient`, golden `WcfSqlResultProtocolTests`, gated live tests. See `docs/reverse-engineering/wcf-exec-sql.md`. | | ~~R1.2~~ | ~~`GetRuntimeParameterAsync`~~ | `Status.GetRuntimeParameter` (`aa/Stat/GETRP`) | ✅ **DONE (2026-06-20), live-verified.** Captured (`scripts/Capture-RuntimeParam.ps1`): GETRP is a **`string`-handle** op (GETHI's shape), but reachable from the managed client using the Open2 storage-session GUID as an **uppercase** string handle (`ToString("D").ToUpperInvariant()`). Returns `HistorianVersion` = `20,0,000,000` live. pRequestBuff = `54 67 01 00` + uint nameCount + per-name(uint charCount + UTF-16); pResponseBuff = version + uint resultCount + CRetVariant(`0x43` VT_BSTR + uint16 len + uint16 charCount + UTF-16). Single string-valued param only (multi-name framing inferred, not captured). Shipped: `HistorianClient.GetRuntimeParameterAsync(name)`; golden `WcfRuntimeParameterProtocolTests`. **Note:** GETRP punching through the string-handle wall with the uppercase storage GUID is a strong lead that GETHI/ExeC may be a handle-*format* issue — see `wcf-string-handle-wall.md` §Update. | | ~~R1.3~~ | ~~`GetServerTimeZoneAsync`~~ | `Status.GetSystemTimeZoneName` | ⚠ **gRPC/2023R2-only.** Verified 2026-06-20: over **2020 WCF** this op is a stub (rc=0, empty value) in the `GetServerTime` family — not shippable here. Build+verify only against a live 2023 R2 server. See `docs/reverse-engineering/wcf-status-localhost.md`. | diff --git a/docs/reverse-engineering/wcf-exec-sql.md b/docs/reverse-engineering/wcf-exec-sql.md new file mode 100644 index 0000000..263e0d9 --- /dev/null +++ b/docs/reverse-engineering/wcf-exec-sql.md @@ -0,0 +1,86 @@ +# SQL command execution over 2020 WCF — ExeC + GetR (HCAL R1.1) + +**Status: ✅ DONE + live-verified (2026-06-20).** `HistorianClient.ExecuteSqlCommandAsync(sql)` runs a +SQL command against the Historian over the 2020 WCF ops `aa/Retr/ExeC` (ExecuteSqlCommand) + +`aa/Retr/GetR` (GetRecordSetByteStream) and returns the record set as a `HistorianSqlResult` (the +managed equivalent of the native `DataTable`). Live-verified end-to-end from the pure-managed .NET 10 +client against the local 2020 Historian (single-cell, multi-column/multi-row, and NULL cases). + +## The ops + +Both are on the Retrieval service (`IRetrievalServiceContract3`), **string-handle** ops reached with +the Open2 storage-session GUID formatted `storageSessionId.ToString("D").ToUpperInvariant()` (the same +uppercase handle that unlocked GETRP/GETHI; see `wcf-string-handle-wall.md`). `Retr.GetV` is primed +first. + +``` +bool ExecuteSqlCommand(string handle, string command, uint option, + ref uint queryHandle, out int retValue, out uint errorSize, out byte[] errorBuffer) + +bool GetRecordSetByteStream(string handle, uint queryHandle, ref uint sequence, + out uint resultSize, out byte[] pResultBuff, out uint errorSize, out byte[] errorBuffer) +``` + +- **`command`** is sent as a plain string (MDAS-encoded ASCII), e.g. `SELECT 1 AS ProbeValue`. +- **`option`** = `HistorianSqlExecuteOption` (`ExecuteRecord=0` is the captured/proven record-set path). +- **`ExeC`** returns the assigned `queryHandle` (and `retValue`); pass it to `GetR`. +- **`GetR` returns `false` even on success** — the byte stream is in `pResultBuff` regardless; a + `false` result just signals the final page. So the orchestrator always consumes `pResultBuff`, then + stops on a `false` result or an empty page. (`sequence` is the paging cursor; small record sets + return everything in one call.) + +## The result stream — NRBF-wrapped DataTable (no BinaryFormatter) + +`GetR`'s `pResultBuff` is a **.NET Remoting Binary Format (NRBF)** stream wrapping a +`System.Data.DataTable` serialized with `SerializationFormat.Xml`. Stream shape (captured): + +``` +SerializationHeader (00 01 00 00 00 FF FF FF FF 01 00 00 00 00 00 00 00) +BinaryLibrary (0C): "System.Data, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089" +ClassWithMembersAndTypes (05): System.Data.DataTable, members: + DataTable.RemotingVersion -> System.Version object + XmlSchema -> BinaryObjectString (XSD: column names + xs types) + XmlDiffGram -> BinaryObjectString (ADO.NET diffgram: the rows) +System.Version object (_Major/_Minor/_Build/_Revision) +MessageEnd (0B) +``` + +**BinaryFormatter is removed from .NET 10**, so the stream is decoded read-only with +`System.Formats.Nrbf.NrbfDecoder` (the sanctioned successor — it parses records without instantiating +or executing any payload type; added as a managed `PackageReference`). The two embedded XML strings are +then parsed with `XDocument`: + +- **XmlSchema** → columns (name + XSD type) under the `msdata:MainDataTable` element's sequence. +- **XmlDiffGram** → rows (each row element under the dataset; cells are child elements or attributes + matching column names). NULL cells are simply absent → parsed as `null`. + +Values are typed per the XSD type (`xs:int`→int, `xs:string`→string, `xs:dateTime`→DateTime, …), +falling back to the raw string for unrecognized types. Only the `SerializationFormat.Xml` DataTable +shape is evidence-backed; a stream whose root is not a DataTable class record, or that lacks the two +XML members, throws `ProtocolEvidenceMissingException`. + +## Capture / decode tooling + +`scripts/Capture-ExecSql.ps1` (NativeTraceHarness `exec-sql` scenario + instrument-wcf-{write,read}message) +captures the ExeC/GetR exchange. ⚠️ A **raw** instrument-wcf capture interleaves MDAS transport chunk +markers (`0x9F`/`0x9E`) into a large `pResultBuff`, so raw byte-slicing yields a corrupted NRBF stream. +The **clean** contract-level byte[] (what the WCF channel reassembles) is dumped via the +`AVEVA_HISTORIAN_SQL_DUMP` env var on `HistorianWcfSqlClient` — that is the golden fixture in +`WcfSqlResultProtocolTests` (the benign `SELECT 1 AS ProbeValue`, no sensitive data). + +## Shipped surface + +- `HistorianClient.ExecuteSqlCommandAsync(command, option = ExecuteRecord)` → `HistorianSqlResult` + (`Columns` name+type, `Rows` typed values, `ReturnValue`). +- Models `HistorianSqlResult` / `HistorianSqlColumn` / `HistorianSqlExecuteOption`; + `HistorianSqlResultProtocol` (NRBF + diffgram parser); `HistorianWcfSqlClient` (ExeC/GetR + orchestration); golden `WcfSqlResultProtocolTests`; gated live tests + (`ExecuteSqlCommandAsync_AgainstLocalHistorian_ReturnsRecordSet` and `_MultiColumnMultiRow`). + +## Scope notes + +- `ExecuteRecord` (record set) is the evidence-backed path. `ExecuteNonQuery`/`ExecuteScalar`/ + `ExecuteRecordDirect` are accepted via the option enum but their non-record-set return shapes are not + separately captured — a non-record result yields empty `Columns`/`Rows` with the `ReturnValue` set. +- The command is whatever the Historian's SQL surface accepts (it routes to the Runtime DB). No + client-side SQL validation is performed. diff --git a/docs/reverse-engineering/wcf-string-handle-wall.md b/docs/reverse-engineering/wcf-string-handle-wall.md index 6fdd4fb..d2f97c9 100644 --- a/docs/reverse-engineering/wcf-string-handle-wall.md +++ b/docs/reverse-engineering/wcf-string-handle-wall.md @@ -100,14 +100,24 @@ GETHI probe is the **handle string format**: the native client sends the GUID lowercase, so a probe that passed the GUID without upcasing would not byte-match what the server's session table is keyed on. -**Implication / open lead (not yet retested):** the GETHI/ExeC/QTB/QTG family failures -may be (at least partly) a **handle-format** issue, not (only) a missing native -registration step. The highest-value cheap follow-up is to **re-probe GETHI and ExeC -with the uppercased storage-session GUID** before assuming the registration wall. If -they also return data, the "wall" collapses to a formatting bug and R1.4/R1.5/R1.6/R1.1 -may be reachable without any new RE. This has **not** been done yet — do not reclassify -those items until it is. GETRP is shipped because it was directly captured + live-verified -end-to-end; the rest remain `ProtocolEvidenceMissingException`/unprobed until tested. +**Implication — CONFIRMED, the wall is largely a handle-format bug.** The follow-up was done: +GETHI and **ExeC both return data with the uppercased storage-session GUID**. + +- **R1.1 `ExecuteSqlCommandAsync` (ExeC + GetR) — SHIPPED + live-verified (2026-06-20).** + `ExecuteSqlCommandAsync(sql)` → `HistorianSqlResult`. `Retr.GetV` prime → `ExeC(handle, + sql, option=0, ref queryHandle)` → `GetR` loop. Note: **`GetR` returns false even on + success** (the byte stream is in `pResultBuff` regardless; false = final page). `pResultBuff` + is an **NRBF `DataTable`** (`SerializationFormat.Xml`: `XmlSchema` + `XmlDiffGram`), decoded + read-only with `System.Formats.Nrbf` + `XDocument` (BinaryFormatter is gone from .NET 10). + Shipped: `HistorianSqlResultProtocol`, `HistorianWcfSqlClient`, golden `WcfSqlResultProtocolTests`, + gated live tests. See `docs/reverse-engineering/wcf-exec-sql.md`. +- **GETHI (R1.4)** also returns data with the uppercase handle (probe; public API not yet shipped). + +So the "wall" collapses to the handle **format** for the Retrieval/Status string-handle ops. +**Exception — QTB/QTG:** `StartTagQuery` does *not* punch through; captured with a correctly +uppercase handle it still fails `error 1` **server-side** (`CMdServer::StartActiveTagnamesQuery` +over `\\.\pipe\aahMetadataServer\console`) — a metadata-server blocker, independent of handle +format. Name-based ops route around it. See `HistorianRuntimeParameterProtocol`, `IStatusServiceContract2.GetRuntimeParameter`, golden `WcfRuntimeParameterProtocolTests`, and capture tooling diff --git a/src/AVEVA.Historian.Client/AVEVA.Historian.Client.csproj b/src/AVEVA.Historian.Client/AVEVA.Historian.Client.csproj index 405d5d6..f4cf6d4 100644 --- a/src/AVEVA.Historian.Client/AVEVA.Historian.Client.csproj +++ b/src/AVEVA.Historian.Client/AVEVA.Historian.Client.csproj @@ -7,6 +7,7 @@ + diff --git a/src/AVEVA.Historian.Client/HistorianClient.cs b/src/AVEVA.Historian.Client/HistorianClient.cs index 2a24954..284cd50 100644 --- a/src/AVEVA.Historian.Client/HistorianClient.cs +++ b/src/AVEVA.Historian.Client/HistorianClient.cs @@ -166,6 +166,22 @@ public sealed class HistorianClient : IAsyncDisposable return _protocol.GetRuntimeParameterAsync(name, cancellationToken); } + /// + /// Executes a SQL command against the Historian over the WCF ExeC/GetR ops and + /// returns the record set as a (the managed equivalent of the + /// native DataTable). The record-set path (, + /// the default) is the evidence-backed surface; the result is decoded from the server's + /// NRBF-serialized DataTable without BinaryFormatter. See HistorianSqlResultProtocol. + /// + public Task ExecuteSqlCommandAsync( + string command, + HistorianSqlExecuteOption option = HistorianSqlExecuteOption.ExecuteRecord, + CancellationToken cancellationToken = default) + { + ArgumentException.ThrowIfNullOrWhiteSpace(command); + return _protocol.ExecuteSqlCommandAsync(command, option, cancellationToken); + } + /// /// Creates or updates the named tag in the Historian Runtime database via /// EnsureTags2. Currently only is diff --git a/src/AVEVA.Historian.Client/Models/HistorianSqlExecuteOption.cs b/src/AVEVA.Historian.Client/Models/HistorianSqlExecuteOption.cs new file mode 100644 index 0000000..d249eb1 --- /dev/null +++ b/src/AVEVA.Historian.Client/Models/HistorianSqlExecuteOption.cs @@ -0,0 +1,21 @@ +namespace AVEVA.Historian.Client.Models; + +/// +/// Execution mode for ExecuteSqlCommandAsync, mirroring the native +/// HistorianSqlExecuteOption enum passed to the ExeC op. Only +/// (the record-set path) is evidence-backed end-to-end. +/// +public enum HistorianSqlExecuteOption +{ + /// Execute and return a record set (the captured/proven path). + ExecuteRecord = 0, + + /// Execute without returning a record set; the result carries only the return value. + ExecuteNonQuery = 1, + + /// Execute and return a single scalar value. + ExecuteScalar = 2, + + /// Execute a record set directly (server-side direct path). + ExecuteRecordDirect = 3, +} diff --git a/src/AVEVA.Historian.Client/Models/HistorianSqlResult.cs b/src/AVEVA.Historian.Client/Models/HistorianSqlResult.cs new file mode 100644 index 0000000..c811bb4 --- /dev/null +++ b/src/AVEVA.Historian.Client/Models/HistorianSqlResult.cs @@ -0,0 +1,35 @@ +namespace AVEVA.Historian.Client.Models; + +/// +/// A column in a : its name and the XSD type from the result-set +/// schema (e.g. xs:int, xs:string, xs:dateTime). +/// +public sealed record HistorianSqlColumn(string Name, string SchemaType); + +/// +/// The result of ExecuteSqlCommandAsync — the managed equivalent of the DataTable +/// the native client returns. Columns carry name + XSD type; each row is a list of values aligned +/// to (typed per the schema where the XSD type is recognized, otherwise the +/// raw string; null for a SQL NULL / absent cell). +/// +public sealed class HistorianSqlResult +{ + public HistorianSqlResult( + IReadOnlyList columns, + IReadOnlyList> rows, + int returnValue) + { + Columns = columns; + Rows = rows; + ReturnValue = returnValue; + } + + /// The result-set columns, in order. + public IReadOnlyList Columns { get; } + + /// The rows; each row's values align positionally to . + public IReadOnlyList> Rows { get; } + + /// The native retValue from ExeC (e.g. rows affected for non-queries). + public int ReturnValue { get; } +} diff --git a/src/AVEVA.Historian.Client/Protocol/Historian2020ProtocolDialect.cs b/src/AVEVA.Historian.Client/Protocol/Historian2020ProtocolDialect.cs index eb9d70e..c8066e4 100644 --- a/src/AVEVA.Historian.Client/Protocol/Historian2020ProtocolDialect.cs +++ b/src/AVEVA.Historian.Client/Protocol/Historian2020ProtocolDialect.cs @@ -74,6 +74,13 @@ internal sealed class Historian2020ProtocolDialect return Wcf.HistorianWcfStatusClient.GetRuntimeParameterAsync(_options, name, cancellationToken); } + public Task ExecuteSqlCommandAsync(string command, HistorianSqlExecuteOption option, CancellationToken cancellationToken) + { + cancellationToken.ThrowIfCancellationRequested(); + ArgumentException.ThrowIfNullOrWhiteSpace(command); + return Wcf.HistorianWcfSqlClient.ExecuteSqlCommandAsync(_options, command, option, cancellationToken); + } + private static async IAsyncEnumerable Missing( string operation, [System.Runtime.CompilerServices.EnumeratorCancellation] CancellationToken cancellationToken) diff --git a/src/AVEVA.Historian.Client/Wcf/HistorianSqlResultProtocol.cs b/src/AVEVA.Historian.Client/Wcf/HistorianSqlResultProtocol.cs new file mode 100644 index 0000000..50fc7f2 --- /dev/null +++ b/src/AVEVA.Historian.Client/Wcf/HistorianSqlResultProtocol.cs @@ -0,0 +1,221 @@ +using System.Formats.Nrbf; +using System.Globalization; +using System.Xml.Linq; +using AVEVA.Historian.Client.Models; +using AVEVA.Historian.Client.Protocol; + +namespace AVEVA.Historian.Client.Wcf; + +/// +/// Parses the GetR (GetRecordSetByteStream) result buffer returned by the 2020 WCF +/// ExeC/GetR SQL surface into a . +/// +/// +/// Captured from the native client (scripts/Capture-ExecSql.ps1 + +/// instrument-wcf-{write,read}message; golden-pinned in WcfSqlResultProtocolTests). The +/// GetR pResultBuff is a .NET Remoting Binary Format (NRBF) stream wrapping a +/// System.Data.DataTable serialized with SerializationFormat.Xml: its data lives in +/// two string members — XmlSchema (an XSD describing the columns + types) and +/// XmlDiffGram (an ADO.NET diffgram carrying the rows). +/// +/// BinaryFormatter is removed from .NET 10, so the stream is decoded read-only with +/// (the sanctioned successor — it parses records without instantiating +/// or executing any payload type). The two embedded XML strings are then parsed with +/// . +/// +/// Only the SerializationFormat.Xml DataTable shape is evidence-backed; a stream whose +/// root is not a DataTable class record, or that lacks the two XML members, throws +/// . +/// +internal static class HistorianSqlResultProtocol +{ + private const string DataTableTypeName = "System.Data.DataTable"; + private const string XmlSchemaMember = "XmlSchema"; + private const string XmlDiffGramMember = "XmlDiffGram"; + + private static readonly XNamespace Xsd = "http://www.w3.org/2001/XMLSchema"; + private static readonly XNamespace MsData = "urn:schemas-microsoft-com:xml-msdata"; + private static readonly XNamespace DiffGr = "urn:schemas-microsoft-com:xml-diffgram-v1"; + + public static HistorianSqlResult Parse(ReadOnlyMemory resultBuffer, int returnValue) + { + (string? schemaXml, string? diffGramXml) = ExtractXmlMembers(resultBuffer); + + if (string.IsNullOrEmpty(schemaXml)) + { + // No record set (e.g. a non-query, or an empty stream): return columns/rows empty. + return new HistorianSqlResult([], [], returnValue); + } + + (string? tableName, List columns) = ParseSchema(schemaXml); + IReadOnlyList> rows = ParseDiffGram(diffGramXml, tableName, columns); + return new HistorianSqlResult(columns, rows, returnValue); + } + + private static (string? schemaXml, string? diffGramXml) ExtractXmlMembers(ReadOnlyMemory resultBuffer) + { + if (resultBuffer.Length == 0) + { + return (null, null); + } + + SerializationRecord root; + try + { + using MemoryStream stream = new(resultBuffer.ToArray(), writable: false); + root = NrbfDecoder.Decode(stream); + } + catch (Exception ex) when (ex is not ProtocolEvidenceMissingException) + { + throw new ProtocolEvidenceMissingException( + $"GetR result buffer is not a decodable NRBF stream: {ex.Message}"); + } + + if (root is not ClassRecord classRecord || !classRecord.TypeNameMatches(typeof(System.Data.DataTable))) + { + // TypeNameMatches needs the real type; fall back to the raw type name when the assembly + // identity differs from the local one. + if (root is not ClassRecord cr || !string.Equals(cr.TypeName.FullName, DataTableTypeName, StringComparison.Ordinal)) + { + throw new ProtocolEvidenceMissingException( + $"GetR result root record is not a {DataTableTypeName} (got '{(root as ClassRecord)?.TypeName.FullName ?? root.GetType().Name}')."); + } + classRecord = cr; + } + + if (!classRecord.HasMember(XmlSchemaMember) || !classRecord.HasMember(XmlDiffGramMember)) + { + throw new ProtocolEvidenceMissingException( + $"GetR DataTable record is missing the {XmlSchemaMember}/{XmlDiffGramMember} members (non-Xml SerializationFormat not supported)."); + } + + return (classRecord.GetString(XmlSchemaMember), classRecord.GetString(XmlDiffGramMember)); + } + + private static (string? tableName, List columns) ParseSchema(string schemaXml) + { + XDocument doc = XDocument.Parse(schemaXml); + XElement? schema = doc.Root; + List columns = []; + if (schema is null) + { + return (null, columns); + } + + // The dataset element advertises the main data table via msdata:MainDataTable. + XElement? dataSet = schema.Elements(Xsd + "element") + .FirstOrDefault(e => (string?)e.Attribute(MsData + "IsDataSet") == "true"); + string? mainTable = (string?)dataSet?.Attribute(MsData + "MainDataTable"); + + // The table element (named by MainDataTable, else the first non-dataset top-level element). + XElement? tableElement = schema.Elements(Xsd + "element") + .FirstOrDefault(e => mainTable is not null && (string?)e.Attribute("name") == mainTable) + ?? schema.Elements(Xsd + "element").FirstOrDefault(e => (string?)e.Attribute(MsData + "IsDataSet") != "true"); + + string? tableName = (string?)tableElement?.Attribute("name"); + + // Columns are the xs:element children of the table's complexType/sequence. + IEnumerable columnElements = tableElement + ?.Descendants(Xsd + "element") + .Where(e => e.Attribute("name") is not null) + ?? []; + foreach (XElement column in columnElements) + { + string name = (string)column.Attribute("name")!; + string type = (string?)column.Attribute("type") ?? InlineSimpleType(column) ?? "xs:string"; + columns.Add(new HistorianSqlColumn(name, type)); + } + + return (tableName, columns); + } + + private static string? InlineSimpleType(XElement column) + { + XElement? restriction = column.Element(Xsd + "simpleType")?.Element(Xsd + "restriction"); + return (string?)restriction?.Attribute("base"); + } + + private static IReadOnlyList> ParseDiffGram( + string? diffGramXml, + string? tableName, + List columns) + { + if (string.IsNullOrEmpty(diffGramXml) || columns.Count == 0) + { + return []; + } + + XDocument doc = XDocument.Parse(diffGramXml); + XElement? diffgram = doc.Root; + if (diffgram is null) + { + return []; + } + + // The dataset element is the diffgram's child that is not in the diffgr: namespace + // (diffgr:before / diffgr:errors are change-tracking sections we don't need for a SELECT). + XElement? dataSet = diffgram.Elements().FirstOrDefault(e => e.Name.Namespace != DiffGr); + if (dataSet is null) + { + return []; + } + + // Row elements are the dataset's children whose local name matches the table (or any child + // when the table name is unknown). + IEnumerable rowElements = dataSet.Elements() + .Where(e => tableName is null || e.Name.LocalName == tableName); + + List> rows = []; + foreach (XElement rowElement in rowElements) + { + object?[] values = new object?[columns.Count]; + for (int i = 0; i < columns.Count; i++) + { + HistorianSqlColumn column = columns[i]; + // ADO.NET maps a column either as a child element or as an attribute on the row. + XElement? cell = rowElement.Elements().FirstOrDefault(e => e.Name.LocalName == column.Name); + string? raw = cell?.Value + ?? rowElement.Attributes().FirstOrDefault(a => a.Name.LocalName == column.Name)?.Value; + values[i] = raw is null ? null : ConvertValue(raw, column.SchemaType); + } + rows.Add(values); + } + + return rows; + } + + private static object ConvertValue(string raw, string schemaType) + { + string type = schemaType.Contains(':') ? schemaType[(schemaType.IndexOf(':') + 1)..] : schemaType; + try + { + return type switch + { + "int" or "integer" => int.Parse(raw, CultureInfo.InvariantCulture), + "long" => long.Parse(raw, CultureInfo.InvariantCulture), + "short" => short.Parse(raw, CultureInfo.InvariantCulture), + "byte" => sbyte.Parse(raw, CultureInfo.InvariantCulture), + "unsignedByte" => byte.Parse(raw, CultureInfo.InvariantCulture), + "unsignedShort" => ushort.Parse(raw, CultureInfo.InvariantCulture), + "unsignedInt" => uint.Parse(raw, CultureInfo.InvariantCulture), + "unsignedLong" => ulong.Parse(raw, CultureInfo.InvariantCulture), + "double" => double.Parse(raw, CultureInfo.InvariantCulture), + "float" => float.Parse(raw, CultureInfo.InvariantCulture), + "decimal" => decimal.Parse(raw, CultureInfo.InvariantCulture), + "boolean" => ParseXsdBoolean(raw), + "dateTime" => DateTime.Parse(raw, CultureInfo.InvariantCulture, DateTimeStyles.RoundtripKind), + "base64Binary" => Convert.FromBase64String(raw), + _ => raw, + }; + } + catch (Exception ex) when (ex is FormatException or OverflowException) + { + // Keep the raw string rather than throwing — the column type is informational and the + // server's textual rendering is always available. + return raw; + } + } + + private static bool ParseXsdBoolean(string raw) => + raw is "true" or "1" || (raw is not ("false" or "0") && bool.Parse(raw)); +} diff --git a/src/AVEVA.Historian.Client/Wcf/HistorianWcfSqlClient.cs b/src/AVEVA.Historian.Client/Wcf/HistorianWcfSqlClient.cs new file mode 100644 index 0000000..59b99d8 --- /dev/null +++ b/src/AVEVA.Historian.Client/Wcf/HistorianWcfSqlClient.cs @@ -0,0 +1,132 @@ +using System.ServiceModel; +using System.ServiceModel.Channels; +using AVEVA.Historian.Client.Models; +using AVEVA.Historian.Client.Wcf.Contracts; + +namespace AVEVA.Historian.Client.Wcf; + +/// +/// Executes SQL commands (HCAL R1.1) over the 2020 WCF aa/Retr/ExeC + aa/Retr/GetR +/// ops. Both are string-handle ops reached with the Open2 storage-session GUID formatted UPPERCASE +/// (the same handle format that unlocked GETRP/GETHI; see wcf-string-handle-wall.md). The +/// record set comes back from GetR as an NRBF-serialized DataTable, parsed by +/// . +/// +internal static class HistorianWcfSqlClient +{ + // GetR is byte-stream-paged. A small record set returns the whole stream in the first call; + // larger ones chunk across calls accumulated here. The cap is a runaway guard. + private const int MaxPages = 4096; + + /// Diagnostic: the ExeC/GetR error from the last call (set only on server rejection). + public static string? LastError { get; private set; } + + public static Task ExecuteSqlCommandAsync( + HistorianClientOptions options, + string command, + HistorianSqlExecuteOption option, + CancellationToken cancellationToken) + { + ArgumentException.ThrowIfNullOrWhiteSpace(command); + return Task.Run(() => ExecuteSqlCommand(options, command, option), cancellationToken); + } + + private static HistorianSqlResult ExecuteSqlCommand( + HistorianClientOptions options, + string command, + HistorianSqlExecuteOption option) + { + Guid contextKey = Guid.NewGuid(); + var (histBinding, histEndpoint, retrBinding, retrEndpoint) = HistorianWcfBindingFactory.CreateBindingPair(options); + + LastError = null; + HistorianSqlResult? result = null; + HistorianWcfAuthChainHelper.OpenAuthenticatedConnection( + options, histBinding, histEndpoint, contextKey, CancellationToken.None, + additionalSetup: (_, context) => + result = RunSql(retrBinding, retrEndpoint, options, context.StorageSessionId, command, option)); + return result ?? new HistorianSqlResult([], [], 0); + } + + private static HistorianSqlResult RunSql( + Binding retrBinding, + EndpointAddress retrEndpoint, + HistorianClientOptions options, + Guid storageSessionId, + string command, + HistorianSqlExecuteOption option) + { + // ExeC/GetR take the storage-session GUID as a string handle, uppercase dash-no-braces. + string handle = storageSessionId.ToString("D").ToUpperInvariant(); + + ChannelFactory factory = new(retrBinding, retrEndpoint); + HistorianWcfClientCredentialsHelper.Configure(factory, options); + IRetrievalServiceContract3 channel = factory.CreateChannel(); + ICommunicationObject co = (ICommunicationObject)channel; + try + { + // Prime the Retrieval service version handshake (Retr.GetV). + channel.GetInterfaceVersion(out _); + + uint queryHandle = 0; + bool execOk = channel.ExecuteSqlCommand( + handle, command, (uint)option, ref queryHandle, + out int returnValue, out uint errSize, out byte[] errBuf); + if (!execOk) + { + LastError = $"ExeC returned false (retValue={returnValue}, errSize={errSize}, errLen={errBuf?.Length ?? 0})."; + return new HistorianSqlResult([], [], returnValue); + } + + using MemoryStream accumulated = new(); + uint sequence = 0; + for (int page = 0; page < MaxPages; page++) + { + bool getrOk = channel.GetRecordSetByteStream( + handle, queryHandle, ref sequence, + out uint resultSize, out byte[] resultBuffer, out uint gErrSize, out byte[] gErrBuf); + + // GetRResult is false even on a successful single-page read — the byte stream is in + // resultBuffer regardless, and false signals "this is the final page". So always + // consume the buffer first, then stop on a false result or an empty page. + if (resultBuffer is { Length: > 0 }) + { + accumulated.Write(resultBuffer, 0, resultBuffer.Length); + } + + if (!getrOk || resultBuffer is null || resultBuffer.Length == 0) + { + if (page == 0 && accumulated.Length == 0 && gErrBuf is { Length: > 0 }) + { + LastError = $"GetR error (resultSize={resultSize}, errSize={gErrSize}, errLen={gErrBuf.Length})."; + } + break; + } + } + + byte[] stream = accumulated.ToArray(); + DumpStreamIfRequested(stream); + return HistorianSqlResultProtocol.Parse(stream, returnValue); + } + finally + { + try { if (co.State == CommunicationState.Faulted) co.Abort(); else co.Close(); } catch { try { co.Abort(); } catch { } } + try { if (factory.State == CommunicationState.Faulted) factory.Abort(); else factory.Close(); } catch { try { factory.Abort(); } catch { } } + } + } + + /// + /// Reverse-engineering aid: when AVEVA_HISTORIAN_SQL_DUMP is set to a path, writes the + /// base64 of the reassembled GetR NRBF stream there (the clean contract-level byte[], free of + /// the MDAS transport chunk markers that corrupt a raw instrument-wcf capture). Used once to + /// produce the golden fixture for WcfSqlResultProtocolTests. + /// + private static void DumpStreamIfRequested(byte[] stream) + { + string? path = Environment.GetEnvironmentVariable("AVEVA_HISTORIAN_SQL_DUMP"); + if (!string.IsNullOrWhiteSpace(path) && stream.Length > 0) + { + try { File.WriteAllText(path, Convert.ToBase64String(stream)); } catch { /* diagnostic only */ } + } + } +} diff --git a/tests/AVEVA.Historian.Client.Tests/HistorianClientIntegrationTests.cs b/tests/AVEVA.Historian.Client.Tests/HistorianClientIntegrationTests.cs index 6b38821..6860159 100644 --- a/tests/AVEVA.Historian.Client.Tests/HistorianClientIntegrationTests.cs +++ b/tests/AVEVA.Historian.Client.Tests/HistorianClientIntegrationTests.cs @@ -296,6 +296,65 @@ public sealed class HistorianClientIntegrationTests Assert.False(string.IsNullOrWhiteSpace(value)); } + [Fact] + public async Task ExecuteSqlCommandAsync_AgainstLocalHistorian_ReturnsRecordSet() + { + string? host = Environment.GetEnvironmentVariable("HISTORIAN_HOST"); + if (string.IsNullOrWhiteSpace(host) || !string.Equals(host, "localhost", StringComparison.OrdinalIgnoreCase) || !OperatingSystem.IsWindows()) + { + return; + } + + HistorianClient client = new(new HistorianClientOptions + { + Host = host, + IntegratedSecurity = true, + Transport = HistorianTransport.LocalPipe + }); + + // ExeC/GetR ride the storage-session GUID as an uppercase string handle. A constant SELECT + // returns a single int column; the DataTable is decoded from the NRBF stream (no BinaryFormatter). + AVEVA.Historian.Client.Models.HistorianSqlResult result = + await client.ExecuteSqlCommandAsync("SELECT 1 AS ProbeValue", cancellationToken: CancellationToken.None); + + AVEVA.Historian.Client.Models.HistorianSqlColumn column = Assert.Single(result.Columns); + Assert.Equal("ProbeValue", column.Name); + IReadOnlyList row = Assert.Single(result.Rows); + Assert.Equal(1, Assert.Single(row)); + } + + [Fact] + public async Task ExecuteSqlCommandAsync_AgainstLocalHistorian_MultiColumnMultiRow() + { + string? host = Environment.GetEnvironmentVariable("HISTORIAN_HOST"); + if (string.IsNullOrWhiteSpace(host) || !string.Equals(host, "localhost", StringComparison.OrdinalIgnoreCase) || !OperatingSystem.IsWindows()) + { + return; + } + + HistorianClient client = new(new HistorianClientOptions + { + Host = host, + IntegratedSecurity = true, + Transport = HistorianTransport.LocalPipe + }); + + // Fully synthetic query (no server data): two columns (int + string), two rows, one NULL — + // exercises the schema/diffgram parser beyond the single-cell case. + AVEVA.Historian.Client.Models.HistorianSqlResult result = await client.ExecuteSqlCommandAsync( + "SELECT 10 AS Num, 'alpha' AS Word UNION ALL SELECT 20, NULL", + cancellationToken: CancellationToken.None); + + Assert.Equal(2, result.Columns.Count); + Assert.Equal("Num", result.Columns[0].Name); + Assert.Equal("Word", result.Columns[1].Name); + Assert.Equal(2, result.Rows.Count); + Assert.Equal(10, result.Rows[0][0]); + Assert.Equal("alpha", result.Rows[0][1]); + Assert.Equal(20, result.Rows[1][0]); + Assert.Null(result.Rows[1][1]); + } + [Fact] public async Task GetConnectionStatusAsync_AgainstLocalHistorian_ReportsConnectedToServer() { diff --git a/tests/AVEVA.Historian.Client.Tests/WcfSqlResultProtocolTests.cs b/tests/AVEVA.Historian.Client.Tests/WcfSqlResultProtocolTests.cs new file mode 100644 index 0000000..614d03e --- /dev/null +++ b/tests/AVEVA.Historian.Client.Tests/WcfSqlResultProtocolTests.cs @@ -0,0 +1,79 @@ +using AVEVA.Historian.Client.Models; +using AVEVA.Historian.Client.Protocol; +using AVEVA.Historian.Client.Wcf; + +namespace AVEVA.Historian.Client.Tests; + +/// +/// Golden tests for the ExeC/GetR SQL result codec. The fixture is the real, clean GetR +/// pResultBuff captured from the live server for the benign query +/// SELECT 1 AS ProbeValue (no sensitive data) — an NRBF stream wrapping a +/// System.Data.DataTable serialized with SerializationFormat.Xml. It is the +/// contract-level byte[] the WCF channel reassembles (free of MDAS transport chunk markers). +/// +public sealed class WcfSqlResultProtocolTests +{ + // GetR pResultBuff for "SELECT 1 AS ProbeValue" (1232 bytes), base64. Captured via the + // AVEVA_HISTORIAN_SQL_DUMP diagnostic in HistorianWcfSqlClient. + private const string GetRStreamBase64 = + "AAEAAAD/////AQAAAAAAAAAMAgAAAE5TeXN0ZW0uRGF0YSwgVmVyc2lvbj00LjAuMC4wLCBDdWx0dXJlPW5ldXRyYWwsIFB1Ymxp" + + "Y0tleVRva2VuPWI3N2E1YzU2MTkzNGUwODkFAQAAABVTeXN0ZW0uRGF0YS5EYXRhVGFibGUDAAAAGURhdGFUYWJsZS5SZW1vdGlu" + + "Z1ZlcnNpb24JWG1sU2NoZW1hC1htbERpZmZHcmFtAwEBDlN5c3RlbS5WZXJzaW9uAgAAAAkDAAAABgQAAACJBTw/eG1sIHZlcnNp" + + "b249IjEuMCIgZW5jb2Rpbmc9InV0Zi0xNiI/Pg0KPHhzOnNjaGVtYSB4bWxucz0iIiB4bWxuczp4cz0iaHR0cDovL3d3dy53My5v" + + "cmcvMjAwMS9YTUxTY2hlbWEiIHhtbG5zOm1zZGF0YT0idXJuOnNjaGVtYXMtbWljcm9zb2Z0LWNvbTp4bWwtbXNkYXRhIj4NCiAg" + + "PHhzOmVsZW1lbnQgbmFtZT0iVGFibGUxIj4NCiAgICA8eHM6Y29tcGxleFR5cGU+DQogICAgICA8eHM6c2VxdWVuY2U+DQogICAg" + + "ICAgIDx4czplbGVtZW50IG5hbWU9IlByb2JlVmFsdWUiIG1zZGF0YTpSZWFkT25seT0idHJ1ZSIgdHlwZT0ieHM6aW50IiBtc2Rh" + + "dGE6dGFyZ2V0TmFtZXNwYWNlPSIiIC8+DQogICAgICA8L3hzOnNlcXVlbmNlPg0KICAgIDwveHM6Y29tcGxleFR5cGU+DQogIDwv" + + "eHM6ZWxlbWVudD4NCiAgPHhzOmVsZW1lbnQgbmFtZT0idG1wRGF0YVNldCIgbXNkYXRhOklzRGF0YVNldD0idHJ1ZSIgbXNkYXRh" + + "Ok1haW5EYXRhVGFibGU9IlRhYmxlMSIgbXNkYXRhOlVzZUN1cnJlbnRMb2NhbGU9InRydWUiPg0KICAgIDx4czpjb21wbGV4VHlw" + + "ZT4NCiAgICAgIDx4czpjaG9pY2UgbWluT2NjdXJzPSIwIiBtYXhPY2N1cnM9InVuYm91bmRlZCIgLz4NCiAgICA8L3hzOmNvbXBs" + + "ZXhUeXBlPg0KICA8L3hzOmVsZW1lbnQ+DQo8L3hzOnNjaGVtYT4GBQAAAJoCPGRpZmZncjpkaWZmZ3JhbSB4bWxuczptc2RhdGE9" + + "InVybjpzY2hlbWFzLW1pY3Jvc29mdC1jb206eG1sLW1zZGF0YSIgeG1sbnM6ZGlmZmdyPSJ1cm46c2NoZW1hcy1taWNyb3NvZnQt" + + "Y29tOnhtbC1kaWZmZ3JhbS12MSI+DQogIDx0bXBEYXRhU2V0Pg0KICAgIDxUYWJsZTEgZGlmZmdyOmlkPSJUYWJsZTExIiBtc2Rh" + + "dGE6cm93T3JkZXI9IjAiPg0KICAgICAgPFByb2JlVmFsdWU+MTwvUHJvYmVWYWx1ZT4NCiAgICA8L1RhYmxlMT4NCiAgPC90bXBE" + + "YXRhU2V0Pg0KPC9kaWZmZ3I6ZGlmZmdyYW0+BAMAAAAOU3lzdGVtLlZlcnNpb24EAAAABl9NYWpvcgZfTWlub3IGX0J1aWxkCV9S" + + "ZXZpc2lvbgAAAAAICAgIAgAAAAAAAAD//////////ws="; + + [Fact] + public void Parse_DataTableStream_ExtractsColumnAndTypedRow() + { + byte[] stream = Convert.FromBase64String(GetRStreamBase64); + + HistorianSqlResult result = HistorianSqlResultProtocol.Parse(stream, returnValue: 0); + + HistorianSqlColumn column = Assert.Single(result.Columns); + Assert.Equal("ProbeValue", column.Name); + Assert.Equal("xs:int", column.SchemaType); + + IReadOnlyList row = Assert.Single(result.Rows); + object? value = Assert.Single(row); + Assert.Equal(1, value); // typed to int per the xs:int schema + Assert.IsType(value); + Assert.Equal(0, result.ReturnValue); + } + + [Fact] + public void Parse_PreservesReturnValue() + { + byte[] stream = Convert.FromBase64String(GetRStreamBase64); + HistorianSqlResult result = HistorianSqlResultProtocol.Parse(stream, returnValue: 7); + Assert.Equal(7, result.ReturnValue); + } + + [Fact] + public void Parse_EmptyBuffer_ReturnsEmptyResult() + { + HistorianSqlResult result = HistorianSqlResultProtocol.Parse(ReadOnlyMemory.Empty, returnValue: 3); + Assert.Empty(result.Columns); + Assert.Empty(result.Rows); + Assert.Equal(3, result.ReturnValue); + } + + [Fact] + public void Parse_NonNrbfBuffer_Throws() + { + byte[] garbage = [0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, 0x08]; + Assert.Throws( + () => HistorianSqlResultProtocol.Parse(garbage, returnValue: 0)); + } +}