R1.1 ExecuteSqlCommandAsync (ExeC + GetR, NRBF DataTable, no BinaryFormatter)

Ship SQL command execution over the 2020 WCF aa/Retr/ExeC + aa/Retr/GetR ops:
HistorianClient.ExecuteSqlCommandAsync(sql) -> HistorianSqlResult (columns +
typed rows). String-handle ops reached with the Open2 storage-session GUID
formatted uppercase (the handle format that unlocked GETRP/GETHI).

Chain: Retr.GetV prime -> ExeC(handle, sql, option=0, ref queryHandle) ->
GetR loop. Key gotcha captured: GetR returns FALSE even on success -- the byte
stream is in pResultBuff regardless; false just signals the final page. So the
orchestrator consumes the buffer first, then stops on a false result / empty page.

GetR's pResultBuff is an NRBF-serialized System.Data.DataTable
(SerializationFormat.Xml: members XmlSchema (XSD) + XmlDiffGram (rows)).
BinaryFormatter is removed from .NET 10, so the stream is decoded read-only with
the System.Formats.Nrbf package (NrbfDecoder) + XDocument -- no BinaryFormatter,
no code execution. Values are typed per the XSD type, falling back to string.

Adds: HistorianSqlResult / HistorianSqlColumn / HistorianSqlExecuteOption models,
HistorianSqlResultProtocol (NRBF + diffgram parser), HistorianWcfSqlClient
(ExeC/GetR orchestration with an AVEVA_HISTORIAN_SQL_DUMP diagnostic), dialect +
public API. Golden WcfSqlResultProtocolTests pinned to the real clean GetR stream
for the benign "SELECT 1 AS ProbeValue" (no sensitive data); gated live tests
(single cell + multi-column/multi-row/NULL). Doc: wcf-exec-sql.md; roadmap R1.1
DONE; wall doc + memory updated (incl. the QTB-server-side nuance). 229 tests green.

Note: a raw instrument-wcf capture corrupts a large pResultBuff with MDAS
transport chunk markers (0x9F); the clean contract-level byte[] is dumped via the
AVEVA_HISTORIAN_SQL_DUMP env var for the golden fixture.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B6mcaT2PjRFKcogzp9UkfC
This commit is contained in:
Joseph Doherty
2026-06-20 23:16:06 -04:00
parent 4da5287d01
commit 1a539882d0
12 changed files with 676 additions and 9 deletions
+86
View File
@@ -0,0 +1,86 @@
# SQL command execution over 2020 WCF — ExeC + GetR (HCAL R1.1)
**Status: ✅ DONE + live-verified (2026-06-20).** `HistorianClient.ExecuteSqlCommandAsync(sql)` runs a
SQL command against the Historian over the 2020 WCF ops `aa/Retr/ExeC` (ExecuteSqlCommand) +
`aa/Retr/GetR` (GetRecordSetByteStream) and returns the record set as a `HistorianSqlResult` (the
managed equivalent of the native `DataTable`). Live-verified end-to-end from the pure-managed .NET 10
client against the local 2020 Historian (single-cell, multi-column/multi-row, and NULL cases).
## The ops
Both are on the Retrieval service (`IRetrievalServiceContract3`), **string-handle** ops reached with
the Open2 storage-session GUID formatted `storageSessionId.ToString("D").ToUpperInvariant()` (the same
uppercase handle that unlocked GETRP/GETHI; see `wcf-string-handle-wall.md`). `Retr.GetV` is primed
first.
```
bool ExecuteSqlCommand(string handle, string command, uint option,
ref uint queryHandle, out int retValue, out uint errorSize, out byte[] errorBuffer)
bool GetRecordSetByteStream(string handle, uint queryHandle, ref uint sequence,
out uint resultSize, out byte[] pResultBuff, out uint errorSize, out byte[] errorBuffer)
```
- **`command`** is sent as a plain string (MDAS-encoded ASCII), e.g. `SELECT 1 AS ProbeValue`.
- **`option`** = `HistorianSqlExecuteOption` (`ExecuteRecord=0` is the captured/proven record-set path).
- **`ExeC`** returns the assigned `queryHandle` (and `retValue`); pass it to `GetR`.
- **`GetR` returns `false` even on success** — the byte stream is in `pResultBuff` regardless; a
`false` result just signals the final page. So the orchestrator always consumes `pResultBuff`, then
stops on a `false` result or an empty page. (`sequence` is the paging cursor; small record sets
return everything in one call.)
## The result stream — NRBF-wrapped DataTable (no BinaryFormatter)
`GetR`'s `pResultBuff` is a **.NET Remoting Binary Format (NRBF)** stream wrapping a
`System.Data.DataTable` serialized with `SerializationFormat.Xml`. Stream shape (captured):
```
SerializationHeader (00 01 00 00 00 FF FF FF FF 01 00 00 00 00 00 00 00)
BinaryLibrary (0C): "System.Data, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089"
ClassWithMembersAndTypes (05): System.Data.DataTable, members:
DataTable.RemotingVersion -> System.Version object
XmlSchema -> BinaryObjectString (XSD: column names + xs types)
XmlDiffGram -> BinaryObjectString (ADO.NET diffgram: the rows)
System.Version object (_Major/_Minor/_Build/_Revision)
MessageEnd (0B)
```
**BinaryFormatter is removed from .NET 10**, so the stream is decoded read-only with
`System.Formats.Nrbf.NrbfDecoder` (the sanctioned successor — it parses records without instantiating
or executing any payload type; added as a managed `PackageReference`). The two embedded XML strings are
then parsed with `XDocument`:
- **XmlSchema** → columns (name + XSD type) under the `msdata:MainDataTable` element's sequence.
- **XmlDiffGram** → rows (each row element under the dataset; cells are child elements or attributes
matching column names). NULL cells are simply absent → parsed as `null`.
Values are typed per the XSD type (`xs:int`→int, `xs:string`→string, `xs:dateTime`→DateTime, …),
falling back to the raw string for unrecognized types. Only the `SerializationFormat.Xml` DataTable
shape is evidence-backed; a stream whose root is not a DataTable class record, or that lacks the two
XML members, throws `ProtocolEvidenceMissingException`.
## Capture / decode tooling
`scripts/Capture-ExecSql.ps1` (NativeTraceHarness `exec-sql` scenario + instrument-wcf-{write,read}message)
captures the ExeC/GetR exchange. ⚠️ A **raw** instrument-wcf capture interleaves MDAS transport chunk
markers (`0x9F`/`0x9E`) into a large `pResultBuff`, so raw byte-slicing yields a corrupted NRBF stream.
The **clean** contract-level byte[] (what the WCF channel reassembles) is dumped via the
`AVEVA_HISTORIAN_SQL_DUMP` env var on `HistorianWcfSqlClient` — that is the golden fixture in
`WcfSqlResultProtocolTests` (the benign `SELECT 1 AS ProbeValue`, no sensitive data).
## Shipped surface
- `HistorianClient.ExecuteSqlCommandAsync(command, option = ExecuteRecord)``HistorianSqlResult`
(`Columns` name+type, `Rows` typed values, `ReturnValue`).
- Models `HistorianSqlResult` / `HistorianSqlColumn` / `HistorianSqlExecuteOption`;
`HistorianSqlResultProtocol` (NRBF + diffgram parser); `HistorianWcfSqlClient` (ExeC/GetR
orchestration); golden `WcfSqlResultProtocolTests`; gated live tests
(`ExecuteSqlCommandAsync_AgainstLocalHistorian_ReturnsRecordSet` and `_MultiColumnMultiRow`).
## Scope notes
- `ExecuteRecord` (record set) is the evidence-backed path. `ExecuteNonQuery`/`ExecuteScalar`/
`ExecuteRecordDirect` are accepted via the option enum but their non-record-set return shapes are not
separately captured — a non-record result yields empty `Columns`/`Rows` with the `ReturnValue` set.
- The command is whatever the Historian's SQL surface accepts (it routes to the Runtime DB). No
client-side SQL validation is performed.
@@ -100,14 +100,24 @@ GETHI probe is the **handle string format**: the native client sends the GUID
lowercase, so a probe that passed the GUID without upcasing would not byte-match what
the server's session table is keyed on.
**Implication / open lead (not yet retested):** the GETHI/ExeC/QTB/QTG family failures
may be (at least partly) a **handle-format** issue, not (only) a missing native
registration step. The highest-value cheap follow-up is to **re-probe GETHI and ExeC
with the uppercased storage-session GUID** before assuming the registration wall. If
they also return data, the "wall" collapses to a formatting bug and R1.4/R1.5/R1.6/R1.1
may be reachable without any new RE. This has **not** been done yet — do not reclassify
those items until it is. GETRP is shipped because it was directly captured + live-verified
end-to-end; the rest remain `ProtocolEvidenceMissingException`/unprobed until tested.
**Implication — CONFIRMED, the wall is largely a handle-format bug.** The follow-up was done:
GETHI and **ExeC both return data with the uppercased storage-session GUID**.
- **R1.1 `ExecuteSqlCommandAsync` (ExeC + GetR) — SHIPPED + live-verified (2026-06-20).**
`ExecuteSqlCommandAsync(sql)``HistorianSqlResult`. `Retr.GetV` prime → `ExeC(handle,
sql, option=0, ref queryHandle)``GetR` loop. Note: **`GetR` returns false even on
success** (the byte stream is in `pResultBuff` regardless; false = final page). `pResultBuff`
is an **NRBF `DataTable`** (`SerializationFormat.Xml`: `XmlSchema` + `XmlDiffGram`), decoded
read-only with `System.Formats.Nrbf` + `XDocument` (BinaryFormatter is gone from .NET 10).
Shipped: `HistorianSqlResultProtocol`, `HistorianWcfSqlClient`, golden `WcfSqlResultProtocolTests`,
gated live tests. See `docs/reverse-engineering/wcf-exec-sql.md`.
- **GETHI (R1.4)** also returns data with the uppercase handle (probe; public API not yet shipped).
So the "wall" collapses to the handle **format** for the Retrieval/Status string-handle ops.
**Exception — QTB/QTG:** `StartTagQuery` does *not* punch through; captured with a correctly
uppercase handle it still fails `error 1` **server-side** (`CMdServer::StartActiveTagnamesQuery`
over `\\.\pipe\aahMetadataServer\console`) — a metadata-server blocker, independent of handle
format. Name-based ops route around it.
See `HistorianRuntimeParameterProtocol`, `IStatusServiceContract2.GetRuntimeParameter`,
golden `WcfRuntimeParameterProtocolTests`, and capture tooling