R1.1 ExecuteSqlCommandAsync (ExeC + GetR, NRBF DataTable, no BinaryFormatter)
Ship SQL command execution over the 2020 WCF aa/Retr/ExeC + aa/Retr/GetR ops: HistorianClient.ExecuteSqlCommandAsync(sql) -> HistorianSqlResult (columns + typed rows). String-handle ops reached with the Open2 storage-session GUID formatted uppercase (the handle format that unlocked GETRP/GETHI). Chain: Retr.GetV prime -> ExeC(handle, sql, option=0, ref queryHandle) -> GetR loop. Key gotcha captured: GetR returns FALSE even on success -- the byte stream is in pResultBuff regardless; false just signals the final page. So the orchestrator consumes the buffer first, then stops on a false result / empty page. GetR's pResultBuff is an NRBF-serialized System.Data.DataTable (SerializationFormat.Xml: members XmlSchema (XSD) + XmlDiffGram (rows)). BinaryFormatter is removed from .NET 10, so the stream is decoded read-only with the System.Formats.Nrbf package (NrbfDecoder) + XDocument -- no BinaryFormatter, no code execution. Values are typed per the XSD type, falling back to string. Adds: HistorianSqlResult / HistorianSqlColumn / HistorianSqlExecuteOption models, HistorianSqlResultProtocol (NRBF + diffgram parser), HistorianWcfSqlClient (ExeC/GetR orchestration with an AVEVA_HISTORIAN_SQL_DUMP diagnostic), dialect + public API. Golden WcfSqlResultProtocolTests pinned to the real clean GetR stream for the benign "SELECT 1 AS ProbeValue" (no sensitive data); gated live tests (single cell + multi-column/multi-row/NULL). Doc: wcf-exec-sql.md; roadmap R1.1 DONE; wall doc + memory updated (incl. the QTB-server-side nuance). 229 tests green. Note: a raw instrument-wcf capture corrupts a large pResultBuff with MDAS transport chunk markers (0x9F); the clean contract-level byte[] is dumped via the AVEVA_HISTORIAN_SQL_DUMP env var for the golden fixture. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01B6mcaT2PjRFKcogzp9UkfC
This commit is contained in:
@@ -0,0 +1,86 @@
|
||||
# SQL command execution over 2020 WCF — ExeC + GetR (HCAL R1.1)
|
||||
|
||||
**Status: ✅ DONE + live-verified (2026-06-20).** `HistorianClient.ExecuteSqlCommandAsync(sql)` runs a
|
||||
SQL command against the Historian over the 2020 WCF ops `aa/Retr/ExeC` (ExecuteSqlCommand) +
|
||||
`aa/Retr/GetR` (GetRecordSetByteStream) and returns the record set as a `HistorianSqlResult` (the
|
||||
managed equivalent of the native `DataTable`). Live-verified end-to-end from the pure-managed .NET 10
|
||||
client against the local 2020 Historian (single-cell, multi-column/multi-row, and NULL cases).
|
||||
|
||||
## The ops
|
||||
|
||||
Both are on the Retrieval service (`IRetrievalServiceContract3`), **string-handle** ops reached with
|
||||
the Open2 storage-session GUID formatted `storageSessionId.ToString("D").ToUpperInvariant()` (the same
|
||||
uppercase handle that unlocked GETRP/GETHI; see `wcf-string-handle-wall.md`). `Retr.GetV` is primed
|
||||
first.
|
||||
|
||||
```
|
||||
bool ExecuteSqlCommand(string handle, string command, uint option,
|
||||
ref uint queryHandle, out int retValue, out uint errorSize, out byte[] errorBuffer)
|
||||
|
||||
bool GetRecordSetByteStream(string handle, uint queryHandle, ref uint sequence,
|
||||
out uint resultSize, out byte[] pResultBuff, out uint errorSize, out byte[] errorBuffer)
|
||||
```
|
||||
|
||||
- **`command`** is sent as a plain string (MDAS-encoded ASCII), e.g. `SELECT 1 AS ProbeValue`.
|
||||
- **`option`** = `HistorianSqlExecuteOption` (`ExecuteRecord=0` is the captured/proven record-set path).
|
||||
- **`ExeC`** returns the assigned `queryHandle` (and `retValue`); pass it to `GetR`.
|
||||
- **`GetR` returns `false` even on success** — the byte stream is in `pResultBuff` regardless; a
|
||||
`false` result just signals the final page. So the orchestrator always consumes `pResultBuff`, then
|
||||
stops on a `false` result or an empty page. (`sequence` is the paging cursor; small record sets
|
||||
return everything in one call.)
|
||||
|
||||
## The result stream — NRBF-wrapped DataTable (no BinaryFormatter)
|
||||
|
||||
`GetR`'s `pResultBuff` is a **.NET Remoting Binary Format (NRBF)** stream wrapping a
|
||||
`System.Data.DataTable` serialized with `SerializationFormat.Xml`. Stream shape (captured):
|
||||
|
||||
```
|
||||
SerializationHeader (00 01 00 00 00 FF FF FF FF 01 00 00 00 00 00 00 00)
|
||||
BinaryLibrary (0C): "System.Data, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089"
|
||||
ClassWithMembersAndTypes (05): System.Data.DataTable, members:
|
||||
DataTable.RemotingVersion -> System.Version object
|
||||
XmlSchema -> BinaryObjectString (XSD: column names + xs types)
|
||||
XmlDiffGram -> BinaryObjectString (ADO.NET diffgram: the rows)
|
||||
System.Version object (_Major/_Minor/_Build/_Revision)
|
||||
MessageEnd (0B)
|
||||
```
|
||||
|
||||
**BinaryFormatter is removed from .NET 10**, so the stream is decoded read-only with
|
||||
`System.Formats.Nrbf.NrbfDecoder` (the sanctioned successor — it parses records without instantiating
|
||||
or executing any payload type; added as a managed `PackageReference`). The two embedded XML strings are
|
||||
then parsed with `XDocument`:
|
||||
|
||||
- **XmlSchema** → columns (name + XSD type) under the `msdata:MainDataTable` element's sequence.
|
||||
- **XmlDiffGram** → rows (each row element under the dataset; cells are child elements or attributes
|
||||
matching column names). NULL cells are simply absent → parsed as `null`.
|
||||
|
||||
Values are typed per the XSD type (`xs:int`→int, `xs:string`→string, `xs:dateTime`→DateTime, …),
|
||||
falling back to the raw string for unrecognized types. Only the `SerializationFormat.Xml` DataTable
|
||||
shape is evidence-backed; a stream whose root is not a DataTable class record, or that lacks the two
|
||||
XML members, throws `ProtocolEvidenceMissingException`.
|
||||
|
||||
## Capture / decode tooling
|
||||
|
||||
`scripts/Capture-ExecSql.ps1` (NativeTraceHarness `exec-sql` scenario + instrument-wcf-{write,read}message)
|
||||
captures the ExeC/GetR exchange. ⚠️ A **raw** instrument-wcf capture interleaves MDAS transport chunk
|
||||
markers (`0x9F`/`0x9E`) into a large `pResultBuff`, so raw byte-slicing yields a corrupted NRBF stream.
|
||||
The **clean** contract-level byte[] (what the WCF channel reassembles) is dumped via the
|
||||
`AVEVA_HISTORIAN_SQL_DUMP` env var on `HistorianWcfSqlClient` — that is the golden fixture in
|
||||
`WcfSqlResultProtocolTests` (the benign `SELECT 1 AS ProbeValue`, no sensitive data).
|
||||
|
||||
## Shipped surface
|
||||
|
||||
- `HistorianClient.ExecuteSqlCommandAsync(command, option = ExecuteRecord)` → `HistorianSqlResult`
|
||||
(`Columns` name+type, `Rows` typed values, `ReturnValue`).
|
||||
- Models `HistorianSqlResult` / `HistorianSqlColumn` / `HistorianSqlExecuteOption`;
|
||||
`HistorianSqlResultProtocol` (NRBF + diffgram parser); `HistorianWcfSqlClient` (ExeC/GetR
|
||||
orchestration); golden `WcfSqlResultProtocolTests`; gated live tests
|
||||
(`ExecuteSqlCommandAsync_AgainstLocalHistorian_ReturnsRecordSet` and `_MultiColumnMultiRow`).
|
||||
|
||||
## Scope notes
|
||||
|
||||
- `ExecuteRecord` (record set) is the evidence-backed path. `ExecuteNonQuery`/`ExecuteScalar`/
|
||||
`ExecuteRecordDirect` are accepted via the option enum but their non-record-set return shapes are not
|
||||
separately captured — a non-record result yields empty `Columns`/`Rows` with the `ReturnValue` set.
|
||||
- The command is whatever the Historian's SQL surface accepts (it routes to the Runtime DB). No
|
||||
client-side SQL validation is performed.
|
||||
@@ -100,14 +100,24 @@ GETHI probe is the **handle string format**: the native client sends the GUID
|
||||
lowercase, so a probe that passed the GUID without upcasing would not byte-match what
|
||||
the server's session table is keyed on.
|
||||
|
||||
**Implication / open lead (not yet retested):** the GETHI/ExeC/QTB/QTG family failures
|
||||
may be (at least partly) a **handle-format** issue, not (only) a missing native
|
||||
registration step. The highest-value cheap follow-up is to **re-probe GETHI and ExeC
|
||||
with the uppercased storage-session GUID** before assuming the registration wall. If
|
||||
they also return data, the "wall" collapses to a formatting bug and R1.4/R1.5/R1.6/R1.1
|
||||
may be reachable without any new RE. This has **not** been done yet — do not reclassify
|
||||
those items until it is. GETRP is shipped because it was directly captured + live-verified
|
||||
end-to-end; the rest remain `ProtocolEvidenceMissingException`/unprobed until tested.
|
||||
**Implication — CONFIRMED, the wall is largely a handle-format bug.** The follow-up was done:
|
||||
GETHI and **ExeC both return data with the uppercased storage-session GUID**.
|
||||
|
||||
- **R1.1 `ExecuteSqlCommandAsync` (ExeC + GetR) — SHIPPED + live-verified (2026-06-20).**
|
||||
`ExecuteSqlCommandAsync(sql)` → `HistorianSqlResult`. `Retr.GetV` prime → `ExeC(handle,
|
||||
sql, option=0, ref queryHandle)` → `GetR` loop. Note: **`GetR` returns false even on
|
||||
success** (the byte stream is in `pResultBuff` regardless; false = final page). `pResultBuff`
|
||||
is an **NRBF `DataTable`** (`SerializationFormat.Xml`: `XmlSchema` + `XmlDiffGram`), decoded
|
||||
read-only with `System.Formats.Nrbf` + `XDocument` (BinaryFormatter is gone from .NET 10).
|
||||
Shipped: `HistorianSqlResultProtocol`, `HistorianWcfSqlClient`, golden `WcfSqlResultProtocolTests`,
|
||||
gated live tests. See `docs/reverse-engineering/wcf-exec-sql.md`.
|
||||
- **GETHI (R1.4)** also returns data with the uppercase handle (probe; public API not yet shipped).
|
||||
|
||||
So the "wall" collapses to the handle **format** for the Retrieval/Status string-handle ops.
|
||||
**Exception — QTB/QTG:** `StartTagQuery` does *not* punch through; captured with a correctly
|
||||
uppercase handle it still fails `error 1` **server-side** (`CMdServer::StartActiveTagnamesQuery`
|
||||
over `\\.\pipe\aahMetadataServer\console`) — a metadata-server blocker, independent of handle
|
||||
format. Name-based ops route around it.
|
||||
|
||||
See `HistorianRuntimeParameterProtocol`, `IStatusServiceContract2.GetRuntimeParameter`,
|
||||
golden `WcfRuntimeParameterProtocolTests`, and capture tooling
|
||||
|
||||
Reference in New Issue
Block a user