Merge re/r1.4-gethi-finding: R1.1 ExecuteSqlCommand + R1.4 GetHistorianInfo (bounded)
# Conflicts: # docs/plans/hcal-roadmap.md # src/AVEVA.Historian.Client/HistorianClient.cs # src/AVEVA.Historian.Client/Protocol/Historian2020ProtocolDialect.cs # tests/AVEVA.Historian.Client.Tests/HistorianClientIntegrationTests.cs # tools/AVEVA.Historian.NativeTraceHarness/Program.cs
This commit is contained in:
Joseph Doherty
@@ -0,0 +1,86 @@
|
||||
# SQL command execution over 2020 WCF — ExeC + GetR (HCAL R1.1)
|
||||
|
||||
**Status: ✅ DONE + live-verified (2026-06-20).** `HistorianClient.ExecuteSqlCommandAsync(sql)` runs a
|
||||
SQL command against the Historian over the 2020 WCF ops `aa/Retr/ExeC` (ExecuteSqlCommand) +
|
||||
`aa/Retr/GetR` (GetRecordSetByteStream) and returns the record set as a `HistorianSqlResult` (the
|
||||
managed equivalent of the native `DataTable`). Live-verified end-to-end from the pure-managed .NET 10
|
||||
client against the local 2020 Historian (single-cell, multi-column/multi-row, and NULL cases).
|
||||
|
||||
## The ops
|
||||
|
||||
Both are on the Retrieval service (`IRetrievalServiceContract3`), **string-handle** ops reached with
|
||||
the Open2 storage-session GUID formatted `storageSessionId.ToString("D").ToUpperInvariant()` (the same
|
||||
uppercase handle that unlocked GETRP/GETHI; see `wcf-string-handle-wall.md`). `Retr.GetV` is primed
|
||||
first.
|
||||
|
||||
```
|
||||
bool ExecuteSqlCommand(string handle, string command, uint option,
|
||||
ref uint queryHandle, out int retValue, out uint errorSize, out byte[] errorBuffer)
|
||||
|
||||
bool GetRecordSetByteStream(string handle, uint queryHandle, ref uint sequence,
|
||||
out uint resultSize, out byte[] pResultBuff, out uint errorSize, out byte[] errorBuffer)
|
||||
```
|
||||
|
||||
- **`command`** is sent as a plain string (MDAS-encoded ASCII), e.g. `SELECT 1 AS ProbeValue`.
|
||||
- **`option`** = `HistorianSqlExecuteOption` (`ExecuteRecord=0` is the captured/proven record-set path).
|
||||
- **`ExeC`** returns the assigned `queryHandle` (and `retValue`); pass it to `GetR`.
|
||||
- **`GetR` returns `false` even on success** — the byte stream is in `pResultBuff` regardless; a
|
||||
`false` result just signals the final page. So the orchestrator always consumes `pResultBuff`, then
|
||||
stops on a `false` result or an empty page. (`sequence` is the paging cursor; small record sets
|
||||
return everything in one call.)
|
||||
|
||||
## The result stream — NRBF-wrapped DataTable (no BinaryFormatter)
|
||||
|
||||
`GetR`'s `pResultBuff` is a **.NET Remoting Binary Format (NRBF)** stream wrapping a
|
||||
`System.Data.DataTable` serialized with `SerializationFormat.Xml`. Stream shape (captured):
|
||||
|
||||
```
|
||||
SerializationHeader (00 01 00 00 00 FF FF FF FF 01 00 00 00 00 00 00 00)
|
||||
BinaryLibrary (0C): "System.Data, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089"
|
||||
ClassWithMembersAndTypes (05): System.Data.DataTable, members:
|
||||
DataTable.RemotingVersion -> System.Version object
|
||||
XmlSchema -> BinaryObjectString (XSD: column names + xs types)
|
||||
XmlDiffGram -> BinaryObjectString (ADO.NET diffgram: the rows)
|
||||
System.Version object (_Major/_Minor/_Build/_Revision)
|
||||
MessageEnd (0B)
|
||||
```
|
||||
|
||||
**BinaryFormatter is removed from .NET 10**, so the stream is decoded read-only with
|
||||
`System.Formats.Nrbf.NrbfDecoder` (the sanctioned successor — it parses records without instantiating
|
||||
or executing any payload type; added as a managed `PackageReference`). The two embedded XML strings are
|
||||
then parsed with `XDocument`:
|
||||
|
||||
- **XmlSchema** → columns (name + XSD type) under the `msdata:MainDataTable` element's sequence.
|
||||
- **XmlDiffGram** → rows (each row element under the dataset; cells are child elements or attributes
|
||||
matching column names). NULL cells are simply absent → parsed as `null`.
|
||||
|
||||
Values are typed per the XSD type (`xs:int`→int, `xs:string`→string, `xs:dateTime`→DateTime, …),
|
||||
falling back to the raw string for unrecognized types. Only the `SerializationFormat.Xml` DataTable
|
||||
shape is evidence-backed; a stream whose root is not a DataTable class record, or that lacks the two
|
||||
XML members, throws `ProtocolEvidenceMissingException`.
|
||||
|
||||
## Capture / decode tooling
|
||||
|
||||
`scripts/Capture-ExecSql.ps1` (NativeTraceHarness `exec-sql` scenario + instrument-wcf-{write,read}message)
|
||||
captures the ExeC/GetR exchange. ⚠️ A **raw** instrument-wcf capture interleaves MDAS transport chunk
|
||||
markers (`0x9F`/`0x9E`) into a large `pResultBuff`, so raw byte-slicing yields a corrupted NRBF stream.
|
||||
The **clean** contract-level byte[] (what the WCF channel reassembles) is dumped via the
|
||||
`AVEVA_HISTORIAN_SQL_DUMP` env var on `HistorianWcfSqlClient` — that is the golden fixture in
|
||||
`WcfSqlResultProtocolTests` (the benign `SELECT 1 AS ProbeValue`, no sensitive data).
|
||||
|
||||
## Shipped surface
|
||||
|
||||
- `HistorianClient.ExecuteSqlCommandAsync(command, option = ExecuteRecord)` → `HistorianSqlResult`
|
||||
(`Columns` name+type, `Rows` typed values, `ReturnValue`).
|
||||
- Models `HistorianSqlResult` / `HistorianSqlColumn` / `HistorianSqlExecuteOption`;
|
||||
`HistorianSqlResultProtocol` (NRBF + diffgram parser); `HistorianWcfSqlClient` (ExeC/GetR
|
||||
orchestration); golden `WcfSqlResultProtocolTests`; gated live tests
|
||||
(`ExecuteSqlCommandAsync_AgainstLocalHistorian_ReturnsRecordSet` and `_MultiColumnMultiRow`).
|
||||
|
||||
## Scope notes
|
||||
|
||||
- `ExecuteRecord` (record set) is the evidence-backed path. `ExecuteNonQuery`/`ExecuteScalar`/
|
||||
`ExecuteRecordDirect` are accepted via the option enum but their non-record-set return shapes are not
|
||||
separately captured — a non-record result yields empty `Columns`/`Rows` with the `ReturnValue` set.
|
||||
- The command is whatever the Historian's SQL surface accepts (it routes to the Runtime DB). No
|
||||
client-side SQL validation is performed.
|
||||
@@ -0,0 +1,68 @@
|
||||
# GetHistorianInfo over 2020 WCF — GETHI is named-value-only (HCAL R1.4)
|
||||
|
||||
**Status: ⚠ Bounded out on 2020 WCF (2026-06-20).** `GetHistorianInfoAsync` is **not shipped**:
|
||||
the one field that motivates it — `EventStorageMode` — is **not on the 2020 WCF wire**. The
|
||||
version field that GETHI *does* return over WCF is already exposed (`ProbeAsync`,
|
||||
`GetRuntimeParameterAsync("HistorianVersion")`), so there is nothing new to ship here without a
|
||||
2023 R2 gRPC server. This parallels R1.3 (`GetServerTimeZone`), which is likewise 2023R2-only.
|
||||
|
||||
## What the capture showed
|
||||
|
||||
`scripts/Capture-HistorianInfo.ps1` drives the native `HistorianAccess.GetHistorianInfo(out
|
||||
HistorianInfo, out error)` through the instrumented (`instrument-wcf-{write,read}message`)
|
||||
`current/aahClientManaged.dll`. The native call **succeeds** and returns
|
||||
`EventStorageMode = Blocks`, `ServerVersion = 20,0,000,000`, no error.
|
||||
|
||||
But the wire tells a different story (`scripts/decode-historian-info-capture.py`):
|
||||
|
||||
- The only `GETHI` op on the wire is **`aa/Stat/GETHI(handle, pRequestBuff)`** with
|
||||
`pRequestBuff = 53 67 02 00` (sig `0x6753` + version `2`) `+ uint charCount(16) + UTF-16
|
||||
"HistorianVersion"` — i.e. the **named-value request**, identical to the GETRP/version shape.
|
||||
- Its response `pResponseBuff` is **~30 bytes**: `uint charCount(12) + UTF-16 "20,0,000,000"`
|
||||
(+ a `02 00 01 00` trailer). **Just the version** — not a 518-byte struct.
|
||||
- The post-GETHI ops in the same capture are `Hist/UpdC3` + a run of `Stat/GetSystemParameter`
|
||||
(`AllowOriginals`, `HistorianPartner`, `HistorianVersion`, `MaxCyclicStorageTimeout`,
|
||||
`RealTimeWindow`, `FutureTimeThreshold`, `AllowRenameTags`). **None carries a storage-mode
|
||||
value.** So the native wrapper's `EventStorageMode` is derived by the C++ HCAL **outside the
|
||||
WCF wire**, not fetched over it.
|
||||
|
||||
## Probe: does GETHI expose storage mode under any name?
|
||||
|
||||
`StringHandleProbeDiagnosticTests.GETHI_CandidateInfoNames_AgainstLocalHistorian` (gated on
|
||||
`HISTORIAN_HOST=localhost`) issues GETHI for `HistorianVersion` plus seven storage-mode name
|
||||
guesses. Result on the live 2020 server:
|
||||
|
||||
| GETHI parameter name | result |
|
||||
|---|---|
|
||||
| `HistorianVersion` | **ok=True**, respLen=32 (version) |
|
||||
| `EventStorageMode`, `EventStorageType`, `StorageType`, `HistorianEventStorageMode`, `EventStorage`, `StorageMode`, `HistorianInfo` | **ok=False**, errLen=5, empty |
|
||||
|
||||
So GETHI on 2020 WCF is a strict named-value lookup with exactly one known-good key
|
||||
(`HistorianVersion`). There is no storage-mode key, no full-struct request.
|
||||
|
||||
## Why the 518-byte struct doesn't apply here
|
||||
|
||||
The 2023 R2 decompiled `ArchestrA.HistorianAccess.GetHistorianInfo` (analysis folder) allocates
|
||||
a **518-byte `HISTORIAN_INFO`** struct, pre-inits `int32 @514` to `-1`, calls native HCAL
|
||||
(vtable+648) which fills it, then reads version (UTF-16 @0) + `EventStorageMode` (`@514`:
|
||||
`-1`=Unsupported, `0`=Database, else=Blocks). That is the **HCAL-native / 2023R2 gRPC**
|
||||
front-door model (`StatusService.GetHistorianInfo` returns `bytes btHistorianInfo`). On **2020
|
||||
WCF** that struct is never marshaled across the wire — only the version named-value is. The
|
||||
native client's `EventStorageMode` therefore comes from C++-internal state the managed WCF
|
||||
replay cannot observe or reproduce.
|
||||
|
||||
## Conclusion / where it lands
|
||||
|
||||
- **2020 WCF:** `GetHistorianInfoAsync` would add nothing over existing surface (version only) and
|
||||
could not report a real `EventStorageMode` — so it is intentionally **not shipped** (no hollow
|
||||
`Unsupported`-returning API; project discipline: don't ship misleading behavior).
|
||||
- **2023 R2 gRPC:** `Status.GetHistorianInfo` returns the full 518-byte `btHistorianInfo`; decode
|
||||
version@0 + `EventStorageMode`@514 there. Build + verify against a live 2023 R2 server. The
|
||||
`HistorianInfo` / `HistorianEventStorageMode` public types should land alongside that path.
|
||||
|
||||
## Tooling kept as RE aids
|
||||
|
||||
- `tools/AVEVA.Historian.NativeTraceHarness` `historian-info` scenario (drives the native call).
|
||||
- `scripts/Capture-HistorianInfo.ps1` + `scripts/decode-historian-info-capture.py`.
|
||||
- `StringHandleProbeDiagnosticTests.GETHI_CandidateInfoNames_AgainstLocalHistorian` (locks the
|
||||
named-value-only finding; gated).
|
||||
@@ -110,14 +110,28 @@ GETHI probe is the **handle string format**: the native client sends the GUID
|
||||
lowercase, so a probe that passed the GUID without upcasing would not byte-match what
|
||||
the server's session table is keyed on.
|
||||
|
||||
**Implication / open lead (not yet retested):** the GETHI/ExeC/QTB/QTG family failures
|
||||
may be (at least partly) a **handle-format** issue, not (only) a missing native
|
||||
registration step. The highest-value cheap follow-up is to **re-probe GETHI and ExeC
|
||||
with the uppercased storage-session GUID** before assuming the registration wall. If
|
||||
they also return data, the "wall" collapses to a formatting bug and R1.4/R1.5/R1.6/R1.1
|
||||
may be reachable without any new RE. This has **not** been done yet — do not reclassify
|
||||
those items until it is. GETRP is shipped because it was directly captured + live-verified
|
||||
end-to-end; the rest remain `ProtocolEvidenceMissingException`/unprobed until tested.
|
||||
**Implication — CONFIRMED, the wall is largely a handle-format bug.** The follow-up was done:
|
||||
GETHI and **ExeC both return data with the uppercased storage-session GUID**.
|
||||
|
||||
- **R1.1 `ExecuteSqlCommandAsync` (ExeC + GetR) — SHIPPED + live-verified (2026-06-20).**
|
||||
`ExecuteSqlCommandAsync(sql)` → `HistorianSqlResult`. `Retr.GetV` prime → `ExeC(handle,
|
||||
sql, option=0, ref queryHandle)` → `GetR` loop. Note: **`GetR` returns false even on
|
||||
success** (the byte stream is in `pResultBuff` regardless; false = final page). `pResultBuff`
|
||||
is an **NRBF `DataTable`** (`SerializationFormat.Xml`: `XmlSchema` + `XmlDiffGram`), decoded
|
||||
read-only with `System.Formats.Nrbf` + `XDocument` (BinaryFormatter is gone from .NET 10).
|
||||
Shipped: `HistorianSqlResultProtocol`, `HistorianWcfSqlClient`, golden `WcfSqlResultProtocolTests`,
|
||||
gated live tests. See `docs/reverse-engineering/wcf-exec-sql.md`.
|
||||
- **GETHI (R1.4)** returns data with the uppercase handle, **but only the named `HistorianVersion`
|
||||
value** — over 2020 WCF GETHI is a named-value query (the only working key), *not* a full-struct
|
||||
read. `EventStorageMode` (the 518-byte-struct `@514` field) is **not on the 2020 WCF wire**; it is
|
||||
the 2023R2 HCAL-native/gRPC model. So R1.4 is **bounded out on WCF / gRPC-2023R2-only** and the
|
||||
public API is intentionally not shipped. Full analysis: `docs/reverse-engineering/wcf-historian-info.md`.
|
||||
|
||||
So the "wall" collapses to the handle **format** for the Retrieval/Status string-handle ops.
|
||||
**Exception — QTB/QTG:** `StartTagQuery` does *not* punch through; captured with a correctly
|
||||
uppercase handle it still fails `error 1` **server-side** (`CMdServer::StartActiveTagnamesQuery`
|
||||
over `\\.\pipe\aahMetadataServer\console`) — a metadata-server blocker, independent of handle
|
||||
format. Name-based ops route around it.
|
||||
|
||||
See `HistorianRuntimeParameterProtocol`, `IStatusServiceContract2.GetRuntimeParameter`,
|
||||
golden `WcfRuntimeParameterProtocolTests`, and capture tooling
|
||||
|
||||
Reference in New Issue
Block a user