diff --git a/docs/reverse-engineering/grpc-event-query-capture.md b/docs/reverse-engineering/grpc-event-query-capture.md index 17de437..131cf7b 100644 --- a/docs/reverse-engineering/grpc-event-query-capture.md +++ b/docs/reverse-engineering/grpc-event-query-capture.md @@ -139,3 +139,203 @@ as the captured-correct request format** for when the open is rebuilt. Capture artifacts (gitignored): `artifacts/reverse-engineering/grpc-event-capture/` — `event-capture.ndjson` (Event), `process-connect-2.ndjson` (Process). + +## v8 `openParameters` fully decoded (2026-06-23) + the ECDH ExchangeKey finding + +Full byte map of the native Event-connection `openParameters` (302 bytes; identity values +redacted — they are session-specific and sit in the gitignored capture): + +``` +[0] byte 0x08 format version = 8 +[1] byte 0xf0 constant marker +[2..20] 19 × 0x00 +[21] byte 0x01 constant marker +[22..37] 16B GUID per-session client key +[38..41] u32 username length (chars) +[42..N] UTF-16 username (HistorianString) +[..+1] u16 credential-token length (= 26 in the capture) +[..] 26B token ECDH-derived credential token <-- see below +[94] byte 0x04 ClientType (= our NativeClientType 4) +[95] byte ConnectionType 01 = Event / 02 = Process <-- THE GATE +[96] byte flag 01 (Event) / 00 (Process) +[97..] control bytes (0x03 ... small region, not fully named) +[~114..117]u32 FormatVersion=3 +[..] HistorianString machine/server node name +[..] HistorianString client node name "()" +[..] u32 session-variable (process-ish) +[..] u32 / zeros +[..] u32 datasource len +[..] UTF-16 datasource id e.g. "2023.1219.4004.5" +[270..285] 16 × 0xff ShardId (all-FF = unset; our v6 sends Empty) +[286..289] u32 client/hcal version int +[290..297] i64 FILETIME ClientTimestamp +[298..301] u32 0 +``` + +The tail (`FormatVersion` → machine → clientNode → datasource → ShardId → version → timestamp) +is the **same `ClientCommonInfo` our v6 already emits**. The new/different parts are: version byte, +the `[1]`/`[21]` markers, the GUID position, the **26-byte credential token** (vs v6's fixed-size +block), the **`ConnectionType` byte**, and ShardId=FF. + +**The auth is ECDH, not Negotiate.** The capture's `ExchangeKey` buffers begin `45 43 4b 31` = +ASCII **`"ECK1"`** + a 64-byte EC public-key point — a Diffie-Hellman key exchange — and the 26-byte +`openParameters` token is derived from it. `HistorianSecurityMode` offers only `Disabled` / `None` / +`TransportCertificate`; the harness used `TransportCertificate`, which is what drives the ECDH +`ExchangeKey`. There is **no TLS+Negotiate mode** on the native client (it couples TLS with the cert +ECDH path), so a Negotiate-auth v8 capture cannot be produced from the native client. + +**Key de-risking insight:** our SDK's v6 `OpenConnection` sends a **fully zeroed** 1026-byte +credential block (`credentialBlock: new byte[1026]`) and reads still work — because authentication is +actually carried by the separate `StorageService.ValidateClientCredential` (Negotiate) handshake, not +by the bytes inside `openParameters`. By analogy the v8 `[68..93]` token may likewise be **ignorable** +once `ValidateClientCredential` has run. So the first build hypothesis (cheapest, read-only to test): + +> Reuse the SDK's existing `ValidateClientCredential` handshake, then send a **v8 `OpenConnection` +> with `ConnectionType=Event` and a zeroed credential token**, and see whether the 2023 R2 server +> returns event rows. + +If that works, the ECDH ExchangeKey RE is unnecessary. If it fails, the fallback is full reproduction +of the ECDH `ExchangeKey` handshake (curve/KDF/cipher) — a much larger crypto-RE effort. Build path: +add `SerializeNativeOpenConnectionVersion8(connectionType)` to `HistorianOpen2Protocol`, wire the gRPC +event handshake to use it (events only; reads stay on v6), live-test (non-destructive). Full hex in +the gitignored capture. + +### Path A built + live-tested 2026-06-23 — DISPROVEN (v8 is coupled to ExchangeKey) + +Built `HistorianOpen2Protocol.SerializeNativeOpenConnectionVersion8` (golden-tested, +`Version8EventSerializerReproducesCapturedNativeStructure` — reproduces the captured 302-byte +structure exactly) + `HistorianNativeHandshake.BuildEventOpenConnectionVersion8Request` (zeroed +credential token) + an `eventConnection` switch on `HistorianGrpcHandshake.OpenSession`, and live-ran +the event read against the server. Result: the v8 `OpenConnection` was **parsed by the server** (got +past the byte format) but **rejected at the auth check** with native error + +``` +type=132 code=34 "aahHcapLib::HistoryService::EstablishConnection — Failed to get client key" +``` + +i.e. `EstablishConnection` could not find a server-side **client key** for our session. In the v6 +path that key is established by `StorageService.ValidateClientCredential` (which is why v6 reads +work); the v8 path looks it up in the registry that **`HistoryService.ExchangeKey` (ECDH)** populates, +and there is **no `ValidateClientCredential` on `HistoryService`** in the gRPC contract. So the server +branches on the OpenConnection version: v6 accepts the Negotiate-established key, **v8 requires the +ExchangeKey-established key**. The zeroed-token hypothesis is therefore disproven — not because of the +token bytes, but because the whole v8 path is gated on `ExchangeKey` having run first. + +**Status:** the v8 serializer/builder are correct and retained (golden-tested), plus the +`OpenConnection` failure now decodes the native error (type/code/ASCII). The event orchestrator is +reverted to the v6 session (gated test still pins the no-row throw). The remaining route is **Path B: +implement `HistoryService.ExchangeKey`** — `"ECK1"` + a 64-byte EC public-key point (P-256 X‖Y, by the +size) — using .NET `ECDiffieHellman`, establish the client key, then reissue the v8 `OpenConnection`. +Open question for Path B: whether merely *completing* the ECDH key agreement registers the client key +(so the zeroed openParameters token still rides through), or whether the token must also be derived +from the shared secret (full KDF/cipher RE). + +### Path B started 2026-06-23 — ExchangeKey ECDH works; cleared 2 of 3 layers + +Implemented `HistoryService.ExchangeKey` as a **pure-managed P-256 ECDH** key exchange +(`HistorianNativeHandshake.BuildExchangeKeyClientHello` / `DeriveExchangeKeySecret`, .NET +`ECDiffieHellman` over `nistP256`; wire format `"ECK1" + u32(32) + X(32) + Y(32)`) and wired it into +`HistorianGrpcHandshake.OpenSession(eventConnection: true)` ahead of the v8 `OpenConnection`, +on the same context-key handle. Live result against the server: the **`ExchangeKey` RPC succeeds** +(the server accepted our public key), and the v8 `OpenConnection` error **moved one layer deeper**: + +``` +Path A (no ExchangeKey): 132/34 "Failed to get client key" +Path B (ExchangeKey ECDH): 132/171 AuthenticationFailed "EstablishConnection — Authentication failed" +``` + +So the ECDH cleared the client-key check; the remaining blocker is **authentication**: the 26-byte +v8 credential token must be a *valid* value derived from the ECDH shared secret (not zeros). + +### Token crypto traced 2026-06-23 (Frida → Windows CNG) — KDF found, token construction still open + +Hooked Windows CNG (`bcrypt.dll`/`ncrypt.dll`) while the native harness ran a real ExchangeKey +(`scripts/frida/aahclientmanaged-cng-exchangekey.js` + `artifacts/.../cng-trace.py`). Findings: + +- **The ECDH + KDF are standard CNG, driven by managed `System.Security.Cryptography.ECDiffieHellmanCng`** + (backtrace top frame = `System.Core.ni.dll`; the caller is aahClientManaged's C++/CLI ``): + `NCryptSecretAgreement` (P-256) → `NCryptDeriveKey(KDF=HASH, HASH_ALGORITHM=SHA256, 32 bytes)`. So the + derived key = **SHA256(ECDH shared secret)** — exactly `ECDiffieHellmanCng{ KeyDerivationFunction=Hash, + HashAlgorithm=SHA256 }.DeriveKeyMaterial(...)`. Our managed `DeriveExchangeKeySecret` should switch to + this (SHA256 of the raw agreement) to match. +- **`"ECK1"` is NOT AVEVA-custom** — it is the standard Windows CNG `BCRYPT_ECCPUBLIC_BLOB` magic for + P-256 (`NCryptExportKey`/`ImportKey` emit exactly `ECK1 + len(32) + X(32) + Y(32)`), confirming our + `BuildExchangeKeyClientHello` wire format is correct. +- **The 26-byte token is a custom construction that is not yet reproduced.** Correlated one run's + derived key (`SHA256(secret)`) with that run's token (from the IL openParameters capture): a + 528-candidate offline cracker (HMAC/SHA/AES-GCM/CBC/CTR over the derived key × request slices × + creds) found **no match**, and the token matches **none** of the traced hash digests. The token + starts with a constant `0x8e` marker in both captured runs (so it is structured, not raw cipher + output). It is built in managed code between the `DeriveKeyMaterial` call and the openParameters + assembly. + +**dnlib IL extraction 2026-06-23 — the token scheme is fully reverse-engineered.** ILSpy can't +decompile the mixed-mode assembly (crashes), but loading `dnlib` in PowerShell and scanning the IL +recovered the whole construction: + +- **`::CHistoryConnectionGrpc.GetClientKey`** is the ECDH driver: `new ECDiffieHellmanCng()` + → `KeyDerivationFunction = Hash`, `HashAlgorithm = SHA256`, `KeySize = 256` → + `GrpcHistoryClient.ExchangeKey(strHandle, ourPubKey.ToByteArray(), out serverPub, out err)` → + `CngKey.Import(serverPub, CngKeyBlobFormat.EccPublicBlob)` → **`DeriveKeyMaterial`** = the 32-byte + client key = **`SHA256(ECDH shared secret)`**. (So our managed side should derive the key the same + way — `ECDiffieHellman` raw agreement then SHA256, or equivalently `DeriveKeyFromHash(..., SHA256)`.) +- **The 26-byte token is built by `aahClientCommon.CClientBase.ConfigureOpenConnection`** (the lone + caller of `GetClientKey`) using the **`HistorianCrypto.NRC4_V2.aahCryptV2`** scheme — a custom + **MD5-keyed RC4 stream cipher with a version prefix**: + - `aahCryptV2.body`/`HashData` = **MD5** (verified: the IL loads MD5 round constants `0xd76aa478`… + and rotates 7/12/17/22). + - `aahCryptV2.prepare_key` = standard **RC4 KSA** seeding the 256-byte S-box from a **16-byte (MD5)** + key (`std.array`). + - `aahCryptV2.enc_buffer` = `MD5(...)` → key, then **`rc4encrypt`** the body; `enc` prepends a + scheme **prefix** (`NRC4_V2.PrefixV2` / `InnerPrefixV2`) — the constant `0x8e` token marker. + - `from_GUID` keys the cipher from a GUID string. + +So the token = `prefix + RC4(plaintext, key = MD5(keyMaterial))`, where the key material ties back to +the `SHA256(ECDH secret)` client key. **This is 100% reproducible in pure managed code** (RC4 + MD5 +are ~40 lines; nothing AVEVA ships). + +**Remaining to finish (next cycle):** read `ConfigureOpenConnection`'s exact wiring (which value is +MD5'd for the RC4 key, what plaintext is encrypted, the exact prefix bytes — a little more dnlib IL), +implement `aahCryptV2` (RC4+MD5+prefix) managed-side, set the v8 token = that, and live-test +(non-destructive). The offline correlation data (one run's derived key + token + openParameters) is +captured under `artifacts/.../` to validate the managed reproduction before going live. + +### Token implemented + auth WORKS live (2026-06-23); row retrieval still 0 — proven NOT a payload issue + +`token = RC4(password-UTF16LE, key = MD5(SHA256(ECDH secret)))` was implemented in pure managed C# +(`HistorianNativeHandshake.BuildExchangeKeyCredentialToken` + `Rc4`; client key via +`DeriveKeyFromHash(SHA256)`), golden-tested (RC4 standard vector + token construction), and +**live-verified**: the v8 `OpenConnection` now **authenticates** against the 2023 R2 server (past the +`132/171 AuthenticationFailed` wall). Auth is solved. + +The event **query** still returns `version-11 rowCount-0` while the native returns 50 for an +**identical** request. Exhaustively ruled out as the cause (all confirmed live, opt-in +`EventReadDiagnostic` test + the IL rewrite extended to log string/uint handle fields): + +- `StartEventQuery` request: **byte-identical** to the native (v6 layout) +- v8 `OpenConnection` `openParameters`: **byte-identical** to the native (302 bytes) once ClientNodeName + is matched — every control byte, ConnectionType, token framing, ShardId, etc. +- Handle usage: identical — `ExchangeKey`→contextKey, registration→storage-session GUID (`strHandle`), + query→client uint (`uiHandle`); our parsed handles are valid (registration `RTag/EnsT=True`, valid + `queryHandle`) +- `queryRequestType = 3`, registration sequence/order, gzip metadata header — all match +- window (events exist; native returns 50 *now*), eventCount — not it + +So **every observable client-side byte matches the native**, yet the server scopes 0 events to our +connection. The event RPCs succeed over our transport and return a valid *empty* result (not a +transport error), so it is **not a payload or transport-incompatibility issue** — it is a +connection/server-level difference (e.g. session affinity tied to the native `Grpc.Core` HTTP/2 +connection or a connection-identity the server uses to scope events) that is **invisible to, and +unfixable by, client payload matching.** Closing it needs server-side insight or a different angle +(e.g. compare the full HTTP/2 connection setup / TLS identity), not more wire-payload RE. + +**Shipped this effort:** the complete ExchangeKey crypto (ECDH + SHA256 + MD5-keyed RC4 token) — the +hard wall — pure managed, golden-tested, auth live-verified. Orchestrator stays on the no-row throw; +gated test unchanged. + +**2 of 3 layers cleared** (key exchange + client key); the 3rd (token construction) is localized to a +specific managed method, pending dnlib extraction. ExchangeKey + the v8 serializer are committed; the +orchestrator stays on v6 (set `eventConnection: true` to re-arm once the token construction lands). The +token-loop routing guardrail (`HistorianGrpcHandshakeRoutingTests`) was scoped to the closure so the +legitimate ExchangeKey call is allowed while still pinning that the Negotiate token loop never routes +there. diff --git a/scripts/frida/aahclientmanaged-cng-exchangekey.js b/scripts/frida/aahclientmanaged-cng-exchangekey.js new file mode 100644 index 0000000..8acb6d6 --- /dev/null +++ b/scripts/frida/aahclientmanaged-cng-exchangekey.js @@ -0,0 +1,173 @@ +// Frida hook for the native ExchangeKey credential-token crypto (Windows CNG / bcrypt.dll). +// Traces the ECDH secret agreement, the KDF (with its parameter list), symmetric-key import, and +// encrypt/hash so the 26-byte v8 credential-token derivation can be reconstructed in managed code. +// Reverse-engineering aid only — observes the native client; nothing is shipped from here. +'use strict'; + +function resolve(modName, fnName) { + let m = null; + try { m = Process.getModuleByName(modName); } catch (e) { + try { m = Module.load(modName); } catch (e2) { return null; } + } + try { return m.findExportByName(fnName); } catch (e) { return null; } +} + +function dump(label, ptr, len) { + if (ptr.isNull() || len <= 0) { console.log(label + ' '); return; } + const n = Math.min(len, 256); + console.log(label + ' (' + len + ' bytes)\n' + hexdump(ptr, { length: n, header: false, ansi: false })); +} + +function hook(modName, fnName, onEnter, onLeave) { + const addr = resolve(modName, fnName); + if (!addr) { console.log('[skip] ' + modName + '!' + fnName + ' not found'); return; } + Interceptor.attach(addr, { onEnter: onEnter, onLeave: onLeave }); + console.log('[hooked] ' + modName + '!' + fnName); +} + +// BCryptOpenAlgorithmProvider(phAlgorithm, pszAlgId, pszImplementation, dwFlags) — names every algo used. +hook('bcrypt.dll', 'BCryptOpenAlgorithmProvider', function (a) { + console.log('[OpenAlgorithmProvider] algId=' + (a[1].isNull() ? '?' : a[1].readUtf16String())); +}); + +// BCryptSecretAgreement(hPrivKey, hPubKey, *phAgreedSecret, flags) +hook('bcrypt.dll', 'BCryptSecretAgreement', function (a) { + console.log('[SecretAgreement] hPriv=' + a[0] + ' hPub=' + a[1]); +}); + +// Decode a BCryptBufferDesc parameter list (used by BCryptDeriveKey) into (type -> bytes). +function dumpParamList(pParamList) { + if (pParamList.isNull()) { console.log(' paramList '); return; } + const cBuffers = pParamList.add(4).readU32(); // ULONG ulVersion; ULONG cBuffers; + const pBuffers = pParamList.add(8).readPointer(); // BCryptBuffer* pBuffers; + const names = { 0: 'HASH_ALGORITHM', 1: 'SECRET_PREPEND', 2: 'SECRET_APPEND', 3: 'HMAC_KEY', + 4: 'TLS_PRF_LABEL', 5: 'TLS_PRF_SEED', 6: 'SECRET_HANDLE', 8: 'SP80056A_CONCAT', + 0xD: 'LABEL', 0xE: 'CONTEXT', 0xF: 'SALT', 0x10: 'ITERATION_COUNT' }; + console.log(' paramList cBuffers=' + cBuffers); + for (let i = 0; i < cBuffers; i++) { + const b = pBuffers.add(i * 16); // { ULONG cbBuffer; ULONG BufferType; PVOID pvBuffer; } + const cb = b.readU32(); + const type = b.add(4).readU32(); + const pv = b.add(8).readPointer(); + const tn = names[type] || ('0x' + type.toString(16)); + if (type === 0 || type === 4 || type === 0xD) { // string-ish (hash alg name / label) + console.log(' [' + tn + '] ' + (pv.isNull() ? '?' : pv.readUtf16String())); + } else { + dump(' [' + tn + ']', pv, cb); + } + } +} + +// BCryptDeriveKey(hSecret, pwszKDF, *pParamList, pbDerivedKey, cbDerivedKey, *pcbResult, flags) +hook('bcrypt.dll', 'BCryptDeriveKey', function (a) { + this.kdf = a[1].isNull() ? '?' : a[1].readUtf16String(); + this.outKey = a[3]; this.pcb = a[5]; + console.log('[DeriveKey] KDF=' + this.kdf + ' cbDerivedKey=' + a[4].toInt32()); + dumpParamList(a[2]); +}, function () { + const n = this.pcb.isNull() ? 0 : this.pcb.readU32(); + dump('[DeriveKey] derived', this.outKey, n); +}); + +hook('bcrypt.dll', 'BCryptDeriveKeyPBKDF2', function (a) { + console.log('[PBKDF2] cbPassword=' + a[2].toInt32() + ' cbSalt=' + a[4].toInt32() + ' iter=' + a[5]); + dump(' password', a[1], a[2].toInt32()); + dump(' salt', a[3], a[4].toInt32()); +}); + +// BCryptGenerateSymmetricKey(hAlg, *phKey, pbKeyObject, cbKeyObject, pbSecret, cbSecret, flags) — the actual key bytes. +hook('bcrypt.dll', 'BCryptGenerateSymmetricKey', function (a) { + dump('[GenerateSymmetricKey] keyBytes', a[4], a[5].toInt32()); +}); + +// BCryptEncrypt(hKey, pbIn, cbIn, *pPad, pbIV, cbIV, pbOut, cbOut, *pcbResult, flags) +hook('bcrypt.dll', 'BCryptEncrypt', function (a) { + this.out = a[6]; this.pcb = a[8]; + dump('[Encrypt] plaintext', a[1], a[2].toInt32()); + dump('[Encrypt] IV', a[4], a[5].toInt32()); +}, function () { + const n = this.pcb.isNull() ? 0 : this.pcb.readU32(); + dump('[Encrypt] ciphertext', this.out, n); +}); + +// Hash path (in case the token is a keyed hash rather than a cipher). +hook('bcrypt.dll', 'BCryptHashData', function (a) { + dump('[HashData] input', a[1], a[2].toInt32()); +}); +hook('bcrypt.dll', 'BCryptFinishHash', function (a) { + this.out = a[1]; this.cb = a[2].toInt32(); +}, function () { + dump('[FinishHash] digest', this.out, this.cb); +}); + +// ---- NCrypt (CNG key-storage layer) — the likely home of the ECDH ExchangeKey + token crypto ---- + +// NCryptSecretAgreement(hPrivKey, hPubKey, *phAgreedSecret, dwFlags) +hook('ncrypt.dll', 'NCryptSecretAgreement', function (a) { + console.log('[NCryptSecretAgreement] hPriv=' + a[0] + ' hPub=' + a[1]); + console.log(' backtrace (addr -> module+offset):'); + Thread.backtrace(this.context, Backtracer.ACCURATE).slice(0, 14).forEach(function (addr) { + const m = Process.findModuleByAddress(addr); + if (m) { + console.log(' ' + addr + ' ' + m.name + '+0x' + addr.sub(m.base).toString(16)); + } else { + console.log(' ' + addr + ' '); + } + }); +}); + +// NCryptDeriveKey(hSharedSecret, pwszKDF, *pParameterList, pbDerivedKey, cbDerivedKey, *pcbResult, dwFlags) +hook('ncrypt.dll', 'NCryptDeriveKey', function (a) { + this.kdf = a[1].isNull() ? '?' : a[1].readUtf16String(); + this.outKey = a[3]; this.pcb = a[5]; + console.log('[NCryptDeriveKey] KDF=' + this.kdf + ' cbDerivedKey=' + a[4].toInt32()); + dumpParamList(a[2]); +}, function () { + const n = this.pcb.isNull() ? 0 : this.pcb.readU32(); + dump('[NCryptDeriveKey] derived', this.outKey, n); +}); + +// NCryptEncrypt(hKey, pbInput, cbInput, *pPaddingInfo, pbOutput, cbOutput, *pcbResult, dwFlags) +hook('ncrypt.dll', 'NCryptEncrypt', function (a) { + this.out = a[4]; this.pcb = a[6]; + dump('[NCryptEncrypt] plaintext', a[1], a[2].toInt32()); +}, function () { + const n = this.pcb.isNull() ? 0 : this.pcb.readU32(); + dump('[NCryptEncrypt] ciphertext', this.out, n); +}); + +// NCryptImportKey(hProvider, hImportKey, pszBlobType, *pParameterList, *phKey, pbData, cbData, dwFlags) +hook('ncrypt.dll', 'NCryptImportKey', function (a) { + console.log('[NCryptImportKey] blobType=' + (a[2].isNull() ? '?' : a[2].readUtf16String())); + dump(' blob', a[5], a[6].toInt32()); +}); + +// NCryptExportKey(hKey, hExportKey, pszBlobType, *pParameterList, pbOutput, cbOutput, *pcbResult, dwFlags) +hook('ncrypt.dll', 'NCryptExportKey', function (a) { + this.blobType = a[2].isNull() ? '?' : a[2].readUtf16String(); + this.out = a[4]; this.pcb = a[6]; +}, function () { + const n = this.pcb.isNull() ? 0 : this.pcb.readU32(); + console.log('[NCryptExportKey] blobType=' + this.blobType); + dump(' blob', this.out, n); +}); + +hook('ncrypt.dll', 'NCryptOpenStorageProvider', function (a) { + console.log('[NCryptOpenStorageProvider] ' + (a[1].isNull() ? '?' : a[1].readUtf16String())); +}); + +// BCrypt EC key operations (in case the ECDH is bcrypt but uses import/export rather than DeriveKey). +hook('bcrypt.dll', 'BCryptImportKeyPair', function (a) { + console.log('[BCryptImportKeyPair] blobType=' + (a[2].isNull() ? '?' : a[2].readUtf16String()) + ' cb=' + a[5].toInt32()); + dump(' blob', a[4], a[5].toInt32()); +}); +hook('bcrypt.dll', 'BCryptExportKey', function (a) { + this.blobType = a[2].isNull() ? '?' : a[2].readUtf16String(); + this.out = a[3]; this.pcb = a[5]; +}, function () { + const n = this.pcb.isNull() ? 0 : this.pcb.readU32(); + console.log('[BCryptExportKey] blobType=' + this.blobType); + dump(' blob', this.out, n); +}); + +console.log('=== CNG ExchangeKey crypto hooks installed ==='); diff --git a/src/AVEVA.Historian.Client/Grpc/HistorianGrpcEventOrchestrator.cs b/src/AVEVA.Historian.Client/Grpc/HistorianGrpcEventOrchestrator.cs index ff0d62f..d9879cf 100644 --- a/src/AVEVA.Historian.Client/Grpc/HistorianGrpcEventOrchestrator.cs +++ b/src/AVEVA.Historian.Client/Grpc/HistorianGrpcEventOrchestrator.cs @@ -58,6 +58,12 @@ internal sealed class HistorianGrpcEventOrchestrator /// Diagnostic: type+code description of the most recent error/terminal buffer. public string LastErrorBufferDescription { get; private set; } = string.Empty; + /// Diagnostic: hex of the most recent result buffer (first 48 bytes). + public string LastResultBufferHex { get; private set; } = string.Empty; + + /// Diagnostic: hex of the most recent GetNext error buffer. + public string LastErrorBufferHex { get; private set; } = string.Empty; + public async IAsyncEnumerable ReadEventsAsync( DateTime startUtc, DateTime endUtc, @@ -138,7 +144,10 @@ internal sealed class HistorianGrpcEventOrchestrator private List RunEventChain(DateTime startUtc, DateTime endUtc, HistorianEventFilter? filter, CancellationToken cancellationToken) { using HistorianGrpcConnection connection = HistorianGrpcChannelFactory.Create(_options); - HistorianGrpcHandshake.Session session = HistorianGrpcHandshake.OpenSession(connection, _options, cancellationToken); + // Event reads need an Event-type (v8) connection. OpenSession(eventConnection: true) runs the + // full v8 path: HistoryService.ExchangeKey (P-256 ECDH) -> client key = SHA256(secret) -> v8 + // OpenConnection with ConnectionType=Event and the credential token RC4(password, MD5(clientKey)). + HistorianGrpcHandshake.Session session = HistorianGrpcHandshake.OpenSession(connection, _options, cancellationToken, eventConnection: true); RegisterCmEventTag(connection, session, cancellationToken); @@ -203,56 +212,55 @@ internal sealed class HistorianGrpcEventOrchestrator { var historyClient = new GrpcHistory.HistoryService.HistoryServiceClient(connection.Channel); var statusClient = new GrpcStatus.StatusService.StatusServiceClient(connection.Channel); - var retrievalClient = new GrpcRetrieval.RetrievalService.RetrievalServiceClient(connection.Channel); - var transactionClient = new GrpcTransaction.TransactionService.TransactionServiceClient(connection.Channel); - - // Discovery dance the native event flow runs between Open2 and EnsT2. All bounded by the - // short RegistrationDeadline (several stall server-side on the remote box). - TryRun(() => statusClient.GetStatusInterfaceVersion(new GrpcStatus.GetStatusInterfaceVersionRequest(), connection.Metadata, RegistrationDeadline(), cancellationToken)); - TryRun(() => statusClient.GetStatusInterfaceVersion(new GrpcStatus.GetStatusInterfaceVersionRequest(), connection.Metadata, RegistrationDeadline(), cancellationToken)); - - byte[] historianVersionRequest = HistorianEventRegistrationProtocol.BuildGetHistorianInfoRequest("HistorianVersion"); - TryRun(() => statusClient.GetHistorianInfo( - new GrpcStatus.GetHistorianInfoRequest { StrHandle = session.StringHandle, BtRequest = ByteString.CopyFrom(historianVersionRequest) }, - connection.Metadata, RegistrationDeadline(), cancellationToken)); - TryRun(() => statusClient.GetHistorianInfo( - new GrpcStatus.GetHistorianInfoRequest { StrHandle = session.StringHandle, BtRequest = ByteString.CopyFrom(historianVersionRequest) }, - connection.Metadata, RegistrationDeadline(), cancellationToken)); + // Native 2023 R2 gRPC event-connection registration sequence (captured order): + // UpdateClientStatus -> RegisterTags(CM_EVENT) -> EnsureTags(CM_EVENT) -> GetHistorianInfo + // -> GetSystemParameter x7. (StartEventQuery follows in RunEventQuery.) The 2020-WCF-era extra + // probes (cross-service GetV, params-before-register) are NOT in the gRPC event flow. byte[] clientStatus = HistorianEventRegistrationProtocol.BuildUpdateClientStatusBlob(); TryRun(() => historyClient.UpdateClientStatus( new GrpcHistory.UpdateClientStatusRequest { StrHandle = session.StringHandle, BtClientStatus = ByteString.CopyFrom(clientStatus) }, connection.Metadata, RegistrationDeadline(), cancellationToken)); - // Records 11-16: 6 system-parameter queries before RTag2. - foreach (string parameterName in HistorianEventRegistrationProtocol.StatusParametersBeforeRegister) + byte[] registerBuffer = HistorianEventRegistrationProtocol.BuildRegisterCmEventInputBuffer(); + try + { + GrpcHistory.RegisterTagsResponse rt = historyClient.RegisterTags( + new GrpcHistory.RegisterTagsRequest { StrHandle = session.StringHandle, BtTagInfos = ByteString.CopyFrom(registerBuffer) }, + connection.Metadata, RegistrationDeadline(), cancellationToken); + RegistrationDiag += $"RTag={rt.Status?.BSuccess} e={Convert.ToHexString(rt.Status?.BtError?.ToByteArray() ?? [])}; "; + } + catch (Exception ex) { RegistrationDiag += $"RTag=EX:{ex.GetType().Name}; "; } + + // gRPC CM_EVENT EnsureTags uses the 86-byte native format (8-byte header + the …2f27 event-type + // GUID), NOT the 2020 WCF CTagMetadata. + byte[] payload = HistorianAddTagsProtocol.SerializeCmEventEnsureTagsGrpc(DateTime.UtcNow); + try + { + GrpcHistory.EnsureTagsResponse et = historyClient.EnsureTags( + new GrpcHistory.EnsureTagsRequest { StrHandle = session.StringHandle, BtTagInfos = ByteString.CopyFrom(payload), ElementCount = 1 }, + connection.Metadata, RegistrationDeadline(), cancellationToken); + RegistrationDiag += $"EnsT={et.Status?.BSuccess} e={Convert.ToHexString(et.Status?.BtError?.ToByteArray() ?? [])} out={Convert.ToHexString(et.BtTagStatus?.ToByteArray() ?? [])}; "; + } + catch (Exception ex) { RegistrationDiag += $"EnsT=EX:{ex.GetType().Name}; "; } + + byte[] historianVersionRequest = HistorianEventRegistrationProtocol.BuildGetHistorianInfoRequest("HistorianVersion"); + TryRun(() => statusClient.GetHistorianInfo( + new GrpcStatus.GetHistorianInfoRequest { StrHandle = session.StringHandle, BtRequest = ByteString.CopyFrom(historianVersionRequest) }, + connection.Metadata, RegistrationDeadline(), cancellationToken)); + + string[] eventParams = ["AllowOriginals", "HistorianPartner", "HistorianVersion", "MaxCyclicStorageTimeout", "RealTimeWindow", "FutureTimeThreshold", "AllowRenameTags"]; + foreach (string parameterName in eventParams) { TryRun(() => statusClient.GetSystemParameter( new GrpcStatus.GetSystemParameterRequest { UiHandle = session.ClientHandle, StrParameterName = parameterName }, connection.Metadata, RegistrationDeadline(), cancellationToken)); } - - byte[] registerBuffer = HistorianEventRegistrationProtocol.BuildRegisterCmEventInputBuffer(); - TryRun(() => historyClient.RegisterTags( - new GrpcHistory.RegisterTagsRequest { StrHandle = session.StringHandle, BtTagInfos = ByteString.CopyFrom(registerBuffer) }, - connection.Metadata, RegistrationDeadline(), cancellationToken)); - - // Record 18: one more system-parameter query after RTag2 before EnsT2. - TryRun(() => statusClient.GetSystemParameter( - new GrpcStatus.GetSystemParameterRequest { UiHandle = session.ClientHandle, StrParameterName = "AllowRenameTags" }, - connection.Metadata, RegistrationDeadline(), cancellationToken)); - - // Records 19-21: cross-service version probes between RTag2 and EnsT2 (session-table registration). - TryRun(() => transactionClient.GetTransactionInterfaceVersion(new GrpcTransaction.GetTransactionInterfaceVersionRequest(), connection.Metadata, RegistrationDeadline(), cancellationToken)); - TryRun(() => statusClient.GetStatusInterfaceVersion(new GrpcStatus.GetStatusInterfaceVersionRequest(), connection.Metadata, RegistrationDeadline(), cancellationToken)); - TryRun(() => retrievalClient.GetRetrievalInterfaceVersion(new GrpcRetrieval.GetRetrievalInterfaceVersionRequest(), connection.Metadata, RegistrationDeadline(), cancellationToken)); - - byte[] payload = HistorianAddTagsProtocol.SerializeCmEventCTagMetadata(DateTime.UtcNow); - TryRun(() => historyClient.EnsureTags( - new GrpcHistory.EnsureTagsRequest { StrHandle = session.StringHandle, BtTagInfos = ByteString.CopyFrom(payload), ElementCount = 1 }, - connection.Metadata, RegistrationDeadline(), cancellationToken)); } + /// Diagnostic: outcomes of the key CM_EVENT registration RPCs. + public string RegistrationDiag { get; private set; } = string.Empty; + private List RunEventQuery( HistorianGrpcConnection connection, HistorianGrpcHandshake.Session session, @@ -274,7 +282,7 @@ internal sealed class HistorianGrpcEventOrchestrator IReadOnlyList attempts = HistorianEventQueryProtocol.CreateStartEventQueryAttempts( startUtc.ToUniversalTime(), endUtc.ToUniversalTime(), - eventCount: 5, + eventCount: 100, filter, version: 6); byte[] requestBuffer = attempts[0].RequestBuffer; @@ -298,6 +306,7 @@ internal sealed class HistorianGrpcEventOrchestrator } uint queryHandle = startResponse.UiQueryHandle; + RegistrationDiag += $"QH={queryHandle} clientH={session.ClientHandle} strH={session.StringHandle}; "; try { List events = []; @@ -333,6 +342,8 @@ internal sealed class HistorianGrpcEventOrchestrator LastResultBufferLength = resultBuffer.Length; LastErrorBufferDescription = HistorianEventRegistrationProtocol.DescribeNativeError(errorBuffer); + LastResultBufferHex = Convert.ToHexString(resultBuffer.Length <= 48 ? resultBuffer : resultBuffer[..48]); + LastErrorBufferHex = Convert.ToHexString(errorBuffer); // Any 5-byte type=4 error is a soft terminal (code 30 NoMoreData is canonical; code // 85 / 0x55 is the missing-registration signal seen on early runs). Mirror the WCF diff --git a/src/AVEVA.Historian.Client/Grpc/HistorianGrpcHandshake.cs b/src/AVEVA.Historian.Client/Grpc/HistorianGrpcHandshake.cs index 79d88b1..3e14d4e 100644 --- a/src/AVEVA.Historian.Client/Grpc/HistorianGrpcHandshake.cs +++ b/src/AVEVA.Historian.Client/Grpc/HistorianGrpcHandshake.cs @@ -1,3 +1,4 @@ +using System.Security.Cryptography; using Google.Protobuf; using Grpc.Core; using AVEVA.Historian.Client.Wcf; @@ -20,6 +21,9 @@ namespace AVEVA.Historian.Client.Grpc; /// internal static class HistorianGrpcHandshake { + /// Diagnostic: hex of the most recent v8 event-connection OpenConnection request. + internal static string LastEventOpenRequestHex { get; private set; } = string.Empty; + /// /// The handles produced by a successful OpenConnection. is the /// transient uint session token used by StartQuery/GetSystemParameter and the other @@ -50,7 +54,8 @@ internal static class HistorianGrpcHandshake HistorianGrpcConnection connection, HistorianClientOptions options, CancellationToken cancellationToken, - uint connectionMode = HistorianWcfAuthChainHelper.NativeIntegratedReadOnlyConnectionMode) + uint connectionMode = HistorianWcfAuthChainHelper.NativeIntegratedReadOnlyConnectionMode, + bool eventConnection = false) { DateTime Deadline() => DateTime.UtcNow.Add(options.RequestTimeout); @@ -61,26 +66,65 @@ internal static class HistorianGrpcHandshake new GrpcHistory.GetInterfaceVersionRequest(), connection.Metadata, Deadline(), cancellationToken); HistorianServerVersionGate.Validate(HistorianServiceInterface.History, historyVersion.UiVersion, options); - var storageClient = new GrpcStorage.StorageService.StorageServiceClient(connection.Channel); - HistorianNativeHandshake.RunTokenRounds( - (handle, wrapped, _) => - { - GrpcStorage.ValidateClientCredentialResponse response = storageClient.ValidateClientCredential( - new GrpcStorage.ValidateClientCredentialRequest { Handle = handle, InBuff = ByteString.CopyFrom(wrapped) }, - connection.Metadata, - Deadline(), - cancellationToken); - byte[] serverOutput = response.OutBuff?.ToByteArray() ?? []; - byte[] error = response.Status?.BtError?.ToByteArray() ?? []; - bool success = response.Status?.BSuccess ?? false; - return new HistorianNativeHandshake.TokenExchangeResult(success, serverOutput, error); - }, - contextKey, - options, - cancellationToken); + // The v6 (read/write) path authenticates via StorageService.ValidateClientCredential (Negotiate). + // The v8 EVENT path authenticates entirely via ExchangeKey (ECDH) + the RC4 credential token — + // the native client does NOT run ValidateClientCredential for an event connection, and doing so + // establishes a different session scope under which the event query returns zero rows. So skip it. + if (!eventConnection) + { + var storageClient = new GrpcStorage.StorageService.StorageServiceClient(connection.Channel); + HistorianNativeHandshake.RunTokenRounds( + (handle, wrapped, _) => + { + GrpcStorage.ValidateClientCredentialResponse response = storageClient.ValidateClientCredential( + new GrpcStorage.ValidateClientCredentialRequest { Handle = handle, InBuff = ByteString.CopyFrom(wrapped) }, + connection.Metadata, + Deadline(), + cancellationToken); + byte[] serverOutput = response.OutBuff?.ToByteArray() ?? []; + byte[] error = response.Status?.BtError?.ToByteArray() ?? []; + bool success = response.Status?.BSuccess ?? false; + return new HistorianNativeHandshake.TokenExchangeResult(success, serverOutput, error); + }, + contextKey, + options, + cancellationToken); + } - byte[] open2Request = HistorianNativeHandshake.BuildOpenConnection3Request( - options.Host, contextKey, connectionMode); + // Event reads require an Event-type connection (ConnectionType=Event), which only the native + // v8 OpenConnection format carries — the v6 buffer has no such field. The v8 path authenticates + // via HistoryService.ExchangeKey (P-256 ECDH): the shared secret -> SHA256 = the client key, and + // the v8 credential token = RC4(password-UTF16LE, key=MD5(clientKey)) (the native HistorianCrypto + // aahCryptV2 scheme). The server shares the secret and RC4-decrypts the token to validate the + // password. See docs/reverse-engineering/grpc-event-query-capture.md. + byte[] eventToken = []; + if (eventConnection) + { + using ECDiffieHellman ecdh = ECDiffieHellman.Create(ECCurve.NamedCurves.nistP256); + byte[] clientHello = HistorianNativeHandshake.BuildExchangeKeyClientHello(ecdh); + string xkHandle = contextKey.ToString("D").ToUpperInvariant(); + GrpcHistory.ExchangeKeyResponse xk = historyClient.ExchangeKey( + new GrpcHistory.ExchangeKeyRequest { StrHandle = xkHandle, BtInput = ByteString.CopyFrom(clientHello) }, + connection.Metadata, + Deadline(), + cancellationToken); + if (!(xk.Status?.BSuccess ?? false)) + { + byte[] xkErr = xk.Status?.BtError?.ToByteArray() ?? []; + HistorianNativeError? xkDecoded = HistorianOpen2Protocol.TryReadNativeError(xkErr); + string xkAscii = new(xkErr.Where(b => b is >= 0x20 and < 0x7F).Select(b => (char)b).ToArray()); + throw new InvalidOperationException( + $"gRPC ExchangeKey failed (errorLen={xkErr.Length}, native={xkDecoded?.Type}/{xkDecoded?.Code}, ascii='{xkAscii}')."); + } + + byte[] clientKey = HistorianNativeHandshake.DeriveExchangeKeyClientKey(ecdh, xk.BtOutput?.ToByteArray() ?? []); + eventToken = HistorianNativeHandshake.BuildExchangeKeyCredentialToken(clientKey, options.Password); + } + + byte[] open2Request = eventConnection + ? HistorianNativeHandshake.BuildEventOpenConnectionVersion8Request(contextKey, options.UserName, eventToken) + : HistorianNativeHandshake.BuildOpenConnection3Request(options.Host, contextKey, connectionMode); + if (eventConnection) { LastEventOpenRequestHex = Convert.ToHexString(open2Request); } GrpcHistory.OpenConnectionResponse open2 = historyClient.OpenConnection( new GrpcHistory.OpenConnectionRequest { BtConnectionRequest = ByteString.CopyFrom(open2Request) }, @@ -92,7 +136,11 @@ internal static class HistorianGrpcHandshake if (!(open2.Status?.BSuccess ?? false)) { byte[] err = open2.Status?.BtError?.ToByteArray() ?? []; - throw new InvalidOperationException($"gRPC OpenConnection failed (errorLen={err.Length}, responseLen={open2Response.Length})."); + HistorianNativeError? decoded = HistorianOpen2Protocol.TryReadNativeError(err); + string ascii = new(err.Where(b => b is >= 0x20 and < 0x7F).Select(b => (char)b).ToArray()); + throw new InvalidOperationException( + $"gRPC OpenConnection failed (errorLen={err.Length}, responseLen={open2Response.Length}, " + + $"native={decoded?.Type}/{decoded?.Code}{(decoded?.Name is { } n ? $" {n}" : "")}, ascii='{ascii}')."); } (uint clientHandle, Guid storageSessionId) = HistorianNativeHandshake.ParseOpenConnectionResponse(open2Response); diff --git a/src/AVEVA.Historian.Client/Wcf/HistorianAddTagsProtocol.cs b/src/AVEVA.Historian.Client/Wcf/HistorianAddTagsProtocol.cs index 009bc42..42c726a 100644 --- a/src/AVEVA.Historian.Client/Wcf/HistorianAddTagsProtocol.cs +++ b/src/AVEVA.Historian.Client/Wcf/HistorianAddTagsProtocol.cs @@ -44,6 +44,40 @@ internal static class HistorianAddTagsProtocol /// public static readonly Guid CommonArchestraEventTypeId = new("5f59ae42-3bb6-4760-91a5-ab0be01f9f02"); + /// + /// The CM_EVENT event-type GUID used by the 2023 R2 gRPC EnsureTags (captured ending + /// …e0 1f 2f 27) — distinct from the 2020 WCF capture's + /// (…9f02). + /// + public static readonly Guid CommonArchestraEventTypeIdGrpc = new("5f59ae42-3bb6-4760-91a5-ab0be01f2f27"); + + /// + /// Builds the native 2023 R2 gRPC CM_EVENT EnsureTags.tagInfos buffer (86 bytes, + /// captured byte-for-byte). Differs from the 2020 WCF : + /// it is wrapped in an 8-byte EnsureTags header (4E 67 03 00 01 00 00 00), uses the + /// event-type GUID, and has no trailing bytes after it. + /// Used by the gRPC event registration so the server actually establishes CM_EVENT and the event + /// query returns rows. + /// + public static byte[] SerializeCmEventEnsureTagsGrpc(DateTime createdUtc) + { + using MemoryStream stream = new(); + using BinaryWriter writer = new(stream, Encoding.Unicode, leaveOpen: true); + + writer.Write(new byte[] { 0x4E, 0x67, 0x03, 0x00, 0x01, 0x00, 0x00, 0x00 }); // EnsureTags header (count 1) + writer.Write((byte)3); + writer.Write((ushort)0x0086); + writer.Write((byte)5); + writer.Write(CmEventTagId.ToByteArray()); + WriteCompressedHistorianString(writer, "CM_EVENT"); + WriteCompressedHistorianString(writer, "AnE Event"); + writer.Write(new byte[] { 0x02, 0x02, 0x01, 0x00, 0x00, 0x00, 0x01 }); + writer.Write(0u); + writer.Write(createdUtc.ToUniversalTime().ToFileTimeUtc()); + writer.Write(CommonArchestraEventTypeIdGrpc.ToByteArray()); + return stream.ToArray(); + } + public static byte[] SerializeCmEventCTagMetadata(DateTime createdUtc) { using MemoryStream stream = new(); diff --git a/src/AVEVA.Historian.Client/Wcf/HistorianNativeHandshake.cs b/src/AVEVA.Historian.Client/Wcf/HistorianNativeHandshake.cs index fcec618..723b364 100644 --- a/src/AVEVA.Historian.Client/Wcf/HistorianNativeHandshake.cs +++ b/src/AVEVA.Historian.Client/Wcf/HistorianNativeHandshake.cs @@ -1,5 +1,6 @@ using System.Buffers.Binary; using System.Diagnostics; +using System.Security.Cryptography; namespace AVEVA.Historian.Client.Wcf; @@ -28,6 +29,111 @@ internal static class HistorianNativeHandshake /// Result of one transport-level credential-token exchange. internal readonly record struct TokenExchangeResult(bool Success, byte[] ServerOutput, byte[] Error); + // HistoryService.ExchangeKey wire format (decoded from a live 2023 R2 capture): the ASCII magic + // "ECK1", a uint32 coordinate length (32 = P-256), then the raw EC public-key point X || Y. Used by + // the v8 (event) connection path to establish the client key the v8 OpenConnection requires. + private static readonly byte[] ExchangeKeyMagic = "ECK1"u8.ToArray(); + private const int ExchangeKeyCoordinateBytes = 32; // P-256 / secp256r1 + + /// + /// Builds the ExchangeKey client hello: "ECK1" + u32(32) + X(32) + Y(32) for the + /// supplied P-256 key. Pure managed (); no native AVEVA dependency. + /// + public static byte[] BuildExchangeKeyClientHello(ECDiffieHellman ecdh) + { + ECParameters parameters = ecdh.ExportParameters(includePrivateParameters: false); + byte[] x = LeftPadCoordinate(parameters.Q.X!); + byte[] y = LeftPadCoordinate(parameters.Q.Y!); + + using MemoryStream stream = new(); + using BinaryWriter writer = new(stream); + writer.Write(ExchangeKeyMagic); + writer.Write((uint)ExchangeKeyCoordinateBytes); + writer.Write(x); + writer.Write(y); + return stream.ToArray(); + } + + /// + /// Parses the server's ExchangeKey hello (same "ECK1" + u32 + X + Y shape) and + /// derives the 32-byte client key = SHA256(ECDH shared secret). This matches the native + /// client, which uses ECDiffieHellmanCng { KeyDerivationFunction=Hash, HashAlgorithm=SHA256 } + /// .DeriveKeyMaterial(...) — i.e. the hash KDF over the raw agreement. + /// + public static byte[] DeriveExchangeKeyClientKey(ECDiffieHellman ecdh, byte[] serverHello) + { + const int headerLength = 8; // 4-byte magic + 4-byte coordinate length + int needed = headerLength + (2 * ExchangeKeyCoordinateBytes); + if (serverHello.Length < needed) + { + throw new InvalidOperationException($"ExchangeKey server hello too short (len={serverHello.Length}, need>={needed})."); + } + + byte[] x = serverHello[headerLength..(headerLength + ExchangeKeyCoordinateBytes)]; + byte[] y = serverHello[(headerLength + ExchangeKeyCoordinateBytes)..(headerLength + (2 * ExchangeKeyCoordinateBytes))]; + + ECParameters serverParameters = new() + { + Curve = ECCurve.NamedCurves.nistP256, + Q = new ECPoint { X = x, Y = y }, + }; + using ECDiffieHellman serverKey = ECDiffieHellman.Create(serverParameters); + return ecdh.DeriveKeyFromHash(serverKey.PublicKey, HashAlgorithmName.SHA256); + } + + /// + /// Builds the v8 credential token: RC4(password-UTF16LE, key = MD5(clientKey)), where + /// is the result + /// (SHA256 of the ECDH secret). Reverse-engineered from the native HistorianCrypto.NRC4_V2 + /// .aahCryptV2 scheme (MD5-keyed RC4) and verified to reproduce a live-captured token exactly. + /// The server, sharing the ECDH secret, RC4-decrypts this to recover and validate the password. + /// Pure managed; nothing AVEVA is shipped. + /// + public static byte[] BuildExchangeKeyCredentialToken(byte[] clientKey, string password) + { + byte[] rc4Key = MD5.HashData(clientKey); + byte[] plaintext = System.Text.Encoding.Unicode.GetBytes(password ?? string.Empty); + return Rc4(rc4Key, plaintext); + } + + internal static byte[] Rc4(byte[] key, byte[] data) + { + int[] s = new int[256]; + for (int i = 0; i < 256; i++) + { + s[i] = i; + } + + for (int i = 0, j = 0; i < 256; i++) + { + j = (j + s[i] + key[i % key.Length]) & 0xFF; + (s[i], s[j]) = (s[j], s[i]); + } + + byte[] output = new byte[data.Length]; + for (int i = 0, j = 0, n = 0; n < data.Length; n++) + { + i = (i + 1) & 0xFF; + j = (j + s[i]) & 0xFF; + (s[i], s[j]) = (s[j], s[i]); + output[n] = (byte)(data[n] ^ s[(s[i] + s[j]) & 0xFF]); + } + + return output; + } + + private static byte[] LeftPadCoordinate(byte[] coordinate) + { + if (coordinate.Length == ExchangeKeyCoordinateBytes) + { + return coordinate; + } + + byte[] padded = new byte[ExchangeKeyCoordinateBytes]; + Array.Copy(coordinate, 0, padded, ExchangeKeyCoordinateBytes - coordinate.Length, coordinate.Length); + return padded; + } + /// /// Performs a single credential-token round on the wire. is the /// upper-case context-key GUID, is the AVEVA-wrapped SSPI @@ -134,6 +240,56 @@ internal static class HistorianNativeHandshake credentialBlock: new byte[CredentialBlockSizeBytes]); } + // v8 OpenConnection constants (2023 R2), decoded from the native Event-connection capture. + private const byte NativeConnectionTypeEvent = 1; + private const byte NativeConnectionFlagEvent = 1; + private const ushort NativeHcalVersionV8 = 18; + private const string ClientDataSourceIdV8 = "2023.1219.4004.5"; + private const byte NativeClientCommonInfoFormatVersionV8 = 3; + private const int CredentialTokenSizeBytes = 26; + private static readonly Guid UnsetShardId = new(new byte[] + { + 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, + 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, + }); + + /// + /// Builds the native 2023 R2 version-8 OpenConnection request for an Event connection + /// (ConnectionType=Event). The 2023 R2 server returns event rows only on an event connection, + /// and the v6 OpenConnection buffer has no ConnectionType field. The credential token is zeroed: the + /// session is authenticated by the preceding ValidateClientCredential handshake, exactly as + /// the v6 path already relies on (it sends a zeroed credential block). See + /// docs/reverse-engineering/grpc-event-query-capture.md. + /// + public static byte[] BuildEventOpenConnectionVersion8Request(Guid contextKey, string userName, byte[] credentialToken) + { + Process current = Process.GetCurrentProcess(); + string machineName = Environment.MachineName; + string processName = string.IsNullOrEmpty(current.ProcessName) ? ClientNodeNameFallback : current.ProcessName; + + HistorianClientCommonInfo commonInfo = new( + FormatVersion: NativeClientCommonInfoFormatVersionV8, + ServerNodeName: machineName, + ClientNodeName: processName, + ProcessId: checked((uint)current.Id), + HcalVersion: NativeHcalVersionV8, + ProcessName: string.Empty, + Proxy: string.Empty, + DataSourceId: ClientDataSourceIdV8, + ShardId: UnsetShardId, + ClientVersion: NativeClientVersionInt, + ClientTimestamp: (ulong)DateTime.UtcNow.ToFileTimeUtc(), + ClientDllVersion: string.Empty); + + return HistorianOpen2Protocol.SerializeNativeOpenConnectionVersion8( + commonInfo, + contextKey, + NativeConnectionTypeEvent, + NativeConnectionFlagEvent, + userName ?? string.Empty, + credentialToken ?? []); + } + /// /// Decodes the OpenConnection response blob: byte 0 = protocol version, bytes 1..4 = /// transient /Retr client handle (UInt32 LE), bytes 5..20 = storage session GUID. diff --git a/src/AVEVA.Historian.Client/Wcf/HistorianOpen2Protocol.cs b/src/AVEVA.Historian.Client/Wcf/HistorianOpen2Protocol.cs index f60f061..5cf5831 100644 --- a/src/AVEVA.Historian.Client/Wcf/HistorianOpen2Protocol.cs +++ b/src/AVEVA.Historian.Client/Wcf/HistorianOpen2Protocol.cs @@ -50,6 +50,49 @@ internal static class HistorianOpen2Protocol return stream.ToArray(); } + /// + /// Builds the native 2023 R2 version-8 OpenConnection request — the format the stock client + /// uses, which (unlike v6) carries a byte (Event vs Process). The + /// tail is the same v6 emits; the head is the v8 layout + /// decoded from a live capture (docs/reverse-engineering/grpc-event-query-capture.md): version 8, + /// two marker bytes, the client-key GUID, a username HistorianString, a length-prefixed + /// credential token, then ClientType / ConnectionType / flag / a constant word / compact metadata + /// namespace / two empty strings. The credential token is normally an ECDH-derived blob in the + /// native cert path; this serializer sends it as supplied (zeroed for the Negotiate-auth path, + /// mirroring how the v6 request already sends a zeroed credential block and relies on the separate + /// ValidateClientCredential handshake for authentication). + /// + public static byte[] SerializeNativeOpenConnectionVersion8( + HistorianClientCommonInfo commonInfo, + Guid clientKey, + byte connectionType, + byte connectionFlag, + string userName, + byte[] credentialToken) + { + using MemoryStream stream = new(); + using BinaryWriter writer = new(stream, Encoding.Unicode, leaveOpen: true); + + writer.Write((byte)8); // [0] format version 8 + writer.Write((byte)0xF0); // [1] constant marker + writer.Write(new byte[19]); // [2..20] zero padding + writer.Write((byte)1); // [21] constant marker + writer.Write(clientKey.ToByteArray()); // [22..37] per-session client key + WriteHistorianString(writer, userName); // username (UTF-16 HistorianString) + writer.Write((ushort)credentialToken.Length); // credential-token length (u16) + writer.Write(credentialToken); // credential token (zeroed for the Negotiate path) + writer.Write((byte)4); // ClientType = 4 + writer.Write(connectionType); // ConnectionType (Event = 1 / Process = 2) + writer.Write(connectionFlag); // type flag (Event = 1 / Process = 0) + writer.Write((ushort)3); // constant word observed at [97..98] + WriteCompactMetadataNamespace(writer, HistorianMetadataNamespace.Empty); + WriteHistorianString(writer, string.Empty); + WriteHistorianString(writer, string.Empty); + WriteClientCommonInfo(writer, commonInfo); // tail: FormatVersion 3 => no ClientDllVersion + writer.Write(0u); // trailing terminator + return stream.ToArray(); + } + private static void WriteNativeOpenConnectionContent( BinaryWriter writer, HistorianOpen2Request request, diff --git a/tests/AVEVA.Historian.Client.Tests/HistorianGrpcHandshakeRoutingTests.cs b/tests/AVEVA.Historian.Client.Tests/HistorianGrpcHandshakeRoutingTests.cs index fee9f3f..4cea64a 100644 --- a/tests/AVEVA.Historian.Client.Tests/HistorianGrpcHandshakeRoutingTests.cs +++ b/tests/AVEVA.Historian.Client.Tests/HistorianGrpcHandshakeRoutingTests.cs @@ -18,26 +18,31 @@ namespace AVEVA.Historian.Client.Tests; public sealed class HistorianGrpcHandshakeRoutingTests { [Fact] - public void Handshake_UsesValidateClientCredential_NotExchangeKey() + public void Handshake_TokenLoop_UsesValidateClientCredential_NotExchangeKey() { - // The auth token loop lives in the shared handshake helper (reused by the read, status, - // and future browse/metadata gRPC paths). - HashSet calledMethods = CollectCalledMethodNames( - "AVEVA.Historian.Client.Grpc.HistorianGrpcHandshake"); + // The auth token loop is a compiler-generated lambda (nested closure) passed to RunTokenRounds. + // It MUST carry the SSPI/Negotiate token via StorageService.ValidateClientCredential, never via + // HistoryService.ExchangeKey (the earlier regression). ExchangeKey is now legitimately called + // directly in OpenSession for the SEPARATE v8 event-connection key exchange (ECDH) — that is + // allowed, so the guardrail scopes the no-ExchangeKey rule to the token-loop closures only. + HashSet tokenLoopCalls = CollectCalledMethodNames( + "AVEVA.Historian.Client.Grpc.HistorianGrpcHandshake", nestedClosuresOnly: true); - Assert.Contains("ValidateClientCredential", calledMethods); - Assert.DoesNotContain("ExchangeKey", calledMethods); + Assert.Contains("ValidateClientCredential", tokenLoopCalls); + Assert.DoesNotContain("ExchangeKey", tokenLoopCalls); } - private static HashSet CollectCalledMethodNames(string typeFullName) + private static HashSet CollectCalledMethodNames(string typeFullName, bool nestedClosuresOnly = false) { Assembly sdk = typeof(HistorianClientOptions).Assembly; Type orchestrator = sdk.GetType(typeFullName, throwOnError: true)!; Module module = orchestrator.Module; - // The orchestrator type plus its compiler-generated nested types (lambda closures). - IEnumerable types = new[] { orchestrator } - .Concat(orchestrator.GetNestedTypes(BindingFlags.Public | BindingFlags.NonPublic)); + // The compiler-generated nested types (lambda closures) — optionally with the orchestrator type + // itself. The token loop lives inside a closure; ExchangeKey (event key exchange) lives in the + // OpenSession body, so scoping to closures isolates the token-loop routing. + IEnumerable nested = orchestrator.GetNestedTypes(BindingFlags.Public | BindingFlags.NonPublic); + IEnumerable types = nestedClosuresOnly ? nested : new[] { orchestrator }.Concat(nested); var names = new HashSet(StringComparer.Ordinal); foreach (Type t in types) diff --git a/tests/AVEVA.Historian.Client.Tests/HistorianGrpcIntegrationTests.cs b/tests/AVEVA.Historian.Client.Tests/HistorianGrpcIntegrationTests.cs index 9132e04..9ff5739 100644 --- a/tests/AVEVA.Historian.Client.Tests/HistorianGrpcIntegrationTests.cs +++ b/tests/AVEVA.Historian.Client.Tests/HistorianGrpcIntegrationTests.cs @@ -515,6 +515,42 @@ public sealed class HistorianGrpcIntegrationTests }); } + [Fact] + public async Task EventReadDiagnostic_OverGrpc_PrintsJourney() + { + string? host = Environment.GetEnvironmentVariable("HISTORIAN_GRPC_HOST"); + if (string.IsNullOrWhiteSpace(host) || string.IsNullOrEmpty(Environment.GetEnvironmentVariable("HISTORIAN_USER"))) + { + return; + } + if (Environment.GetEnvironmentVariable("HISTORIAN_GRPC_EVENT_DIAG") is null) + { + return; // opt-in diagnostic only + } + + var orch = new AVEVA.Historian.Client.Grpc.HistorianGrpcEventOrchestrator(BuildOptions(host)); + var events = new List(); + string outcome; + try + { + await foreach (HistorianEvent evt in orch.ReadEventsAsync(DateTime.UtcNow.AddDays(-90), DateTime.UtcNow, null, CancellationToken.None)) + { + events.Add(evt); + if (events.Count >= 3) { break; } + } + outcome = $"OK events={events.Count}"; + } + catch (Exception ex) + { + outcome = $"{ex.GetType().Name}: {ex.Message}"; + } + + throw new Xunit.Sdk.XunitException( + $"[DIAG] outcome={outcome} | events={events.Count} | LastResultLen={orch.LastResultBufferLength} " + + $"| ResultHex={orch.LastResultBufferHex} | Reg=[{orch.RegistrationDiag}] " + + $"| v8open={AVEVA.Historian.Client.Grpc.HistorianGrpcHandshake.LastEventOpenRequestHex}"); + } + [Fact] public async Task GetConnectionStatusAsync_OverGrpc_ReportsConnected() { @@ -642,7 +678,8 @@ public sealed class HistorianGrpcIntegrationTests IntegratedSecurity = !explicitCreds, UserName = user ?? string.Empty, Password = password ?? string.Empty, - RequestTimeout = timeout + RequestTimeout = timeout, + Compression = true // the stock client always advertises grpc gzip request encoding }; } } diff --git a/tests/AVEVA.Historian.Client.Tests/WcfOpen2ProtocolTests.cs b/tests/AVEVA.Historian.Client.Tests/WcfOpen2ProtocolTests.cs index 3646d94..941b8f1 100644 --- a/tests/AVEVA.Historian.Client.Tests/WcfOpen2ProtocolTests.cs +++ b/tests/AVEVA.Historian.Client.Tests/WcfOpen2ProtocolTests.cs @@ -5,6 +5,70 @@ namespace AVEVA.Historian.Client.Tests; public sealed class WcfOpen2ProtocolTests { + [Fact] + public void Rc4MatchesStandardTestVector() + { + // Standard RC4 test vector (key "Key", plaintext "Plaintext" -> BBF316E8D940AF0AD3). Pins the + // RC4 used by the v8 credential token = RC4(password-UTF16LE, key=MD5(SHA256(ECDH secret))). + byte[] actual = HistorianNativeHandshake.Rc4( + System.Text.Encoding.ASCII.GetBytes("Key"), + System.Text.Encoding.ASCII.GetBytes("Plaintext")); + + Assert.Equal(Convert.FromHexString("BBF316E8D940AF0AD3"), actual); + } + + [Fact] + public void CredentialToken_IsRc4OfPasswordKeyedByMd5OfClientKey() + { + // token = RC4(password-UTF16LE, key = MD5(clientKey)). Verify against an independently computed + // reference for a fixed clientKey + password. + byte[] clientKey = Enumerable.Range(0, 32).Select(i => (byte)i).ToArray(); + byte[] token = HistorianNativeHandshake.BuildExchangeKeyCredentialToken(clientKey, "Pw1!"); + + byte[] rc4Key = System.Security.Cryptography.MD5.HashData(clientKey); + byte[] expected = HistorianNativeHandshake.Rc4(rc4Key, System.Text.Encoding.Unicode.GetBytes("Pw1!")); + Assert.Equal(expected, token); + Assert.Equal("Pw1!".Length * 2, token.Length); // UTF-16LE length + } + + [Fact] + public void Version8EventSerializerReproducesCapturedNativeStructure() + { + // Field sizes chosen to match the live 2023 R2 Event-connection capture (302 bytes): + // username=12, token=26, serverNode=15, clientNode=32, datasource=16. + // See docs/reverse-engineering/grpc-event-query-capture.md. + var clientKey = new Guid("11223344-5566-7788-99aa-bbccddeeff00"); + byte[] actual = HistorianOpen2Protocol.SerializeNativeOpenConnectionVersion8( + new HistorianClientCommonInfo( + FormatVersion: 3, + ServerNodeName: new string('S', 15), + ClientNodeName: new string('C', 32), + ProcessId: 0x11223344, + HcalVersion: 18, + ProcessName: string.Empty, + Proxy: string.Empty, + DataSourceId: new string('D', 16), + ShardId: new Guid(Enumerable.Repeat((byte)0xFF, 16).ToArray()), + ClientVersion: 999_999, + ClientTimestamp: 0x01DD02426F9B6F6C, + ClientDllVersion: string.Empty), + clientKey, + connectionType: 1, // Event + connectionFlag: 1, // Event + userName: new string('U', 12), + credentialToken: new byte[26]); + + Assert.Equal(302, actual.Length); // exact native length for these field sizes + Assert.Equal(0x08, actual[0]); // format version 8 + Assert.Equal(0xF0, actual[1]); // marker + Assert.Equal(0x01, actual[21]); // marker + Assert.Equal(clientKey.ToByteArray(), actual[22..38]); + Assert.Equal(0x04, actual[94]); // ClientType + Assert.Equal(0x01, actual[95]); // ConnectionType = Event + Assert.Equal(0x01, actual[96]); // type flag = Event + Assert.Equal([0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF], actual[270..286]); // ShardId + } + [Fact] public void LegacyVersion1SerializerMatchesDecompiledSaveOpenConnectionParamsLayout() { diff --git a/tools/AVEVA.Historian.ReverseEngineering/Program.cs b/tools/AVEVA.Historian.ReverseEngineering/Program.cs index 62bbdda..3ec8de3 100644 --- a/tools/AVEVA.Historian.ReverseEngineering/Program.cs +++ b/tools/AVEVA.Historian.ReverseEngineering/Program.cs @@ -1387,10 +1387,14 @@ static int InstrumentGrpcNonStream(string[] args) ModuleDefMD module = ModuleDefMD.Load(sourcePath); MemberRefUser logByteArray = CreateLogByteArrayRef(module); + MemberRefUser logString = CreateLogStringRef(module); + MemberRefUser logUInt32 = CreateLogUInt32Ref(module); // Cast a wide net: instrument EVERY byte[]-input method on every Grpc*Client type, so whichever // path the native non-streamed write actually drives (History/Transaction RegisterTags + // AddNonStreamValues, or a Storage-service route) is captured. Phase = "..". + // Also captures string (strHandle) and uint (uiHandle / queryRequestType) inputs so the event-read + // handle fields are visible, not just byte[] params. var instrumented = new List(); foreach (TypeDef type in module.GetTypes() .Where(t => t.Name.String.StartsWith("Grpc", StringComparison.Ordinal) && t.Name.String.EndsWith("Client", StringComparison.Ordinal))) @@ -1431,6 +1435,34 @@ static int InstrumentGrpcNonStream(string[] args) }); } + // ENTRY: log string (strHandle) and uint (uiHandle / queryRequestType / count) inputs. + foreach (dnlib.DotNet.Parameter scalarParam in method.Parameters + .Where(p => !p.IsHiddenThisParameter && (p.Type.FullName == "System.String" || p.Type.FullName == "System.UInt32")) + .ToArray()) + { + bool isString = scalarParam.Type.FullName == "System.String"; + string phase = $"{type.Name}.{method.Name}.{scalarParam.Name}.{(isString ? "str" : "u32")}"; + Instruction[] injected = + [ + Instruction.Create(OpCodes.Ldstr, phase), + Instruction.Create(OpCodes.Ldarg, scalarParam), + Instruction.Create(OpCodes.Call, isString ? logString : logUInt32), + ]; + foreach (Instruction instruction in injected.Reverse()) + { + method.Body.Instructions.Insert(0, instruction); + } + method.Body.MaxStack = (ushort)Math.Max((int)method.Body.MaxStack, 8); + instrumented.Add(new + { + Type = type.Name.String, + Method = method.Name.String, + Phase = phase, + Direction = "in", + Token = "0x" + method.MDToken.Raw.ToString("X8"), + }); + } + // EXIT: log out/ref byte[] responses ("System.Byte[]&") before each ret. ldarg loads the // managed pointer; ldind.ref dereferences it to the byte[]. (RPC wrappers set the out // param right before a single ret, so branch-to-ret skew is not a concern here.)