From 8199dde4528819cb227279ccc4eb089e9d181189 Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Tue, 23 Jun 2026 13:43:21 -0400 Subject: [PATCH] =?UTF-8?q?gRPC=20events:=20capture=20decrypted=20HTTP/1.1?= =?UTF-8?q?=20frames=20native=20vs=20ours=20=E2=80=94=20topology=20found?= =?UTF-8?q?=20+=20tested=20null?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Pursued hypothesis #3 (connection-frame capture). Built a TLS-terminating tee proxy (artifacts/.../httpcap/, gitignored: self-signed server cert, forwards through the loopback tunnel, logs decrypted HTTP/1.1 + gRPC-Web both directions) and ran a native capture-event (returns 50 rows) and our SDK diagnostic (0 rows) through the SAME proxy/upstream for a clean A/B. Findings: - The stock client is gRPC-Web/HTTP-1.1 (alpn empty), and clientCert=none on every connection — confirming (with the decompile) that hypothesis #2 (TLS client cert) is moot: the native presents no client cert. - Connection topology differs: the native opens 5 TLS connections, one per service, and runs the event query (StartEventQuery/GetNext/EndEventQuery) on a DEDICATED RetrievalService connection, separate from the HistoryService connection that opened and registered the session. Our SDK collapses every service onto one connection. (Matches the decompile: the stock client has a separate GrpcClientBase per service.) - Framing differs benignly: native uses content-length + Expect: 100-continue; SDK uses transfer-encoding: chunked. The server accepts both (StartEventQuery returns a valid handle), so framing is not the gate. No hidden header on either side. Tested the topology hypothesis with a new env-gated switch (HISTORIAN_GRPC_EVENT_SPLIT_CHANNEL=1): run StartEventQuery/GetNext/EndEventQuery on a dedicated RetrievalService connection (no re-handshake, reusing the session handle — mirroring native conn4), registration staying on the main connection. Result: still 0B00000000001E000000 (0 rows), QH=1063. Splitting the event query onto its own connection does not make rows flow — the server correlates by session handle, not connection, so topology is not the row-scoping gate. Every angle is now exhausted (payload, transport, metadata/interceptor/cert, topology, data store). The gate is a server-internal per-connection retrieval working-set in the native HistorianClient C++ core, unreachable from a pure-managed client. Conclusion unchanged: auth-solved / retrieval-server-gated; ReadEventsAsync over gRPC keeps the no-row throw, event reads use WCF. 56 offline gRPC/event tests pass. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01B6mcaT2PjRFKcogzp9UkfC --- .../grpc-event-query-capture.md | 78 +++++++++++-------- .../Grpc/HistorianGrpcEventOrchestrator.cs | 27 +++++-- 2 files changed, 67 insertions(+), 38 deletions(-) diff --git a/docs/reverse-engineering/grpc-event-query-capture.md b/docs/reverse-engineering/grpc-event-query-capture.md index 9806dcc..537615b 100644 --- a/docs/reverse-engineering/grpc-event-query-capture.md +++ b/docs/reverse-engineering/grpc-event-query-capture.md @@ -370,29 +370,34 @@ live-server access reference. retained for future connection-level probing. This eliminates the leading hypothesis and tightens the conclusion: the server scopes 0 events to our connection at a layer **above** the gRPC transport. -2. **TLS client identity / certificate.** The native used `SecurityMode=TransportCertificate`. Determine - whether it presents a **client certificate** the server uses to scope events (our SDK presents none — - `AllowUntrustedServerCertificate=true`, server cert only). TEST: capture the TLS handshake (e.g. - `SSLKEYLOGFILE` + Wireshark, or a decrypting proxy) for a native `capture-event` run and check the - Certificate message; if a client cert is presented, replicate it. +2. ~~**TLS client identity / certificate.**~~ **DISPROVEN 2026-06-23 (decompile + capture).** The stock + client's `GrpcClientBase.InitializeBase` creates a bare `HttpClientHandler` and sets only + `ServerCertificateCustomValidationCallback` — it **never adds a client certificate**. The TLS-tee + capture (below) confirms `clientCert=none` on every native connection. So the native presents no client + cert; this is not the gate. -3. **HTTP/2-level capture.** The byte[]/handle capture is RPC-payload only. Capture the actual HTTP/2 - frames (HEADERS/SETTINGS/stream IDs, connection reuse) for the native run vs ours — via a - TLS-decrypting mitm on the loopback forward — to see any connection-level header/affinity our capture - can't see. - -2. **TLS client identity / certificate.** The native used `SecurityMode=TransportCertificate`. Determine - whether it presents a **client certificate** the server uses to scope events (our SDK presents none — - `AllowUntrustedServerCertificate=true`, server cert only). TEST: capture the TLS handshake (e.g. - `SSLKEYLOGFILE` + Wireshark, or a decrypting proxy) for a native `capture-event` run and check the - Certificate message; if a client cert is presented, replicate it. **Lower-probability after #1: the - plain-HTTP/2 path presents no client cert either, yet auth + registration still succeed and the gate - persists — so the gate is not at the TLS-identity layer the cert would affect.** - -3. **HTTP/2-level capture.** The byte[]/handle capture is RPC-payload only. Capture the actual HTTP/2 - frames (HEADERS/SETTINGS/stream IDs, connection reuse) for the native run vs ours — via a - TLS-decrypting mitm on the loopback forward — to see any connection-level header/affinity our capture - can't see. +3. ~~**HTTP/2-level / connection-frame capture.**~~ **DONE 2026-06-23 — topology difference found, tested, + NULL.** Built a TLS-terminating tee proxy (`artifacts/.../httpcap/`, gitignored: self-signed server + cert, forwards through the loopback tunnel, logs decrypted HTTP/1.1 + gRPC-Web both ways) and ran a + **native `capture-event` (returns 50 rows) and our SDK diagnostic (0 rows) through the same + proxy/upstream**. Note: the stock client is gRPC-Web/HTTP-1.1 (not HTTP/2 — `alpn` empty), so the + capture is HTTP/1.1 framing. Findings: + - **Connection topology differs.** The native opens **5 TLS connections, one per service** — + `HistoryService` (ExchangeKey/OpenConnection/Register/EnsureTags), `StatusService` (×2), and + **`RetrievalService` (the event query: GetRetrievalInterfaceVersion → StartEventQuery → GetNext → + EndEventQuery) on its own dedicated connection**. Our SDK collapses **every service onto one + connection**. (Matches the decompile: stock has a separate `GrpcClientBase` per service.) + - **Framing differs** (benign): native uses `content-length` + `Expect: 100-continue`; SDK uses + `transfer-encoding: chunked`. The server accepts both (our `StartEventQuery` returns a valid handle), + so framing is not the gate. No extra/hidden header on either side; `clientCert=none` throughout. + - **TESTED the topology hypothesis (`HISTORIAN_GRPC_EVENT_SPLIT_CHANNEL=1`):** ran + `StartEventQuery`/`GetNext`/`EndEventQuery` on a **dedicated RetrievalService connection** (no + re-handshake, reusing the session handle — exactly mirroring native conn4), registration staying on + the main connection. **Result: still `0B00000000001E000000` (0 rows), `QH=1063`.** Splitting the + event query onto its own connection — the one concrete structural difference the capture revealed — + **does not make rows flow.** So the server correlates by session handle, not by connection, and the + topology is **not** the row-scoping gate. The `CreateHttp2`/`SPLIT_CHANNEL` switches + the + `httpcap` proxy are retained as diagnostics. 4. ~~**Server-side ground truth.**~~ **ANSWERED 2026-06-23 (DISPROVES the data-scoping premise).** Via the SOCKS→SQL relay (read-only; `artifacts/.../sqlschema/`, gitignored), dumped the full event schema @@ -449,17 +454,24 @@ So **every client-controllable layer is now confirmed identical by reading the s by wire match: request bytes, transport, channel options, gRPC metadata, interceptor. The remaining difference is below the managed surface (native core) / server-side. -**Conclusion (after #1 disproven + #4 answered + stock client decompiled).** Four independent angles are -now exhausted: client payload (byte-identical), transport (stock client is *also* gRPC-Web/HTTP-1.1 — -HTTP/2 makes no difference, both 0 rows), client-side metadata/interceptor/channel (decompiled — identical, -no hidden header), and data store (global, unscoped, 71,332 events the engine serves via INSQL but -withholds from our gRPC connection). The gate is a **server-internal per-connection retrieval working-set** -that a pure-managed client cannot reconstruct by matching wire bytes, transport, metadata, or data — and -the establishing logic is in the native `HistorianClient` C++ core, not in any decompilable managed step. -The remaining angle (#3 HTTP/2-frame capture) is low-probability given the stock client uses the same -gRPC-Web/HTTP-1.1 channel. **gRPC event-row retrieval stands documented as auth-solved / -retrieval-server-gated**; `ReadEventsAsync` over gRPC keeps the honest no-row throw, and event reads use -the WCF transport. +**Conclusion (after #1–#4 + stock client decompiled + TLS-tee capture).** Every angle is now exhausted: +- **client payload** — byte-identical (IL capture + decompile); +- **transport** — stock client is *also* gRPC-Web/HTTP-1.1; native HTTP/2 makes no difference, both 0 rows; +- **client metadata/interceptor/channel** — decompiled: identical gzip-only header, no-op interceptor, no + client cert; the TLS-tee capture confirms no hidden header and `clientCert=none`; +- **connection topology** — the native splits services across 5 connections and queries on a dedicated + RetrievalService connection; replicating that (`SPLIT_CHANNEL`) still returns 0 rows → the server + correlates by session handle, not connection; +- **data store** — global, unscoped; 71,332 events the engine serves via INSQL but withholds from our + gRPC connection. + +The gate is a **server-internal per-connection retrieval working-set** that a pure-managed client cannot +reconstruct by matching wire bytes, transport, metadata, topology, or data — and the establishing logic is +in the native `HistorianClient` C++ core, not in any decompilable managed step or observable on the wire. +**gRPC event-row retrieval stands documented as auth-solved / retrieval-server-gated**; `ReadEventsAsync` +over gRPC keeps the honest no-row throw, and event reads use the WCF transport. Diagnostics retained for +any future server-side investigation: the `httpcap` TLS-tee proxy, the `CreateHttp2` / `SPLIT_CHANNEL` +switches, the `EventReadDiagnostic` test, and the `capture-event` harness (native, returns rows). **2 of 3 layers cleared** (key exchange + client key); the 3rd (token construction) is localized to a specific managed method, pending dnlib extraction. ExchangeKey + the v8 serializer are committed; the diff --git a/src/AVEVA.Historian.Client/Grpc/HistorianGrpcEventOrchestrator.cs b/src/AVEVA.Historian.Client/Grpc/HistorianGrpcEventOrchestrator.cs index 8c88fd9..758b527 100644 --- a/src/AVEVA.Historian.Client/Grpc/HistorianGrpcEventOrchestrator.cs +++ b/src/AVEVA.Historian.Client/Grpc/HistorianGrpcEventOrchestrator.cs @@ -281,9 +281,21 @@ internal sealed class HistorianGrpcEventOrchestrator HistorianEventFilter? filter, CancellationToken cancellationToken) { - var retrievalClient = new GrpcRetrieval.RetrievalService.RetrievalServiceClient(connection.Channel); + // HTTP/2-frame capture (grpc-event-query-capture.md #3) showed the stock client runs the event + // query on a DEDICATED RetrievalService TLS connection, separate from the HistoryService + // connection that opened+registered the session (correlated only by the session handle); our SDK + // collapses every service onto one connection. Opt in via HISTORIAN_GRPC_EVENT_SPLIT_CHANNEL=1 to + // run StartEventQuery/GetNext/EndEventQuery on their own connection (mirrors native conn4: no + // re-handshake, just the existing handle), to test whether topology is the row-scoping gate. + bool splitChannel = string.Equals( + Environment.GetEnvironmentVariable("HISTORIAN_GRPC_EVENT_SPLIT_CHANNEL"), "1", StringComparison.Ordinal); + HistorianGrpcConnection rconn = splitChannel ? HistorianGrpcChannelFactory.Create(_options) : connection; + try + { + + var retrievalClient = new GrpcRetrieval.RetrievalService.RetrievalServiceClient(rconn.Channel); GrpcRetrieval.GetRetrievalInterfaceVersionResponse retrievalVersion = retrievalClient.GetRetrievalInterfaceVersion( - new GrpcRetrieval.GetRetrievalInterfaceVersionRequest(), connection.Metadata, Deadline(), cancellationToken); + new GrpcRetrieval.GetRetrievalInterfaceVersionRequest(), rconn.Metadata, Deadline(), cancellationToken); HistorianServerVersionGate.Validate(HistorianServiceInterface.Retrieval, retrievalVersion.UiVersion, _options); // Version 6 envelope: the stock 2023 R2 client sends v6 (the WCF path's v5 request is accepted @@ -306,7 +318,7 @@ internal sealed class HistorianGrpcEventOrchestrator UiQueryRequestType = HistorianEventQueryProtocol.QueryRequestTypeEvent, BtRequest = ByteString.CopyFrom(requestBuffer) }, - connection.Metadata, + rconn.Metadata, Deadline(), cancellationToken); @@ -331,7 +343,7 @@ internal sealed class HistorianGrpcEventOrchestrator { nextResponse = retrievalClient.GetNextEventQueryResultBuffer( new GrpcRetrieval.GetNextEventQueryResultBufferRequest { UiHandle = session.ClientHandle, UiQueryHandle = queryHandle }, - connection.Metadata, + rconn.Metadata, EventPollDeadline(), cancellationToken); } @@ -384,7 +396,12 @@ internal sealed class HistorianGrpcEventOrchestrator } finally { - EndEventQuerySafely(retrievalClient, connection, session.ClientHandle, queryHandle); + EndEventQuerySafely(retrievalClient, rconn, session.ClientHandle, queryHandle); + } + } + finally + { + if (splitChannel) { rconn.Dispose(); } } }