From b0388e7a40b875124f4945e79f8e61b9285be0d2 Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Tue, 23 Jun 2026 12:55:09 -0400 Subject: [PATCH 1/2] gRPC events: disprove transport hypothesis (native HTTP/2 also returns zero rows) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Tested grpc-event-query-capture.md's leading next-session hypothesis — that the native client's Grpc.Core HTTP/2 transport (vs our Grpc.Net.Client + GrpcWebHandler gRPC-Web) is why event reads return zero rows. Added HistorianGrpcChannelFactory .CreateHttp2 (plain HTTP/2 over SocketsHttpHandler, no gRPC-Web wrap) and an HISTORIAN_GRPC_EVENT_HTTP2 switch on the event orchestrator (event path only; reads stay gRPC-Web). Live side-by-side against the event-bearing 2023 R2 server, everything else held constant: the full v8 chain (ExchangeKey auth, CM_EVENT RegisterTags/EnsureTags=True, StartEventQuery with a valid handle) runs end-to-end over BOTH native HTTP/2 and gRPC-Web, and the server returns the byte-identical version-11 rowCount-0 terminal (0B00000000001E000000) on both transports. Transport choice makes no difference — the leading hypothesis is disproven and the zero-row scoping sits above the gRPC transport layer. Also confirmed the native capture-event harness queries a 30-day historical window (returns 50 rows), so the native read is connection-scoped historical retrieval, not a live subscription. CreateHttp2 + the env switch + the EventChannelMode diagnostic are retained for further connection-level probing. 44 offline tests pass; orchestrator stays on the no-row throw. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01B6mcaT2PjRFKcogzp9UkfC --- .../grpc-event-query-capture.md | 25 +++++++--- .../Grpc/HistorianGrpcChannelFactory.cs | 47 ++++++++++++++++++- .../Grpc/HistorianGrpcEventOrchestrator.cs | 14 +++++- .../HistorianGrpcIntegrationTests.cs | 2 +- 4 files changed, 78 insertions(+), 10 deletions(-) diff --git a/docs/reverse-engineering/grpc-event-query-capture.md b/docs/reverse-engineering/grpc-event-query-capture.md index c08c1fb..1acee4c 100644 --- a/docs/reverse-engineering/grpc-event-query-capture.md +++ b/docs/reverse-engineering/grpc-event-query-capture.md @@ -351,13 +351,24 @@ string/uint handle fields too; the CNG Frida hook. Live recipe: set `HISTORIAN_G 32565`/`_TLS true`/`_DNSID` to the 2023 R2 server + domain creds (strip quotes); reach the box per the live-server access reference. -1. **Transport: native `Grpc.Core` HTTP/2 vs our `Grpc.Net.Client` + `GrpcWebHandler` (gRPC-Web).** - This is the leading hypothesis — the strongest remaining difference. Reads work over gRPC-Web *and* - return rows, so gRPC-Web isn't broken in general; but events are connection-scoped and may require a - **native HTTP/2** connection. TEST: build the event channel WITHOUT the `GrpcWebHandler` wrap (plain - HTTP/2 `GrpcChannel`) in `HistorianGrpcChannelFactory` for the event path only, and re-run the - diagnostic. If rows flow → gate found. (Mind TLS/ALPN over the loopback tunnel — may need - `HttpVersion = 2.0`/`HttpVersionPolicy.RequestVersionExact`.) +1. ~~**Transport: native `Grpc.Core` HTTP/2 vs our `Grpc.Net.Client` + `GrpcWebHandler` (gRPC-Web).**~~ + **DISPROVEN 2026-06-23.** Built `HistorianGrpcChannelFactory.CreateHttp2` (plain HTTP/2 over a + `SocketsHttpHandler`, no `GrpcWebHandler` wrap, ALPN `h2` to the TLS server) and wired it into the + event orchestrator behind `HISTORIAN_GRPC_EVENT_HTTP2=1` (event path only; reads stay gRPC-Web). Live + side-by-side against the event-bearing server, **everything else held constant**: + + | channel | auth | registration | queryHandle | result buffer | + |---------|------|--------------|-------------|---------------| + | `http2` (native HTTP/2) | ✓ | `RTag=True EnsT=True` | 1057 | `0B00000000001E000000` | + | `grpc-web` (default) | ✓ | `RTag=True EnsT=True` | 1058 | `0B00000000001E000000` | + + The complete v8 chain — ExchangeKey ECDH auth, CM_EVENT `RegisterTags`/`EnsureTags`, `StartEventQuery` + (valid handle) — runs end-to-end over **plain native HTTP/2**, and the server returns the + **byte-identical** version-11 (`0x0B`) rowCount-0 terminal on both transports. So gRPC-Web vs native + HTTP/2 is **not** the discriminator — the zero-row scoping is identical regardless of transport. The + `CreateHttp2` factory + the `HISTORIAN_GRPC_EVENT_HTTP2` switch + the `EventChannelMode` diagnostic are + retained for future connection-level probing. This eliminates the leading hypothesis and tightens the + conclusion: the server scopes 0 events to our connection at a layer **above** the gRPC transport. 2. **TLS client identity / certificate.** The native used `SecurityMode=TransportCertificate`. Determine whether it presents a **client certificate** the server uses to scope events (our SDK presents none — diff --git a/src/AVEVA.Historian.Client/Grpc/HistorianGrpcChannelFactory.cs b/src/AVEVA.Historian.Client/Grpc/HistorianGrpcChannelFactory.cs index 151033b..8ea93bb 100644 --- a/src/AVEVA.Historian.Client/Grpc/HistorianGrpcChannelFactory.cs +++ b/src/AVEVA.Historian.Client/Grpc/HistorianGrpcChannelFactory.cs @@ -1,3 +1,4 @@ +using System.Net.Http; using System.Net.Security; using System.Security.Cryptography.X509Certificates; using Grpc.Core; @@ -63,6 +64,50 @@ internal static class HistorianGrpcChannelFactory GrpcChannel channel = GrpcChannel.ForAddress(address, channelOptions); + return new HistorianGrpcConnection(channel, BuildMetadata(options)); + } + + /// + /// Builds an event-path channel that speaks native HTTP/2 gRPC (no + /// wrap) — the leading hypothesis for why gRPC-Web event reads return zero rows while the native + /// Grpc.Core HTTP/2 client returns rows for a byte-identical request. The stock 2023 R2 client + /// uses native Grpc.Core (HTTP/2); reads happen to work over gRPC-Web too, but the + /// connection-scoped event query may require a true HTTP/2 connection. Over TLS this depends on the + /// server negotiating the h2 ALPN protocol; is pinned to + /// HTTP/2 exact so the channel does not silently fall back to HTTP/1.1. + /// + public static HistorianGrpcConnection CreateHttp2(HistorianClientOptions options) + { + string address = ResolveAddress(options); + + var socketsHandler = new SocketsHttpHandler + { + EnableMultipleHttp2Connections = true + }; + + if (options.AllowUntrustedServerCertificate && options.GrpcUseTls) + { + socketsHandler.SslOptions = new SslClientAuthenticationOptions + { + RemoteCertificateValidationCallback = (_, _, _, _) => true + }; + } + + var channelOptions = new GrpcChannelOptions + { + HttpHandler = socketsHandler + }; + + // GrpcChannel over a SocketsHttpHandler already issues requests as HTTP/2 with + // RequestVersionExact (no GrpcWebHandler means no HTTP/1.1 fallback to mask a failed h2 + // negotiation — it surfaces instead). + GrpcChannel channel = GrpcChannel.ForAddress(address, channelOptions); + + return new HistorianGrpcConnection(channel, BuildMetadata(options)); + } + + private static Metadata BuildMetadata(HistorianClientOptions options) + { // The stock client always advertises gzip request encoding; honour the option so // bandwidth-limited links can disable it. var metadata = new Metadata(); @@ -71,7 +116,7 @@ internal static class HistorianGrpcChannelFactory metadata.Add("grpc-internal-encoding-request", "gzip"); } - return new HistorianGrpcConnection(channel, metadata); + return metadata; } private static bool AcceptAnyCertificate( diff --git a/src/AVEVA.Historian.Client/Grpc/HistorianGrpcEventOrchestrator.cs b/src/AVEVA.Historian.Client/Grpc/HistorianGrpcEventOrchestrator.cs index d9879cf..8c88fd9 100644 --- a/src/AVEVA.Historian.Client/Grpc/HistorianGrpcEventOrchestrator.cs +++ b/src/AVEVA.Historian.Client/Grpc/HistorianGrpcEventOrchestrator.cs @@ -58,6 +58,9 @@ internal sealed class HistorianGrpcEventOrchestrator /// Diagnostic: type+code description of the most recent error/terminal buffer. public string LastErrorBufferDescription { get; private set; } = string.Empty; + /// Diagnostic: which transport the event channel used (grpc-web or http2). + public string EventChannelMode { get; private set; } = string.Empty; + /// Diagnostic: hex of the most recent result buffer (first 48 bytes). public string LastResultBufferHex { get; private set; } = string.Empty; @@ -143,7 +146,16 @@ internal sealed class HistorianGrpcEventOrchestrator private List RunEventChain(DateTime startUtc, DateTime endUtc, HistorianEventFilter? filter, CancellationToken cancellationToken) { - using HistorianGrpcConnection connection = HistorianGrpcChannelFactory.Create(_options); + // Hypothesis #1 (server-side/connection angle, grpc-event-query-capture.md): the native client + // uses Grpc.Core native HTTP/2, while our default channel wraps gRPC-Web over HTTP/1.1. Reads + // work over gRPC-Web, but the connection-scoped event query may require a true HTTP/2 connection. + // Opt in via HISTORIAN_GRPC_EVENT_HTTP2=1 to use a plain HTTP/2 channel for the event path only. + bool useHttp2 = string.Equals( + Environment.GetEnvironmentVariable("HISTORIAN_GRPC_EVENT_HTTP2"), "1", StringComparison.Ordinal); + EventChannelMode = useHttp2 ? "http2" : "grpc-web"; + using HistorianGrpcConnection connection = useHttp2 + ? HistorianGrpcChannelFactory.CreateHttp2(_options) + : HistorianGrpcChannelFactory.Create(_options); // Event reads need an Event-type (v8) connection. OpenSession(eventConnection: true) runs the // full v8 path: HistoryService.ExchangeKey (P-256 ECDH) -> client key = SHA256(secret) -> v8 // OpenConnection with ConnectionType=Event and the credential token RC4(password, MD5(clientKey)). diff --git a/tests/AVEVA.Historian.Client.Tests/HistorianGrpcIntegrationTests.cs b/tests/AVEVA.Historian.Client.Tests/HistorianGrpcIntegrationTests.cs index 9ff5739..5eaf91e 100644 --- a/tests/AVEVA.Historian.Client.Tests/HistorianGrpcIntegrationTests.cs +++ b/tests/AVEVA.Historian.Client.Tests/HistorianGrpcIntegrationTests.cs @@ -546,7 +546,7 @@ public sealed class HistorianGrpcIntegrationTests } throw new Xunit.Sdk.XunitException( - $"[DIAG] outcome={outcome} | events={events.Count} | LastResultLen={orch.LastResultBufferLength} " + + $"[DIAG] channel={orch.EventChannelMode} outcome={outcome} | events={events.Count} | LastResultLen={orch.LastResultBufferLength} " + $"| ResultHex={orch.LastResultBufferHex} | Reg=[{orch.RegistrationDiag}] " + $"| v8open={AVEVA.Historian.Client.Grpc.HistorianGrpcHandshake.LastEventOpenRequestHex}"); } From f19eb3b821ca409c1709518ed624e7c98f2aef95 Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Tue, 23 Jun 2026 13:05:19 -0400 Subject: [PATCH 2/2] =?UTF-8?q?gRPC=20events:=20answer=20hypothesis=20#4?= =?UTF-8?q?=20(SQL=20ground=20truth)=20=E2=80=94=20event=20store=20is=20gl?= =?UTF-8?q?obal,=20not=20connection-scoped?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Pursued the server-side SQL angle for the gRPC event zero-rows. Built a read-only SOCKS5-relay + Microsoft.Data.SqlClient probe (gitignored, artifacts/.../sqlschema/) and dumped the live Runtime event schema. Findings: - No per-connection / per-client / per-session column exists anywhere in the event store. The only scoping-like columns on Events/EventHistory/snapshots are event content (Source_* origin, User_* acker, Provider_NodeName, SourceServer replication). - The rich Events view is not a relational table — it is served live by the Historian engine via the INSQL OLE DB provider (linked servers INSQL/INSQLD; encrypted remote view). The SQL EventHistory base table holds only 168 rows / 1 tag. - Decisive: for the SAME -90d..now window the gRPC StartEventQuery diagnostic returned 0 rows, the Events view via INSQL returns 71,332 events (most recent Alarm.Set firing seconds before the probe). Same engine, same window — INSQL serves the data, gRPC withholds it from our connection. So there is nothing in the data to scope by: the zero-row gate is the gRPC RetrievalService's per-connection in-process execution state, not data scoping or transport (the same class of wall as DeleteTagExtendedProperties). Combined with the transport disproof, three independent angles are now exhausted — client payload (byte-identical), transport (HTTP/2 == gRPC-Web), and data store (global, unscoped). gRPC event-row retrieval stands documented as auth-solved / retrieval-server-gated; ReadEventsAsync over gRPC keeps the no-row throw and event reads use WCF. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01B6mcaT2PjRFKcogzp9UkfC --- .../grpc-event-query-capture.md | 55 ++++++++++++++++--- 1 file changed, 47 insertions(+), 8 deletions(-) diff --git a/docs/reverse-engineering/grpc-event-query-capture.md b/docs/reverse-engineering/grpc-event-query-capture.md index 1acee4c..b39b510 100644 --- a/docs/reverse-engineering/grpc-event-query-capture.md +++ b/docs/reverse-engineering/grpc-event-query-capture.md @@ -381,15 +381,54 @@ live-server access reference. TLS-decrypting mitm on the loopback forward — to see any connection-level header/affinity our capture can't see. -4. **Server-side ground truth.** Via the SOCKS→SQL relay (user-authorized, read-only), inspect the - `Runtime.dbo.Events` schema for any per-connection / per-client / source-session column that would - explain why the server returns the rows to the native connection but not ours. Also check whether the - StorageService/event-store path has a connection-scoping notion the History-service event query - depends on. +2. **TLS client identity / certificate.** The native used `SecurityMode=TransportCertificate`. Determine + whether it presents a **client certificate** the server uses to scope events (our SDK presents none — + `AllowUntrustedServerCertificate=true`, server cert only). TEST: capture the TLS handshake (e.g. + `SSLKEYLOGFILE` + Wireshark, or a decrypting proxy) for a native `capture-event` run and check the + Certificate message; if a client cert is presented, replicate it. **Lower-probability after #1: the + plain-HTTP/2 path presents no client cert either, yet auth + registration still succeed and the gate + persists — so the gate is not at the TLS-identity layer the cert would affect.** -If 1–4 don't crack it, the realistic conclusion is that gRPC event-row retrieval has a server-side -connection-identity dependency not reachable from a pure-managed client, and it stays documented as -auth-solved / retrieval-connection-gated. +3. **HTTP/2-level capture.** The byte[]/handle capture is RPC-payload only. Capture the actual HTTP/2 + frames (HEADERS/SETTINGS/stream IDs, connection reuse) for the native run vs ours — via a + TLS-decrypting mitm on the loopback forward — to see any connection-level header/affinity our capture + can't see. + +4. ~~**Server-side ground truth.**~~ **ANSWERED 2026-06-23 (DISPROVES the data-scoping premise).** Via + the SOCKS→SQL relay (read-only; `artifacts/.../sqlschema/`, gitignored), dumped the full event schema + on the live `Runtime` DB. Findings: + - **No per-connection / per-client / per-session column exists anywhere in the event store.** The only + "scoping-like" columns on `Events`/`EventHistory`/snapshots are event *content* — `Source_*` (event + origin area/object/PV), `User_*` (who acknowledged), `Provider_NodeName` (alarm provider node), + `SourceServer`/`SourceTag` (cross-server replication). None is "which client connection requested + this." + - **The rich `Events` view is not a relational table — it is served live by the Historian engine via + the `INSQL` OLE DB provider** (`sys.servers` shows linked servers `INSQL` + `INSQLD`; + `OBJECT_DEFINITION('dbo.Events')` is `NULL` = encrypted remote view). The Historian's own + `EventHistory` base table holds just 168 rows / 1 tag (the internal event-tag detector log); the + alarm/event journal the gRPC query reads lives in the engine, surfaced through INSQL. + - **Decisive: same engine, same `-90d..now` window, two paths diverge.** The `Events` view (via INSQL) + returns **71,332 events** for that window — most recent `Alarm.Set` firing seconds before the probe + (live, every few seconds) — while gRPC `StartEventQuery` for **our** connection returns **0**. The + data is global, abundant, recent, and identical-window-addressable; the engine simply does not hand + it to our gRPC connection. + + → There is **nothing in the data to scope by**, so the zero-row gate is **not** data scoping. It is the + gRPC RetrievalService's **per-connection in-process execution state** — the same class of wall as + `DeleteTagExtendedProperties` (server-side native in-process working-set, not reconstructable from + byte-identical wire requests). Reproduce: `artifacts/.../sqlschema/` (Program.cs = SOCKS5 relay + + `Microsoft.Data.SqlClient`; authenticate with the server's SQL login, not the domain Historian acct — + creds in the gitignored creds file). + +**Conclusion (after #1 disproven + #4 answered).** Three independent angles are now exhausted: client +payload (byte-identical), transport (native HTTP/2 == gRPC-Web, both 0 rows), and data store (global, +unscoped, 71,332 events the engine serves via INSQL but withholds from our gRPC connection). The gate is +a **server-internal per-connection retrieval working-set** that a pure-managed client cannot reconstruct +by matching wire bytes, transport, or data. The remaining angles (#2 client-cert, #3 HTTP/2-frame +capture) are low-probability — #1 showed auth+registration succeed with no client cert over plain HTTP/2 +and the gate still holds. **gRPC event-row retrieval stands documented as auth-solved / +retrieval-server-gated**; `ReadEventsAsync` over gRPC keeps the honest no-row throw, and event reads use +the WCF transport. **2 of 3 layers cleared** (key exchange + client key); the 3rd (token construction) is localized to a specific managed method, pending dnlib extraction. ExchangeKey + the v8 serializer are committed; the