gRPC events: disprove transport hypothesis (native HTTP/2 also returns zero rows)

Tested grpc-event-query-capture.md's leading next-session hypothesis — that the
native client's Grpc.Core HTTP/2 transport (vs our Grpc.Net.Client + GrpcWebHandler
gRPC-Web) is why event reads return zero rows. Added HistorianGrpcChannelFactory
.CreateHttp2 (plain HTTP/2 over SocketsHttpHandler, no gRPC-Web wrap) and an
HISTORIAN_GRPC_EVENT_HTTP2 switch on the event orchestrator (event path only; reads
stay gRPC-Web).

Live side-by-side against the event-bearing 2023 R2 server, everything else held
constant: the full v8 chain (ExchangeKey auth, CM_EVENT RegisterTags/EnsureTags=True,
StartEventQuery with a valid handle) runs end-to-end over BOTH native HTTP/2 and
gRPC-Web, and the server returns the byte-identical version-11 rowCount-0 terminal
(0B00000000001E000000) on both transports. Transport choice makes no difference —
the leading hypothesis is disproven and the zero-row scoping sits above the gRPC
transport layer.

Also confirmed the native capture-event harness queries a 30-day historical window
(returns 50 rows), so the native read is connection-scoped historical retrieval,
not a live subscription.

CreateHttp2 + the env switch + the EventChannelMode diagnostic are retained for
further connection-level probing. 44 offline tests pass; orchestrator stays on the
no-row throw.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B6mcaT2PjRFKcogzp9UkfC

docs/reverse-engineering/grpc-event-query-capture.md

@@ -351,13 +351,24 @@ string/uint handle fields too; the CNG Frida hook. Live recipe: set `HISTORIAN_G
 32565`/`_TLS true`/`_DNSID` to the 2023 R2 server + domain creds (strip quotes); reach the box per the
 live-server access reference.
 
-1. **Transport: native `Grpc.Core` HTTP/2 vs our `Grpc.Net.Client` + `GrpcWebHandler` (gRPC-Web).**
-   This is the leading hypothesis — the strongest remaining difference. Reads work over gRPC-Web *and*
-   return rows, so gRPC-Web isn't broken in general; but events are connection-scoped and may require a
-   **native HTTP/2** connection. TEST: build the event channel WITHOUT the `GrpcWebHandler` wrap (plain
-   HTTP/2 `GrpcChannel`) in `HistorianGrpcChannelFactory` for the event path only, and re-run the
-   diagnostic. If rows flow → gate found. (Mind TLS/ALPN over the loopback tunnel — may need
-   `HttpVersion = 2.0`/`HttpVersionPolicy.RequestVersionExact`.)
+1. ~~**Transport: native `Grpc.Core` HTTP/2 vs our `Grpc.Net.Client` + `GrpcWebHandler` (gRPC-Web).**~~
+   **DISPROVEN 2026-06-23.** Built `HistorianGrpcChannelFactory.CreateHttp2` (plain HTTP/2 over a
+   `SocketsHttpHandler`, no `GrpcWebHandler` wrap, ALPN `h2` to the TLS server) and wired it into the
+   event orchestrator behind `HISTORIAN_GRPC_EVENT_HTTP2=1` (event path only; reads stay gRPC-Web). Live
+   side-by-side against the event-bearing server, **everything else held constant**:
+
+   | channel | auth | registration | queryHandle | result buffer |
+   |---------|------|--------------|-------------|---------------|
+   | `http2` (native HTTP/2) | ✓ | `RTag=True EnsT=True` | 1057 | `0B00000000001E000000` |
+   | `grpc-web` (default)    | ✓ | `RTag=True EnsT=True` | 1058 | `0B00000000001E000000` |
+
+   The complete v8 chain — ExchangeKey ECDH auth, CM_EVENT `RegisterTags`/`EnsureTags`, `StartEventQuery`
+   (valid handle) — runs end-to-end over **plain native HTTP/2**, and the server returns the
+   **byte-identical** version-11 (`0x0B`) rowCount-0 terminal on both transports. So gRPC-Web vs native
+   HTTP/2 is **not** the discriminator — the zero-row scoping is identical regardless of transport. The
+   `CreateHttp2` factory + the `HISTORIAN_GRPC_EVENT_HTTP2` switch + the `EventChannelMode` diagnostic are
+   retained for future connection-level probing. This eliminates the leading hypothesis and tightens the
+   conclusion: the server scopes 0 events to our connection at a layer **above** the gRPC transport.
 
 2. **TLS client identity / certificate.** The native used `SecurityMode=TransportCertificate`. Determine
    whether it presents a **client certificate** the server uses to scope events (our SDK presents none —

src/AVEVA.Historian.Client/Grpc/HistorianGrpcChannelFactory.cs

@@ -1,3 +1,4 @@
+using System.Net.Http;
 using System.Net.Security;
 using System.Security.Cryptography.X509Certificates;
 using Grpc.Core;
@@ -63,6 +64,50 @@ internal static class HistorianGrpcChannelFactory
 
         GrpcChannel channel = GrpcChannel.ForAddress(address, channelOptions);
 
+        return new HistorianGrpcConnection(channel, BuildMetadata(options));
+    }
+
+    /// <summary>
+    /// Builds an event-path channel that speaks <b>native HTTP/2 gRPC</b> (no <see cref="GrpcWebHandler"/>
+    /// wrap) — the leading hypothesis for why gRPC-Web event reads return zero rows while the native
+    /// <c>Grpc.Core</c> HTTP/2 client returns rows for a byte-identical request. The stock 2023 R2 client
+    /// uses native <c>Grpc.Core</c> (HTTP/2); reads happen to work over gRPC-Web too, but the
+    /// connection-scoped event query may require a true HTTP/2 connection. Over TLS this depends on the
+    /// server negotiating the <c>h2</c> ALPN protocol; <see cref="SocketsHttpHandler"/> is pinned to
+    /// HTTP/2 exact so the channel does not silently fall back to HTTP/1.1.
+    /// </summary>
+    public static HistorianGrpcConnection CreateHttp2(HistorianClientOptions options)
+    {
+        string address = ResolveAddress(options);
+
+        var socketsHandler = new SocketsHttpHandler
+        {
+            EnableMultipleHttp2Connections = true
+        };
+
+        if (options.AllowUntrustedServerCertificate && options.GrpcUseTls)
+        {
+            socketsHandler.SslOptions = new SslClientAuthenticationOptions
+            {
+                RemoteCertificateValidationCallback = (_, _, _, _) => true
+            };
+        }
+
+        var channelOptions = new GrpcChannelOptions
+        {
+            HttpHandler = socketsHandler
+        };
+
+        // GrpcChannel over a SocketsHttpHandler already issues requests as HTTP/2 with
+        // RequestVersionExact (no GrpcWebHandler means no HTTP/1.1 fallback to mask a failed h2
+        // negotiation — it surfaces instead).
+        GrpcChannel channel = GrpcChannel.ForAddress(address, channelOptions);
+
+        return new HistorianGrpcConnection(channel, BuildMetadata(options));
+    }
+
+    private static Metadata BuildMetadata(HistorianClientOptions options)
+    {
         // The stock client always advertises gzip request encoding; honour the option so
         // bandwidth-limited links can disable it.
         var metadata = new Metadata();
@@ -71,7 +116,7 @@ internal static class HistorianGrpcChannelFactory
             metadata.Add("grpc-internal-encoding-request", "gzip");
         }
 
-        return new HistorianGrpcConnection(channel, metadata);
+        return metadata;
     }
 
     private static bool AcceptAnyCertificate(

src/AVEVA.Historian.Client/Grpc/HistorianGrpcEventOrchestrator.cs

@@ -58,6 +58,9 @@ internal sealed class HistorianGrpcEventOrchestrator
     /// <summary>Diagnostic: type+code description of the most recent error/terminal buffer.</summary>
     public string LastErrorBufferDescription { get; private set; } = string.Empty;
 
+    /// <summary>Diagnostic: which transport the event channel used (<c>grpc-web</c> or <c>http2</c>).</summary>
+    public string EventChannelMode { get; private set; } = string.Empty;
+
     /// <summary>Diagnostic: hex of the most recent result buffer (first 48 bytes).</summary>
     public string LastResultBufferHex { get; private set; } = string.Empty;
 
@@ -143,7 +146,16 @@ internal sealed class HistorianGrpcEventOrchestrator
 
     private List<HistorianEvent> RunEventChain(DateTime startUtc, DateTime endUtc, HistorianEventFilter? filter, CancellationToken cancellationToken)
     {
-        using HistorianGrpcConnection connection = HistorianGrpcChannelFactory.Create(_options);
+        // Hypothesis #1 (server-side/connection angle, grpc-event-query-capture.md): the native client
+        // uses Grpc.Core native HTTP/2, while our default channel wraps gRPC-Web over HTTP/1.1. Reads
+        // work over gRPC-Web, but the connection-scoped event query may require a true HTTP/2 connection.
+        // Opt in via HISTORIAN_GRPC_EVENT_HTTP2=1 to use a plain HTTP/2 channel for the event path only.
+        bool useHttp2 = string.Equals(
+            Environment.GetEnvironmentVariable("HISTORIAN_GRPC_EVENT_HTTP2"), "1", StringComparison.Ordinal);
+        EventChannelMode = useHttp2 ? "http2" : "grpc-web";
+        using HistorianGrpcConnection connection = useHttp2
+            ? HistorianGrpcChannelFactory.CreateHttp2(_options)
+            : HistorianGrpcChannelFactory.Create(_options);
         // Event reads need an Event-type (v8) connection. OpenSession(eventConnection: true) runs the
         // full v8 path: HistoryService.ExchangeKey (P-256 ECDH) -> client key = SHA256(secret) -> v8
         // OpenConnection with ConnectionType=Event and the credential token RC4(password, MD5(clientKey)).

tests/AVEVA.Historian.Client.Tests/HistorianGrpcIntegrationTests.cs

@@ -546,7 +546,7 @@ public sealed class HistorianGrpcIntegrationTests
         }
 
         throw new Xunit.Sdk.XunitException(
-            $"[DIAG] outcome={outcome} | events={events.Count} | LastResultLen={orch.LastResultBufferLength} " +
+            $"[DIAG] channel={orch.EventChannelMode} outcome={outcome} | events={events.Count} | LastResultLen={orch.LastResultBufferLength} " +
             $"| ResultHex={orch.LastResultBufferHex} | Reg=[{orch.RegistrationDiag}] " +
             $"| v8open={AVEVA.Historian.Client.Grpc.HistorianGrpcHandshake.LastEventOpenRequestHex}");
     }