Merge grpc-event-http2-disproof: gRPC event row retrieval — transport + SQL angles exhausted

Pursued the two server-side/connection hypotheses from grpc-event-query-capture.md
for the gRPC event zero-rows blocker:

- #1 transport (DISPROVEN): native HTTP/2 (no GrpcWebHandler) returns the byte-identical
  version-11 rowCount-0 terminal as gRPC-Web; the full v8 chain (ExchangeKey auth,
  CM_EVENT registration, StartEventQuery) runs end-to-end over both. Added
  HistorianGrpcChannelFactory.CreateHttp2 + HISTORIAN_GRPC_EVENT_HTTP2 switch.
- #4 SQL ground truth (ANSWERED): the event store has no per-connection column; the
  rich Events view is served live by the Historian engine via the INSQL OLE DB
  provider. Same engine + same window: 71,332 events via INSQL vs 0 via gRPC for our
  connection. The data is global/unscoped — the gate is the gRPC RetrievalService's
  per-connection in-process execution state, not data scoping or transport.

Three independent angles now exhausted (client payload byte-identical, transport
HTTP/2 == gRPC-Web, data store global). gRPC event-row retrieval stands documented as
auth-solved / retrieval-server-gated; ReadEventsAsync over gRPC keeps the no-row throw
and event reads use WCF. 326 offline tests pass.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B6mcaT2PjRFKcogzp9UkfC

docs/reverse-engineering/grpc-event-query-capture.md

@@ -351,13 +351,24 @@ string/uint handle fields too; the CNG Frida hook. Live recipe: set `HISTORIAN_G
 32565`/`_TLS true`/`_DNSID` to the 2023 R2 server + domain creds (strip quotes); reach the box per the
 live-server access reference.
 
-1. **Transport: native `Grpc.Core` HTTP/2 vs our `Grpc.Net.Client` + `GrpcWebHandler` (gRPC-Web).**
-   This is the leading hypothesis — the strongest remaining difference. Reads work over gRPC-Web *and*
-   return rows, so gRPC-Web isn't broken in general; but events are connection-scoped and may require a
-   **native HTTP/2** connection. TEST: build the event channel WITHOUT the `GrpcWebHandler` wrap (plain
-   HTTP/2 `GrpcChannel`) in `HistorianGrpcChannelFactory` for the event path only, and re-run the
-   diagnostic. If rows flow → gate found. (Mind TLS/ALPN over the loopback tunnel — may need
-   `HttpVersion = 2.0`/`HttpVersionPolicy.RequestVersionExact`.)
+1. ~~**Transport: native `Grpc.Core` HTTP/2 vs our `Grpc.Net.Client` + `GrpcWebHandler` (gRPC-Web).**~~
+   **DISPROVEN 2026-06-23.** Built `HistorianGrpcChannelFactory.CreateHttp2` (plain HTTP/2 over a
+   `SocketsHttpHandler`, no `GrpcWebHandler` wrap, ALPN `h2` to the TLS server) and wired it into the
+   event orchestrator behind `HISTORIAN_GRPC_EVENT_HTTP2=1` (event path only; reads stay gRPC-Web). Live
+   side-by-side against the event-bearing server, **everything else held constant**:
+
+   | channel | auth | registration | queryHandle | result buffer |
+   |---------|------|--------------|-------------|---------------|
+   | `http2` (native HTTP/2) | ✓ | `RTag=True EnsT=True` | 1057 | `0B00000000001E000000` |
+   | `grpc-web` (default)    | ✓ | `RTag=True EnsT=True` | 1058 | `0B00000000001E000000` |
+
+   The complete v8 chain — ExchangeKey ECDH auth, CM_EVENT `RegisterTags`/`EnsureTags`, `StartEventQuery`
+   (valid handle) — runs end-to-end over **plain native HTTP/2**, and the server returns the
+   **byte-identical** version-11 (`0x0B`) rowCount-0 terminal on both transports. So gRPC-Web vs native
+   HTTP/2 is **not** the discriminator — the zero-row scoping is identical regardless of transport. The
+   `CreateHttp2` factory + the `HISTORIAN_GRPC_EVENT_HTTP2` switch + the `EventChannelMode` diagnostic are
+   retained for future connection-level probing. This eliminates the leading hypothesis and tightens the
+   conclusion: the server scopes 0 events to our connection at a layer **above** the gRPC transport.
 
 2. **TLS client identity / certificate.** The native used `SecurityMode=TransportCertificate`. Determine
    whether it presents a **client certificate** the server uses to scope events (our SDK presents none —
@@ -370,15 +381,54 @@ live-server access reference.
    TLS-decrypting mitm on the loopback forward — to see any connection-level header/affinity our capture
    can't see.
 
-4. **Server-side ground truth.** Via the SOCKS→SQL relay (user-authorized, read-only), inspect the
-   `Runtime.dbo.Events` schema for any per-connection / per-client / source-session column that would
-   explain why the server returns the rows to the native connection but not ours. Also check whether the
-   StorageService/event-store path has a connection-scoping notion the History-service event query
-   depends on.
+2. **TLS client identity / certificate.** The native used `SecurityMode=TransportCertificate`. Determine
+   whether it presents a **client certificate** the server uses to scope events (our SDK presents none —
+   `AllowUntrustedServerCertificate=true`, server cert only). TEST: capture the TLS handshake (e.g.
+   `SSLKEYLOGFILE` + Wireshark, or a decrypting proxy) for a native `capture-event` run and check the
+   Certificate message; if a client cert is presented, replicate it. **Lower-probability after #1: the
+   plain-HTTP/2 path presents no client cert either, yet auth + registration still succeed and the gate
+   persists — so the gate is not at the TLS-identity layer the cert would affect.**
 
-If 1–4 don't crack it, the realistic conclusion is that gRPC event-row retrieval has a server-side
-connection-identity dependency not reachable from a pure-managed client, and it stays documented as
-auth-solved / retrieval-connection-gated.
+3. **HTTP/2-level capture.** The byte[]/handle capture is RPC-payload only. Capture the actual HTTP/2
+   frames (HEADERS/SETTINGS/stream IDs, connection reuse) for the native run vs ours — via a
+   TLS-decrypting mitm on the loopback forward — to see any connection-level header/affinity our capture
+   can't see.
+
+4. ~~**Server-side ground truth.**~~ **ANSWERED 2026-06-23 (DISPROVES the data-scoping premise).** Via
+   the SOCKS→SQL relay (read-only; `artifacts/.../sqlschema/`, gitignored), dumped the full event schema
+   on the live `Runtime` DB. Findings:
+   - **No per-connection / per-client / per-session column exists anywhere in the event store.** The only
+     "scoping-like" columns on `Events`/`EventHistory`/snapshots are event *content* — `Source_*` (event
+     origin area/object/PV), `User_*` (who acknowledged), `Provider_NodeName` (alarm provider node),
+     `SourceServer`/`SourceTag` (cross-server replication). None is "which client connection requested
+     this."
+   - **The rich `Events` view is not a relational table — it is served live by the Historian engine via
+     the `INSQL` OLE DB provider** (`sys.servers` shows linked servers `INSQL` + `INSQLD`;
+     `OBJECT_DEFINITION('dbo.Events')` is `NULL` = encrypted remote view). The Historian's own
+     `EventHistory` base table holds just 168 rows / 1 tag (the internal event-tag detector log); the
+     alarm/event journal the gRPC query reads lives in the engine, surfaced through INSQL.
+   - **Decisive: same engine, same `-90d..now` window, two paths diverge.** The `Events` view (via INSQL)
+     returns **71,332 events** for that window — most recent `Alarm.Set` firing seconds before the probe
+     (live, every few seconds) — while gRPC `StartEventQuery` for **our** connection returns **0**. The
+     data is global, abundant, recent, and identical-window-addressable; the engine simply does not hand
+     it to our gRPC connection.
+
+   → There is **nothing in the data to scope by**, so the zero-row gate is **not** data scoping. It is the
+   gRPC RetrievalService's **per-connection in-process execution state** — the same class of wall as
+   `DeleteTagExtendedProperties` (server-side native in-process working-set, not reconstructable from
+   byte-identical wire requests). Reproduce: `artifacts/.../sqlschema/` (Program.cs = SOCKS5 relay +
+   `Microsoft.Data.SqlClient`; authenticate with the server's SQL login, not the domain Historian acct —
+   creds in the gitignored creds file).
+
+**Conclusion (after #1 disproven + #4 answered).** Three independent angles are now exhausted: client
+payload (byte-identical), transport (native HTTP/2 == gRPC-Web, both 0 rows), and data store (global,
+unscoped, 71,332 events the engine serves via INSQL but withholds from our gRPC connection). The gate is
+a **server-internal per-connection retrieval working-set** that a pure-managed client cannot reconstruct
+by matching wire bytes, transport, or data. The remaining angles (#2 client-cert, #3 HTTP/2-frame
+capture) are low-probability — #1 showed auth+registration succeed with no client cert over plain HTTP/2
+and the gate still holds. **gRPC event-row retrieval stands documented as auth-solved /
+retrieval-server-gated**; `ReadEventsAsync` over gRPC keeps the honest no-row throw, and event reads use
+the WCF transport.
 
 **2 of 3 layers cleared** (key exchange + client key); the 3rd (token construction) is localized to a
 specific managed method, pending dnlib extraction. ExchangeKey + the v8 serializer are committed; the

src/AVEVA.Historian.Client/Grpc/HistorianGrpcChannelFactory.cs

@@ -1,3 +1,4 @@
+using System.Net.Http;
 using System.Net.Security;
 using System.Security.Cryptography.X509Certificates;
 using Grpc.Core;
@@ -63,6 +64,50 @@ internal static class HistorianGrpcChannelFactory
 
         GrpcChannel channel = GrpcChannel.ForAddress(address, channelOptions);
 
+        return new HistorianGrpcConnection(channel, BuildMetadata(options));
+    }
+
+    /// <summary>
+    /// Builds an event-path channel that speaks <b>native HTTP/2 gRPC</b> (no <see cref="GrpcWebHandler"/>
+    /// wrap) — the leading hypothesis for why gRPC-Web event reads return zero rows while the native
+    /// <c>Grpc.Core</c> HTTP/2 client returns rows for a byte-identical request. The stock 2023 R2 client
+    /// uses native <c>Grpc.Core</c> (HTTP/2); reads happen to work over gRPC-Web too, but the
+    /// connection-scoped event query may require a true HTTP/2 connection. Over TLS this depends on the
+    /// server negotiating the <c>h2</c> ALPN protocol; <see cref="SocketsHttpHandler"/> is pinned to
+    /// HTTP/2 exact so the channel does not silently fall back to HTTP/1.1.
+    /// </summary>
+    public static HistorianGrpcConnection CreateHttp2(HistorianClientOptions options)
+    {
+        string address = ResolveAddress(options);
+
+        var socketsHandler = new SocketsHttpHandler
+        {
+            EnableMultipleHttp2Connections = true
+        };
+
+        if (options.AllowUntrustedServerCertificate && options.GrpcUseTls)
+        {
+            socketsHandler.SslOptions = new SslClientAuthenticationOptions
+            {
+                RemoteCertificateValidationCallback = (_, _, _, _) => true
+            };
+        }
+
+        var channelOptions = new GrpcChannelOptions
+        {
+            HttpHandler = socketsHandler
+        };
+
+        // GrpcChannel over a SocketsHttpHandler already issues requests as HTTP/2 with
+        // RequestVersionExact (no GrpcWebHandler means no HTTP/1.1 fallback to mask a failed h2
+        // negotiation — it surfaces instead).
+        GrpcChannel channel = GrpcChannel.ForAddress(address, channelOptions);
+
+        return new HistorianGrpcConnection(channel, BuildMetadata(options));
+    }
+
+    private static Metadata BuildMetadata(HistorianClientOptions options)
+    {
         // The stock client always advertises gzip request encoding; honour the option so
         // bandwidth-limited links can disable it.
         var metadata = new Metadata();
@@ -71,7 +116,7 @@ internal static class HistorianGrpcChannelFactory
             metadata.Add("grpc-internal-encoding-request", "gzip");
         }
 
-        return new HistorianGrpcConnection(channel, metadata);
+        return metadata;
     }
 
     private static bool AcceptAnyCertificate(

src/AVEVA.Historian.Client/Grpc/HistorianGrpcEventOrchestrator.cs

@@ -58,6 +58,9 @@ internal sealed class HistorianGrpcEventOrchestrator
     /// <summary>Diagnostic: type+code description of the most recent error/terminal buffer.</summary>
     public string LastErrorBufferDescription { get; private set; } = string.Empty;
 
+    /// <summary>Diagnostic: which transport the event channel used (<c>grpc-web</c> or <c>http2</c>).</summary>
+    public string EventChannelMode { get; private set; } = string.Empty;
+
     /// <summary>Diagnostic: hex of the most recent result buffer (first 48 bytes).</summary>
     public string LastResultBufferHex { get; private set; } = string.Empty;
 
@@ -143,7 +146,16 @@ internal sealed class HistorianGrpcEventOrchestrator
 
     private List<HistorianEvent> RunEventChain(DateTime startUtc, DateTime endUtc, HistorianEventFilter? filter, CancellationToken cancellationToken)
     {
-        using HistorianGrpcConnection connection = HistorianGrpcChannelFactory.Create(_options);
+        // Hypothesis #1 (server-side/connection angle, grpc-event-query-capture.md): the native client
+        // uses Grpc.Core native HTTP/2, while our default channel wraps gRPC-Web over HTTP/1.1. Reads
+        // work over gRPC-Web, but the connection-scoped event query may require a true HTTP/2 connection.
+        // Opt in via HISTORIAN_GRPC_EVENT_HTTP2=1 to use a plain HTTP/2 channel for the event path only.
+        bool useHttp2 = string.Equals(
+            Environment.GetEnvironmentVariable("HISTORIAN_GRPC_EVENT_HTTP2"), "1", StringComparison.Ordinal);
+        EventChannelMode = useHttp2 ? "http2" : "grpc-web";
+        using HistorianGrpcConnection connection = useHttp2
+            ? HistorianGrpcChannelFactory.CreateHttp2(_options)
+            : HistorianGrpcChannelFactory.Create(_options);
         // Event reads need an Event-type (v8) connection. OpenSession(eventConnection: true) runs the
         // full v8 path: HistoryService.ExchangeKey (P-256 ECDH) -> client key = SHA256(secret) -> v8
         // OpenConnection with ConnectionType=Event and the credential token RC4(password, MD5(clientKey)).

tests/AVEVA.Historian.Client.Tests/HistorianGrpcIntegrationTests.cs

@@ -546,7 +546,7 @@ public sealed class HistorianGrpcIntegrationTests
         }
 
         throw new Xunit.Sdk.XunitException(
-            $"[DIAG] outcome={outcome} | events={events.Count} | LastResultLen={orch.LastResultBufferLength} " +
+            $"[DIAG] channel={orch.EventChannelMode} outcome={outcome} | events={events.Count} | LastResultLen={orch.LastResultBufferLength} " +
             $"| ResultHex={orch.LastResultBufferHex} | Reg=[{orch.RegistrationDiag}] " +
             $"| v8open={AVEVA.Historian.Client.Grpc.HistorianGrpcHandshake.LastEventOpenRequestHex}");
     }