7 Commits

Author SHA1 Message Date
Joseph Doherty 54bcf8a5e0 docs(c2): record 2020-vs-2023R2 positive control for the event-read server gate
Adds the positive control that the prior C2 evidence lacked. The SAME native WCF
event-read client returns real events (5) from a local AVEVA Historian 2020 but 0
from the 2023 R2 server over the identical sequence and window, while both boxes
hold tens of thousands of events in SQL — isolating the zero-rows to the 2023 R2
server, not the client, protocol, or serializers.

- wcf-event-read-spike-results.md: new "2026-06-26 positive control" section
  (2020 vs 2023 R2 A/B from one WCF client; stock-2020-client version-self-block
  caveat; stock-2023R2 gRPC cross-check).
- grpc-event-query-capture.md: re-control note — the 2026-06-22 stock 50-row
  capture did NOT reproduce; the stock 2023 R2 client now also returns 0 rows.
- HistorianGrpcIntegrationTests: correct the stale "capture-gated, NOT
  server-gated" comment to the server-gate conclusion backed by the controls.

Sanitized throughout (counts, native return codes, buffer lengths only).

Claude-Session: https://claude.ai/code/session_012SDSQ3AcaXqPcBtDESBRii
2026-06-26 11:08:20 -04:00
dohertj2 f0a1b04b34 Merge pull request 'feat(wcf): C2 spike + ConnectViaAddress/connmode — WCF transport viable, rows server-gated' (#1) from feat/c2-wcf-event-spike into main 2026-06-26 06:48:01 -04:00
Joseph Doherty f2297315b9 docs(c2): correct the WCF finding — transport+auth viable, row-retrieval server-gated
Overturns the earlier wrong "WCF not served on 2023 R2" conclusion (that was a
test error: wrong port/transport for the reverse tunnel). Corrected: the cert
(TLS) transport + NegotiateAuthentication auth reach the 2023 R2 historian
cross-platform; the 0x501 event connection mode makes CM_EVENT RegisterTags
succeed; yet StartEventQuery returns a 0-row buffer + long-polls over a window
that has events. Registration and window ruled out -> the same server-side
per-connection row gate as gRPC. Event reads stay server-gated over BOTH
transports; not client-fixable. Evidence doc rewritten; gRPC + WCF orchestrator
gating messages corrected.

Claude-Session: https://claude.ai/code/session_012SDSQ3AcaXqPcBtDESBRii
2026-06-26 04:41:21 -04:00
Joseph Doherty de8d5e91ce feat(wcf): EventReadConnectionModeOverride + cross-platform/bounded C2 spike
Live investigation (direct from a VPN host to the 2023 R2 historian's real WCF
port) showed the certificate transport + NegotiateAuthentication auth work
cross-platform, and that the event-read chain needs the 0x501 event connection
mode for CM_EVENT RegisterTags to succeed (0x402/0x401 fail). Even with
registration succeeding over a window that has events, StartEventQuery returns a
0-row header and long-polls — the same server-side per-connection row gate proven
for gRPC. Adds: EventReadConnectionModeOverride (diagnostic), and spike knobs —
cross-platform cert gate, version-check bypass, per-call timeout, overall budget
with phase-diagnostic dump, connection-mode override.

Claude-Session: https://claude.ai/code/session_012SDSQ3AcaXqPcBtDESBRii
2026-06-26 04:38:58 -04:00
Joseph Doherty 8777c0b816 fix(wcf): set Via via CreateChannel(address, via) — ClientViaBehavior absent in .NET WCF
ClientViaBehavior is a .NET Framework type not present in the System.ServiceModel
client libraries. Use the portable ChannelFactory.CreateChannel(EndpointAddress, Uri)
overload instead, via a CreateChannel helper applied at the history-open and
retrieval-query sites (the critical event path). Fixes the build break in 954b9cc.

Claude-Session: https://claude.ai/code/session_012SDSQ3AcaXqPcBtDESBRii
2026-06-25 20:37:38 -04:00
Joseph Doherty 954b9cc9cc feat(wcf): add ConnectViaAddress (WCF Via) for tunneled historian access + wire into C2 spike
When the historian is reached through a port-forward whose local port differs
from the server's real service port, WCF's server-side AddressFilter rejects the
message (To = tunnel port != server port). ConnectViaAddress lets the channel
connect to the tunnel while addressing the SOAP To the real Host/Port endpoint.
Applied in HistorianWcfClientCredentialsHelper.Configure (the critical event
factories already call it). The C2 spike reads HISTORIAN_WCF_EVENT_VIA.

Claude-Session: https://claude.ai/code/session_012SDSQ3AcaXqPcBtDESBRii
2026-06-25 20:35:46 -04:00
Joseph Doherty 7992e43908 test(c2): make WCF spike transport-selectable (integrated|certificate) + opt-in verbose
The first live run used the wrong port (32568 direct vs the 42568 WCF tunnel) and
hardcoded RemoteTcpIntegrated; via the tunnel the error advanced from socket-RST
to ProtocolException (binding/security mismatch). Add HISTORIAN_WCF_EVENT_TRANSPORT
(certificate), _DNSID, _ALLOW_UNTRUSTED, and an opt-in _VERBOSE for live binding
diagnosis. Default output stays sanitized; still Windows-only, never fails the suite.

Claude-Session: https://claude.ai/code/session_012SDSQ3AcaXqPcBtDESBRii
2026-06-25 20:18:23 -04:00
9 changed files with 246 additions and 86 deletions
@@ -4,6 +4,16 @@ Captured the stock 2023 R2 client performing a **gRPC event read** that returns
the open item "gRPC event ROW retrieval returns zero rows" (handoff §Current Status item 1). This
closes the capture-gate: the working request shape is now known.
> **Re-control 2026-06-26 — this 50-row result did NOT reproduce.** Re-running the stock 2023 R2
> client (same `capture-event` harness, native Event connection) against the same server returned
> **0 rows** over 30d / 90d / 365d / 3yr, even though `Runtime.dbo.Events` held ~71.5k events in the
> window (newest minutes old). So the native gRPC event read is **server-gated**, not merely
> request-shape/"capture" gated — the gate also stops the stock client now. Positive control the same
> day: the same native **WCF** event-read client returns real events (5) from a local AVEVA Historian
> 2020 but 0 from this 2023 R2 server (see `wcf-event-read-spike-results.md` §"2026-06-26 — positive
> control"). The 50-row capture below remains valid as the *captured-correct request format*; treat its
> "returns rows" framing as a point-in-time observation that the server no longer honors.
## How it was captured
`tools/AVEVA.Historian.Grpc2023CaptureHarness` gained a `capture-event` scenario. It loads the
@@ -1,63 +1,105 @@
# WCF event-read spike — live result (2026-06-25): WCF transport not served on 2023 R2
# WCF event-read spike — live result (2026-06-25/26): transport+auth viable, row-retrieval server-gated
Settles the open question behind **C2** ("event reads over gRPC are gated; the only listed unblock is
*route event reads via WCF*"). The gRPC event-read path is a proven server-side dead-end
(`grpc-event-query-capture.md`: auth fully solved, every client-controllable layer byte-matched to the
stock client, yet the server scopes 0 rows to our connection). This spike tested the **WCF** leg.
stock client, yet the server scopes 0 rows to our connection). This spike resolved the **WCF** leg.
> **Correction to an earlier draft of this doc.** A first pass concluded "the 2023 R2 historian does not
> serve the legacy WCF transport (connection reset at framing)." **That was a test error, not a server
> fact.** It connected to the historian's real WCF port `32568` *directly* and used the Windows-integrated
> transport. In this environment the historian is reached through a **reverse SSH tunnel** (local
> `42568` → historian `32568`), and integrated/Kerberos auth does not work through that tunnel. The
> socket-RST was the tunnel/transport mismatch, not an absent listener. Corrected below.
## What was run
A Windows-only, env-gated diagnostic (`tests/AVEVA.Historian.Client.Tests/WcfEventReadSpikeTests.cs`,
gated by `HISTORIAN_WCF_EVENT_HOST`) drove `HistorianWcfEventOrchestrator.ReadEventsAsync` directly
over `RemoteTcpIntegrated` (WCF `net.tcp`, port 32568) against the **live 2023 R2 historian**, with a
−90d window (the engine holds tens of thousands of events in that range), run from the native Windows
capture rig over VPN. Auth supplied as explicit domain credentials (consumed by the app-level
`ValidateClientCredential` SSPI rounds).
A Windows-only-by-default, env-gated diagnostic (`tests/AVEVA.Historian.Client.Tests/WcfEventReadSpikeTests.cs`)
drives `HistorianWcfEventOrchestrator.ReadEventsAsync` directly. The decisive run was **cross-platform,
direct** (no tunnel): from the VPN-holding host straight to the historian's real WCF endpoint
`net.tcp://<historian>:32568/HistCert`, using the **certificate transport** (`RemoteTcpCertificate`,
TLS, `AllowUntrustedServerCertificate`) and `NegotiateAuthentication` (cross-platform, explicit domain
credentials). The SDK's interface-version gate was bypassed (`VerifyServerInterfaceVersion=false`) —
the 2023 R2 WCF **History interface reports version 13** (this SDK's serializers target 11/12).
## Result — RED (transport not served), sanitized
## Result — transport+auth viable; row-retrieval server-gated (sanitized)
Event spike:
Progression of the live errors as the addressing/transport was corrected:
| field | value |
| attempt | error |
|---|---|
| outcome | `THREW System.ServiceModel.CommunicationException` ("The socket connection was aborted") |
| inner | `System.Net.Sockets.SocketException` — "An existing connection was forcibly closed by the remote host" |
| events observed | 0 |
| LastUpdC3ReturnCode / LastRTag2ReturnCode / LastAddReturnCode(EnsT2) | 0 / 0 / 0 |
| LastEnsT2PayloadSha256 | empty |
| LastResultBufferLength | 0 |
| direct `:32568`, integrated | `SocketException` "forcibly closed" (wrong port + transport for the tunnel) |
| tunnel `:42568`, integrated | `ProtocolException` at the security UpgradeResponse (integrated can't negotiate through the tunnel) |
| tunnel `:42568`, certificate | reached the WCF dispatcher → `AddressFilter` mismatch (tunnel rewrites the port) |
| **direct `:32568`, certificate, cross-platform** | **past auth** → `ProtocolEvidenceMissingException`: History interface version **13** |
| + `VerifyServerInterfaceVersion=false` | **full chain runs**; query returns a 10-byte **0-row** header, then `GetNext` long-polls |
All native return codes are `0` and the EnsT2 payload sha256 is empty: the chain failed at the **first
WCF call** (`GetInterfaceVersion`), *before* any auth token round or CM_EVENT registration ran.
Connection-mode experiment (certificate transport, direct, version-bypassed, a 1-day window that holds
events), comparing the native OpenConnection mode used for the event-read chain:
Corroboration — a basic (non-event) `RemoteTcpIntegrated` `ProbeAsync` + `ReadRawAsync` (the committed
`RemoteTcpIntegrationTests`) throws the **identical** exception, with the stack landing in
`System.ServiceModel.Channels.SocketConnection.WriteAsync` — i.e. the failure is **transport-wide**, not
event-specific, and not auth-specific (it never reaches auth).
Phase 0 (reachability) had confirmed TCP 32568 is **open** (the connect succeeds). So the port accepts a
socket, but the moment the SDK writes its `net.tcp` binary-SOAP framing the server **resets the
connection** (RST at the socket-write layer).
| connMode | RegisterTags (RTag2) | EnsureTags (EnsT2) | result buffer | events |
|---|---|---|---|---|
| `0x501` (event) | **0 — success** | 1 (benign-false, as in the 2020 flow) | 10 bytes (0-row header) | **0** |
| `0x401` (write) | 1 (fail) | 1 | 10 bytes | 0 |
| `0x402` (read-only, default) | 1 (fail) | 1 | 10 bytes | 0 |
## Conclusion
The **2023 R2 historian does not serve the legacy WCF NetTcp transport.** A raw RST at the first socket
write — before any security negotiation, SOAP fault, or auth exchange — is the signature of a listener
that does not speak `net.tcp` binary SOAP, not of an auth/SPN problem or event-row scoping. (The earlier
WCF event-chain native return codes 76/85 documented in `HistorianWcfEventOrchestrator` were only ever
observed against a **2020** historian; against 2023 R2 there is no WCF endpoint to reach at all.)
1. **WCF transport + auth ARE viable on 2023 R2.** The certificate (TLS) transport negotiates and the
`NegotiateAuthentication` app-level handshake authenticates — **cross-platform** (proven from a
non-Windows VPN host). The earlier "WCF not served" conclusion was wrong. (Integrated/Windows
transport security is not usable through the reverse tunnel — `net.tcp` Kerberos does not tunnel.)
2. **The event-read chain needs the `0x501` event connection mode.** With it, CM_EVENT `RegisterTags`
**succeeds** (it fails on `0x402`/`0x401`). `EnsureTags` returns false, but that is documented as
benign in the 2020 flow that *did* return rows.
3. **Row retrieval is server-gated — same as gRPC.** Even with auth solved and `RegisterTags` succeeding,
over a window that holds events, `StartEventQuery` succeeds but `GetNextEventQueryResultBuffer` returns
a **0-row** header (10 bytes) and long-polls. Registration and window are ruled out as the cause; the
server simply does not scope event rows to a managed connection. This is the **identical** server-side
per-connection retrieval working-set gate proven for gRPC in `grpc-event-query-capture.md`.
Therefore **C2's "route event reads via WCF" unblock is moot on 2023 R2** — there is no WCF endpoint to
route to. Event reads are unavailable on the 2023 R2 historian over **both** transports:
**Therefore event reads do not return rows on the 2023 R2 historian over either transport** — gRPC
(retrieval-server-gated) and WCF (transport+auth work, but the same server-side row gate). The only
remaining theoretical unblock is server-side (AVEVA exposing event-row retrieval to a managed
connection) — not client-fixable. **C2 stays closed won't-fix**, for this (corrected) reason.
- **gRPC** — auth-solved but retrieval-server-gated (server scopes 0 rows to our connection;
`grpc-event-query-capture.md`).
- **WCF (`net.tcp`)** — transport not served on 2023 R2 (connection reset at framing).
## SDK additions from this investigation (retained, build-clean, golden where applicable)
The WCF event-read managed path would only ever apply to a legacy **2020** historian, which the gateway
does not target (the gateway runs `RemoteGrpc` against 2023 R2). The only remaining theoretical unblock
is server-side (AVEVA exposing event-row retrieval to a managed gRPC connection) — not client-fixable.
- `HistorianClientOptions.ConnectViaAddress` — WCF `Via` (connect to a tunnel/proxy while addressing the
SOAP `To` the real endpoint), so a port-forward whose local port differs from the server's real port
satisfies the server-side WCF AddressFilter.
- `HistorianClientOptions.EventReadConnectionModeOverride` — diagnostic override of the event-read
OpenConnection mode (the `0x501` finding above).
- The C2 spike is now transport-selectable (integrated|certificate), cross-platform for the cert
transport, bounded (per-call timeout + overall budget with a phase-diagnostic dump), and version-gate
bypassable. Output stays sanitized (counts, native return codes, buffer lengths, sha256).
**C2 is closed won't-fix** for the gateway's target (2023 R2). `ReadEventsAsync` over gRPC keeps its
honest no-row throw; the gating messages are corrected so they no longer point operators at the WCF
transport as a live fallback on 2023 R2.
## 2026-06-26 — positive control: same WCF client, 2020 historian vs 2023 R2
The earlier evidence triangulated the gate but lacked a clean *positive* control — proof that the
native event-query path returns rows for **some** historian, so that the 0-row 2023 R2 result can be
attributed to the server rather than to the client/protocol. This run supplies it, A/B against two
historians from the **same** WCF event-read client (`HistorianWcfEventOrchestrator`, whose wire
protocol is byte-replayed from stock 2020 captures), same 365-day window:
| target historian | transport | RegisterTags (RTag2) | result buffer | events |
|---|---|---|---|---|
| **local AVEVA Historian 2020** | WCF, integrated | **0 — success** | terminal after rows | **5** |
| **2023 R2** (the C2 server) | WCF, certificate | (gate, as documented) | 10-byte 0-row header → long-poll | **0** |
SQL ground truth (`Runtime.dbo.Events`) for the same two boxes: the 2020 historian holds ~51.6k events
over the window, the 2023 R2 holds ~71.5k — both populated. So the **identical native WCF event-read
client returns real events from a 2020 historian and zero from the 2023 R2 server**. That isolates the
zero-rows to the 2023 R2 server: not the client, not the protocol, not our serializers.
Notes / honesty caveats:
- The genuinely-**stock** 2020 client (`aahClientManaged.dll` v2020.0406.2652.2, driven by reflection)
could **not** be run end-to-end here: against the local 2020 historian (services patched to build
3383.3) it self-blocks at `StartEventQuery` with `Invalid InterfaceVersion` (242) — a client-side
build/version gate, and the stock client has no version-bypass. Our client (which *does* bypass the
version check and byte-replays the same native sequence) is the faithful proxy that reaches the rows.
- Cross-check on the gRPC leg the same day: the **stock 2023 R2** client (native Event connection, its
own correct event query) returned **0 rows** over 30d/90d/365d/3yr against the 2023 R2 server; the
2026-06-22 "50 rows" stock capture did not reproduce. Same server-gate, both transports, both clients.
- Output sanitized throughout (counts, native return codes, buffer lengths only — no event identity,
host, or credentials).
@@ -104,11 +104,12 @@ internal sealed class HistorianGrpcEventOrchestrator
{
throw new ProtocolEvidenceMissingException(
$"ReadEvents over gRPC did not return rows within {OverallBudget.TotalSeconds:0}s: StartEventQuery " +
"succeeds but GetNextEventQueryResultBuffer long-polls to the no-data terminal. Event-row retrieval " +
"over gRPC is auth-solved but server-gated — the 2023 R2 server scopes 0 rows to a managed connection " +
"(see docs/reverse-engineering/grpc-event-query-capture.md). The legacy WCF transport is NOT a fallback " +
"on 2023 R2 (live-disproven 2026-06-25: net.tcp is reset at the framing layer — see " +
"docs/reverse-engineering/wcf-event-read-spike-results.md), so there is no event-read path on a 2023 R2 historian.");
"succeeds but GetNextEventQueryResultBuffer long-polls to the no-data terminal. Event-row retrieval is " +
"auth-solved but SERVER-GATED on 2023 R2 over both transports — the server scopes 0 rows to a managed " +
"connection (gRPC: docs/reverse-engineering/grpc-event-query-capture.md). The WCF transport reaches the " +
"2023 R2 historian (certificate transport + auth work, CM_EVENT registration succeeds on the 0x501 event " +
"connection) but hits the SAME server-side row gate — 0-row buffer + long-poll (see " +
"docs/reverse-engineering/wcf-event-read-spike-results.md). Not client-fixable on either transport.");
}
foreach (HistorianEvent evt in events)
@@ -175,18 +176,20 @@ internal sealed class HistorianGrpcEventOrchestrator
// returning the WCF code-85 terminal), we cannot distinguish "genuinely no events in range"
// from "the CM_EVENT registration replay didn't fully land over gRPC" — so we refuse to return
// a possibly-false empty list and surface the gated state instead. Proven server-gated: the live
// 2023 R2 server holds tens of thousands of events yet scopes 0 to a managed gRPC connection
// (grpc-event-query-capture.md); WCF is not a 2023 R2 fallback (wcf-event-read-spike-results.md).
// 2023 R2 server holds tens of thousands of events yet scopes 0 to a managed connection
// (grpc-event-query-capture.md). WCF reaches the same historian (cert transport + auth work,
// CM_EVENT registers on the 0x501 event connection) but hits the SAME row gate — not a fallback
// (wcf-event-read-spike-results.md).
if (events.Count == 0)
{
throw new ProtocolEvidenceMissingException(
"ReadEvents over gRPC: the chain completes and StartEventQuery succeeds, but " +
"GetNextEventQueryResultBuffer returns no rows (it long-polls to the no-data terminal " +
$"after the CM_EVENT registration replay; last={LastErrorBufferDescription}). Event-row retrieval " +
"over gRPC is auth-solved but server-gated — the 2023 R2 server scopes 0 rows to a managed connection " +
"(see docs/reverse-engineering/grpc-event-query-capture.md). The legacy WCF transport is NOT a fallback " +
"on 2023 R2 (live-disproven 2026-06-25: net.tcp is reset at the framing layer — see " +
"docs/reverse-engineering/wcf-event-read-spike-results.md).");
$"after the CM_EVENT registration replay; last={LastErrorBufferDescription}). Event-row retrieval is " +
"auth-solved but SERVER-GATED on 2023 R2 over both transports — the server scopes 0 rows to a managed " +
"connection (gRPC: docs/reverse-engineering/grpc-event-query-capture.md; WCF reaches the historian and " +
"registers on the 0x501 event connection yet hits the same row gate: " +
"docs/reverse-engineering/wcf-event-read-spike-results.md). Not client-fixable on either transport.");
}
return events;
@@ -53,6 +53,25 @@ public sealed class HistorianClientOptions
/// </summary>
public string? ServerDnsIdentity { get; init; }
/// <summary>
/// Optional WCF "Via" address (e.g. <c>net.tcp://host:42568</c>). When set, the SDK's WCF
/// channel factories <b>connect</b> to this address while still addressing the SOAP message
/// <c>To</c> the logical endpoint built from <see cref="Host"/>/<see cref="Port"/>. Use this when
/// the Historian is reached through a port-forwarding tunnel or proxy whose local port differs
/// from the server's real service port: point <see cref="Host"/>/<see cref="Port"/> at the
/// server's real endpoint (so the server's WCF AddressFilter matches) and set this to the tunnel
/// endpoint. Has no effect on the gRPC transport. Default null (connect == address).
/// </summary>
public string? ConnectViaAddress { get; init; }
/// <summary>
/// Diagnostic override for the native OpenConnection mode the WCF event-read chain uses (default
/// <c>0x402</c>, read-only process). Set to e.g. <c>0x501</c> (event) or <c>0x401</c> (write-enabled)
/// to probe whether CM_EVENT registration / event-row retrieval needs a different connection type on a
/// 2023 R2 server. Null = the default read-only process mode. Intended for protocol investigation.
/// </summary>
public uint? EventReadConnectionModeOverride { get; init; }
/// <summary>
/// For <see cref="HistorianTransport.RemoteGrpc"/>: when true the channel uses TLS
/// (<c>https://</c>); when false it uses plaintext (<c>http://</c>). Matches the stock
@@ -41,7 +41,7 @@ internal static class HistorianWcfAuthChainHelper
try
{
IHistoryServiceContract2 historyChannel = historyFactory.CreateChannel();
IHistoryServiceContract2 historyChannel = HistorianWcfClientCredentialsHelper.CreateChannel(historyFactory, options);
ICommunicationObject historyChannelCo = (ICommunicationObject)historyChannel;
try
{
@@ -30,6 +30,24 @@ internal static class HistorianWcfClientCredentialsHelper
}
}
/// <summary>
/// Creates a channel from <paramref name="factory"/>, honoring
/// <see cref="HistorianClientOptions.ConnectViaAddress"/> when set: the channel <b>connects</b> to
/// the Via address while still addressing the SOAP message <c>To</c> the factory's logical endpoint.
/// This lets a port-forward whose local port differs from the server's real service port satisfy the
/// server-side WCF AddressFilter (which validates the To header). Use this in place of
/// <c>factory.CreateChannel()</c> at every WCF event/read channel-creation site.
/// </summary>
public static TChannel CreateChannel<TChannel>(ChannelFactory<TChannel> factory, HistorianClientOptions options)
{
ArgumentNullException.ThrowIfNull(factory);
ArgumentNullException.ThrowIfNull(options);
return string.IsNullOrWhiteSpace(options.ConnectViaAddress)
? factory.CreateChannel()
: factory.CreateChannel(factory.Endpoint.Address, new Uri(options.ConnectViaAddress));
}
private sealed class AcceptAnyCertificateValidator : X509CertificateValidator
{
public static readonly AcceptAnyCertificateValidator Instance = new();
@@ -8,13 +8,15 @@ using AVEVA.Historian.Client.Wcf.Contracts;
namespace AVEVA.Historian.Client.Wcf;
/// <remarks>
/// Mirrors HistorianWcfReadOrchestrator but targets IRetrievalServiceContract4 for the event flow.
/// Applies to <b>legacy 2020-era WCF (net.tcp) historians only</b>. The event row-buffer layout is now
/// decoded (<see cref="HistorianEventRowProtocol"/>; verified against real captured rows). Note: a
/// <b>2023 R2</b> historian does NOT serve this WCF transport at all — net.tcp is reset at the framing
/// layer before any auth (live-disproven 2026-06-25; see
/// <c>docs/reverse-engineering/wcf-event-read-spike-results.md</c>), so this orchestrator is not a
/// fallback for 2023 R2 deployments. The native return codes 76/85 noted below were 2020-historian
/// Mirrors HistorianWcfReadOrchestrator but targets IRetrievalServiceContract4 for the event flow. The
/// event row-buffer layout is decoded (<see cref="HistorianEventRowProtocol"/>; verified against real
/// captured rows). A <b>2023 R2</b> historian <i>does</i> serve this transport via the <b>certificate</b>
/// (TLS) endpoint (the cert transport + <c>NegotiateAuthentication</c> auth work cross-platform; the
/// integrated/Windows transport does not tunnel). With the <c>0x501</c> event connection mode CM_EVENT
/// registration succeeds — but <c>StartEventQuery</c> still returns a 0-row buffer and long-polls: event
/// rows are <b>server-gated</b> per connection on 2023 R2, the same wall as the gRPC path, and not
/// client-fixable (see <c>docs/reverse-engineering/wcf-event-read-spike-results.md</c> and
/// <c>grpc-event-query-capture.md</c>). The native return codes 76/85 noted below were 2020-historian
/// observations.
/// </remarks>
internal sealed class HistorianWcfEventOrchestrator
@@ -147,7 +149,7 @@ internal sealed class HistorianWcfEventOrchestrator
EndpointAddress transactionEndpoint = HistorianWcfBindingFactory.CreateAuxiliaryEndpointAddress(_options, HistorianWcfServiceNames.Transaction);
uint clientHandle = HistorianWcfAuthChainHelper.OpenAuthenticatedConnection(
_options, histBinding, histEndpoint, contextKey, cancellationToken,
connectionMode: HistorianWcfAuthChainHelper.NativeIntegratedReadOnlyConnectionMode,
connectionMode: _options.EventReadConnectionModeOverride ?? HistorianWcfAuthChainHelper.NativeIntegratedReadOnlyConnectionMode,
additionalSetup: (historyChannel, context) =>
AddCmEventTagViaAddT(historyChannel, context, auxBinding, statusEndpoint, transactionEndpoint, retrBinding, retrEndpoint));
return RunEventQuery(retrBinding, retrEndpoint, clientHandle, startUtc, endUtc, filter, cancellationToken);
@@ -167,7 +169,7 @@ internal sealed class HistorianWcfEventOrchestrator
try
{
IRetrievalServiceContract4 channel = factory.CreateChannel();
IRetrievalServiceContract4 channel = HistorianWcfClientCredentialsHelper.CreateChannel(factory, _options);
ICommunicationObject channelCo = (ICommunicationObject)channel;
try
{
@@ -517,19 +517,26 @@ public sealed class HistorianGrpcIntegrationTests
}
// Plan #2: ReadEvents over gRPC. The chain runs end-to-end and StartEventQuery succeeds
// (no InvalidOperationException), but — confirmed live 2026-06-22 — GetNextEventQueryResultBuffer
// LONG-POLLS and returns zero rows: the gRPC server blocks to the deadline instead of
// returning the synchronous 5-byte code-85 terminal the 2020 WCF op returns, so the orchestrator
// reaches its no-data terminal with zero rows and (rather than assert a possibly-false "no events"
// empty) throws ProtocolEvidenceMissingException.
// (no InvalidOperationException), but GetNextEventQueryResultBuffer LONG-POLLS and returns zero
// rows: the gRPC server blocks to the deadline instead of returning the synchronous 5-byte
// code-85 terminal the 2020 WCF op returns, so the orchestrator reaches its no-data terminal
// with zero rows and (rather than assert a possibly-false "no events" empty) throws
// ProtocolEvidenceMissingException.
//
// IMPORTANT (2026-06-22): the zero rows are NOT "no events on the server". Verified against the
// live 2023 R2 box, which holds 19,356 events in the last 30 days (SQL ground truth via the INSQL
// linked server, Runtime.dbo.Events). The EMPTY-FILTER gRPC event query simply does not match
// them. So the gate here is the empty-filter request shape (filter / namespace / event-tag
// registration), NOT data availability — this is capture-gated (needs a native gRPC event-query
// capture), not server-gated. Flip to asserting parsed rows once that capture lands and the
// request is corrected. The chain stays BOUNDED (no multi-minute hang) via the short
// SERVER-GATED on 2023 R2 (settled 2026-06-26; supersedes a 2026-06-22 draft that read the zero
// rows as a client-side empty-filter/request-shape "capture gate"). Three live controls against
// the same event-bearing 2023 R2 server retire the request-shape theory:
// 1. The STOCK AVEVA 2023 R2 client (aahClientManaged, native Event connection, its OWN correct
// event query — not our empty-filter shape) returns 0 rows over 30d / 90d / 365d / 3yr.
// The 2026-06-22 "50 rows" stock capture did NOT reproduce.
// 2. SQL ground truth (Runtime.dbo.Events) shows the server holds tens of thousands of events
// in those windows (newest minutes old) — so it is emphatically NOT "no events".
// 3. POSITIVE CONTROL: the same native WCF event-read client (protocol byte-replayed from stock
// 2020 captures) returns real events (5) from a local AVEVA Historian 2020 over the identical
// sequence, but 0 (10-byte 0-row header + long-poll) from this 2023 R2 server.
// The protocol works; the 2023 R2 server simply does not scope event rows to a managed connection.
// Not client-fixable (see docs/reverse-engineering/grpc-event-query-capture.md +
// wcf-event-read-spike-results.md). The chain stays BOUNDED (no multi-minute hang) via the short
// registration + poll deadlines. (Set a small HISTORIAN_GRPC_TIMEOUT to keep this snappy.)
HistorianClient client = new(BuildOptions(host));
@@ -36,9 +36,14 @@ public sealed class WcfEventReadSpikeTests
public async Task WcfEventRead_DiagnosticDump_AgainstRemoteHistorian()
{
string? host = Environment.GetEnvironmentVariable("HISTORIAN_WCF_EVENT_HOST");
if (string.IsNullOrWhiteSpace(host) || !OperatingSystem.IsWindows())
bool certificate = string.Equals(
Environment.GetEnvironmentVariable("HISTORIAN_WCF_EVENT_TRANSPORT"), "certificate", StringComparison.OrdinalIgnoreCase);
// The certificate (TLS) transport and NegotiateAuthentication-based app auth are cross-platform;
// the integrated (Windows SSPI transport-security) transport is Windows-only. Skip when the gate
// is unconfigured, or when an integrated run is requested off Windows.
if (string.IsNullOrWhiteSpace(host) || (!certificate && !OperatingSystem.IsWindows()))
{
return; // inert without the dedicated gate / off Windows
return;
}
int port = int.TryParse(Environment.GetEnvironmentVariable("HISTORIAN_WCF_EVENT_PORT"), out int p)
@@ -48,41 +53,95 @@ public sealed class WcfEventReadSpikeTests
string? user = Environment.GetEnvironmentVariable("HISTORIAN_WCF_EVENT_USER");
string? password = Environment.GetEnvironmentVariable("HISTORIAN_WCF_EVENT_PASSWORD");
string? spn = Environment.GetEnvironmentVariable("HISTORIAN_WCF_EVENT_SPN");
bool integrated = string.IsNullOrEmpty(user);
string? dnsId = Environment.GetEnvironmentVariable("HISTORIAN_WCF_EVENT_DNSID");
string? via = Environment.GetEnvironmentVariable("HISTORIAN_WCF_EVENT_VIA");
bool bypassVersion = string.Equals(
Environment.GetEnvironmentVariable("HISTORIAN_WCF_EVENT_NOVERSIONCHECK"), "1", StringComparison.Ordinal);
int timeoutSec = int.TryParse(Environment.GetEnvironmentVariable("HISTORIAN_WCF_EVENT_TIMEOUT_SEC"), out int ts) && ts > 0
? ts : 30;
string? connModeRaw = Environment.GetEnvironmentVariable("HISTORIAN_WCF_EVENT_CONNMODE");
uint? connModeOverride = null;
if (!string.IsNullOrWhiteSpace(connModeRaw))
{
string hex = connModeRaw.StartsWith("0x", StringComparison.OrdinalIgnoreCase) ? connModeRaw[2..] : connModeRaw;
if (uint.TryParse(hex, System.Globalization.NumberStyles.HexNumber, null, out uint cm)) { connModeOverride = cm; }
}
bool allowUntrusted = string.Equals(
Environment.GetEnvironmentVariable("HISTORIAN_WCF_EVENT_ALLOW_UNTRUSTED"), "1", StringComparison.Ordinal);
// Certificate transport carries no Windows transport credential (cert-validated TLS channel);
// integrated derives one from the logged-in identity unless an explicit DOMAIN\user is supplied.
// App-level ValidateClientCredential still uses UserName/Password (or the process identity).
bool integrated = !certificate && string.IsNullOrEmpty(user);
HistorianClientOptions options = new()
{
Host = host,
Port = port,
Transport = HistorianTransport.RemoteTcpIntegrated,
Transport = certificate ? HistorianTransport.RemoteTcpCertificate : HistorianTransport.RemoteTcpIntegrated,
IntegratedSecurity = integrated,
UserName = user ?? string.Empty,
Password = password ?? string.Empty,
TargetSpn = string.IsNullOrWhiteSpace(spn) ? "NT SERVICE\\aahClientAccessPoint" : spn,
ServerDnsIdentity = string.IsNullOrWhiteSpace(dnsId) ? null : dnsId,
AllowUntrustedServerCertificate = allowUntrusted,
ConnectViaAddress = string.IsNullOrWhiteSpace(via) ? null : via,
VerifyServerInterfaceVersion = !bypassVersion,
ConnectTimeout = TimeSpan.FromSeconds(timeoutSec),
RequestTimeout = TimeSpan.FromSeconds(timeoutSec),
EventReadConnectionModeOverride = connModeOverride,
};
HistorianWcfEventOrchestrator orchestrator = new(options);
DateTime endUtc = DateTime.UtcNow;
DateTime startUtc = endUtc - TimeSpan.FromDays(days);
int budgetSec = int.TryParse(Environment.GetEnvironmentVariable("HISTORIAN_WCF_EVENT_BUDGET_SEC"), out int bgs) && bgs > 0
? bgs : 60;
int observed = 0;
bool hasFirstEvent = false;
string outcome = "completed";
try
{
await foreach (HistorianEvent evt in orchestrator.ReadEventsAsync(startUtc, endUtc, filter: null, CancellationToken.None))
// Race the read against an overall budget. The chain is synchronous WCF, so a stuck call
// can't be token-cancelled — but the orchestrator's static/instance diagnostics are set as it
// progresses, so on a budget timeout we still dump them to see which phase (auth / CM_EVENT
// registration / query) is hanging. The abandoned task keeps running in the background harness.
using var budget = new CancellationTokenSource(TimeSpan.FromSeconds(budgetSec));
Task<int> readTask = Task.Run(async () =>
{
observed++;
hasFirstEvent = true;
_ = evt; // event identity intentionally NOT logged (sanitized)
int n = 0;
await foreach (HistorianEvent evt in orchestrator.ReadEventsAsync(startUtc, endUtc, filter: null, budget.Token))
{
n++;
_ = evt; // event identity intentionally NOT logged (sanitized)
}
return n;
});
Task finished = await Task.WhenAny(readTask, Task.Delay(TimeSpan.FromSeconds(budgetSec + 5)));
if (finished == readTask)
{
observed = await readTask;
hasFirstEvent = observed > 0;
}
else
{
outcome = $"TIMED OUT after {budgetSec}s (chain still running — see return codes for phase reached)";
}
}
catch (Exception ex)
{
outcome = $"threw {ex.GetType().Name}"; // message omitted — may carry host/credential text
// Default: type name only (sanitized). Opt into messages for live binding/protocol debugging
// via HISTORIAN_WCF_EVENT_VERBOSE=1 — binding/protocol errors may carry the endpoint host
// (already known to the operator) but never credentials; still off by default.
outcome = string.Equals(Environment.GetEnvironmentVariable("HISTORIAN_WCF_EVENT_VERBOSE"), "1", StringComparison.Ordinal)
? $"threw {ex.GetType().FullName} :: {ex.Message} | inner={ex.InnerException?.GetType().FullName} :: {ex.InnerException?.Message}"
: $"threw {ex.GetType().Name}";
}
// Sanitized diagnostic dump — counts, native return codes, buffer lengths, sha256 ONLY.
_output.WriteLine($"[C2 WCF spike] transport: {options.Transport} (integratedSec={integrated}, allowUntrusted={allowUntrusted}, connMode={(connModeOverride.HasValue ? "0x" + connModeOverride.Value.ToString("X") : "default-0x402")})");
_output.WriteLine($"[C2 WCF spike] outcome: {outcome}");
_output.WriteLine($"[C2 WCF spike] events observed: {observed}");
_output.WriteLine($"[C2 WCF spike] hasFirstEvent: {hasFirstEvent}");