From cac81c7e5c80c8101c43f403191a100409ab0968 Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Tue, 23 Jun 2026 14:40:57 -0400 Subject: [PATCH 1/4] handoff: refresh date + offline test count after event-parser merge Bring the doc header current with the merged event-row parser fix: bump "Last updated" to 2026-06-23 and correct the Build And Test known-good result from 321/321 to the actual 328/328 (build clean 0/0), noting the +7 are the HistorianEventRowProtocolTests golden + gated-capture coverage. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01B6mcaT2PjRFKcogzp9UkfC --- docs/reverse-engineering/handoff.md | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/docs/reverse-engineering/handoff.md b/docs/reverse-engineering/handoff.md index aa6bf87..5010545 100644 --- a/docs/reverse-engineering/handoff.md +++ b/docs/reverse-engineering/handoff.md @@ -1,6 +1,6 @@ # AVEVA Historian Managed Driver Handoff -Last updated: 2026-06-22 (roadmap exhausted — no actionable pure-code tasks remain) +Last updated: 2026-06-23 (event-row parser fix merged; roadmap still exhausted — no actionable pure-code tasks remain) > **Current status supersedes the historical blocker narrative below.** The > sections from "Active Blocker" onward are a preserved reverse-engineering @@ -172,12 +172,14 @@ dotnet build .\Histsdk.slnx --no-restore dotnet test .\Histsdk.slnx --no-build --logger "console;verbosity=minimal" ``` -Current known-good result (2026-06-22): +Current known-good result (2026-06-23): - Build succeeds (0 warnings / 0 errors). -- Offline tests pass: 321/321 (live gRPC/integration tests skip cleanly without +- Offline tests pass: 328/328 (live gRPC/integration tests skip cleanly without their env vars). Gated live tests add to this when `HISTORIAN_*` / - `HISTORIAN_GRPC_*` are set. + `HISTORIAN_GRPC_*` are set. The +7 over the prior 321 are the event-row parser + fix's golden + gated-capture coverage (`HistorianEventRowProtocolTests`: + markerless multi-row, the v11 gRPC header, and the 50-event stock-client capture). The workspace is a Git working tree (origin: gitea.dohertylan.com). Use normal git workflow for change tracking; the prior "no working tree, use From ae536bb4b84bb710ff9c8e513ed2d945ffd16925 Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Tue, 23 Jun 2026 15:13:11 -0400 Subject: [PATCH 2/4] Record 2023 R2 binary-dive verdicts; fix revision-probe scope comment MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Mined the full decompiled stock 2023 R2 managed client as the oracle for every still-pending gRPC item. Governing fact: ArchestrA.HistorianAccess is a C++/CLI shim into native HistorianClient; the managed Grpc*Client wrappers have zero call sites, so buffer-building/dispatch for the pending items is native (absent from the binaries). Sharpened verdicts into handoff.md: - Items 4/5/6 + OpenStorageConnection: hard-confirmed walled, with real reasons (SQL gated out client-side via IsManagedHistorian; no Revision RPC in the gRPC contract; LoadBlocks response is a native blob behind the storage console handle). - Items 3 (SendEvent) and 7 (DeleteTEP): moved from vague to precise, LOCAL-box capturable targets (PackToVtq btValues / DeleteTagExtendedPropertiesByName BtInput with deleteFromServer=true). Also correct the HistorianGrpcRevisionProbe doc comment: it probes the non-streamed ORIGINAL-insert path (AddNonStreamValues), a distinct capability from revision EDITS (native AddRevisionValues trio, no gRPC RPC) — the prior comment conflated them. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01B6mcaT2PjRFKcogzp9UkfC --- docs/reverse-engineering/handoff.md | 68 +++++++++++++++++++ .../Grpc/HistorianGrpcRevisionProbe.cs | 12 ++++ 2 files changed, 80 insertions(+) diff --git a/docs/reverse-engineering/handoff.md b/docs/reverse-engineering/handoff.md index 5010545..d619bcd 100644 --- a/docs/reverse-engineering/handoff.md +++ b/docs/reverse-engineering/handoff.md @@ -101,6 +101,74 @@ Live-server gRPC probe recipe: set quotes — `reference_wonder_sql_vd03_credentials`) and run the gated `HistorianGrpcIntegrationTests`. +### 2023 R2 stock-client binary dive (2026-06-23) — sharpened verdicts + +Re-read the full decompiled stock 2023 R2 managed client +(`histsdk-2023r2-analysis/decompiled/`: `Archestra.Historian.GrpcClient`, +`ArchestrA.HistorianAccess`, `Archestra.Grpc.Contract`, `HistorianEvent`, +`HistorianAccessUtil`) as the oracle for every still-pending item. **Governing +fact:** `ArchestrA.HistorianAccess.dll` is a C++/CLI mixed assembly — every +data/config/write method is a thin shim into native `.HistorianClient.*`, +and the managed `Grpc*Client` wrappers are instantiated by **nothing** in the +decompiled set (`new Grpc*Client(` → zero call sites). So the buffer-building and +RPC-dispatch sequencing for these items lives in native C++ not present in the +binaries. That confirms the "gated" calls were not from missing managed steps — +with these refinements: + +- **Item 1 (gRPC event rows)** — **confirmed native/server-side.** Stock event + call graph is provably identical to ours (transport, per-service channels, + gzip-only metadata, CM_EVENT registration, v8 ECDH Event-open, `StartEventQuery` + request bytes). `EventQuery.StartQuery`/`MoveNext` dispatch straight into native + `HistorianClient.StartEventQuery`/`GetNextRow`; the query orchestration that + would differ is native and not on the wire. One untested low-effort check + remains: byte-diff a captured **Event-connection** EnsureTags/RegisterTags + against our replay (the 83-vs-86-byte EnsT gap was never actually compared). +- **Item 3 (SendEvent over gRPC)** — **sharpened from "maybe no RPC" to a precise + capture.** RPC **confirmed** = `HistoryService.AddStreamValues` (the "no distinct + RPC" note is TRUE; an event rides the same RPC as a streamed sample, discriminated + inside `btValues`). Public API `HistorianAccess.AddStreamedValue(HistorianEvent)` + → native `AddHistorianValue`; prereqs known (write-enabled Event conn, CM_EVENT + tag handle, quality 192); field set/order recovered from `HistorianEvent.PackToVtq`. + **Only the `btValues` VTQ byte layout is missing** — built by native + `CCommonArchestraEventValue::PackToVtq` and copied out as an opaque `CDataChunk`. + Our read parser already decodes the inverse property-bag format. **Capturable + against the local Historian** (instrument `PackToVtq` output / the `AddStreamValues` + body) → then build `HistorianEventWriteProtocol` and reuse the + `HistorianGrpcHistoricalWriteOrchestrator` plumbing. +- **Item 4 (ExecuteSql over gRPC)** — **confirmed walled + explained.** The stock + client gates SQL **out client-side**: `HistorianAccess.ExecuteSqlCommand` returns + `OperationNotSupported` when `IsManagedHistorian(node)` or `!IsProcessConnectionRequested()` + (decompile ~:6198/:6214) and never sends the RPC. SQL-over-gRPC is unsupported by + design on a managed/gRPC historian; our `ProtocolEvidenceMissingException` is correct. +- **Item 5 (R4.2 revision edits)** — **confirmed HARD.** There is **no Revision RPC + in the gRPC contract** (zero "Revision" message types); the stock client reaches a + revision edit only via the native `HistorianClient.AddRevisionValuesBegin/AddRevisionValue/ + AddRevisionValuesEnd` transaction trio over the storage-engine channel. NOTE: this is + a **distinct capability** from `AddNonStreamValues` (non-streamed original insert) — + `HistorianGrpcRevisionProbe` probes the latter; its doc comment was corrected to say so. +- **Item 6 (ReadBlocks/LoadBlocks)** — `LoadBlocks` request is a trivial + handle+sequence cursor but the `historyBlocks` response is a native blob with no + managed decoder, and it needs the D2-blocked `OpenStorageConnection` console handle. + Walled. +- **Item 7 (DeleteTagExtendedProperties)** — **reframed to a capturable lead.** RPC + + string handle are **correct** in our SDK; ADD and DELETE are structurally identical + and **neither** routes through `StartJob`. The differentiator is the `deleteFromServer` + flag carried inside the native-built `BtInput` plus the native HCAL **cache-sync + background worker** that actually propagates the delete server-side (config writes hit + the in-process cache first, then sync). **Capturable**: capture native + `DeleteTagExtendedPropertiesByName(deleteFromServer=true)`'s `BtInput` to learn whether + one well-formed RPC durably deletes (→ shippable) or whether it genuinely depends on + the cache-sync worker (→ walled). +- **SF/snapshot/shard/ForwardSnapshot ops** — only `Get/SetSFParameter` are managed-built + (typed strings); all others carry opaque native buffers and need the storage console + handle. Walled / tooling-internal. + +**Net:** 3 items hard-confirmed walled with real explanations (4, 5, 6 + OpenStorageConnection), +and **2 moved to a precise, local-box-capturable target**: **SendEvent** (`PackToVtq` output) +and **DeleteTEP** (`BtInput` with `deleteFromServer=true`). Both need native instrumentation of +`aahClientManaged.dll` (Frida / IL-rewrite — repo tooling exists under +`tools/AVEVA.Historian.NativeTraceHarness` + `scripts/frida/`), not a special server. + ## Project Direction The project goal is still a fully managed .NET 10 C# AVEVA Historian client. diff --git a/src/AVEVA.Historian.Client/Grpc/HistorianGrpcRevisionProbe.cs b/src/AVEVA.Historian.Client/Grpc/HistorianGrpcRevisionProbe.cs index e82e287..8269607 100644 --- a/src/AVEVA.Historian.Client/Grpc/HistorianGrpcRevisionProbe.cs +++ b/src/AVEVA.Historian.Client/Grpc/HistorianGrpcRevisionProbe.cs @@ -20,6 +20,18 @@ namespace AVEVA.Historian.Client.Grpc; /// write-enabled session and calls AddNonStreamValuesBegin, surfacing whatever the server /// returns. It writes NO data — if Begin succeeds it immediately calls AddNonStreamValuesEnd /// with bCommit=false to discard the transaction. +/// +/// Scope note (corrected 2026-06-23 after a 2023 R2 binary re-read). Despite the type +/// name, this probes the non-streamed ORIGINAL / backfill insert capability +/// (AddNonStreamValues), which is a distinct capability from a revision EDIT +/// (overwriting an existing historized value with a new revision). The stock high-level client +/// reaches a revision edit via a separate native transaction trio +/// HistorianClient.AddRevisionValuesBegin/AddRevisionValue/AddRevisionValuesEnd +/// (ArchestrA.HistorianAccess.AddRevisionValues, REVISION_MODE ∈ InsertLatest/UpdateSingle/ +/// UpdateMultiple). That trio has NO corresponding RPC in the gRPC contract (no "Revision" +/// message type exists in Archestra.Grpc.Contract) — it rides the native storage-engine +/// transaction channel only. So R4.2 revision edits remain unreachable over gRPC regardless of this +/// probe's outcome; this probe neither covers nor unblocks them. /// internal sealed class HistorianGrpcRevisionProbe { From afc7c4bf9699038ea936fdba952a69a1792688f7 Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Tue, 23 Jun 2026 15:37:22 -0400 Subject: [PATCH 3/4] SendEvent over gRPC: implement + live-validate (was capture-gated) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Captured the native 2023 R2 client's gRPC event send (new capture-send-event harness scenario): it rides HistoryService.AddStreamValues with the SAME "OS" (0x534F) storage-sample buffer the WCF path already uses (HistorianEventWrite- Protocol) — confirming "no distinct RPC", and that it is NOT the historical write's "ON" buffer. Diffed the write-enabled vs read-only v8 Event open: byte- identical apart from per-session crypto, so the existing OpenSession event path is reused unchanged. So SendEvent-over-gRPC was pure assembly of proven parts: - HistorianGrpcEventWriteOrchestrator = v8 Event open + CM_EVENT registration (UpdC3/RegisterTags/EnsureTags) + AddStreamValues(OS buffer). - HistorianClient.SendEventAsync now routes to it for RemoteGrpc (WCF otherwise). Live-validated end-to-end against the 2023 R2 server: pure-managed SDK send → AddStreamValues BSuccess=true → the event reads back from the server (markers confirmed in returned event rows). The native gRPC RegisterTags(24B) + EnsureTags(86B) byte-match our serializers (new GrpcEventSendProtocolTests golden, closing the 83-vs-86 EnsureTags question). Gated live test SendEventAsync_OverGrpc_AcceptsEvent (opt-in HISTORIAN_GRPC_EVENT_SEND=1). 331 offline tests pass. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01B6mcaT2PjRFKcogzp9UkfC --- docs/reverse-engineering/handoff.md | 35 +++-- .../HistorianGrpcEventWriteOrchestrator.cs | 136 ++++++++++++++++++ src/AVEVA.Historian.Client/HistorianClient.cs | 20 ++- .../GrpcEventSendProtocolTests.cs | 53 +++++++ .../HistorianGrpcIntegrationTests.cs | 31 ++++ .../Program.cs | 130 ++++++++++++++++- 6 files changed, 384 insertions(+), 21 deletions(-) create mode 100644 src/AVEVA.Historian.Client/Grpc/HistorianGrpcEventWriteOrchestrator.cs create mode 100644 tests/AVEVA.Historian.Client.Tests/GrpcEventSendProtocolTests.cs diff --git a/docs/reverse-engineering/handoff.md b/docs/reverse-engineering/handoff.md index d619bcd..812a14f 100644 --- a/docs/reverse-engineering/handoff.md +++ b/docs/reverse-engineering/handoff.md @@ -76,7 +76,17 @@ reuses the proven 2020 WCF byte serializers/parsers unchanged inside protobuf `capture-event` harness (native, returns rows). 2. **R4.3 active-SF magnitude** — needs an **SF-active server** (D2 storage-engine console handle). -3. **SendEvent over gRPC** — **capture-gated**: no distinct RPC, framing uncaptured. +3. **SendEvent over gRPC** — ✅ **SHIPPED + LIVE-VALIDATED 2026-06-23.** `SendEventAsync` + now routes over `RemoteGrpc` (`HistorianGrpcEventWriteOrchestrator`). Captured the native + client live (`capture-send-event` harness scenario): the send rides + `HistoryService.AddStreamValues` with the **same "OS" (0x534F) buffer the WCF path uses** + (`HistorianEventWriteProtocol` — "no distinct RPC" confirmed true), on a v8 Event session + + CM_EVENT registration. The write-enabled Event open is **byte-identical** to the read-only one + (diffed live — only per-session crypto differs), so the existing event-open path is reused + unchanged. End-to-end: pure-managed SDK send → `BSuccess=true` → event read back from the live + server (markers `SdkSendProbe`/`SdkCaptureProbe` confirmed in returned rows). Golden-tested + (`GrpcEventSendProtocolTests`) + gated live test (`SendEventAsync_OverGrpc_AcceptsEvent`, + opt-in `HISTORIAN_GRPC_EVENT_SEND=1`). 4. **ExecuteSqlCommand over gRPC** — **server-walled** (`CSrvDbConnection`; RegisterTags prime doesn't help). Use WCF for SQL. 5. **R4.2 revision EDITS** — storage-engine-pipe-only on BOTH transports (the D2 wall). @@ -123,18 +133,17 @@ with these refinements: would differ is native and not on the wire. One untested low-effort check remains: byte-diff a captured **Event-connection** EnsureTags/RegisterTags against our replay (the 83-vs-86-byte EnsT gap was never actually compared). -- **Item 3 (SendEvent over gRPC)** — **sharpened from "maybe no RPC" to a precise - capture.** RPC **confirmed** = `HistoryService.AddStreamValues` (the "no distinct - RPC" note is TRUE; an event rides the same RPC as a streamed sample, discriminated - inside `btValues`). Public API `HistorianAccess.AddStreamedValue(HistorianEvent)` - → native `AddHistorianValue`; prereqs known (write-enabled Event conn, CM_EVENT - tag handle, quality 192); field set/order recovered from `HistorianEvent.PackToVtq`. - **Only the `btValues` VTQ byte layout is missing** — built by native - `CCommonArchestraEventValue::PackToVtq` and copied out as an opaque `CDataChunk`. - Our read parser already decodes the inverse property-bag format. **Capturable - against the local Historian** (instrument `PackToVtq` output / the `AddStreamValues` - body) → then build `HistorianEventWriteProtocol` and reuse the - `HistorianGrpcHistoricalWriteOrchestrator` plumbing. +- **Item 3 (SendEvent over gRPC)** — ✅ **SHIPPED + LIVE-VALIDATED 2026-06-23** (was + "capturable"). RPC confirmed = `HistoryService.AddStreamValues` (the "no distinct RPC" + note is TRUE). The `btValues` VTQ buffer turned out to be already-owned: our M2 + `HistorianEventWriteProtocol.SerializeAddStreamValuesBuffer` ("OS" buffer, decoded from + the WCF event-send) is the transport-independent `PackToVtq` equivalent and the gRPC send + uses it **verbatim** (live capture: sig `OS`/0x534F, CM_EVENT GUID, identical framing — NOT + the historical write's "ON" buffer). The write-enabled Event open is byte-identical to the + read-only one (live diff). So SendEvent-over-gRPC was pure assembly: + `HistorianGrpcEventWriteOrchestrator` = existing v8 Event open + existing CM_EVENT + registration + `AddStreamValues`(OS buffer). End-to-end live-validated (send → `BSuccess` + → read back from the live server). Golden-tested + gated live test. - **Item 4 (ExecuteSql over gRPC)** — **confirmed walled + explained.** The stock client gates SQL **out client-side**: `HistorianAccess.ExecuteSqlCommand` returns `OperationNotSupported` when `IsManagedHistorian(node)` or `!IsProcessConnectionRequested()` diff --git a/src/AVEVA.Historian.Client/Grpc/HistorianGrpcEventWriteOrchestrator.cs b/src/AVEVA.Historian.Client/Grpc/HistorianGrpcEventWriteOrchestrator.cs new file mode 100644 index 0000000..6971e60 --- /dev/null +++ b/src/AVEVA.Historian.Client/Grpc/HistorianGrpcEventWriteOrchestrator.cs @@ -0,0 +1,136 @@ +using Google.Protobuf; +using AVEVA.Historian.Client.Models; +using AVEVA.Historian.Client.Wcf; +using GrpcHistory = ArchestrA.Grpc.Contract.History; +using GrpcStatus = ArchestrA.Grpc.Contract.Status; + +namespace AVEVA.Historian.Client.Grpc; + +/// +/// 2023 R2 gRPC orchestrator for the event SEND (). +/// Captured live from the native 2023 R2 client (capture-send-event scenario, +/// 2026-06-23): an event send rides HistoryService.AddStreamValues with the SAME +/// "OS" (0x534F) storage-sample buffer the WCF AddS2 path uses +/// () — NOT a distinct event RPC and NOT the historical +/// write's "ON" buffer. The native client's write-enabled Event OpenConnection +/// request is byte-identical to the read-only event open (the ReadOnly arg does not change the v8 +/// open buffer; diffed live — only the per-session client key + credential token differ), so the +/// existing event path is reused unchanged. The +/// chain on a single Event session: +/// +/// OpenConnection (v8 Event, ExchangeKey ECDH auth) → string storage handle +/// CM_EVENT registration: UpdateClientStatus → RegisterTags → EnsureTags (the same +/// buffers the gRPC event READ replays — verified byte-identical to the capture) +/// HistoryService.AddStreamValues(strHandle, "OS" event buffer) +/// +/// Only original events ( = 0) with string-valued +/// properties have a captured encoding; others throw +/// from . +/// +internal sealed class HistorianGrpcEventWriteOrchestrator +{ + private readonly HistorianClientOptions _options; + + public HistorianGrpcEventWriteOrchestrator(HistorianClientOptions options) + { + _options = options ?? throw new ArgumentNullException(nameof(options)); + } + + /// Diagnostic: type+code description of the most recent AddStreamValues error buffer. + public string LastSendErrorDescription { get; private set; } = string.Empty; + + /// Diagnostic: outcomes of the CM_EVENT registration RPCs. + public string RegistrationDiag { get; private set; } = string.Empty; + + public Task SendEventAsync(HistorianEvent evt, CancellationToken cancellationToken) + { + ArgumentNullException.ThrowIfNull(evt); + if (!_options.IntegratedSecurity && string.IsNullOrEmpty(_options.UserName)) + { + throw new ProtocolEvidenceMissingException( + "Managed gRPC event send currently requires IntegratedSecurity or an explicit UserName + Password."); + } + + if (evt.RevisionVersion != 0) + { + throw new ProtocolEvidenceMissingException( + "Only original events (RevisionVersion = 0) have a captured send encoding; " + + "revision/update/delete event sends are not yet supported."); + } + + return Task.Run(() => Run(evt, cancellationToken), cancellationToken); + } + + private bool Run(HistorianEvent evt, CancellationToken cancellationToken) + { + using HistorianGrpcConnection connection = HistorianGrpcChannelFactory.Create(_options); + + // The event SEND uses the same v8 Event connection as the event READ. The write-enabled + // open buffer is byte-identical to the read-only one (verified live), so OpenSession's + // event path is reused unchanged. + HistorianGrpcHandshake.Session session = HistorianGrpcHandshake.OpenSession( + connection, _options, cancellationToken, eventConnection: true); + + RegisterCmEventTag(connection, session, cancellationToken); + + var historyClient = new GrpcHistory.HistoryService.HistoryServiceClient(connection.Channel); + byte[] pBuf = HistorianEventWriteProtocol.SerializeAddStreamValuesBuffer(evt, DateTime.UtcNow); + + GrpcHistory.AddStreamValuesResponse response = historyClient.AddStreamValues( + new GrpcHistory.AddStreamValuesRequest + { + StrHandle = session.StringHandle, + BtValues = ByteString.CopyFrom(pBuf), + }, + connection.Metadata, + DateTime.UtcNow.Add(_options.RequestTimeout), + cancellationToken); + + byte[] error = response.Status?.BtError?.ToByteArray() ?? []; + LastSendErrorDescription = HistorianEventRegistrationProtocol.DescribeNativeError(error); + return response.Status?.BSuccess ?? false; + } + + /// + /// Replays the CM_EVENT registration the native event connection performs before a send: + /// UpdateClientStatus → RegisterTags(CM_EVENT) → EnsureTags(CM_EVENT). The buffers are shared + /// with the gRPC event READ path ( + + /// ) and were verified + /// byte-identical to the live capture. Best-effort: an individual rejection does not abort the + /// send (the server may already hold CM_EVENT registered for the session). + /// + private void RegisterCmEventTag(HistorianGrpcConnection connection, HistorianGrpcHandshake.Session session, CancellationToken cancellationToken) + { + var historyClient = new GrpcHistory.HistoryService.HistoryServiceClient(connection.Channel); + DateTime Deadline() => DateTime.UtcNow.Add(_options.RequestTimeout); + + byte[] clientStatus = HistorianEventRegistrationProtocol.BuildUpdateClientStatusBlob(); + try + { + historyClient.UpdateClientStatus( + new GrpcHistory.UpdateClientStatusRequest { StrHandle = session.StringHandle, BtClientStatus = ByteString.CopyFrom(clientStatus) }, + connection.Metadata, Deadline(), cancellationToken); + } + catch { /* best-effort */ } + + byte[] registerBuffer = HistorianEventRegistrationProtocol.BuildRegisterCmEventInputBuffer(); + try + { + GrpcHistory.RegisterTagsResponse rt = historyClient.RegisterTags( + new GrpcHistory.RegisterTagsRequest { StrHandle = session.StringHandle, BtTagInfos = ByteString.CopyFrom(registerBuffer) }, + connection.Metadata, Deadline(), cancellationToken); + RegistrationDiag += $"RTag={rt.Status?.BSuccess}; "; + } + catch (Exception ex) { RegistrationDiag += $"RTag=EX:{ex.GetType().Name}; "; } + + byte[] payload = HistorianAddTagsProtocol.SerializeCmEventEnsureTagsGrpc(DateTime.UtcNow); + try + { + GrpcHistory.EnsureTagsResponse et = historyClient.EnsureTags( + new GrpcHistory.EnsureTagsRequest { StrHandle = session.StringHandle, BtTagInfos = ByteString.CopyFrom(payload), ElementCount = 1 }, + connection.Metadata, Deadline(), cancellationToken); + RegistrationDiag += $"EnsT={et.Status?.BSuccess}; "; + } + catch (Exception ex) { RegistrationDiag += $"EnsT=EX:{ex.GetType().Name}; "; } + } +} diff --git a/src/AVEVA.Historian.Client/HistorianClient.cs b/src/AVEVA.Historian.Client/HistorianClient.cs index d2974bb..73c9b26 100644 --- a/src/AVEVA.Historian.Client/HistorianClient.cs +++ b/src/AVEVA.Historian.Client/HistorianClient.cs @@ -114,18 +114,24 @@ public sealed class HistorianClient : IAsyncDisposable } /// - /// Sends a single to the Historian's built-in CM_EVENT tag - /// over the WCF event pipeline (Open2 event mode → CM_EVENT registration → AddS2). The - /// event is appended to the historian's event history and is readable back via - /// / the v_AlarmEventHistory2 view. Only original - /// events ( = 0) with string-valued properties - /// are supported; other property value types and revision/update/delete events throw + /// Sends a single to the Historian's built-in CM_EVENT tag. + /// Over WCF this runs Open2 event mode → CM_EVENT registration → AddS2; over the 2023 R2 + /// transport it runs the captured-equivalent + /// v8 Event OpenConnection → CM_EVENT registration → HistoryService.AddStreamValues + /// with the same "OS" event buffer (live-captured 2026-06-23 — the send rides the same RPC + /// and buffer as the WCF path, not a distinct event RPC). The event is appended to the + /// historian's event history and is readable back via / + /// the v_AlarmEventHistory2 view. Only original events + /// ( = 0) with string-valued properties are + /// supported; other property value types and revision/update/delete events throw /// until their wire encoding is captured. /// public Task SendEventAsync(HistorianEvent historianEvent, CancellationToken cancellationToken = default) { ArgumentNullException.ThrowIfNull(historianEvent); - return new HistorianWcfEventOrchestrator(_options).SendEventAsync(historianEvent, cancellationToken); + return _options.Transport == HistorianTransport.RemoteGrpc + ? new Grpc.HistorianGrpcEventWriteOrchestrator(_options).SendEventAsync(historianEvent, cancellationToken) + : new HistorianWcfEventOrchestrator(_options).SendEventAsync(historianEvent, cancellationToken); } /// diff --git a/tests/AVEVA.Historian.Client.Tests/GrpcEventSendProtocolTests.cs b/tests/AVEVA.Historian.Client.Tests/GrpcEventSendProtocolTests.cs new file mode 100644 index 0000000..ac85a95 --- /dev/null +++ b/tests/AVEVA.Historian.Client.Tests/GrpcEventSendProtocolTests.cs @@ -0,0 +1,53 @@ +using System; +using System.Buffers.Binary; +using AVEVA.Historian.Client.Wcf; +using Xunit; + +namespace AVEVA.Historian.Client.Tests; + +/// +/// Golden-byte coverage for the 2023 R2 gRPC event-SEND registration buffers, pinned against a live +/// native capture (capture-send-event scenario, 2026-06-23). The send itself rides +/// HistoryService.AddStreamValues with the same "OS" buffer the WCF path uses +/// (, already golden-tested in +/// WcfEventWriteProtocolTests); what is gRPC-specific is the CM_EVENT registration the event +/// connection performs first (RegisterTags + the 86-byte gRPC EnsureTags). These fixtures are the raw +/// bytes the native client sent on the wire — they carry no identity (CM_EVENT / "AnE Event" / +/// constant tag + event-type GUIDs / a registration FILETIME). +/// +public class GrpcEventSendProtocolTests +{ + // GrpcHistoryClient.RegisterTags.tagInfos captured from the native event connection: the packet + // header 50 67 02 00 + count(1) + the 16-byte CM_EVENT tag GUID. + private const string CapturedRegisterTagsHex = + "506702000100000045813b35f05d464da253871aef49b321"; + + // GrpcHistoryClient.EnsureTags.tagInfos captured from the native event connection (86 bytes): the + // 8-byte EnsureTags header + CM_EVENT CTagMetadata + a registration FILETIME + the …e01f2f27 + // event-type GUID. + private const string CapturedEnsureTagsHex = + "4e670300010000000386000545813b35f05d464da253871aef49b321090800434d5f4556454e54090900416e45204576656e7402020100000001000000004e18d6bd4503dd0142ae595fb63b604791a5ab0be01f2f27"; + + [Fact] + public void BuildRegisterCmEventInputBuffer_MatchesNativeGrpcCapture() + { + byte[] expected = Convert.FromHexString(CapturedRegisterTagsHex); + byte[] actual = HistorianEventRegistrationProtocol.BuildRegisterCmEventInputBuffer(); + Assert.Equal(expected, actual); + } + + [Fact] + public void SerializeCmEventEnsureTagsGrpc_MatchesNativeGrpcCapture() + { + byte[] expected = Convert.FromHexString(CapturedEnsureTagsHex); + Assert.Equal(86, expected.Length); + + // The only run-varying field is the registration FILETIME (the 8 bytes immediately before the + // trailing 16-byte event-type GUID). Feed the captured time back so the comparison is exact. + long filetime = BinaryPrimitives.ReadInt64LittleEndian(expected.AsSpan(expected.Length - 24, 8)); + DateTime createdUtc = DateTime.FromFileTimeUtc(filetime); + + byte[] actual = HistorianAddTagsProtocol.SerializeCmEventEnsureTagsGrpc(createdUtc); + Assert.Equal(expected, actual); + } +} diff --git a/tests/AVEVA.Historian.Client.Tests/HistorianGrpcIntegrationTests.cs b/tests/AVEVA.Historian.Client.Tests/HistorianGrpcIntegrationTests.cs index 5eaf91e..ce47b55 100644 --- a/tests/AVEVA.Historian.Client.Tests/HistorianGrpcIntegrationTests.cs +++ b/tests/AVEVA.Historian.Client.Tests/HistorianGrpcIntegrationTests.cs @@ -233,6 +233,37 @@ public sealed class HistorianGrpcIntegrationTests Assert.Contains(samples, s => s.NumericValue is { } v && Math.Abs(v - expected) < 0.01); } + [Fact] + public async Task SendEventAsync_OverGrpc_AcceptsEvent() + { + string? host = Environment.GetEnvironmentVariable("HISTORIAN_GRPC_HOST"); + // Gated additionally on a dedicated opt-in so this WRITE test never runs by accident — it + // appends a clearly-marked test event to the server's event history. Captured 2026-06-23: + // the gRPC event send rides HistoryService.AddStreamValues with the same "OS" buffer the WCF + // path uses (HistorianEventWriteProtocol), on a v8 Event session + CM_EVENT registration. + if (string.IsNullOrWhiteSpace(host) + || string.IsNullOrEmpty(Environment.GetEnvironmentVariable("HISTORIAN_USER")) + || Environment.GetEnvironmentVariable("HISTORIAN_GRPC_EVENT_SEND") is null) + { + return; + } + + HistorianClient client = new(BuildOptions(host)); + + var evt = new HistorianEvent( + Id: Guid.NewGuid(), + EventTimeUtc: DateTime.UtcNow, + ReceivedTimeUtc: DateTime.UtcNow, + Type: "SdkSendProbe", + SourceName: "SdkSendProbe", + Namespace: "SdkCapture", + RevisionVersion: 0, + Properties: new Dictionary { ["SdkProbeProp"] = "SdkProbeValue" }); + + bool sent = await client.SendEventAsync(evt, CancellationToken.None); + Assert.True(sent, "gRPC SendEvent should be accepted by the server (AddStreamValues BSuccess)."); + } + [Fact] public async Task ReadAggregateAsync_OverGrpc_ReturnsTimeWeightedAverageRows() { diff --git a/tools/AVEVA.Historian.Grpc2023CaptureHarness/Program.cs b/tools/AVEVA.Historian.Grpc2023CaptureHarness/Program.cs index 323c465..fbd6616 100644 --- a/tools/AVEVA.Historian.Grpc2023CaptureHarness/Program.cs +++ b/tools/AVEVA.Historian.Grpc2023CaptureHarness/Program.cs @@ -92,8 +92,10 @@ namespace AVEVA.Historian.Grpc2023CaptureHarness return DeleteTag(managedDll, args); case "capture-event": return CaptureEvent(managedDll, args); + case "capture-send-event": + return CaptureSendEvent(managedDll, args); default: - Console.Error.WriteLine($"Unknown scenario '{scenario}'. Supported: load-check, connect, capture-write, delete-tag, capture-event."); + Console.Error.WriteLine($"Unknown scenario '{scenario}'. Supported: load-check, connect, capture-write, delete-tag, capture-event, capture-send-event."); return 1; } } @@ -603,6 +605,132 @@ namespace AVEVA.Historian.Grpc2023CaptureHarness } } + /// + /// Drives the native 2023 R2 client through an event SEND so the IL-rewritten GrpcClient dumps + /// the AddStreamValues.btValues (the event VTQ storage-sample buffer — resolves whether a gRPC + /// event send uses the "OS" or "ON" outer signature) AND the Event-connection EnsureTags.btTagInfos + /// (the 83-vs-86-byte CM_EVENT registration byte-diff). Opens a WRITE-ENABLED Event connection, + /// builds a clearly-marked test HistorianEvent, calls AddStreamedValue, then CloseConnection to + /// flush the queued event onto the wire. WRITES a real test event into the server's event history. + /// Run with --grpc-rewrite pointing at the instrumented copy and AVEVA_HISTORIAN_RE_CAPTURE set. + /// Usage: capture-send-event [--server ] [--port 32565] [--cert ] + /// [--event-type SdkCaptureProbe] [--flush-seconds 6] + /// + private static int CaptureSendEvent(string managedDll, string[] args) + { + Assembly asm = Assembly.LoadFrom(managedDll); + Type accessType = Req(asm, "ArchestrA.HistorianAccess"); + Type connArgsType = Req(asm, "ArchestrA.HistorianConnectionArgs"); + Type connModeType = Req(asm, "ArchestrA.HistorianConnectionMode"); + Type connTypeType = Req(asm, "ArchestrA.HistorianConnectionType"); + Type errorType = Req(asm, "ArchestrA.HistorianAccessError"); + Type eventType = Req(asm, "ArchestrA.HistorianEvent"); + Type propTypeEnum = Req(asm, "ArchestrA.HistorianEventPropertyType"); + Type certInfoType = Req(asm, "ArchestrA.CertificateInfo"); + Type secModeType = Req(asm, "ArchestrA.HistorianSecurityMode"); + + string server = GetOption(args, "--server") ?? Environment.GetEnvironmentVariable("HISTORIAN_GRPC_HOST") ?? "localhost"; + int port = int.TryParse(GetOption(args, "--port"), out int p) ? p : 32565; + string certName = GetOption(args, "--cert") ?? server; + string evtTypeName = GetOption(args, "--event-type") ?? "SdkCaptureProbe"; + int flushSeconds = int.TryParse(GetOption(args, "--flush-seconds"), out int fs) ? fs : 6; + string? user = Environment.GetEnvironmentVariable("HISTORIAN_USER"); + string? password = Environment.GetEnvironmentVariable("HISTORIAN_PASSWORD"); + if (string.IsNullOrEmpty(user)) + { + Console.Error.WriteLine("Set HISTORIAN_USER/HISTORIAN_PASSWORD."); + return 1; + } + + if (string.IsNullOrWhiteSpace(Environment.GetEnvironmentVariable("AVEVA_HISTORIAN_RE_CAPTURE"))) + { + string defaultCap = Path.GetFullPath(Path.Combine( + "artifacts", "reverse-engineering", "grpc-event-capture", "send-event-capture.ndjson")); + Environment.SetEnvironmentVariable("AVEVA_HISTORIAN_RE_CAPTURE", defaultCap); + } + Console.WriteLine($"Capture sink: {Environment.GetEnvironmentVariable("AVEVA_HISTORIAN_RE_CAPTURE")}"); + + object connArgs = Activator.CreateInstance(connArgsType)!; + SetProp(connArgs, "ServerName", server); + SetProp(connArgs, "TcpPort", checked((ushort)port)); + SetProp(connArgs, "ConnectionMode", Enum.Parse(connModeType, "Historian")); + SetProp(connArgs, "ConnectionType", Enum.Parse(connTypeType, "Event")); // EVENT connection + SetProp(connArgs, "ReadOnly", false); // WRITE-enabled + SetProp(connArgs, "IntegratedSecurity", false); + SetProp(connArgs, "AllowUnTrustedConnection", true); + SetProp(connArgs, "UserName", user); + SetProp(connArgs, "Password", password ?? string.Empty); + object certInfo = Activator.CreateInstance(certInfoType)!; + TrySetProp(certInfo, "CertificateName", certName); + TrySetProp(certInfo, "SecurityMode", Enum.Parse(secModeType, "TransportCertificate")); + TrySetProp(connArgs, "SecurityInfo", certInfo); + + object access = Activator.CreateInstance(accessType)!; + object?[] openArgs = { connArgs, Activator.CreateInstance(errorType) }; + Console.WriteLine($"OpenConnection: server={server} port={port} type=Event readonly=false"); + bool opened; + try + { + opened = (bool)accessType.GetMethod("OpenConnection", new[] { connArgsType, errorType.MakeByRefType() })! + .Invoke(access, openArgs)!; + } + catch (TargetInvocationException tie) + { + Console.Error.WriteLine($"OpenConnection threw: {tie.InnerException?.GetType().Name}: {tie.InnerException?.Message}"); + return 2; + } + Console.WriteLine($"OpenConnection returned: {opened} err={DescribeError(openArgs[1])}"); + if (!opened) { return 2; } + + try + { + // Build a clearly-marked test event. Required: Type (≤32 chars), Id, EventTime. + object evt = Activator.CreateInstance(eventType)!; + SetProp(evt, "Type", evtTypeName); + TrySetProp(evt, "Id", Guid.NewGuid()); + TrySetProp(evt, "EventTime", DateTime.SpecifyKind(DateTime.UtcNow, DateTimeKind.Utc)); + TrySetProp(evt, "Namespace", "SdkCapture"); + TrySetProp(evt, "Source", "SdkCaptureProbe"); + + // One string property to exercise the property-bag framing. + MethodInfo? addProp = eventType.GetMethods().FirstOrDefault(m => + m.Name == "AddProperty" && m.GetParameters().Length == 4); + if (addProp != null) + { + try + { + object strEnum = Enum.Parse(propTypeEnum, "String", true); + object?[] apArgs = { "SdkProbeProp", "SdkProbeValue", strEnum, Activator.CreateInstance(errorType) }; + addProp.Invoke(evt, apArgs); + Console.WriteLine($"AddProperty: err={DescribeError(apArgs[3])}"); + } + catch (Exception ex) { Console.WriteLine($"AddProperty skipped: {ex.GetType().Name}"); } + } + + MethodInfo addStreamed = accessType.GetMethods().First(m => + m.Name == "AddStreamedValue" && m.GetParameters().Length == 2 + && m.GetParameters()[0].ParameterType == eventType); + object?[] asArgs = { evt, Activator.CreateInstance(errorType) }; + bool sent = (bool)addStreamed.Invoke(access, asArgs)!; + Console.WriteLine($"AddStreamedValue({evtTypeName}): {sent} err={DescribeError(asArgs[1])}"); + + // Let the native delivery queue flush the event onto the wire (AddStreamValues). + System.Threading.Thread.Sleep(flushSeconds * 1000); + Console.WriteLine(sent ? "CAPTURE-SEND-EVENT: AddStreamedValue accepted (buffer captured on flush)" : "CAPTURE-SEND-EVENT: AddStreamedValue rejected"); + return sent ? 0 : 3; + } + finally + { + try + { + // CloseConnection flushes any remaining queued values before teardown. + MethodInfo? close = accessType.GetMethod("CloseConnection", new[] { errorType.MakeByRefType() }); + if (close != null) close.Invoke(access, new object?[] { Activator.CreateInstance(errorType) }); + } + catch { /* best-effort */ } + } + } + /// /// Read-only gRPC connect probe: opens a 2023 R2 Historian (mode=Historian) connection via the /// native client and reports the resulting connection status. Proves the mixed-mode client can From dd57d212f88dd4468fde5d0b491c3647212f0ae9 Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Tue, 23 Jun 2026 15:44:53 -0400 Subject: [PATCH 4/4] DeleteTagExtendedProperties: confirm walled via native capture (do not ship) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Tested the binary-dive "use deleteFromServer=true" hypothesis directly against the native client (local 2020 WCF box, Capture-DeleteTagExtendedProperties.ps1 cross-session sync trick). Result: the native DeleteTagExtendedPropertiesByName with deleteFromServer=true returns Success=true, but the property is re-fetchable and re-deletable across repeated fresh sessions — it is NEVER durably removed. So the native client itself only performs an optimistic client-side cache delete the server does not durably honor (the HCAL cache-sync model the decompile shows). This supersedes the earlier "code=1, prop survives" note (that was the same-session sync-gate failure; with proper cross-session sync it returns Success yet still does not durably delete). A managed DeleteTagExtendedPropertiesAsync would return a misleading success, so it correctly stays unshipped. Handoff item 7 updated. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01B6mcaT2PjRFKcogzp9UkfC --- docs/reverse-engineering/handoff.md | 37 ++++++++++++++++------------- 1 file changed, 21 insertions(+), 16 deletions(-) diff --git a/docs/reverse-engineering/handoff.md b/docs/reverse-engineering/handoff.md index 812a14f..4811321 100644 --- a/docs/reverse-engineering/handoff.md +++ b/docs/reverse-engineering/handoff.md @@ -91,13 +91,18 @@ reuses the proven 2020 WCF byte serializers/parsers unchanged inside protobuf RegisterTags prime doesn't help). Use WCF for SQL. 5. **R4.2 revision EDITS** — storage-engine-pipe-only on BOTH transports (the D2 wall). 6. **ReadBlocks** (`StartBlockRetrievalQuery`) — never captured on either transport. -7. **DeleteTagExtendedProperties** — server-blocked on BOTH transports. The gRPC - multiplexed-channel hypothesis was **PROBED + DISPROVEN 2026-06-22** (merge - `c88260c`): GetTgByNm + GetTepByNm primes succeed on one shared write-enabled - gRPC channel, yet DelTep is still rejected (native code=1) and the property - survives — the working set is native in-process registration state, not the - wire session. Pinned by gated negative test - `DeleteTagExtendedProperties_OverGrpc_ProbeMultiplexedChannel`. +7. **DeleteTagExtendedProperties** — server-blocked; **confirmed walled by capturing the NATIVE + client 2026-06-23.** The agent's "use `deleteFromServer=true`" angle is moot: the native + `HistorianAccess.DeleteTagExtendedPropertiesByName(...,deleteFromServer:true)`, driven with the + cross-session sync trick that gets it past the client-side err-229 sync gate + (`Capture-DeleteTagExtendedProperties.ps1`), returns **`Success=true` / ErrorCode=Success** — + yet across repeated fresh sessions the property is **re-fetchable and re-deletable every time**, + i.e. it is **never durably removed**. So the native client itself only performs an optimistic + client-side cache delete; the server does not durably honor it (matches the HCAL cache-sync + model the decompile shows). Shipping a `DeleteTagExtendedPropertiesAsync` would return a + misleading success while the property persists, so it correctly stays **unshipped**. (Earlier + gRPC multiplexed-channel hypothesis also PROBED + DISPROVEN 2026-06-22, merge `c88260c`; pinned + by `DeleteTagExtendedProperties_OverGrpc_ProbeMultiplexedChannel`.) 8. **Deferred-by-design** items (`write-commands` D1–D3, non-analog tag create, etc.) — bounded out until an explicit customer/user demand signal. @@ -159,15 +164,15 @@ with these refinements: handle+sequence cursor but the `historyBlocks` response is a native blob with no managed decoder, and it needs the D2-blocked `OpenStorageConnection` console handle. Walled. -- **Item 7 (DeleteTagExtendedProperties)** — **reframed to a capturable lead.** RPC + - string handle are **correct** in our SDK; ADD and DELETE are structurally identical - and **neither** routes through `StartJob`. The differentiator is the `deleteFromServer` - flag carried inside the native-built `BtInput` plus the native HCAL **cache-sync - background worker** that actually propagates the delete server-side (config writes hit - the in-process cache first, then sync). **Capturable**: capture native - `DeleteTagExtendedPropertiesByName(deleteFromServer=true)`'s `BtInput` to learn whether - one well-formed RPC durably deletes (→ shippable) or whether it genuinely depends on - the cache-sync worker (→ walled). +- **Item 7 (DeleteTagExtendedProperties)** — **capture done 2026-06-23; CONFIRMED WALLED, don't + ship.** RPC + string handle are correct; ADD/DELETE are structurally identical and neither uses + `StartJob`. The `deleteFromServer`-flag hypothesis is now tested and moot: the native + `DeleteTagExtendedPropertiesByName(...,deleteFromServer:true)`, driven past the err-229 client + sync gate with the cross-session trick (`Capture-DeleteTagExtendedProperties.ps1`), returns + `Success=true` — yet the property is **re-fetchable + re-deletable across repeated fresh sessions + (never durably removed)**. So the native client only does an optimistic client-side cache delete + the server doesn't durably honor (the HCAL cache-sync model). Shipping + `DeleteTagExtendedPropertiesAsync` would return a misleading success, so it stays unshipped. - **SF/snapshot/shard/ForwardSnapshot ops** — only `Get/SetSFParameter` are managed-built (typed strings); all others carry opaque native buffers and need the storage console handle. Walled / tooling-internal.