From 23798db1ef6b45acdd07564091ab916a73e659fa Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Sun, 21 Jun 2026 17:51:17 -0400 Subject: [PATCH 1/4] M3 probe: non-streamed write transaction reachable over 2023 R2 gRPC (Begin/End live-verified) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The D2 storage-engine-pipe wall is WCF-transport-specific. On the 2023 R2 gRPC front door, TransactionService is a first-class service AND the gateway to the storage engine, so the Open2 storage-session GUID (uppercase) is accepted directly as strHandle with no legacy pipe. Live-verified against the real 2023 R2 server over a write-enabled (0x401) gRPC session: AddNonStreamValuesBegin returns a real strTransactionId, and AddNonStreamValuesEnd(bCommit=false) discards it cleanly (no data written). On 2020 WCF the same op returns UnknownClient(51) for every handle + priming chain. - HistorianGrpcRevisionProbe + grpc-revision-probe CLI command + gated test NonStreamedWriteTransaction_OverGrpc_BeginsAndDiscards (live pass). - HistorianGrpcHandshake.OpenSession gains an optional connectionMode param (default read-only 0x402; pass 0x401 for write-enabled) — non-breaking. - Docs: revision-write-path.md "the wall is gone" section; roadmap M3 section, R3.1-R3.3 rows, one-glance table, and status note updated honestly. Not yet shipped: the AddNonStreamValues btInput VTQ buffer is uncaptured (never guess wire bytes), so no value-commit is implemented. Scope is non-streamed ORIGINAL backfill; revision EDITS (R4.2) remain pipe-only even on gRPC. 272 unit tests pass; sanitization scan clean. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01B6mcaT2PjRFKcogzp9UkfC --- docs/plans/hcal-roadmap.md | 43 +++-- docs/plans/revision-write-path.md | 62 ++++++- .../Grpc/HistorianGrpcHandshake.cs | 11 +- .../Grpc/HistorianGrpcRevisionProbe.cs | 157 ++++++++++++++++++ .../HistorianGrpcIntegrationTests.cs | 25 +++ .../Program.cs | 38 +++++ 6 files changed, 319 insertions(+), 17 deletions(-) create mode 100644 src/AVEVA.Historian.Client/Grpc/HistorianGrpcRevisionProbe.cs diff --git a/docs/plans/hcal-roadmap.md b/docs/plans/hcal-roadmap.md index bc8dc3d..fe8e2bd 100644 --- a/docs/plans/hcal-roadmap.md +++ b/docs/plans/hcal-roadmap.md @@ -166,8 +166,8 @@ read/browse/status surface is Windows-free and the gRPC stack is the default pat | ~~R1.5~~ | Extended-property **read** | `Retrieval.GetTagExtendedPropertiesFromName` (`GetTepByNm`) | ✅ **DONE (2026-06-20), live-verified.** `GetTagExtendedPropertiesAsync(tag)` → name/value pairs. String-handle op via the uppercase storage GUID; name-based path (`GetTagExtendedPropertiesByName`, not the QTB-gated TagQuery path). Request `tagNames` = `uint count` + per-name(`uint charCount`+UTF-16); response = `uint tagCount` + per-tag(marker + compact-ASCII name + `uint propCount` + per-prop(marker + compact-ASCII name + `0x43` VT_BSTR value) + trailer). Sequence-paged. Shipped: `HistorianTagExtendedPropertyProtocol`, golden `WcfTagExtendedPropertyProtocolTests`, gated live test. See `docs/reverse-engineering/wcf-tag-extended-properties.md`. | uppercase string handle | | ~~R1.6~~ | Localized-property **read** | (no op) | ⛔ **No distinct op on 2020 — collapses into R1.5.** There is no `GetTagLocalizedPropertiesFromName`/`GetTlpByNm` or `GetTagLocalizedPropertiesByName` in `current/aahClientManaged.dll`; the only "localized" surfaces are error-message/UI-text localization. Extended properties (R1.5) are the user-defined tag-property read surface. Closed, not throwing. | — | | ~~R1.7~~ | Event **filters** | filter bytes in `Retrieval.StartEventQuery` | ✅ **DONE (2026-06-20), live-honored.** `ReadEventsAsync(start, end, HistorianEventFilter)`. The filter rides `StartEventQuery`'s `pRequestBuff` (captured via `EventQuery.AddEventFilter` + instrument-wcf-writemessage; Equal vs Contains diffed to isolate the op). Filter block: `ushort 0 + uint filterCount + uint condCount + uint nameLen + name(UTF-16) + uint 1 + ushort op + uint 1 + value(0x09-len-0x00 compact-ASCII) + byte 0`. **REAL, not inert** (a non-matching predicate returns 0 events; matching returns the subset). Single string-valued predicate only; multi-filter (OR) / multi-condition (AND via `AddEventFilterCondition`) framing not yet fully captured. See `HistorianEventFilter`, golden `WcfEventQueryProtocolTests`. | — | -| R1.8 | Analog-summary query | `Retrieval.StartQuery` (summary mode) | summary row layout — **`uint`-handle, reachable. Scoped + decode targets located** (`CAnalogSummaryValue.UnpackFromValueBuffer`, fields Min/Max/First/Last/ValueCount/Integral/…). Plan: [`r1.8-r1.9-summary-queries.md`](r1.8-r1.9-summary-queries.md) | — | -| R1.9 | State-summary query | `Retrieval.StartQuery` (state mode) | state-summary row layout — **`uint`-handle, reachable. Scoped** (`CStateSummaryStruct`: MinContained/MaxContained/TotalContained/PartialStart/PartialEnd/StateEntryCount). Plan: [`r1.8-r1.9-summary-queries.md`](r1.8-r1.9-summary-queries.md) | — | +| ~~R1.8~~ | Analog-summary query | `Retrieval.StartQuery` (summary mode) | ✅ **RESOLVED (2026-06-21) — no new code; == existing `ReadAggregateAsync`.** Request + response both captured (`scripts/Capture-SummaryRequest.ps1 -WithResponse`): the `GetNextQueryResultBuffer2` response is the **ordinary version-9 row buffer** the raw/aggregate parser already handles (decoded 7 rows = SQL ground truth exactly). There is **no rich `CAnalogSummaryValue` struct on the wire** — each row carries a *single* value selected by `RetrievalMode`/QueryType (Integral→8, TimeWeightedAverage→5, …), not an all-aggregates-in-one row; `ValueSelector`/`AggregationType`/`MaxStates` are **inert** on the WCF retrieval path (they configure the SQL provider, not this query). The all-aggregates-at-once shape is the SQL/OLEDB provider's, or the gRPC front door — not 2020 WCF binary. Plan + capture evidence: [`r1.8-r1.9-summary-queries.md`](r1.8-r1.9-summary-queries.md). | — | +| ~~R1.9~~ | State-summary query | `Retrieval.StartQuery` (state mode) | ✅ **RESOLVED (2026-06-21) — same finding as R1.8.** State-summary is the **same `StartQuery2` request** (only `MaxStates`/defaults differ on the wire); the response carries no distinct `CStateSummaryStruct` on the 2020 WCF binary path. Covered by the existing aggregate read; no new `src/` code warranted. Plan: [`r1.8-r1.9-summary-queries.md`](r1.8-r1.9-summary-queries.md). | — | ### 1c. Bounded config writes (S–M each) | ID | Capability | gRPC op | Payload | Notes | @@ -221,9 +221,21 @@ byte-correct `AddS2` (✅). Appears-and-reads-back is environment-gated on event *Goal: insert original historical VTQs (backfill), the path that is NOT the gated cache push.* +> ✅ **gRPC UNLOCK (2026-06-21, LIVE-VERIFIED): the transaction lifecycle is REACHABLE over the +> 2023 R2 gRPC front door.** The `grpc-revision-probe` opened a **write-enabled** (`0x401`) gRPC +> session and drove `TransactionService.AddNonStreamValuesBegin(storage-GUID **uppercase**)` → +> real `strTransactionId` → `AddNonStreamValuesEnd(bCommit=false)` (discarded, no data written). +> Where 2020 WCF returns `UnknownClient (51)`, the gRPC `TransactionService` is itself the gateway +> to the storage engine, so the Open2 session GUID is accepted directly — **no legacy pipe**. This +> answers the M3-over-gRPC question below: **yes**, the non-streamed *original* write transaction is +> reachable from the pure-managed SDK. **Not yet shipped:** the `AddNonStreamValues` `btInput` VTQ +> buffer must be captured before any value-commit (never guess wire bytes); revision *edits* (R4.2) +> remain pipe-only even on gRPC. Full detail + decompile basis: +> [`revision-write-path.md`](revision-write-path.md) §"2023 R2 gRPC — the wall is gone". +> > ⛔ **BLOCKED on 2020 WCF — re-confirmed by the D2 probe (2026-05-05), see > [`revision-write-path.md`](revision-write-path.md).** The premise above ("the path that is NOT -> the gated cache push") was **disproved**: R3.1's op +> the gated cache push") was **disproved** *on WCF*: R3.1's op > (`Transaction.AddNonStreamValuesBegin/AddNonStreamValues/End`) is the **same** > `ITransactionServiceContract2.AddNonStreamValuesBegin2` D2 probed, and over WCF it returns > `04 33 00 00 00` = `UnknownClient (51)` for every handle format **and** the full priming chain @@ -242,12 +254,13 @@ byte-correct `AddS2` (✅). Appears-and-reads-back is environment-gated on event | ID | Work | gRPC op | Status | |---|---|---|---| -| R3.1 | Decode non-streamed VTQ packet | `Transaction.AddNonStreamValuesBegin/AddNonStreamValues/End` | ⛔ WCF blocked (storage-engine pipe — D2). gRPC: untested | -| R3.2 | `AddHistoricalValuesAsync` | batched begin→values→end | ⛔ gated on R3.1 | -| R3.3 | Ingest-permission validation | confirm the target accepts original-data insert (distinct from `AddS2` cache wall) | ⛔ proven to share the same gate, not distinct | +| R3.1 | Decode non-streamed VTQ packet | `Transaction.AddNonStreamValuesBegin/AddNonStreamValues/End` | 🟡 **gRPC transaction Begin/End LIVE-VERIFIED 2026-06-21** (WCF still blocked — D2). Remaining: capture the `AddNonStreamValues` `btInput` VTQ buffer (don't guess) | +| R3.2 | `AddHistoricalValuesAsync` | batched begin→values→end | 🟡 unblocked by R3.1's gRPC proof; needs the `btInput` serializer + a real `bCommit=true` write/read-back | +| R3.3 | Ingest-permission validation | confirm the target accepts original-data insert (distinct from `AddS2` cache wall) | ✅ **distinct on gRPC** — Begin succeeded against a real write-enabled session (the WCF/native cache gate does not apply here) | -**Acceptance:** historical points inserted and read back. **WCF path closed (D2);** would require -the gRPC write path (live 2023 R2 server + capture) to reopen. +**Acceptance:** historical points inserted and read back. **WCF path closed (D2).** gRPC path: +**transaction lifecycle proven (Begin/End live)**; full insert+read-back pending the `btInput` +capture + serializer. --- @@ -301,11 +314,13 @@ event-send). M3/M4 as demand dictates. > **Status 2026-06-21:** sprints 1 + 2 are **complete** (M0 gRPC parity, the reachable M1 surface, > and M2 event-send all shipped + live-verified; remaining M1 items are evidence-bounded-out). The -> reachable surface on the **available 2020 WCF infrastructure is exhausted** — every remaining -> roadmap item is now either (a) blocked by the storage-engine-pipe architecture (**M3-WCF**, R4.2), -> (b) **gRPC/2023R2-only** and needs the live 2023 R2 server for a native capture (R1.3 timezone, -> R1.4 EventStorageMode, M3/revisions over gRPC), or (c) a HARD deferred subsystem (M4). No further -> work lands without one of: a live-2023R2 capture session, or a customer-demand trigger. +> reachable surface on the **available 2020 WCF infrastructure is exhausted**. **M3 update +> (2026-06-21):** with the live 2023 R2 server, the **M3 non-streamed write transaction is now +> proven reachable over gRPC** — `TransactionService.AddNonStreamValuesBegin/End` round-trips live +> (the D2 storage-engine-pipe wall is WCF-only). The remaining M3 work is bounded and concrete: +> capture the `AddNonStreamValues` `btInput` VTQ buffer → golden-tested serializer → real +> commit+read-back → public `AddHistoricalValuesAsync`. The other levers are unchanged: R4.2 revision +> *edits* stay pipe-only even on gRPC, and M4 (SF / redundancy) is a HARD deferred subsystem. ## One-glance status @@ -314,5 +329,5 @@ event-send). M3/M4 as demand dictates. | M0 gRPC parity + capture tooling | foundation | M | unblocks everything, Windows-free | ✅ **done** | | M1 cheap surface | TRIVIAL/BOUNDED | M–L | most remaining read/config | ✅ **done** (reachable surface; rest bounded out) | | M2 event send | CAPTURE | S–M | headline write capability | ✅ **done** | -| M3 historical writes | BOUNDED | M | backfill | ⛔ WCF blocked (D2); gRPC = on-demand + live 2023R2 | +| M3 historical writes | BOUNDED | M | backfill | 🟡 **gRPC transaction Begin/End live-verified (2026-06-21)**; WCF blocked (D2). Remaining: `btInput` capture → commit+read-back | | M4 SF / revisions / redundancy | HARD | L×N | parity completeness | defer (R4.2 = same pipe wall) | diff --git a/docs/plans/revision-write-path.md b/docs/plans/revision-write-path.md index b326689..66065aa 100644 --- a/docs/plans/revision-write-path.md +++ b/docs/plans/revision-write-path.md @@ -1,6 +1,66 @@ # Plan: Revision-Write Path (`AddRevisionValuesBegin/Value/End`) -Status: **ARCHITECTURALLY BLOCKED — verified 2026-05-05.** Same root +Status: **WCF: ARCHITECTURALLY BLOCKED (verified 2026-05-05).** **gRPC (2023 R2): the +non-streamed-original transaction is REACHABLE — Begin/End round-trip LIVE-VERIFIED 2026-06-21.** +Same root cause on WCF as `AddS2`: the `TransactionService` relay needs a pre-existing +storage-engine *pipe* session no WCF op can create. The 2023 R2 gRPC front door removes that wall +(see the §"2023 R2 gRPC — the wall is gone" section immediately below); the legacy WCF analysis is +preserved unchanged after it. + +## 2023 R2 gRPC — the wall is gone (non-streamed original writes), LIVE-VERIFIED 2026-06-21 + +The whole D2 WCF blocker was: `ITransactionServiceContract2.AddNonStreamValuesBegin2` returns +`04 33 00 00 00` = `UnknownClient (51)` because the server-side Trx relay requires a storage-engine +pipe session (`STransactPipeClient2` → `aaStorageEngine.exe`) that no WCF op establishes. On the +**2023 R2 gRPC** transport that relay is replaced by a first-class `TransactionService` gRPC +service, and the gRPC server is itself the gateway to the storage engine — so the client passes the +**HistoryService Open2 storage-session GUID** straight in as `strHandle` and the transaction opens. + +**Live probe (`grpc-revision-probe` CLI command / `HistorianGrpcRevisionProbe`):** against the real +2023 R2 server (History iface 12), over a **write-enabled** (`0x401`) gRPC session — + +| step | result | +|---|---| +| `HistoryService.OpenConnection` (write-enabled `0x401`) | ✅ `OpenSucceeded`, client handle + storage GUID returned | +| `TransactionService.GetTransactionInterfaceVersion` | ✅ error 0, **version 2** | +| `TransactionService.AddNonStreamValuesBegin(strHandle = storage GUID **UPPERCASE**)` | ✅ **`BeginSucceeded`** — returns a real `strTransactionId` (e.g. `…-FE0A-4822-…`) on the **first** handle format tried | +| `TransactionService.AddNonStreamValuesEnd(handle, txId, bCommit=**false**)` | ✅ `EndDiscardSucceeded` — transaction discarded, **no data written** | + +So the answer to the roadmap's open M3-over-gRPC question ("does the 2023 R2 gRPC front door expose +a non-streamed write that bypasses the legacy storage-engine pipe?") is **YES** — Begin/End is +reachable from the pure-managed SDK with no pipe, no native wrapper. The probe is committed as the +`grpc-revision-probe` CLI command + the gated test +`HistorianGrpcIntegrationTests.NonStreamedWriteTransaction_OverGrpc_BeginsAndDiscards`; re-run any +time to confirm the path is still open. + +### Decompile basis (handle + op group) + +`Archestra.Historian.GrpcClient.GrpcHistoryClient` drives the identical three-phase sequence +(`AddNonStreamValuesBegin(strHandle) → strTransactionId`; `AddNonStreamValues(strHandle, +strTransactionId, btInput)`; `AddNonStreamValuesEnd(strHandle, strTransactionId, bCommit)`), passing +the Open2 session GUID as `strHandle`. `btInput` is the **same opaque native VTQ buffer** the 2020 +path uses. Proto: `src/AVEVA.Historian.Client/Grpc/Protos/TransactionService.proto`. + +### What is proven vs. what remains (do NOT ship yet) + +- ✅ **Proven:** the transaction lifecycle (Begin → End/rollback) is reachable over gRPC. The D2 + architectural wall is specific to the WCF transport. +- ⛔ **Not yet captured:** the `AddNonStreamValues` **`btInput` VTQ buffer byte layout**. Per project + discipline ("never guess wire bytes; capture first") no value-commit is implemented. The next step + to actually *ship* M3 (`AddHistoricalValuesAsync`) is to capture the native gRPC `AddNonStreamValues` + `btInput` (or decode the `GrpcHistoryClient` serializer), build a golden-tested serializer, then do a + real `bCommit=true` write + SQL read-back against a sandbox tag created by `EnsureTagAsync`. +- 🔒 **Scope:** this is **non-streamed ORIGINAL backfill** (`HistorianDataCategory.NonStreamedOriginal` + → `TransactionService.AddNonStreamValues*`). **Revision EDITS** (`AddRevisionValue(s)` / + `RevisionInsert*`, the R4.2 path) are NOT on the gRPC contract even in 2023 R2 — the capability + matrix confirms they still ride the storage-engine pipe. The gRPC unlock here is original backfill, + not after-the-fact edits. + +--- + +## Legacy WCF analysis (preserved — still accurate for the 2020 WCF transport) + +Status (WCF only): **ARCHITECTURALLY BLOCKED — verified 2026-05-05.** Same root cause as `AddS2`: client-side cache rejects values for tags that weren't registered through a configured IO server / Application Server pipeline. Documented below; implementation deferred until / unless that diff --git a/src/AVEVA.Historian.Client/Grpc/HistorianGrpcHandshake.cs b/src/AVEVA.Historian.Client/Grpc/HistorianGrpcHandshake.cs index 787adf0..79d88b1 100644 --- a/src/AVEVA.Historian.Client/Grpc/HistorianGrpcHandshake.cs +++ b/src/AVEVA.Historian.Client/Grpc/HistorianGrpcHandshake.cs @@ -40,10 +40,17 @@ internal static class HistorianGrpcHandshake CancellationToken cancellationToken) => OpenSession(connection, options, cancellationToken).ClientHandle; + /// + /// The native Open2 connection mode. Defaults to read-only (0x402); pass + /// + /// (0x401) for write-enabled sessions (e.g. the non-streamed/revision Transaction path, + /// which the read-only mode silently rejects with err 132 OperationNotEnabled). + /// public static Session OpenSession( HistorianGrpcConnection connection, HistorianClientOptions options, - CancellationToken cancellationToken) + CancellationToken cancellationToken, + uint connectionMode = HistorianWcfAuthChainHelper.NativeIntegratedReadOnlyConnectionMode) { DateTime Deadline() => DateTime.UtcNow.Add(options.RequestTimeout); @@ -73,7 +80,7 @@ internal static class HistorianGrpcHandshake cancellationToken); byte[] open2Request = HistorianNativeHandshake.BuildOpenConnection3Request( - options.Host, contextKey, HistorianWcfAuthChainHelper.NativeIntegratedReadOnlyConnectionMode); + options.Host, contextKey, connectionMode); GrpcHistory.OpenConnectionResponse open2 = historyClient.OpenConnection( new GrpcHistory.OpenConnectionRequest { BtConnectionRequest = ByteString.CopyFrom(open2Request) }, diff --git a/src/AVEVA.Historian.Client/Grpc/HistorianGrpcRevisionProbe.cs b/src/AVEVA.Historian.Client/Grpc/HistorianGrpcRevisionProbe.cs new file mode 100644 index 0000000..cbff48e --- /dev/null +++ b/src/AVEVA.Historian.Client/Grpc/HistorianGrpcRevisionProbe.cs @@ -0,0 +1,157 @@ +using Google.Protobuf; +using AVEVA.Historian.Client.Wcf; +using GrpcTransaction = ArchestrA.Grpc.Contract.Transaction; + +namespace AVEVA.Historian.Client.Grpc; + +/// +/// Live probe for the M3 (historical / non-streamed original-value write) path over the 2023 R2 +/// gRPC front door. On 2020 WCF this op group is architecturally blocked: the +/// ITransactionServiceContract2.AddNonStreamValuesBegin2 relay returns +/// UnknownClient (51) because it requires a pre-existing storage-engine pipe session +/// (STransactPipeClient2 → aaStorageEngine.exe) that no WCF op can establish — see +/// docs/plans/revision-write-path.md (the D2 finding). +/// +/// The 2023 R2 decompile shows the native gRPC client driving the SAME op group over +/// TransactionService.AddNonStreamValuesBegin/AddNonStreamValues/AddNonStreamValuesEnd and +/// passing the HistoryService Open2 session GUID directly as strHandle — i.e. the gRPC +/// server is the gateway to the storage engine, so the client never touches the legacy pipe. This +/// probe tests whether the SDK's pure-managed handshake can reproduce that: it opens a +/// write-enabled session and calls AddNonStreamValuesBegin, surfacing whatever the server +/// returns. It writes NO data — if Begin succeeds it immediately calls AddNonStreamValuesEnd +/// with bCommit=false to discard the transaction. +/// +internal sealed class HistorianGrpcRevisionProbe +{ + private readonly HistorianClientOptions _options; + + public HistorianGrpcRevisionProbe(HistorianClientOptions options) + { + _options = options ?? throw new ArgumentNullException(nameof(options)); + } + + public Task ProbeBeginAsync(CancellationToken cancellationToken) + => Task.Run(() => ProbeBegin(cancellationToken), cancellationToken); + + private HistorianGrpcRevisionProbeResult ProbeBegin(CancellationToken cancellationToken) + { + var result = new HistorianGrpcRevisionProbeResult(); + + using HistorianGrpcConnection connection = HistorianGrpcChannelFactory.Create(_options); + + HistorianGrpcHandshake.Session session = HistorianGrpcHandshake.OpenSession( + connection, + _options, + cancellationToken, + connectionMode: HistorianWcfAuthChainHelper.NativeIntegratedWriteEnabledConnectionMode); + + result.OpenSucceeded = true; + result.ClientHandle = session.ClientHandle; + result.StorageSessionId = session.StorageSessionId; + + var transactionClient = new GrpcTransaction.TransactionService.TransactionServiceClient(connection.Channel); + DateTime Deadline() => DateTime.UtcNow.Add(_options.RequestTimeout); + + // Register the client with the Transaction service's session table (matches the + // cross-service GetV priming the WCF write path uses). + try + { + GrpcTransaction.GetTransactionInterfaceVersionResponse version = transactionClient.GetTransactionInterfaceVersion( + new GrpcTransaction.GetTransactionInterfaceVersionRequest(), connection.Metadata, Deadline(), cancellationToken); + result.TrxInterfaceVersionError = version.Error; + result.TrxInterfaceVersion = version.Version; + } + catch (Exception ex) + { + result.TrxInterfaceVersionException = $"{ex.GetType().Name}: {ex.Message}"; + } + + // The decompiled native client passes the Open2 storage-session GUID (string) as strHandle. + // Try that first (uppercase "D" form, as the other string-handle ops require), then a couple + // of fallbacks mirroring the WCF probe, so a wrong-format rejection is distinguishable from a + // genuine server-side block. + foreach ((string label, string handle) in new[] + { + ("storageSessionId-upper", session.StringHandle), + ("storageSessionId-lower", session.StorageSessionId.ToString("D")), + ("clientHandle-as-string", session.ClientHandle.ToString()), + }) + { + var attempt = new HistorianGrpcRevisionBeginAttempt { HandleLabel = label, HandleSent = handle }; + try + { + GrpcTransaction.AddNonStreamValuesBeginResponse begin = transactionClient.AddNonStreamValuesBegin( + new GrpcTransaction.AddNonStreamValuesBeginRequest { StrHandle = handle }, + connection.Metadata, Deadline(), cancellationToken); + + attempt.Succeeded = begin.Status?.BSuccess ?? false; + attempt.TransactionId = begin.StrTransactionId; + byte[] error = begin.Status?.BtError?.ToByteArray() ?? []; + attempt.ErrorHex = error.Length == 0 ? null : Convert.ToHexString(error); + result.BeginAttempts.Add(attempt); + + if (attempt.Succeeded && !string.IsNullOrEmpty(attempt.TransactionId)) + { + result.BeginSucceeded = true; + result.BeginTransactionId = attempt.TransactionId; + + // Discard immediately — bCommit=false writes nothing. This keeps the probe + // read-only against the live (production) server. + try + { + GrpcTransaction.AddNonStreamValuesEndResponse end = transactionClient.AddNonStreamValuesEnd( + new GrpcTransaction.AddNonStreamValuesEndRequest + { + StrHandle = handle, + StrTransactionId = attempt.TransactionId, + BCommit = false, + }, + connection.Metadata, Deadline(), cancellationToken); + result.EndDiscardSucceeded = end.Status?.BSuccess ?? false; + byte[] endError = end.Status?.BtError?.ToByteArray() ?? []; + result.EndDiscardErrorHex = endError.Length == 0 ? null : Convert.ToHexString(endError); + } + catch (Exception ex) + { + result.EndDiscardException = $"{ex.GetType().Name}: {ex.Message}"; + } + + break; + } + } + catch (Exception ex) + { + attempt.Exception = $"{ex.GetType().Name}: {ex.Message}"; + result.BeginAttempts.Add(attempt); + } + } + + return result; + } +} + +internal sealed class HistorianGrpcRevisionProbeResult +{ + public bool OpenSucceeded { get; set; } + public uint ClientHandle { get; set; } + public Guid StorageSessionId { get; set; } + public uint? TrxInterfaceVersionError { get; set; } + public uint? TrxInterfaceVersion { get; set; } + public string? TrxInterfaceVersionException { get; set; } + public bool BeginSucceeded { get; set; } + public string? BeginTransactionId { get; set; } + public bool EndDiscardSucceeded { get; set; } + public string? EndDiscardErrorHex { get; set; } + public string? EndDiscardException { get; set; } + public List BeginAttempts { get; } = new(); +} + +internal sealed class HistorianGrpcRevisionBeginAttempt +{ + public string HandleLabel { get; set; } = ""; + public string HandleSent { get; set; } = ""; + public bool Succeeded { get; set; } + public string? TransactionId { get; set; } + public string? ErrorHex { get; set; } + public string? Exception { get; set; } +} diff --git a/tests/AVEVA.Historian.Client.Tests/HistorianGrpcIntegrationTests.cs b/tests/AVEVA.Historian.Client.Tests/HistorianGrpcIntegrationTests.cs index facb60c..6a805f3 100644 --- a/tests/AVEVA.Historian.Client.Tests/HistorianGrpcIntegrationTests.cs +++ b/tests/AVEVA.Historian.Client.Tests/HistorianGrpcIntegrationTests.cs @@ -1,3 +1,4 @@ +using AVEVA.Historian.Client.Grpc; using AVEVA.Historian.Client.Models; namespace AVEVA.Historian.Client.Tests; @@ -123,6 +124,30 @@ public sealed class HistorianGrpcIntegrationTests Assert.All(names, n => Assert.StartsWith("Sys", n, StringComparison.Ordinal)); } + [Fact] + public async Task NonStreamedWriteTransaction_OverGrpc_BeginsAndDiscards() + { + string? host = Environment.GetEnvironmentVariable("HISTORIAN_GRPC_HOST"); + if (string.IsNullOrWhiteSpace(host) || string.IsNullOrEmpty(Environment.GetEnvironmentVariable("HISTORIAN_USER"))) + { + return; + } + + // M3 reachability probe: on 2020 WCF this op group is walled (TransactionService relay + // returns UnknownClient(51) — the storage-engine-pipe requirement, see + // docs/plans/revision-write-path.md). On the 2023 R2 gRPC front door the native client + // passes the Open2 storage-session GUID straight to TransactionService and it works. + // This asserts the wall is gone: a write-enabled session opens and AddNonStreamValuesBegin + // returns a transaction id, which we immediately End with bCommit=false (writes nothing). + var probe = new HistorianGrpcRevisionProbe(BuildOptions(host)); + HistorianGrpcRevisionProbeResult result = await probe.ProbeBeginAsync(CancellationToken.None); + + Assert.True(result.OpenSucceeded); + Assert.True(result.BeginSucceeded, "AddNonStreamValuesBegin should return a transaction id over gRPC."); + Assert.False(string.IsNullOrEmpty(result.BeginTransactionId)); + Assert.True(result.EndDiscardSucceeded, "AddNonStreamValuesEnd(bCommit:false) should discard cleanly."); + } + private static HistorianClientOptions BuildOptions(string host) { string? user = Environment.GetEnvironmentVariable("HISTORIAN_USER"); diff --git a/tools/AVEVA.Historian.ReverseEngineering/Program.cs b/tools/AVEVA.Historian.ReverseEngineering/Program.cs index 2b64c15..76168f8 100644 --- a/tools/AVEVA.Historian.ReverseEngineering/Program.cs +++ b/tools/AVEVA.Historian.ReverseEngineering/Program.cs @@ -13,6 +13,8 @@ using System.Runtime.Versioning; using System.Text; using System.Text.Json; using AVEVA.Historian.Client; +using AVEVA.Historian.Client.Grpc; +using AVEVA.Historian.Client.Models; using AVEVA.Historian.Client.Wcf; using AVEVA.Historian.Client.Wcf.Contracts; using AVEVA.Historian.ReverseEngineering.Capture; @@ -72,6 +74,7 @@ try "wcf-register-event-tag" => RegisterEventTagAndStartQuery(args), "wcf-add-event-tag" => AddEventTagAndStartQuery(args), "capture-tag-info" => CaptureTagInfo(args), + "grpc-revision-probe" => ProbeGrpcRevision(args), _ => UnknownCommand(args[0]) }; } @@ -3209,6 +3212,41 @@ static int WriteMarker(string[] args) return 0; } +static int ProbeGrpcRevision(string[] args) +{ + // Usage: grpc-revision-probe [port] [--tls] [--dnsid ] [--insecure-cert] + // Reads HISTORIAN_USER / HISTORIAN_PASSWORD from the environment for explicit creds; + // falls back to IntegratedSecurity when unset. + string host = args.Length > 1 ? args[1] : "localhost"; + int port = args.Length > 2 && int.TryParse(args[2], out int parsedPort) + ? parsedPort + : HistorianClientOptions.DefaultGrpcPort; + bool tls = HasOption(args, "--tls"); + string? dnsId = GetOption(args, "--dnsid"); + + string? user = Environment.GetEnvironmentVariable("HISTORIAN_USER"); + string? password = Environment.GetEnvironmentVariable("HISTORIAN_PASSWORD"); + bool explicitCreds = !string.IsNullOrEmpty(user); + + var options = new HistorianClientOptions + { + Host = host, + Port = port, + Transport = HistorianTransport.RemoteGrpc, + GrpcUseTls = tls, + AllowUntrustedServerCertificate = tls, + ServerDnsIdentity = dnsId, + IntegratedSecurity = !explicitCreds, + UserName = user ?? string.Empty, + Password = password ?? string.Empty, + }; + + var probe = new HistorianGrpcRevisionProbe(options); + HistorianGrpcRevisionProbeResult result = probe.ProbeBeginAsync(CancellationToken.None).GetAwaiter().GetResult(); + Console.WriteLine(JsonSerializer.Serialize(result, CreateJsonOptions())); + return result.BeginSucceeded ? 0 : 2; +} + static int ProbeWcf(string[] args) { string host = args.Length > 1 ? args[1] : "localhost"; From 8fbb86881399b2570472224c57ad604eb5c517a2 Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Sun, 21 Jun 2026 18:08:27 -0400 Subject: [PATCH 2/4] M3 R3.1 decode: AddNonStreamValues reaches server StoreNonStreamValues (storage-engine console pipe) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Empirically decoded the AddNonStreamValues btInput framing against the live 2023 R2 server (grpc-nonstream-decode command + ProbeNonStreamedBuffersAsync driver). Every transaction rolled back (bCommit=false) — no data written. Finding: the btInput is assembled native-C++-side (not in any decompile), so 6 evidence-based framings (44-54B, packed HISTORIAN_VALUE2 variants) were probed. All 6 returned the IDENTICAL server error while an empty buffer returned a different InvalidParameter — so non-empty buffers pass parameter validation into CHistStorageConnection::StoreNonStreamValues, which routes to the \.\pipe\aahStorageEngine\console pipe server-side. Identical-across-framings => the blocker is NOT the btInput layout but a missing storage-engine console session / tag-registration precondition for the connection. Next step (untested): StorageService.OpenStorageConnection + tag registration (RegisterTags/AddTagidPairs/AddShardTagids) before AddNonStreamValues, then commit + read-back on a sandbox tag. Documented in revision-write-path.md (R3.1 decode section); raw artifact gitignored. 272 unit tests pass. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01B6mcaT2PjRFKcogzp9UkfC --- docs/plans/revision-write-path.md | 34 +++++ .../Grpc/HistorianGrpcRevisionProbe.cs | 110 +++++++++++++++ .../Program.cs | 128 ++++++++++++++++++ 3 files changed, 272 insertions(+) diff --git a/docs/plans/revision-write-path.md b/docs/plans/revision-write-path.md index 66065aa..922a238 100644 --- a/docs/plans/revision-write-path.md +++ b/docs/plans/revision-write-path.md @@ -56,6 +56,40 @@ path uses. Proto: `src/AVEVA.Historian.Client/Grpc/Protos/TransactionService.pro matrix confirms they still ride the storage-engine pipe. The gRPC unlock here is original backfill, not after-the-fact edits. +### R3.1 decode probe (2026-06-21): `AddNonStreamValues` reaches the server-side storage-engine console pipe + +The `btInput` VTQ buffer is assembled in native C++ (`SendNonStreamedValues(batchID)` → a vtable +call after values are pooled via native `AddNonStreamedValueAsync(&HISTORIAN_VALUE2)`) and is **not +visible in any decompile** — only the 44-byte packed `HISTORIAN_VALUE2` struct is (TagKey@0, +FILETIME@4, OpcQuality@20, Type@24=7 numeric, value@33, bVersioned@41, VersionStatus@42). So the +framing was probed empirically against the live server with `grpc-nonstream-decode` (every +transaction `bCommit=false` → rolled back, nothing written; tag key from `SysTimeSec`). + +**Result — the failure is NOT a buffer-format problem:** six different framings (44–54 bytes: +count-prefixed packed struct, struct-only, version+count, OS-wrapped) all returned the **identical** +`AddNonStreamValues` error, while an empty buffer returned a *different* error (`04 01 00 00 00`, +InvalidParameter). The shared error is a nested `SError` whose detail strings are decisive: + +``` +aahClientAccessPoint::CHistStorageConnection::StoreNonStreamValues::StoreNonStreamValues +\\.\pipe\aahStorageEngine\console,sid() +``` + +So non-empty buffers get **past parameter validation into `StoreNonStreamValues`**, which routes to +the **`aahStorageEngine` console named pipe** server-side (the same storage engine as D2 — but the +gRPC *server* now holds the pipe, not the client). Because the error is identical across every +framing, the blocker is **not** the `btInput` layout — it is a **missing storage-engine console +session / tag-registration precondition** for the connection. + +**Next step to finish M3 (untested):** establish the StorageService side **before** +`AddNonStreamValues` — `StorageService.OpenStorageConnection`/`OpenStorageConnection2` to open the +console session, then register the tag→storage mapping (`RegisterTags` / `AddTagidPairs` / +`AddShardTagids`), then retry `AddNonStreamValues` and finally `End(bCommit=true)` + SQL read-back on +a sandbox tag. Each of those StorageService ops has its own buffer format to RE. Raw decode artifact: +`artifacts/reverse-engineering/grpc-nonstream-decode/batch1-decode.txt` (gitignored). Probe command: +`grpc-nonstream-decode`; driver: `HistorianGrpcRevisionProbe.ProbeNonStreamedBuffersAsync` (candidate +guess-bytes live in the RE tool, not `src/`). + --- ## Legacy WCF analysis (preserved — still accurate for the 2020 WCF transport) diff --git a/src/AVEVA.Historian.Client/Grpc/HistorianGrpcRevisionProbe.cs b/src/AVEVA.Historian.Client/Grpc/HistorianGrpcRevisionProbe.cs index cbff48e..e82e287 100644 --- a/src/AVEVA.Historian.Client/Grpc/HistorianGrpcRevisionProbe.cs +++ b/src/AVEVA.Historian.Client/Grpc/HistorianGrpcRevisionProbe.cs @@ -33,6 +33,106 @@ internal sealed class HistorianGrpcRevisionProbe public Task ProbeBeginAsync(CancellationToken cancellationToken) => Task.Run(() => ProbeBegin(cancellationToken), cancellationToken); + /// + /// Empirical-decode driver for the AddNonStreamValues btInput buffer (R3.1). For + /// each candidate buffer it opens a fresh transaction, sends the buffer, records the server's + /// accept/reject, and ALWAYS ends with bCommit=false (rollback) so nothing persists. + /// The candidate buffers are supplied by the caller (the RE tool) — this method does not invent + /// wire bytes, it just reports what the live server says about each. Safe against a real tag key + /// because every transaction is discarded. + /// + public Task> ProbeNonStreamedBuffersAsync( + IReadOnlyList<(string Label, byte[] Buffer)> candidates, + CancellationToken cancellationToken) + => Task.Run>( + () => ProbeNonStreamedBuffers(candidates, cancellationToken), cancellationToken); + + private List ProbeNonStreamedBuffers( + IReadOnlyList<(string Label, byte[] Buffer)> candidates, + CancellationToken cancellationToken) + { + var results = new List(); + + using HistorianGrpcConnection connection = HistorianGrpcChannelFactory.Create(_options); + HistorianGrpcHandshake.Session session = HistorianGrpcHandshake.OpenSession( + connection, _options, cancellationToken, + connectionMode: HistorianWcfAuthChainHelper.NativeIntegratedWriteEnabledConnectionMode); + + var transactionClient = new GrpcTransaction.TransactionService.TransactionServiceClient(connection.Channel); + string handle = session.StringHandle; + DateTime Deadline() => DateTime.UtcNow.Add(_options.RequestTimeout); + + // Prime the Transaction service session table. + try + { + transactionClient.GetTransactionInterfaceVersion( + new GrpcTransaction.GetTransactionInterfaceVersionRequest(), connection.Metadata, Deadline(), cancellationToken); + } + catch { /* version prime is best-effort */ } + + foreach ((string label, byte[] buffer) in candidates) + { + var candidate = new HistorianGrpcNonStreamedCandidateResult { Label = label, BufferLength = buffer.Length }; + string? transactionId = null; + try + { + GrpcTransaction.AddNonStreamValuesBeginResponse begin = transactionClient.AddNonStreamValuesBegin( + new GrpcTransaction.AddNonStreamValuesBeginRequest { StrHandle = handle }, + connection.Metadata, Deadline(), cancellationToken); + if (!(begin.Status?.BSuccess ?? false) || string.IsNullOrEmpty(begin.StrTransactionId)) + { + candidate.BeginFailed = true; + byte[] be = begin.Status?.BtError?.ToByteArray() ?? []; + candidate.AddErrorHex = be.Length == 0 ? null : Convert.ToHexString(be); + results.Add(candidate); + continue; + } + + transactionId = begin.StrTransactionId; + + GrpcTransaction.AddNonStreamValuesResponse add = transactionClient.AddNonStreamValues( + new GrpcTransaction.AddNonStreamValuesRequest + { + StrHandle = handle, + StrTransactionId = transactionId, + BtInput = ByteString.CopyFrom(buffer), + }, + connection.Metadata, Deadline(), cancellationToken); + + candidate.AddSucceeded = add.Status?.BSuccess ?? false; + byte[] ae = add.Status?.BtError?.ToByteArray() ?? []; + candidate.AddErrorHex = ae.Length == 0 ? null : Convert.ToHexString(ae); + } + catch (Exception ex) + { + candidate.Exception = $"{ex.GetType().Name}: {ex.Message}"; + } + finally + { + // Always roll back — bCommit=false writes nothing. + if (!string.IsNullOrEmpty(transactionId)) + { + try + { + transactionClient.AddNonStreamValuesEnd( + new GrpcTransaction.AddNonStreamValuesEndRequest + { + StrHandle = handle, + StrTransactionId = transactionId, + BCommit = false, + }, + connection.Metadata, Deadline(), cancellationToken); + } + catch { /* rollback best-effort */ } + } + } + + results.Add(candidate); + } + + return results; + } + private HistorianGrpcRevisionProbeResult ProbeBegin(CancellationToken cancellationToken) { var result = new HistorianGrpcRevisionProbeResult(); @@ -155,3 +255,13 @@ internal sealed class HistorianGrpcRevisionBeginAttempt public string? ErrorHex { get; set; } public string? Exception { get; set; } } + +internal sealed class HistorianGrpcNonStreamedCandidateResult +{ + public string Label { get; set; } = ""; + public int BufferLength { get; set; } + public bool BeginFailed { get; set; } + public bool AddSucceeded { get; set; } + public string? AddErrorHex { get; set; } + public string? Exception { get; set; } +} diff --git a/tools/AVEVA.Historian.ReverseEngineering/Program.cs b/tools/AVEVA.Historian.ReverseEngineering/Program.cs index 76168f8..ea6b57d 100644 --- a/tools/AVEVA.Historian.ReverseEngineering/Program.cs +++ b/tools/AVEVA.Historian.ReverseEngineering/Program.cs @@ -75,6 +75,7 @@ try "wcf-add-event-tag" => AddEventTagAndStartQuery(args), "capture-tag-info" => CaptureTagInfo(args), "grpc-revision-probe" => ProbeGrpcRevision(args), + "grpc-nonstream-decode" => ProbeGrpcNonStreamedDecode(args), _ => UnknownCommand(args[0]) }; } @@ -3247,6 +3248,133 @@ static int ProbeGrpcRevision(string[] args) return result.BeginSucceeded ? 0 : 2; } +static int ProbeGrpcNonStreamedDecode(string[] args) +{ + // Usage: grpc-nonstream-decode [port] [--tls] [--dnsid ] [--tag ] + // Empirically decodes the AddNonStreamValues btInput framing: looks up a real tag key, then + // sends evidence-based candidate buffers over a live write-enabled gRPC transaction and reports + // the server's accept/reject for each. Every transaction is rolled back (bCommit=false) — no + // data is written. Candidates are derived from the decompiled 44-byte packed HISTORIAN_VALUE2. + string host = args.Length > 1 ? args[1] : "localhost"; + int port = args.Length > 2 && int.TryParse(args[2], out int parsedPort) + ? parsedPort + : HistorianClientOptions.DefaultGrpcPort; + bool tls = HasOption(args, "--tls"); + string? dnsId = GetOption(args, "--dnsid"); + string tagName = GetOption(args, "--tag") ?? "SysTimeSec"; + + string? user = Environment.GetEnvironmentVariable("HISTORIAN_USER"); + string? password = Environment.GetEnvironmentVariable("HISTORIAN_PASSWORD"); + bool explicitCreds = !string.IsNullOrEmpty(user); + + var options = new HistorianClientOptions + { + Host = host, + Port = port, + Transport = HistorianTransport.RemoteGrpc, + GrpcUseTls = tls, + AllowUntrustedServerCertificate = tls, + ServerDnsIdentity = dnsId, + IntegratedSecurity = !explicitCreds, + UserName = user ?? string.Empty, + Password = password ?? string.Empty, + }; + + var client = new HistorianClient(options); + AVEVA.Historian.Client.Models.HistorianTagMetadata? metadata = + client.GetTagMetadataAsync(tagName, CancellationToken.None).GetAwaiter().GetResult(); + if (metadata is null) + { + Console.Error.WriteLine($"Tag '{tagName}' not found on the server; cannot resolve a tag key."); + return 1; + } + + if (metadata.Key is not uint tagKey) + { + Console.Error.WriteLine($"Tag '{tagName}' metadata has no tag key."); + return 1; + } + + // A historical timestamp ~2 hours in the past (non-streamed = backfill of past data). + long fileTime = DateTime.UtcNow.AddHours(-2).ToFileTimeUtc(); + const short opcQualityGood = 192; + double sampleValue = 123.0; + + byte[] BuildHistorianValue2(byte[] value8) + { + byte[] v = new byte[44]; + BinaryPrimitives.WriteUInt32LittleEndian(v.AsSpan(0, 4), tagKey); + BinaryPrimitives.WriteInt64LittleEndian(v.AsSpan(4, 8), fileTime); + BinaryPrimitives.WriteInt16LittleEndian(v.AsSpan(20, 2), opcQualityGood); + BinaryPrimitives.WriteInt32LittleEndian(v.AsSpan(24, 4), 7); // Type = numeric + // @28 u32 MaxLength = 0 (numeric); @32 ApplyScaling = 0 + value8.AsSpan(0, 8).CopyTo(v.AsSpan(33, 8)); // @33 value (8 bytes, unaligned) + // @41 bVersioned = 0; @42 VersionStatus = 0 + return v; + } + + byte[] valueAsDoubleBits = BitConverter.GetBytes(sampleValue); // 8 bytes, double + byte[] valueAsFloatLow = new byte[8]; + BitConverter.GetBytes((float)sampleValue).CopyTo(valueAsFloatLow, 0); // float in low 4 + + byte[] structDouble = BuildHistorianValue2(valueAsDoubleBits); + byte[] structFloat = BuildHistorianValue2(valueAsFloatLow); + + byte[] WithCountU32(byte[] body, uint count) + { + byte[] b = new byte[4 + body.Length]; + BinaryPrimitives.WriteUInt32LittleEndian(b.AsSpan(0, 4), count); + body.CopyTo(b.AsSpan(4)); + return b; + } + + byte[] WithVersionAndCount(byte[] body, ushort version, uint count) + { + byte[] b = new byte[2 + 4 + body.Length]; + BinaryPrimitives.WriteUInt16LittleEndian(b.AsSpan(0, 2), version); + BinaryPrimitives.WriteUInt32LittleEndian(b.AsSpan(2, 4), count); + body.CopyTo(b.AsSpan(6)); + return b; + } + + // "OS"-style storage-sample header (as AddS2 uses), wrapping the packed struct as the blob. + byte[] OsWrap(byte[] body) + { + byte[] b = new byte[10 + body.Length]; + BinaryPrimitives.WriteUInt16LittleEndian(b.AsSpan(0, 2), 0x534F); // "OS" + BinaryPrimitives.WriteUInt16LittleEndian(b.AsSpan(2, 2), 1); // sampleCount + BinaryPrimitives.WriteUInt32LittleEndian(b.AsSpan(4, 4), (uint)(body.Length + 1)); + BinaryPrimitives.WriteUInt16LittleEndian(b.AsSpan(8, 2), (ushort)body.Length); + body.CopyTo(b.AsSpan(10)); + return b; + } + + var candidates = new List<(string Label, byte[] Buffer)> + { + ("count(u32)+struct[double@33]", WithCountU32(structDouble, 1)), + ("count(u32)+struct[float@33]", WithCountU32(structFloat, 1)), + ("struct-only[double@33]", structDouble), + ("ver(u16=0)+count(u32)+struct[double]", WithVersionAndCount(structDouble, 0, 1)), + ("ver(u16=2)+count(u32)+struct[double]", WithVersionAndCount(structDouble, 2, 1)), + ("OS-wrap(struct[double])", OsWrap(structDouble)), + ("empty", Array.Empty()), + }; + + var probe = new HistorianGrpcRevisionProbe(options); + IReadOnlyList results = + probe.ProbeNonStreamedBuffersAsync(candidates, CancellationToken.None).GetAwaiter().GetResult(); + + Console.WriteLine(JsonSerializer.Serialize(new + { + Tag = tagName, + TagKey = tagKey, + FileTimeUtc = DateTime.FromFileTimeUtc(fileTime).ToString("o"), + Candidates = results, + }, CreateJsonOptions())); + + return results.Any(static r => r.AddSucceeded) ? 0 : 2; +} + static int ProbeWcf(string[] args) { string host = args.Length > 1 ? args[1] : "localhost"; From ac28679a1fea1cf11a11fc7500807c104802794c Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Sun, 21 Jun 2026 18:12:40 -0400 Subject: [PATCH 3/4] M3 R3.1: map the required non-streamed write sequence (OpenStorageConnection is the missing step) Static decompile mining of the 2023 R2 client corroborates the live R3.1 error: the AddNonStreamValues failure is the missing StorageService.OpenStorageConnection, which creates exactly the \.\pipe\aahStorageEngine\console,sid(...) session named in the server error. Mapped the full native sequence: HistoryService.OpenConnection (have) -> StorageService.OpenStorageConnection (MISSING) -> StorageService.RegisterTags -> AddNonStreamValuesBegin (works) -> AddNonStreamValues(btInput) (fails - no console session) -> End(commit). Two hard parts remain, each a live-production decode loop with no static shortcut: (1) reproduce the 12-arg OpenStorageConnection handshake (several args inferred); (2) decode the AddNonStreamValues btInput (C++-built, absent from decompiles; only the 44-byte packed HISTORIAN_VALUE2 is known). Documented in revision-write-path.md. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01B6mcaT2PjRFKcogzp9UkfC --- docs/plans/revision-write-path.md | 44 +++++++++++++++++++++++++------ 1 file changed, 36 insertions(+), 8 deletions(-) diff --git a/docs/plans/revision-write-path.md b/docs/plans/revision-write-path.md index 922a238..c8cbd13 100644 --- a/docs/plans/revision-write-path.md +++ b/docs/plans/revision-write-path.md @@ -81,14 +81,42 @@ gRPC *server* now holds the pipe, not the client). Because the error is identica framing, the blocker is **not** the `btInput` layout — it is a **missing storage-engine console session / tag-registration precondition** for the connection. -**Next step to finish M3 (untested):** establish the StorageService side **before** -`AddNonStreamValues` — `StorageService.OpenStorageConnection`/`OpenStorageConnection2` to open the -console session, then register the tag→storage mapping (`RegisterTags` / `AddTagidPairs` / -`AddShardTagids`), then retry `AddNonStreamValues` and finally `End(bCommit=true)` + SQL read-back on -a sandbox tag. Each of those StorageService ops has its own buffer format to RE. Raw decode artifact: -`artifacts/reverse-engineering/grpc-nonstream-decode/batch1-decode.txt` (gitignored). Probe command: -`grpc-nonstream-decode`; driver: `HistorianGrpcRevisionProbe.ProbeNonStreamedBuffersAsync` (candidate -guess-bytes live in the RE tool, not `src/`). +**Required call sequence (mapped from the 2023 R2 decompile, corroborates the error above):** the +missing precondition is **`StorageService.OpenStorageConnection`** — it creates exactly the +`\\.\pipe\aahStorageEngine\console,sid(...)` console session named in the failure. The native +non-streamed write path is: + +``` +HistoryService.OpenConnection (✅ have it — the Open2 handshake) + → StorageService.OpenStorageConnection (⛔ MISSING — opens the console sid session; SEPARATE + storage session, returns its own uint handle + new GUID) + → StorageService.RegisterTags (register the tag→storage mapping for the session) + → TransactionService.AddNonStreamValuesBegin (✅ works) + → TransactionService.AddNonStreamValues(btInput) (⛔ currently fails here — no console session yet) + → TransactionService.AddNonStreamValuesEnd(bCommit=true) + → StorageService.CloseStorageConnection / HistoryService.CloseConnection +``` + +`OpenStorageConnection` (gRPC `StorageService`) takes 12 args — HostName, EnginePath +(`\\.\pipe\aahStorageEngine\console`), FreeDiskSpace, ProcessName, ProcessId, UserName, Password(+len), +ClientType, ClientVersion, ConnectionMode, ConnectionTimeout, StorageSessionId(in/out) — and returns a +**new** storage `Handle` (uint) + a **new** StorageSessionId GUID (distinct from the Open2 GUID). + +**Two hard parts remain, each a separate live-production decode loop (no static shortcut):** +1. **Reproduce the `OpenStorageConnection` handshake** — several of the 12 args are only inferable from + the decompile (ProcessId, ClientType/Version, ConnectionMode, the password-bytes framing), so the + exact values must be confirmed against the live server. +2. **Decode the `AddNonStreamValues` `btInput`** — built in C++ (`SendNonStreamedValues` vtable call), + **absent from every decompile**; only the 44-byte packed `HISTORIAN_VALUE2` struct is known. Must be + decoded empirically once the console session exists (the batch-1 identical-error result could not + distinguish framings precisely *because* there was no session — with a session, framings should + diverge and the correct one becomes findable). + +Raw decode artifact: `artifacts/reverse-engineering/grpc-nonstream-decode/batch1-decode.txt` +(gitignored). Probe command: `grpc-nonstream-decode`; driver: +`HistorianGrpcRevisionProbe.ProbeNonStreamedBuffersAsync` (candidate guess-bytes live in the RE tool, +not `src/`). **Status: M3 transaction lifecycle proven; full insert blocked on the +OpenStorageConnection handshake + btInput decode — a focused follow-up, each step a live probe.** --- From 1a08dd9ec25e84f39404f8197a4a7c5a11c2f917 Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Sun, 21 Jun 2026 18:20:35 -0400 Subject: [PATCH 4/4] M3: roadmap reflects mapped non-streamed sequence (OpenStorageConnection follow-up) Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01B6mcaT2PjRFKcogzp9UkfC --- docs/plans/hcal-roadmap.md | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/docs/plans/hcal-roadmap.md b/docs/plans/hcal-roadmap.md index fe8e2bd..32dcfac 100644 --- a/docs/plans/hcal-roadmap.md +++ b/docs/plans/hcal-roadmap.md @@ -254,13 +254,14 @@ byte-correct `AddS2` (✅). Appears-and-reads-back is environment-gated on event | ID | Work | gRPC op | Status | |---|---|---|---| -| R3.1 | Decode non-streamed VTQ packet | `Transaction.AddNonStreamValuesBegin/AddNonStreamValues/End` | 🟡 **gRPC transaction Begin/End LIVE-VERIFIED 2026-06-21** (WCF still blocked — D2). Remaining: capture the `AddNonStreamValues` `btInput` VTQ buffer (don't guess) | -| R3.2 | `AddHistoricalValuesAsync` | batched begin→values→end | 🟡 unblocked by R3.1's gRPC proof; needs the `btInput` serializer + a real `bCommit=true` write/read-back | +| R3.1 | Decode non-streamed VTQ packet | `Transaction.AddNonStreamValuesBegin/AddNonStreamValues/End` | 🟡 **gRPC Begin/End LIVE-VERIFIED 2026-06-21; full sequence MAPPED** (WCF still blocked — D2). Live decode showed `AddNonStreamValues` reaches server `StoreNonStreamValues` → `\\.\pipe\aahStorageEngine\console` and fails for lack of a console session. Remaining (follow-up): `StorageService.OpenStorageConnection` handshake + `RegisterTags`, THEN the `btInput` decode. See [`revision-write-path.md`](revision-write-path.md) §R3.1. | +| R3.2 | `AddHistoricalValuesAsync` | batched begin→values→end | 🟡 unblocked architecturally; needs R3.1's two live decode loops (OpenStorageConnection handshake + `btInput` serializer) then a real `bCommit=true` write/read-back | | R3.3 | Ingest-permission validation | confirm the target accepts original-data insert (distinct from `AddS2` cache wall) | ✅ **distinct on gRPC** — Begin succeeded against a real write-enabled session (the WCF/native cache gate does not apply here) | **Acceptance:** historical points inserted and read back. **WCF path closed (D2).** gRPC path: -**transaction lifecycle proven (Begin/End live)**; full insert+read-back pending the `btInput` -capture + serializer. +**transaction lifecycle proven (Begin/End live) + full sequence mapped**; the remaining insert is a +focused follow-up — reproduce `StorageService.OpenStorageConnection` (+ `RegisterTags`), then decode +the `btInput` VTQ buffer, each a live-production probe loop. --- @@ -329,5 +330,5 @@ event-send). M3/M4 as demand dictates. | M0 gRPC parity + capture tooling | foundation | M | unblocks everything, Windows-free | ✅ **done** | | M1 cheap surface | TRIVIAL/BOUNDED | M–L | most remaining read/config | ✅ **done** (reachable surface; rest bounded out) | | M2 event send | CAPTURE | S–M | headline write capability | ✅ **done** | -| M3 historical writes | BOUNDED | M | backfill | 🟡 **gRPC transaction Begin/End live-verified (2026-06-21)**; WCF blocked (D2). Remaining: `btInput` capture → commit+read-back | +| M3 historical writes | BOUNDED | M | backfill | 🟡 **gRPC Begin/End live-verified + full sequence mapped (2026-06-21)**; WCF blocked (D2). Follow-up: OpenStorageConnection handshake + `btInput` decode → commit+read-back | | M4 SF / revisions / redundancy | HARD | L×N | parity completeness | defer (R4.2 = same pipe wall) |