dohertj2/histsdk
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

M3 probe: non-streamed write transaction reachable over 2023 R2 gRPC (Begin/End live-verified)

Browse Source 
The D2 storage-engine-pipe wall is WCF-transport-specific. On the 2023 R2 gRPC
front door, TransactionService is a first-class service AND the gateway to the
storage engine, so the Open2 storage-session GUID (uppercase) is accepted
directly as strHandle with no legacy pipe.

Live-verified against the real 2023 R2 server over a write-enabled (0x401) gRPC
session: AddNonStreamValuesBegin returns a real strTransactionId, and
AddNonStreamValuesEnd(bCommit=false) discards it cleanly (no data written). On
2020 WCF the same op returns UnknownClient(51) for every handle + priming chain.

- HistorianGrpcRevisionProbe + grpc-revision-probe CLI command + gated test
  NonStreamedWriteTransaction_OverGrpc_BeginsAndDiscards (live pass).
- HistorianGrpcHandshake.OpenSession gains an optional connectionMode param
  (default read-only 0x402; pass 0x401 for write-enabled) — non-breaking.
- Docs: revision-write-path.md "the wall is gone" section; roadmap M3 section,
  R3.1-R3.3 rows, one-glance table, and status note updated honestly.

Not yet shipped: the AddNonStreamValues btInput VTQ buffer is uncaptured (never
guess wire bytes), so no value-commit is implemented. Scope is non-streamed
ORIGINAL backfill; revision EDITS (R4.2) remain pipe-only even on gRPC.

272 unit tests pass; sanitization scan clean.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B6mcaT2PjRFKcogzp9UkfC
This commit is contained in:
Joseph Doherty
2026-06-21 17:51:17 -04:00
parent 04ea0b9a1f
commit 23798db1ef
6 changed files with 319 additions and 17 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/plans/hcal-roadmap.md
+29 -14
View File
@@ -166,8 +166,8 @@ read/browse/status surface is Windows-free and the gRPC stack is the default pat
| ~~R1.5~~ | Extended-property **read** | `Retrieval.GetTagExtendedPropertiesFromName` (`GetTepByNm`) | ✅ **DONE (2026-06-20), live-verified.** `GetTagExtendedPropertiesAsync(tag)` → name/value pairs. String-handle op via the uppercase storage GUID; name-based path (`GetTagExtendedPropertiesByName`, not the QTB-gated TagQuery path). Request `tagNames` = `uint count` + per-name(`uint charCount`+UTF-16); response = `uint tagCount` + per-tag(marker + compact-ASCII name + `uint propCount` + per-prop(marker + compact-ASCII name + `0x43` VT_BSTR value) + trailer). Sequence-paged. Shipped: `HistorianTagExtendedPropertyProtocol`, golden `WcfTagExtendedPropertyProtocolTests`, gated live test. See `docs/reverse-engineering/wcf-tag-extended-properties.md`. | uppercase string handle |
| ~~R1.6~~ | Localized-property **read** | (no op) | ⛔ **No distinct op on 2020 — collapses into R1.5.** There is no `GetTagLocalizedPropertiesFromName`/`GetTlpByNm` or `GetTagLocalizedPropertiesByName` in `current/aahClientManaged.dll`; the only "localized" surfaces are error-message/UI-text localization. Extended properties (R1.5) are the user-defined tag-property read surface. Closed, not throwing. | — |
| ~~R1.7~~ | Event **filters** | filter bytes in `Retrieval.StartEventQuery` | ✅ **DONE (2026-06-20), live-honored.** `ReadEventsAsync(start, end, HistorianEventFilter)`. The filter rides `StartEventQuery`'s `pRequestBuff` (captured via `EventQuery.AddEventFilter` + instrument-wcf-writemessage; Equal vs Contains diffed to isolate the op). Filter block: `ushort 0 + uint filterCount + uint condCount + uint nameLen + name(UTF-16) + uint 1 + ushort op + uint 1 + value(0x09-len-0x00 compact-ASCII) + byte 0`. **REAL, not inert** (a non-matching predicate returns 0 events; matching returns the subset). Single string-valued predicate only; multi-filter (OR) / multi-condition (AND via `AddEventFilterCondition`) framing not yet fully captured. See `HistorianEventFilter`, golden `WcfEventQueryProtocolTests`. | — |
| R1.8 | Analog-summary query | `Retrieval.StartQuery` (summary mode) | summary row layout — **`uint`-handle, reachable. Scoped + decode targets located** (`CAnalogSummaryValue.UnpackFromValueBuffer`, fields Min/Max/First/Last/ValueCount/Integral/…). Plan: [`r1.8-r1.9-summary-queries.md`](r1.8-r1.9-summary-queries.md) | — |
| R1.9 | State-summary query | `Retrieval.StartQuery` (state mode) | state-summary row layout — **`uint`-handle, reachable. Scoped** (`CStateSummaryStruct`: MinContained/MaxContained/TotalContained/PartialStart/PartialEnd/StateEntryCount). Plan: [`r1.8-r1.9-summary-queries.md`](r1.8-r1.9-summary-queries.md) | — |
| ~~R1.8~~ | Analog-summary query | `Retrieval.StartQuery` (summary mode) | ✅ **RESOLVED (2026-06-21) — no new code; == existing `ReadAggregateAsync`.** Request + response both captured (`scripts/Capture-SummaryRequest.ps1 -WithResponse`): the `GetNextQueryResultBuffer2` response is the **ordinary version-9 row buffer** the raw/aggregate parser already handles (decoded 7 rows = SQL ground truth exactly). There is **no rich `CAnalogSummaryValue` struct on the wire** — each row carries a *single* value selected by `RetrievalMode`/QueryType (Integral→8, TimeWeightedAverage→5, …), not an all-aggregates-in-one row; `ValueSelector`/`AggregationType`/`MaxStates` are **inert** on the WCF retrieval path (they configure the SQL provider, not this query). The all-aggregates-at-once shape is the SQL/OLEDB provider's, or the gRPC front door — not 2020 WCF binary. Plan + capture evidence: [`r1.8-r1.9-summary-queries.md`](r1.8-r1.9-summary-queries.md). | — |
| ~~R1.9~~ | State-summary query | `Retrieval.StartQuery` (state mode) | ✅ **RESOLVED (2026-06-21) — same finding as R1.8.** State-summary is the **same `StartQuery2` request** (only `MaxStates`/defaults differ on the wire); the response carries no distinct `CStateSummaryStruct` on the 2020 WCF binary path. Covered by the existing aggregate read; no new `src/` code warranted. Plan: [`r1.8-r1.9-summary-queries.md`](r1.8-r1.9-summary-queries.md). | — |
### 1c. Bounded config writes (S–M each)
| ID | Capability | gRPC op | Payload | Notes |
@@ -221,9 +221,21 @@ byte-correct `AddS2` (✅). Appears-and-reads-back is environment-gated on event
*Goal: insert original historical VTQs (backfill), the path that is NOT the gated cache push.*
> ✅ **gRPC UNLOCK (2026-06-21, LIVE-VERIFIED): the transaction lifecycle is REACHABLE over the
> 2023 R2 gRPC front door.** The `grpc-revision-probe` opened a **write-enabled** (`0x401`) gRPC
> session and drove `TransactionService.AddNonStreamValuesBegin(storage-GUID **uppercase**)` →
> real `strTransactionId` → `AddNonStreamValuesEnd(bCommit=false)` (discarded, no data written).
> Where 2020 WCF returns `UnknownClient (51)`, the gRPC `TransactionService` is itself the gateway
> to the storage engine, so the Open2 session GUID is accepted directly — **no legacy pipe**. This
> answers the M3-over-gRPC question below: **yes**, the non-streamed *original* write transaction is
> reachable from the pure-managed SDK. **Not yet shipped:** the `AddNonStreamValues` `btInput` VTQ
> buffer must be captured before any value-commit (never guess wire bytes); revision *edits* (R4.2)
> remain pipe-only even on gRPC. Full detail + decompile basis:
> [`revision-write-path.md`](revision-write-path.md) §"2023 R2 gRPC — the wall is gone".
>
> ⛔ **BLOCKED on 2020 WCF — re-confirmed by the D2 probe (2026-05-05), see
> [`revision-write-path.md`](revision-write-path.md).** The premise above ("the path that is NOT
> the gated cache push") was **disproved**: R3.1's op
> the gated cache push") was **disproved** *on WCF*: R3.1's op
> (`Transaction.AddNonStreamValuesBegin/AddNonStreamValues/End`) is the **same**
> `ITransactionServiceContract2.AddNonStreamValuesBegin2` D2 probed, and over WCF it returns
> `04 33 00 00 00` = `UnknownClient (51)` for every handle format **and** the full priming chain
@@ -242,12 +254,13 @@ byte-correct `AddS2` (✅). Appears-and-reads-back is environment-gated on event
| ID | Work | gRPC op | Status |
|---|---|---|---|
| R3.1 | Decode non-streamed VTQ packet | `Transaction.AddNonStreamValuesBegin/AddNonStreamValues/End` | ⛔ WCF blocked (storage-engine pipe — D2). gRPC: untested |
| R3.2 | `AddHistoricalValuesAsync` | batched begin→values→end | ⛔ gated on R3.1 |
| R3.3 | Ingest-permission validation | confirm the target accepts original-data insert (distinct from `AddS2` cache wall) | ⛔ proven to share the same gate, not distinct |
| R3.1 | Decode non-streamed VTQ packet | `Transaction.AddNonStreamValuesBegin/AddNonStreamValues/End` | 🟡 **gRPC transaction Begin/End LIVE-VERIFIED 2026-06-21** (WCF still blocked — D2). Remaining: capture the `AddNonStreamValues` `btInput` VTQ buffer (don't guess) |
| R3.2 | `AddHistoricalValuesAsync` | batched begin→values→end | 🟡 unblocked by R3.1's gRPC proof; needs the `btInput` serializer + a real `bCommit=true` write/read-back |
| R3.3 | Ingest-permission validation | confirm the target accepts original-data insert (distinct from `AddS2` cache wall) | ✅ **distinct on gRPC** — Begin succeeded against a real write-enabled session (the WCF/native cache gate does not apply here) |
**Acceptance:** historical points inserted and read back. **WCF path closed (D2);** would require
the gRPC write path (live 2023 R2 server + capture) to reopen.
**Acceptance:** historical points inserted and read back. **WCF path closed (D2).** gRPC path:
**transaction lifecycle proven (Begin/End live)**; full insert+read-back pending the `btInput`
capture + serializer.
---
@@ -301,11 +314,13 @@ event-send). M3/M4 as demand dictates.
> **Status 2026-06-21:** sprints 1 + 2 are **complete** (M0 gRPC parity, the reachable M1 surface,
> and M2 event-send all shipped + live-verified; remaining M1 items are evidence-bounded-out). The
> reachable surface on the **available 2020 WCF infrastructure is exhausted** — every remaining
> roadmap item is now either (a) blocked by the storage-engine-pipe architecture (**M3-WCF**, R4.2),
> (b) **gRPC/2023R2-only** and needs the live 2023 R2 server for a native capture (R1.3 timezone,
> R1.4 EventStorageMode, M3/revisions over gRPC), or (c) a HARD deferred subsystem (M4). No further
> work lands without one of: a live-2023R2 capture session, or a customer-demand trigger.
> reachable surface on the **available 2020 WCF infrastructure is exhausted**. **M3 update
> (2026-06-21):** with the live 2023 R2 server, the **M3 non-streamed write transaction is now
> proven reachable over gRPC** — `TransactionService.AddNonStreamValuesBegin/End` round-trips live
> (the D2 storage-engine-pipe wall is WCF-only). The remaining M3 work is bounded and concrete:
> capture the `AddNonStreamValues` `btInput` VTQ buffer → golden-tested serializer → real
> commit+read-back → public `AddHistoricalValuesAsync`. The other levers are unchanged: R4.2 revision
> *edits* stay pipe-only even on gRPC, and M4 (SF / redundancy) is a HARD deferred subsystem.
## One-glance status
@@ -314,5 +329,5 @@ event-send). M3/M4 as demand dictates.
| M0 gRPC parity + capture tooling | foundation | M | unblocks everything, Windows-free | ✅ **done** |
| M1 cheap surface | TRIVIAL/BOUNDED | M–L | most remaining read/config | ✅ **done** (reachable surface; rest bounded out) |
| M2 event send | CAPTURE | S–M | headline write capability | ✅ **done** |
| M3 historical writes | BOUNDED | M | backfill | ⛔ WCF blocked (D2); gRPC = on-demand + live 2023R2 |
| M3 historical writes | BOUNDED | M | backfill | 🟡 **gRPC transaction Begin/End live-verified (2026-06-21)**; WCF blocked (D2). Remaining: `btInput` capture → commit+read-back |
| M4 SF / revisions / redundancy | HARD | L×N | parity completeness | defer (R4.2 = same pipe wall) |