From 0e78d638d0ac94e420a7b793a0d8a2ffc0429210 Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Sun, 21 Jun 2026 21:04:17 -0400 Subject: [PATCH] M3 R3.1: document the captured + validated AddStreamValues "ON" write path MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit revision-write-path.md Β§"R3.1 CAPTURED" + roadmap R3.1/R3.2/one-glance now record the validated finding: the historical write is HistoryService.AddStreamValues ("ON" storage-sample buffer, AddS2 "OS" family) + EnsureTags, not AddNonStreamValues/TransactionService. Includes the decoded 56-byte "ON" buffer layout, the working priming/batch sequence, the tag-GUID keying, and that the D2 cache gate does not block the primed 2023 R2 client. Remaining work to ship AddHistoricalValuesAsync is the managed "ON" serializer (adapt HistorianEventWriteProtocol) + gRPC orchestrator wiring. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01B6mcaT2PjRFKcogzp9UkfC --- docs/plans/hcal-roadmap.md | 6 ++--- docs/plans/revision-write-path.md | 37 +++++++++++++++++++++++++++++++ 2 files changed, 40 insertions(+), 3 deletions(-) diff --git a/docs/plans/hcal-roadmap.md b/docs/plans/hcal-roadmap.md index 9528c50..f06788d 100644 --- a/docs/plans/hcal-roadmap.md +++ b/docs/plans/hcal-roadmap.md @@ -254,8 +254,8 @@ byte-correct `AddS2` (βœ…). Appears-and-reads-back is environment-gated on event | ID | Work | gRPC op | Status | |---|---|---|---| -| R3.1 | Decode non-streamed VTQ packet | `Transaction.AddNonStreamValuesBegin/AddNonStreamValues/End` | 🟑 **gRPC Begin/End LIVE-VERIFIED 2026-06-21; precondition CORRECTED 2026-06-21** (WCF still blocked β€” D2). The earlier "missing console session" step was **disproved live**: `StorageService.OpenStorageConnection` returns `type=4 code=85` ("session not registered") for every param β€” it's the storage engine's SF/snapshot channel (separate `GrpcStorageClient`/service identity), not a front-door client op. The real precondition is **front-door `HistoryService.RegisterTags`** (RTag2-family) for the target tag β€” the R3.1 batch failed at `AddNonStreamValues` *because the tag wasn't registered*. Remaining: capture the regular-tag `RegisterTags` `btTagInfos` (only CM_EVENT's tag-GUID form is known) + the `AddNonStreamValues` `btInput`. See [`revision-write-path.md`](revision-write-path.md) Β§R3.1 follow-up. | -| R3.2 | `AddHistoricalValuesAsync` | batched beginβ†’valuesβ†’end | 🟑 architecturally unblocked; precondition now correctly identified (front-door `RegisterTags`, not `OpenStorageConnection`); needs a native gRPC capture of the regular-tag `RegisterTags` `btTagInfos` + the `AddNonStreamValues` `btInput`, then a real `bCommit=true` write/read-back | +| R3.1 | Decode non-streamed VTQ packet | `History.AddStreamValues` ("ON" buffer) + `EnsureTags` | βœ… **CAPTURED + VALIDATED 2026-06-21.** Drove the native 2023 R2 client through a committed historical write (sandbox tag) with the IL-rewritten gRPC client dumping every `byte[]`; the value **read back over gRPC**. The path is **NOT** `AddNonStreamValues`/TransactionService β€” it's **`HistoryService.AddStreamValues`** with an **"ON" storage-sample buffer** (AddS2 "OS" family) + `EnsureTags`. Buffer decoded: `"ON"(0x4E4F) + u16 count + u32 totalLen + u16 payloadLen + 16B tag GUID + FILETIME + u16 quality + u32 type + FILETIME + 8B double`. D2 cache gate does NOT block the primed 2023 R2 client. See [`revision-write-path.md`](revision-write-path.md) Β§"R3.1 CAPTURED". | +| R3.2 | `AddHistoricalValuesAsync` | `History.AddStreamValues` ("ON") + `EnsureTags` | 🟑 **buffers captured + validated**; remaining: build the managed "ON" serializer in `src/` (adapt `HistorianEventWriteProtocol` "OS"), resolve tag GUID, reuse `EnsureTagAsync` CTagMetadata, wire `AddStreamValues` over the gRPC orchestrator, golden-test, then ship | | R3.3 | Ingest-permission validation | confirm the target accepts original-data insert (distinct from `AddS2` cache wall) | βœ… **distinct on gRPC** β€” Begin succeeded against a real write-enabled session (the WCF/native cache gate does not apply here) | **Acceptance:** historical points inserted and read back. **WCF path closed (D2).** gRPC path: @@ -330,5 +330,5 @@ event-send). M3/M4 as demand dictates. | M0 gRPC parity + capture tooling | foundation | M | unblocks everything, Windows-free | βœ… **done** | | M1 cheap surface | TRIVIAL/BOUNDED | M–L | most remaining read/config | βœ… **done** (reachable surface; rest bounded out) | | M2 event send | CAPTURE | S–M | headline write capability | βœ… **done** | -| M3 historical writes | BOUNDED | M | backfill | 🟑 **gRPC Begin/End live-verified (2026-06-21); precondition corrected** β€” front-door `HistoryService.RegisterTags` (not `OpenStorageConnection`, which is the SF-channel dead end / error 85). WCF blocked (D2). Follow-up: capture regular-tag `RegisterTags` `btTagInfos` + `AddNonStreamValues` `btInput` β†’ commit+read-back | +| M3 historical writes | BOUNDED | M | backfill | 🟑 **wire path CAPTURED + VALIDATED (2026-06-21)** β€” native historical write = `HistoryService.AddStreamValues` ("ON" buffer) + `EnsureTags` (NOT AddNonStreamValues/Transaction; NOT OpenStorageConnection). Committed write read back over gRPC. Remaining: build the "ON" serializer in `src/` + ship `AddHistoricalValuesAsync`. WCF still blocked (D2) | | M4 SF / revisions / redundancy | HARD | LΓ—N | parity completeness | defer (R4.2 = same pipe wall) | diff --git a/docs/plans/revision-write-path.md b/docs/plans/revision-write-path.md index 519195f..0dac986 100644 --- a/docs/plans/revision-write-path.md +++ b/docs/plans/revision-write-path.md @@ -213,6 +213,43 @@ IL-rewrite `Archestra.Historian.GrpcClient.dll`; (3) write-enabled run β†’ captu `tagInfos` (+ `btInput` if the gate passes); (4) build golden serializer(s) in `src/`; (5) real `bCommit=true` write + SQL read-back on a sandbox tag β†’ ship `AddHistoricalValuesAsync`. +### R3.1 CAPTURED + VALIDATED (2026-06-21): the write rides `HistoryService.AddStreamValues` ("ON" buffer) + +The capture ran end-to-end against the live server (`AVEVA.Historian.Grpc2023CaptureHarness`, +`capture-write` scenario, sandbox tag created by the harness, IL-rewritten `GrpcClient` dumping every +`byte[]`). The committed write **persisted and read back over gRPC** (SDK `ReadRawAsync` returned the +sample) β€” fully validated. + +**The roadmap's assumption was wrong.** The native non-streamed (historical backfill) write does **not** +use `AddNonStreamValues` / the TransactionService at all. The native `HistorianAccess.AddNonStreamedValue +β†’ SendValues` routes over gRPC as **`HistoryService.AddStreamValues`** carrying an **"ON" +storage-sample buffer** (structurally the AddS2 **"OS"** family β€” same serializer pattern the SDK already +has in `HistorianEventWriteProtocol`), preceded by **`EnsureTags`** to register the tag: + +``` +EnsureTags.tagInfos (144B) = the analog CTagMetadata the SDK's EnsureTagAsync already builds + (0x4E marker … fe 00 trailer) +AddStreamValues.values (56B) = "ON" (0x4E4F) + u16 sampleCount(1) + u32 totalLen(56) + + u16 payloadLen(46) + 16B tag GUID + FILETIME(sample) + + u16 OpcQuality(192=Good) + u32 type/descriptor + + FILETIME(received/version) + 8B double value +``` + +The full priming/write sequence that works from the native client (write-enabled session): `OpenConnection` +β†’ `UpdateClientStatus` Γ—N β†’ `EnsureTags` β†’ `GetTagInfosFromName` (resolve identity) β†’ `AddStreamValues` +("ON" buffer). Notes: (a) the **D2 cache gate (err 129) does NOT block** the primed 2023 R2 client β€” +`AddNonStreamedValue` returned success once the session was primed (via `AddTag`/`GetTagInfoByName`) and +the server had assigned the tag key; (b) the value is keyed by a **16-byte tag GUID**, not the uint +`tagKey` (so the SDK serializer needs the tag's GUID, available from EnsureTags/GetTagInfo, not just +`HistorianTagMetadata.Key`); (c) batch lifecycle is `NonStreamedValuesBegin β†’ AddNonStreamedValue β†’ +SendValues β†’ AddNonStreamedValuesEnd` (End-before-Send returns err 160 InvalidBatchId). + +**Remaining to ship `AddHistoricalValuesAsync`:** build the managed "ON" `AddStreamValues` serializer in +`src/` (adapt `HistorianEventWriteProtocol`'s "OS" builder), resolve the tag GUID, reuse the existing +`EnsureTagAsync` CTagMetadata, wire `HistoryService.AddStreamValues` over the gRPC orchestrator, golden-test +the buffer, then a real write + read-back on a sandbox tag. Capture artifacts (gitignored): +`artifacts/reverse-engineering/grpc-nonstream-capture/captureB4.ndjson`. + --- ## Legacy WCF analysis (preserved β€” still accurate for the 2020 WCF transport)