docs/plans: import 2023 R2 gRPC analysis + HCAL reimpl roadmap
Version-control the planning docs alongside the code they describe: - grpc-transport.md — 2023 R2 gRPC transport analysis (sanitized source path) - hcal-capability-matrix.md — HistorianAccess surface x gRPC ops x histsdk status x feasibility tiers - hcal-roadmap.md — ordered build plan M0-M4 + cross-cutting workstreams - histevents.md — how a HistorianEvent reaches the DB (client->wire->server) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Joseph Doherty
@@ -0,0 +1,166 @@
|
||||
# HCAL → modern-.NET reimplementation — capability matrix
|
||||
|
||||
Feasibility map for a clean managed-.NET client that replaces the AVEVA Historian
|
||||
SDK (`aahClientManaged` / HCAL). Grounded in: the real `ArchestrA.HistorianAccess`
|
||||
public surface (`aahClientManaged.xml`), the recovered **2023 R2 gRPC contract**, the
|
||||
existing **histsdk** reimplementation, and the event/storage analysis in
|
||||
[`histevents.md`](histevents.md).
|
||||
|
||||
## Legend
|
||||
|
||||
**Status (histsdk today)** — ✅ implemented + live-verified · 🟗 partial · ⬜ not yet
|
||||
|
||||
**Feasibility tier**
|
||||
| Tier | Meaning | Effort |
|
||||
|---|---|---|
|
||||
| **DONE** | already working in histsdk | 0 |
|
||||
| **TRIVIAL** | gRPC op known, payload already decoded or empty | XS (hrs) |
|
||||
| **CAPTURE** | one instrument-and-capture of a native payload, then serialize + golden-byte test | S (days) |
|
||||
| **BOUNDED** | gRPC op exists; decode one proprietary `bytes` payload | S–M |
|
||||
| **HARD** | whole subsystem to reimplement | L (weeks) |
|
||||
| **GATED** | blocked server-side — client effort doesn't unblock it | n/a |
|
||||
|
||||
Effort = incremental work on top of histsdk's existing infrastructure (auth chain,
|
||||
transport, frame/byte primitives, test harness). All non-DONE items assume the
|
||||
**gRPC transport** as the foundation (clean protobuf envelope; only the inner byte
|
||||
blob needs RE).
|
||||
|
||||
---
|
||||
|
||||
## 1. Connection & session
|
||||
|
||||
| Capability | HCAL API | 2023 R2 gRPC op | histsdk | Tier | Notes |
|
||||
|---|---|---|---|---|---|
|
||||
| Probe / version | `TestConnection`, GetV | `*Service.GetInterfaceVersion` | ✅ | DONE | |
|
||||
| Open connection (Process) | `OpenConnection` | `History.OpenConnection` (+ `ExchangeKey` auth) | ✅ | DONE | full auth chain works |
|
||||
| Open connection (Event) | `OpenConnection` (Event type) | `History.OpenConnection` event mode | 🟗 | TRIVIAL | read path already opens it; flag = ConnectionType.Event |
|
||||
| Close connection | `CloseConnection` | `History.CloseConnection` | ✅ | DONE | |
|
||||
| Connection status | `GetConnectionStatus` | `Status.GetHistorianConsoleStatus` | ✅ | DONE | |
|
||||
| Open/close **storage** connection | `OpenStorageConnection`, `CloseStorageConnection` | `Storage.OpenStorageConnection2` | ⬜ | BOUNDED | needed for any data-write path; storage-engine session |
|
||||
|
||||
## 2. Reads — process data
|
||||
|
||||
| Capability | HCAL API | 2023 R2 gRPC op | histsdk | Tier | Notes |
|
||||
|---|---|---|---|---|---|
|
||||
| Raw / full history | `CreateHistoryQuery` → Start/MoveNext/End | `Retrieval.StartQuery`→`GetNextQueryResultBuffer`→`EndQuery` | ✅ | DONE | row buffer parsed |
|
||||
| Aggregate (interp/avg/min/max/…) | `CreateHistoryQuery` (RetrievalMode) | same | ✅ | DONE | all 15 RetrievalModes mapped |
|
||||
| At-time / value-at | (interp window) | same | ✅ | DONE | |
|
||||
| Analog summary | `CreateAnalogSummaryQuery` | `Retrieval.StartQuery` (summary mode) | 🟗 | BOUNDED | mode variant of existing query |
|
||||
| State summary | `CreateStateSummaryQuery` | `Retrieval.StartQuery` (state mode) | ⬜ | BOUNDED | extra row layout to decode |
|
||||
| Block read | `ReadBlocks` | `Storage.LoadBlocks` | ⬜ | BOUNDED | low-level; rarely needed |
|
||||
|
||||
## 3. Reads — events
|
||||
|
||||
| Capability | HCAL API | 2023 R2 gRPC op | histsdk | Tier | Notes |
|
||||
|---|---|---|---|---|---|
|
||||
| Event query | `CreateEventQuery` → Start/MoveNext/End | `Retrieval.StartEventQuery`→`GetNextEventQueryResultBuffer`→`EndEventQuery` | ✅ | DONE | rows + typed property bag parsed; CM_EVENT registration done |
|
||||
| Event filters | `EventQuery.AddEventFilter` / `AddEventFilterCondition` | filter bytes in StartEventQuery request | ⬜ | BOUNDED | encode filter predicate into request buffer |
|
||||
|
||||
## 4. Browse & metadata
|
||||
|
||||
| Capability | HCAL API | 2023 R2 gRPC op | histsdk | Tier | Notes |
|
||||
|---|---|---|---|---|---|
|
||||
| Tag name browse | `CreateTagQuery` → `GetTagNames` | `Retrieval.StartTagQuery`/`QueryTag` (or LikeTagnames) | ✅ | DONE | wildcard works |
|
||||
| Tag metadata | `GetTagInfoByName`, `TagQuery.GetTagInfo` | `Retrieval.GetTagInfosFromName` | ✅ | DONE | |
|
||||
| Extended properties (read) | `GetTagExtendedPropertiesByName` | `Retrieval.GetTagExtendedPropertiesFromName` | ⬜ | BOUNDED | TEP buffer decode |
|
||||
| Localized properties (read) | `GetTagLocalizedPropertiesByName` | `Retrieval.GetTagLocalizedPropertiesFromName` | ⬜ | BOUNDED | |
|
||||
| SQL passthrough | `ExecuteSqlCommand` | `Retrieval.ExecuteSqlCommand` | ⬜ | TRIVIAL | thin string-in / status-out |
|
||||
|
||||
## 5. Tag configuration (writes)
|
||||
|
||||
| Capability | HCAL API | 2023 R2 gRPC op | histsdk | Tier | Notes |
|
||||
|---|---|---|---|---|---|
|
||||
| Create analog tag | `AddTag` | `History.EnsureTags` (EnsT2) | ✅ | DONE | Float/Double/Int2/Int4/UInt2/UInt4 + scaling |
|
||||
| Create string/discrete tag | `AddTag` | `History.EnsureTags` | ⬜ | GATED/BOUNDED | native AddTag rejects these types server-side; needs different metadata path |
|
||||
| Delete tag(s) | `DeleteTags` | `History.DeleteTags` | ✅ | DONE | |
|
||||
| Rename tag(s) | `RenameTags` | (History op) | ⬜ | BOUNDED | `AllowRenameTags` param already probed |
|
||||
| Add/Delete extended properties | `AddTagExtendedProperties`, `DeleteTagExtendedPropertiesByName` | `History.AddTagExtendedProperties` / `DeleteTagExtendedProperties` | ⬜ | BOUNDED | gRPC op + TEP serialize |
|
||||
| Add/Delete localized properties | `AddTagLocalizedProperties`, `DeleteTagLocalizedPropertiesByName` | `History.AddTagLocalizedProperties` / `DeleteTagLocalizedProperties` | ⬜ | BOUNDED | |
|
||||
|
||||
## 6. Data writes — values
|
||||
|
||||
| Capability | HCAL API | 2023 R2 gRPC op | histsdk | Tier | Notes |
|
||||
|---|---|---|---|---|---|
|
||||
| Stream process values | `AddStreamedValue(HistorianDataValue)` | `Storage.AddStreamValues` | ⬜ | **GATED** | runtime cache only ingests from IOServer/AppServer pipelines (`129 Tag not found in cache`). Not a client bug |
|
||||
| Stream **events** | `AddStreamedValue(HistorianEvent)` | `Storage.AddStreamValues` (event VTQ) | ⬜ | **CAPTURE** | full path mapped; need `CCommonArchestraEventValue::PackToVtq` blob bytes. See histevents.md |
|
||||
| Non-streamed / historical insert | `AddNonStreamedValue`, `SendNonStreamedValues` | `Transaction.AddNonStreamValues(Begin/End)` | ⬜ | BOUNDED | explicit original-data insert via Transaction svc; verify ingest permission on target |
|
||||
| Versioned streamed value | `AddVersionedStreamedValue` | `Storage.AddStreamValues2` | ⬜ | CAPTURE | revision flag on the VTQ |
|
||||
|
||||
## 7. Revisions / edits (modify stored data)
|
||||
|
||||
| Capability | HCAL API | 2023 R2 gRPC op | histsdk | Tier | Notes |
|
||||
|---|---|---|---|---|---|
|
||||
| Insert/update/delete revision values | `AddRevisionValue(s)`, `AddRevisionValuesBegin/End` | (storage-engine / transaction path) | ⬜ | HARD | prior RE: revision-write needs the non-WCF **storage-engine pipe** (`STransactPipeClient2`), not the WCF/gRPC surface |
|
||||
| Event update/delete (revise) | `HistorianEvent.Update/.Delete` | `UpdateEventStatus` (+ revised VTQ) | ⬜ | CAPTURE | RevisionVersion + Update/Delete flags in the event VTQ |
|
||||
|
||||
## 8. Status & system info
|
||||
|
||||
| Capability | HCAL API | 2023 R2 gRPC op | histsdk | Tier | Notes |
|
||||
|---|---|---|---|---|---|
|
||||
| System parameter | `GetSystemParameter` | `Status.GetSystemParameter` | ✅ | DONE | |
|
||||
| Runtime parameter | `GetRuntimeParameter` | `Status.GetRuntimeParameter` | ⬜ | TRIVIAL | same shape as GetSystemParameter |
|
||||
| Historian info | `GetHistorianInfo` | `Status.GetHistorianInfo` | 🟗 | BOUNDED | GETHI buffer; partially decoded (incl. EventStorageMode @ offset 514) |
|
||||
| Server timezone | `GetSystemTimeZoneInfo` | `Status.GetSystemTimeZoneName` | ⬜ | TRIVIAL | |
|
||||
| Historization status | `GetHistorizationStatus` | `Status` op | ⬜ | BOUNDED | |
|
||||
| Store-and-forward status | `GetStoreForwardStatus` | (push events / pull GETHI) | 🟗 | HARD | currently synthesized; real read needs duplex push or a decoded pull endpoint — see store-forward plan |
|
||||
|
||||
## 9. Store-and-forward (offline buffering)
|
||||
|
||||
| Capability | HCAL API | 2023 R2 gRPC op | histsdk | Tier | Notes |
|
||||
|---|---|---|---|---|---|
|
||||
| SF buffering + replay | (implicit on write conns) | `Storage`/`Transaction` `*Snapshot` + `Forward*Snapshot` | ⬜ | HARD | full subsystem: local cache format, snapshot framing, recovery log, forward-on-reconnect. Pragmatic alt: a simpler local queue, not bit-faithful SF |
|
||||
| Event SF | (event conn) | `Forward**Event**SnapshotBegin/…/End` | ⬜ | HARD | dedicated event-snapshot SF stream |
|
||||
| SF parameters | Get/Set SFP | `Storage.GetSFParameter`/`SetSFParameter` | ⬜ | BOUNDED | |
|
||||
|
||||
## 10. Redundancy / multi-historian
|
||||
|
||||
| Capability | HCAL API | 2023 R2 gRPC op | histsdk | Tier | Notes |
|
||||
|---|---|---|---|---|---|
|
||||
| Tiered/redundant access, failover | `MultiHistorianAccess.*` (OpenConnectionToAll, AddSecondaries, partner watchdog, ReSyncTags) | N×single-historian sessions + client logic | ⬜ | HARD | mostly client-side orchestration over §1–§6; build last |
|
||||
| Replication config | (server `aahReplication`) | — | ⬜ | GATED | server-side concern |
|
||||
|
||||
---
|
||||
|
||||
## Roll-up & recommended cut line
|
||||
|
||||
**Phase 0 — already DONE (✅):** probe · open/close · raw+aggregate+at-time reads ·
|
||||
event reads · tag browse · tag metadata · system parameter · connection status ·
|
||||
create/delete analog tag. This is a usable modern client **today**.
|
||||
|
||||
**Phase 1 — TRIVIAL/BOUNDED, high value (S–M each):** ExecuteSqlCommand ·
|
||||
runtime parameter · server timezone · extended/localized property read · event
|
||||
filters · summary/state-summary queries · rename tags · ext/localized property
|
||||
writes · GetHistorianInfo. Each is "gRPC op exists, decode one buffer, golden-byte
|
||||
test." Knocks out most of the remaining read/config surface.
|
||||
|
||||
**Phase 2 — CAPTURE (one native capture each, S):** **event sending** (the headline
|
||||
gap — fully mapped, one `PackToVtq` capture away) · versioned/non-streamed value
|
||||
writes. Now feasible locally since the Historian is installed.
|
||||
|
||||
**Defer / simplify (HARD):** store-and-forward (do a pragmatic local queue instead of
|
||||
bit-faithful SF) · revision/edit writes (separate storage-engine pipe) · multi-
|
||||
historian redundancy (client orchestration, build last).
|
||||
|
||||
**Won't unblock from the client (GATED):** streaming **process-sample** writes
|
||||
(`AddS2`) — server cache only ingests from IOServer/AppServer pipelines; confirm your
|
||||
ingestion model rather than chasing this. Non-analog tag creation likely needs a
|
||||
distinct server path.
|
||||
|
||||
## Cross-cutting realities (apply to every non-DONE row)
|
||||
|
||||
- **Inner payloads stay proprietary** even under gRPC — the `bytes` fields carry
|
||||
native VTQ / CTagMetadata / event-value formats. These are **version-sensitive**;
|
||||
pin to the server version probed at connect and fail closed on mismatch.
|
||||
- **Validation needs a live Historian** — now available locally, which is what makes
|
||||
the CAPTURE-tier items practical.
|
||||
- **Support tradeoff** — you take on maintenance across Historian versions in exchange
|
||||
for shedding the stock SDK's bugs (mixed-mode marshaling, WCF quirks, global state)
|
||||
for the surface you cover.
|
||||
|
||||
## Bottom line
|
||||
|
||||
A modern-.NET HCAL replacement is **feasible and ~60–70% done** for a typical
|
||||
read+browse+config+event-read workload. The remaining high-value surface is mostly
|
||||
**BOUNDED/CAPTURE** (incremental, well-understood), with only store-and-forward,
|
||||
revision-edit, and redundancy being genuine **HARD** subsystems — and one true wall
|
||||
(**GATED** process-sample writes) that no client can remove.
|
||||
Reference in New Issue
Block a user