Joseph Doherty
08b950caee
Adds user-defined extended properties to an existing tag via the 2020 WCF
AddTEx (AddTagExtendedProperties) op. Write-enabled connection + uppercase
storage-session GUID handle; reuses the write orchestrator open/priming chain.
The AddTEx inBuff is the exact inverse of the R1.5 GetTepByNm read-response
framing, so the serializer mirrors the read parser:
uint32 groupCount + 0x01(group) + [0x09+u16+ASCII tag] + uint32 propCount
+ per prop{ 0x02 + [0x09+u16+ASCII name] + 0x43 VT_BSTR + u16 payloadLen
+ u16 charCount + UTF-16 value } + 0x01(group trailer) + 0x00(terminator).
The trailing 0x00 is required — without it inBuff is one byte short and the
server throws SErrorException in CHistStorage::AddTagExtendedProperties. The
golden fixture pins the clean inBuff the live server accepted (dumped via
AVEVA_HISTORIAN_TEP_DUMP); read-back verified via R1.5. String (0x43) values only.
Delete (DelTep) is deferred: the native DeleteTagExtendedPropertiesByName does a
client-side sync check and returns err 229 for a just-added property, so the
DelTep request never reaches the wire and its inBuff can't be captured yet.
Shipped: HistorianClient.AddTagExtendedPropertiesAsync/AddTagExtendedPropertyAsync;
HistorianTagExtendedPropertyProtocol.SerializeAddRequest; orchestrator path;
golden WcfTagExtendedPropertyWriteProtocolTests (4); gated live write/read-back test;
native-harness `add-tep` scenario + Capture-AddTagExtendedProperties.ps1 +
decode-add-tep-capture.py. Doc: wcf-add-tag-extended-properties.md. 233 tests green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B6mcaT2PjRFKcogzp9UkfC
167 lines
11 KiB
Markdown
167 lines
11 KiB
Markdown
# HCAL → modern-.NET reimplementation — capability matrix
|
||
|
||
Feasibility map for a clean managed-.NET client that replaces the AVEVA Historian
|
||
SDK (`aahClientManaged` / HCAL). Grounded in: the real `ArchestrA.HistorianAccess`
|
||
public surface (`aahClientManaged.xml`), the recovered **2023 R2 gRPC contract**, the
|
||
existing **histsdk** reimplementation, and the event/storage analysis in
|
||
[`histevents.md`](histevents.md).
|
||
|
||
## Legend
|
||
|
||
**Status (histsdk today)** — ✅ implemented + live-verified · 🟗 partial · ⬜ not yet
|
||
|
||
**Feasibility tier**
|
||
| Tier | Meaning | Effort |
|
||
|---|---|---|
|
||
| **DONE** | already working in histsdk | 0 |
|
||
| **TRIVIAL** | gRPC op known, payload already decoded or empty | XS (hrs) |
|
||
| **CAPTURE** | one instrument-and-capture of a native payload, then serialize + golden-byte test | S (days) |
|
||
| **BOUNDED** | gRPC op exists; decode one proprietary `bytes` payload | S–M |
|
||
| **HARD** | whole subsystem to reimplement | L (weeks) |
|
||
| **GATED** | blocked server-side — client effort doesn't unblock it | n/a |
|
||
|
||
Effort = incremental work on top of histsdk's existing infrastructure (auth chain,
|
||
transport, frame/byte primitives, test harness). All non-DONE items assume the
|
||
**gRPC transport** as the foundation (clean protobuf envelope; only the inner byte
|
||
blob needs RE).
|
||
|
||
---
|
||
|
||
## 1. Connection & session
|
||
|
||
| Capability | HCAL API | 2023 R2 gRPC op | histsdk | Tier | Notes |
|
||
|---|---|---|---|---|---|
|
||
| Probe / version | `TestConnection`, GetV | `*Service.GetInterfaceVersion` | ✅ | DONE | |
|
||
| Open connection (Process) | `OpenConnection` | `History.OpenConnection` (+ `ExchangeKey` auth) | ✅ | DONE | full auth chain works |
|
||
| Open connection (Event) | `OpenConnection` (Event type) | `History.OpenConnection` event mode | 🟗 | TRIVIAL | read path already opens it; flag = ConnectionType.Event |
|
||
| Close connection | `CloseConnection` | `History.CloseConnection` | ✅ | DONE | |
|
||
| Connection status | `GetConnectionStatus` | `Status.GetHistorianConsoleStatus` | ✅ | DONE | |
|
||
| Open/close **storage** connection | `OpenStorageConnection`, `CloseStorageConnection` | `Storage.OpenStorageConnection2` | ⬜ | BOUNDED | needed for any data-write path; storage-engine session |
|
||
|
||
## 2. Reads — process data
|
||
|
||
| Capability | HCAL API | 2023 R2 gRPC op | histsdk | Tier | Notes |
|
||
|---|---|---|---|---|---|
|
||
| Raw / full history | `CreateHistoryQuery` → Start/MoveNext/End | `Retrieval.StartQuery`→`GetNextQueryResultBuffer`→`EndQuery` | ✅ | DONE | row buffer parsed |
|
||
| Aggregate (interp/avg/min/max/…) | `CreateHistoryQuery` (RetrievalMode) | same | ✅ | DONE | all 15 RetrievalModes mapped |
|
||
| At-time / value-at | (interp window) | same | ✅ | DONE | |
|
||
| Analog summary | `CreateAnalogSummaryQuery` | `Retrieval.StartQuery` (summary mode) | 🟗 | BOUNDED | mode variant of existing query |
|
||
| State summary | `CreateStateSummaryQuery` | `Retrieval.StartQuery` (state mode) | ⬜ | BOUNDED | extra row layout to decode |
|
||
| Block read | `ReadBlocks` | `Storage.LoadBlocks` | ⬜ | BOUNDED | low-level; rarely needed |
|
||
|
||
## 3. Reads — events
|
||
|
||
| Capability | HCAL API | 2023 R2 gRPC op | histsdk | Tier | Notes |
|
||
|---|---|---|---|---|---|
|
||
| Event query | `CreateEventQuery` → Start/MoveNext/End | `Retrieval.StartEventQuery`→`GetNextEventQueryResultBuffer`→`EndEventQuery` | ✅ | DONE | rows + typed property bag parsed; CM_EVENT registration done |
|
||
| Event filters | `EventQuery.AddEventFilter` / `AddEventFilterCondition` | filter bytes in StartEventQuery request | ⬜ | BOUNDED | encode filter predicate into request buffer |
|
||
|
||
## 4. Browse & metadata
|
||
|
||
| Capability | HCAL API | 2023 R2 gRPC op | histsdk | Tier | Notes |
|
||
|---|---|---|---|---|---|
|
||
| Tag name browse | `CreateTagQuery` → `GetTagNames` | `Retrieval.StartTagQuery`/`QueryTag` (or LikeTagnames) | ✅ | DONE | wildcard works |
|
||
| Tag metadata | `GetTagInfoByName`, `TagQuery.GetTagInfo` | `Retrieval.GetTagInfosFromName` | ✅ | DONE | |
|
||
| Extended properties (read) | `GetTagExtendedPropertiesByName` | `Retrieval.GetTagExtendedPropertiesFromName` | ⬜ | BOUNDED | TEP buffer decode |
|
||
| Localized properties (read) | `GetTagLocalizedPropertiesByName` | `Retrieval.GetTagLocalizedPropertiesFromName` | ⬜ | BOUNDED | |
|
||
| SQL passthrough | `ExecuteSqlCommand` | `Retrieval.ExecuteSqlCommand` | ⬜ | TRIVIAL | thin string-in / status-out |
|
||
|
||
## 5. Tag configuration (writes)
|
||
|
||
| Capability | HCAL API | 2023 R2 gRPC op | histsdk | Tier | Notes |
|
||
|---|---|---|---|---|---|
|
||
| Create analog tag | `AddTag` | `History.EnsureTags` (EnsT2) | ✅ | DONE | Float/Double/Int2/Int4/UInt2/UInt4 + scaling |
|
||
| Create string/discrete tag | `AddTag` | `History.EnsureTags` | ⬜ | GATED/BOUNDED | native AddTag rejects these types server-side; needs different metadata path |
|
||
| Delete tag(s) | `DeleteTags` | `History.DeleteTags` | ✅ | DONE | |
|
||
| Rename tag(s) | `RenameTags` | (History op) | ⬜ | BOUNDED | `AllowRenameTags` param already probed |
|
||
| Add/Delete extended properties | `AddTagExtendedProperties`, `DeleteTagExtendedPropertiesByName` | `History.AddTagExtendedProperties` / `DeleteTagExtendedProperties` | 🟗 | BOUNDED | **Add DONE** (`AddTagExtendedPropertiesAsync`, AddTEx; inBuff = inverse of R1.5 read framing + trailing `01 00`). Delete (DelTep) deferred — native sync gate (err 229) blocks capturing its inBuff. See `wcf-add-tag-extended-properties.md` |
|
||
| Add/Delete localized properties | `AddTagLocalizedProperties`, `DeleteTagLocalizedPropertiesByName` | `History.AddTagLocalizedProperties` / `DeleteTagLocalizedProperties` | ⬜ | BOUNDED | |
|
||
|
||
## 6. Data writes — values
|
||
|
||
| Capability | HCAL API | 2023 R2 gRPC op | histsdk | Tier | Notes |
|
||
|---|---|---|---|---|---|
|
||
| Stream process values | `AddStreamedValue(HistorianDataValue)` | `Storage.AddStreamValues` | ⬜ | **GATED** | runtime cache only ingests from IOServer/AppServer pipelines (`129 Tag not found in cache`). Not a client bug |
|
||
| Stream **events** | `AddStreamedValue(HistorianEvent)` | `Storage.AddStreamValues` (event VTQ) | ⬜ | **CAPTURE** | full path mapped; need `CCommonArchestraEventValue::PackToVtq` blob bytes. See histevents.md |
|
||
| Non-streamed / historical insert | `AddNonStreamedValue`, `SendNonStreamedValues` | `Transaction.AddNonStreamValues(Begin/End)` | ⬜ | BOUNDED | explicit original-data insert via Transaction svc; verify ingest permission on target |
|
||
| Versioned streamed value | `AddVersionedStreamedValue` | `Storage.AddStreamValues2` | ⬜ | CAPTURE | revision flag on the VTQ |
|
||
|
||
## 7. Revisions / edits (modify stored data)
|
||
|
||
| Capability | HCAL API | 2023 R2 gRPC op | histsdk | Tier | Notes |
|
||
|---|---|---|---|---|---|
|
||
| Insert/update/delete revision values | `AddRevisionValue(s)`, `AddRevisionValuesBegin/End` | (storage-engine / transaction path) | ⬜ | HARD | prior RE: revision-write needs the non-WCF **storage-engine pipe** (`STransactPipeClient2`), not the WCF/gRPC surface |
|
||
| Event update/delete (revise) | `HistorianEvent.Update/.Delete` | `UpdateEventStatus` (+ revised VTQ) | ⬜ | CAPTURE | RevisionVersion + Update/Delete flags in the event VTQ |
|
||
|
||
## 8. Status & system info
|
||
|
||
| Capability | HCAL API | 2023 R2 gRPC op | histsdk | Tier | Notes |
|
||
|---|---|---|---|---|---|
|
||
| System parameter | `GetSystemParameter` | `Status.GetSystemParameter` | ✅ | DONE | |
|
||
| Runtime parameter | `GetRuntimeParameter` | `Status.GetRuntimeParameter` | ⬜ | TRIVIAL | same shape as GetSystemParameter |
|
||
| Historian info | `GetHistorianInfo` | `Status.GetHistorianInfo` | 🟗 | BOUNDED | GETHI buffer; partially decoded (incl. EventStorageMode @ offset 514) |
|
||
| Server timezone | `GetSystemTimeZoneInfo` | `Status.GetSystemTimeZoneName` | ⬜ | TRIVIAL | |
|
||
| Historization status | `GetHistorizationStatus` | `Status` op | ⬜ | BOUNDED | |
|
||
| Store-and-forward status | `GetStoreForwardStatus` | (push events / pull GETHI) | 🟗 | HARD | currently synthesized; real read needs duplex push or a decoded pull endpoint — see store-forward plan |
|
||
|
||
## 9. Store-and-forward (offline buffering)
|
||
|
||
| Capability | HCAL API | 2023 R2 gRPC op | histsdk | Tier | Notes |
|
||
|---|---|---|---|---|---|
|
||
| SF buffering + replay | (implicit on write conns) | `Storage`/`Transaction` `*Snapshot` + `Forward*Snapshot` | ⬜ | HARD | full subsystem: local cache format, snapshot framing, recovery log, forward-on-reconnect. Pragmatic alt: a simpler local queue, not bit-faithful SF |
|
||
| Event SF | (event conn) | `Forward**Event**SnapshotBegin/…/End` | ⬜ | HARD | dedicated event-snapshot SF stream |
|
||
| SF parameters | Get/Set SFP | `Storage.GetSFParameter`/`SetSFParameter` | ⬜ | BOUNDED | |
|
||
|
||
## 10. Redundancy / multi-historian
|
||
|
||
| Capability | HCAL API | 2023 R2 gRPC op | histsdk | Tier | Notes |
|
||
|---|---|---|---|---|---|
|
||
| Tiered/redundant access, failover | `MultiHistorianAccess.*` (OpenConnectionToAll, AddSecondaries, partner watchdog, ReSyncTags) | N×single-historian sessions + client logic | ⬜ | HARD | mostly client-side orchestration over §1–§6; build last |
|
||
| Replication config | (server `aahReplication`) | — | ⬜ | GATED | server-side concern |
|
||
|
||
---
|
||
|
||
## Roll-up & recommended cut line
|
||
|
||
**Phase 0 — already DONE (✅):** probe · open/close · raw+aggregate+at-time reads ·
|
||
event reads · tag browse · tag metadata · system parameter · connection status ·
|
||
create/delete analog tag. This is a usable modern client **today**.
|
||
|
||
**Phase 1 — TRIVIAL/BOUNDED, high value (S–M each):** ExecuteSqlCommand ·
|
||
runtime parameter · server timezone · extended/localized property read · event
|
||
filters · summary/state-summary queries · rename tags · ext/localized property
|
||
writes · GetHistorianInfo. Each is "gRPC op exists, decode one buffer, golden-byte
|
||
test." Knocks out most of the remaining read/config surface.
|
||
|
||
**Phase 2 — CAPTURE (one native capture each, S):** **event sending** (the headline
|
||
gap — fully mapped, one `PackToVtq` capture away) · versioned/non-streamed value
|
||
writes. Now feasible locally since the Historian is installed.
|
||
|
||
**Defer / simplify (HARD):** store-and-forward (do a pragmatic local queue instead of
|
||
bit-faithful SF) · revision/edit writes (separate storage-engine pipe) · multi-
|
||
historian redundancy (client orchestration, build last).
|
||
|
||
**Won't unblock from the client (GATED):** streaming **process-sample** writes
|
||
(`AddS2`) — server cache only ingests from IOServer/AppServer pipelines; confirm your
|
||
ingestion model rather than chasing this. Non-analog tag creation likely needs a
|
||
distinct server path.
|
||
|
||
## Cross-cutting realities (apply to every non-DONE row)
|
||
|
||
- **Inner payloads stay proprietary** even under gRPC — the `bytes` fields carry
|
||
native VTQ / CTagMetadata / event-value formats. These are **version-sensitive**;
|
||
pin to the server version probed at connect and fail closed on mismatch.
|
||
- **Validation needs a live Historian** — now available locally, which is what makes
|
||
the CAPTURE-tier items practical.
|
||
- **Support tradeoff** — you take on maintenance across Historian versions in exchange
|
||
for shedding the stock SDK's bugs (mixed-mode marshaling, WCF quirks, global state)
|
||
for the surface you cover.
|
||
|
||
## Bottom line
|
||
|
||
A modern-.NET HCAL replacement is **feasible and ~60–70% done** for a typical
|
||
read+browse+config+event-read workload. The remaining high-value surface is mostly
|
||
**BOUNDED/CAPTURE** (incremental, well-understood), with only store-and-forward,
|
||
revision-edit, and redundancy being genuine **HARD** subsystems — and one true wall
|
||
(**GATED** process-sample writes) that no client can remove.
|