histsdk/docs/plans/hcal-capability-matrix.md

# HCAL → modern-.NET reimplementation — capability matrix

Feasibility map for a clean managed-.NET client that replaces the AVEVA Historian
SDK (`aahClientManaged` / HCAL). Grounded in: the real `ArchestrA.HistorianAccess`
public surface (`aahClientManaged.xml`), the recovered **2023 R2 gRPC contract**, the
existing **histsdk** reimplementation, and the event/storage analysis in
[`histevents.md`](histevents.md).

## Legend

**Status (histsdk today)** — ✅ implemented + live-verified · 🟗 partial · ⬜ not yet

**Feasibility tier**
| Tier | Meaning | Effort |
|---|---|---|
| **DONE** | already working in histsdk | 0 |
| **TRIVIAL** | gRPC op known, payload already decoded or empty | XS (hrs) |
| **CAPTURE** | one instrument-and-capture of a native payload, then serialize + golden-byte test | S (days) |
| **BOUNDED** | gRPC op exists; decode one proprietary `bytes` payload | S–M |
| **HARD** | whole subsystem to reimplement | L (weeks) |
| **GATED** | blocked server-side — client effort doesn't unblock it | n/a |

Effort = incremental work on top of histsdk's existing infrastructure (auth chain,
transport, frame/byte primitives, test harness). All non-DONE items assume the
**gRPC transport** as the foundation (clean protobuf envelope; only the inner byte
blob needs RE).

---

## 1. Connection & session

| Capability | HCAL API | 2023 R2 gRPC op | histsdk | Tier | Notes |
|---|---|---|---|---|---|
| Probe / version | `TestConnection`, GetV | `*Service.GetInterfaceVersion` | ✅ | DONE | |
| Open connection (Process) | `OpenConnection` | `History.OpenConnection` (+ `ExchangeKey` auth) | ✅ | DONE | full auth chain works |
| Open connection (Event) | `OpenConnection` (Event type) | `History.OpenConnection` event mode | 🟗 | TRIVIAL | read path already opens it; flag = ConnectionType.Event |
| Close connection | `CloseConnection` | `History.CloseConnection` | ✅ | DONE | |
| Connection status | `GetConnectionStatus` | `Status.GetHistorianConsoleStatus` | ✅ | DONE | |
| Open/close **storage** connection | `OpenStorageConnection`, `CloseStorageConnection` | `Storage.OpenStorageConnection2` | ⬜ | BOUNDED | needed for any data-write path; storage-engine session |

## 2. Reads — process data

| Capability | HCAL API | 2023 R2 gRPC op | histsdk | Tier | Notes |
|---|---|---|---|---|---|
| Raw / full history | `CreateHistoryQuery` → Start/MoveNext/End | `Retrieval.StartQuery`→`GetNextQueryResultBuffer`→`EndQuery` | ✅ | DONE | row buffer parsed |
| Aggregate (interp/avg/min/max/…) | `CreateHistoryQuery` (RetrievalMode) | same | ✅ | DONE | all 15 RetrievalModes mapped |
| At-time / value-at | (interp window) | same | ✅ | DONE | |
| Analog summary | `CreateAnalogSummaryQuery` | `Retrieval.StartQuery` (summary mode) | 🟗 | BOUNDED | mode variant of existing query |
| State summary | `CreateStateSummaryQuery` | `Retrieval.StartQuery` (state mode) | ⬜ | BOUNDED | extra row layout to decode |
| Block read | `ReadBlocks` | `Storage.LoadBlocks` | ⬜ | BOUNDED | low-level; rarely needed |

## 3. Reads — events

| Capability | HCAL API | 2023 R2 gRPC op | histsdk | Tier | Notes |
|---|---|---|---|---|---|
| Event query | `CreateEventQuery` → Start/MoveNext/End | `Retrieval.StartEventQuery`→`GetNextEventQueryResultBuffer`→`EndEventQuery` | ✅ | DONE | rows + typed property bag parsed; CM_EVENT registration done |
| Event filters | `EventQuery.AddEventFilter` / `AddEventFilterCondition` | filter bytes in StartEventQuery request | ⬜ | BOUNDED | encode filter predicate into request buffer |

## 4. Browse & metadata

| Capability | HCAL API | 2023 R2 gRPC op | histsdk | Tier | Notes |
|---|---|---|---|---|---|
| Tag name browse | `CreateTagQuery` → `GetTagNames` | `Retrieval.StartTagQuery`/`QueryTag` (or LikeTagnames) | ✅ | DONE | wildcard works |
| Tag metadata | `GetTagInfoByName`, `TagQuery.GetTagInfo` | `Retrieval.GetTagInfosFromName` | ✅ | DONE | |
| Extended properties (read) | `GetTagExtendedPropertiesByName` | `Retrieval.GetTagExtendedPropertiesFromName` | ⬜ | BOUNDED | TEP buffer decode |
| Localized properties (read) | `GetTagLocalizedPropertiesByName` | `Retrieval.GetTagLocalizedPropertiesFromName` | ⬜ | BOUNDED | |
| SQL passthrough | `ExecuteSqlCommand` | `Retrieval.ExecuteSqlCommand` | ⬜ | TRIVIAL | thin string-in / status-out |

## 5. Tag configuration (writes)

| Capability | HCAL API | 2023 R2 gRPC op | histsdk | Tier | Notes |
|---|---|---|---|---|---|
| Create analog tag | `AddTag` | `History.EnsureTags` (EnsT2) | ✅ | DONE | Float/Double/Int2/Int4/UInt2/UInt4 + scaling |
| Create string/discrete tag | `AddTag` | `History.EnsureTags` | ⬜ | GATED/BOUNDED | native AddTag rejects these types server-side; needs different metadata path |
| Delete tag(s) | `DeleteTags` | `History.DeleteTags` | ✅ | DONE | |
| Rename tag(s) | `RenameTags` | (History op) | ⬜ | BOUNDED | `AllowRenameTags` param already probed |
| Add/Delete extended properties | `AddTagExtendedProperties`, `DeleteTagExtendedPropertiesByName` | `History.AddTagExtendedProperties` / `DeleteTagExtendedProperties` | ⬜ | BOUNDED | gRPC op + TEP serialize |
| Add/Delete localized properties | `AddTagLocalizedProperties`, `DeleteTagLocalizedPropertiesByName` | `History.AddTagLocalizedProperties` / `DeleteTagLocalizedProperties` | ⬜ | BOUNDED | |

## 6. Data writes — values

| Capability | HCAL API | 2023 R2 gRPC op | histsdk | Tier | Notes |
|---|---|---|---|---|---|
| Stream process values | `AddStreamedValue(HistorianDataValue)` | `Storage.AddStreamValues` | ⬜ | **GATED** | runtime cache only ingests from IOServer/AppServer pipelines (`129 Tag not found in cache`). Not a client bug |
| Stream **events** | `AddStreamedValue(HistorianEvent)` | `Storage.AddStreamValues` (event VTQ) | ⬜ | **CAPTURE** | full path mapped; need `CCommonArchestraEventValue::PackToVtq` blob bytes. See histevents.md |
| Non-streamed / historical insert | `AddNonStreamedValue`, `SendNonStreamedValues` | `Transaction.AddNonStreamValues(Begin/End)` | ⬜ | BOUNDED | explicit original-data insert via Transaction svc; verify ingest permission on target |
| Versioned streamed value | `AddVersionedStreamedValue` | `Storage.AddStreamValues2` | ⬜ | CAPTURE | revision flag on the VTQ |

## 7. Revisions / edits (modify stored data)

| Capability | HCAL API | 2023 R2 gRPC op | histsdk | Tier | Notes |
|---|---|---|---|---|---|
| Insert/update/delete revision values | `AddRevisionValue(s)`, `AddRevisionValuesBegin/End` | (storage-engine / transaction path) | ⬜ | HARD | prior RE: revision-write needs the non-WCF **storage-engine pipe** (`STransactPipeClient2`), not the WCF/gRPC surface |
| Event update/delete (revise) | `HistorianEvent.Update/.Delete` | `UpdateEventStatus` (+ revised VTQ) | ⬜ | CAPTURE | RevisionVersion + Update/Delete flags in the event VTQ |

## 8. Status & system info

| Capability | HCAL API | 2023 R2 gRPC op | histsdk | Tier | Notes |
|---|---|---|---|---|---|
| System parameter | `GetSystemParameter` | `Status.GetSystemParameter` | ✅ | DONE | |
| Runtime parameter | `GetRuntimeParameter` | `Status.GetRuntimeParameter` | ⬜ | TRIVIAL | same shape as GetSystemParameter |
| Historian info | `GetHistorianInfo` | `Status.GetHistorianInfo` | 🟗 | BOUNDED | GETHI buffer; partially decoded (incl. EventStorageMode @ offset 514) |
| Server timezone | `GetSystemTimeZoneInfo` | `Status.GetSystemTimeZoneName` | ⬜ | TRIVIAL | |
| Historization status | `GetHistorizationStatus` | `Status` op | ⬜ | BOUNDED | |
| Store-and-forward status | `GetStoreForwardStatus` | (push events / pull GETHI) | 🟗 | HARD | currently synthesized; real read needs duplex push or a decoded pull endpoint — see store-forward plan |

## 9. Store-and-forward (offline buffering)

| Capability | HCAL API | 2023 R2 gRPC op | histsdk | Tier | Notes |
|---|---|---|---|---|---|
| SF buffering + replay | (implicit on write conns) | `Storage`/`Transaction` `*Snapshot` + `Forward*Snapshot` | ⬜ | HARD | full subsystem: local cache format, snapshot framing, recovery log, forward-on-reconnect. Pragmatic alt: a simpler local queue, not bit-faithful SF |
| Event SF | (event conn) | `Forward**Event**SnapshotBegin/…/End` | ⬜ | HARD | dedicated event-snapshot SF stream |
| SF parameters | Get/Set SFP | `Storage.GetSFParameter`/`SetSFParameter` | ⬜ | BOUNDED | |

## 10. Redundancy / multi-historian

| Capability | HCAL API | 2023 R2 gRPC op | histsdk | Tier | Notes |
|---|---|---|---|---|---|
| Tiered/redundant access, failover | `MultiHistorianAccess.*` (OpenConnectionToAll, AddSecondaries, partner watchdog, ReSyncTags) | N×single-historian sessions + client logic | ⬜ | HARD | mostly client-side orchestration over §1–§6; build last |
| Replication config | (server `aahReplication`) | — | ⬜ | GATED | server-side concern |

---

## Roll-up & recommended cut line

**Phase 0 — already DONE (✅):** probe · open/close · raw+aggregate+at-time reads ·
event reads · tag browse · tag metadata · system parameter · connection status ·
create/delete analog tag. This is a usable modern client **today**.

**Phase 1 — TRIVIAL/BOUNDED, high value (S–M each):** ExecuteSqlCommand ·
runtime parameter · server timezone · extended/localized property read · event
filters · summary/state-summary queries · rename tags · ext/localized property
writes · GetHistorianInfo. Each is "gRPC op exists, decode one buffer, golden-byte
test." Knocks out most of the remaining read/config surface.

**Phase 2 — CAPTURE (one native capture each, S):** **event sending** (the headline
gap — fully mapped, one `PackToVtq` capture away) · versioned/non-streamed value
writes. Now feasible locally since the Historian is installed.

**Defer / simplify (HARD):** store-and-forward (do a pragmatic local queue instead of
bit-faithful SF) · revision/edit writes (separate storage-engine pipe) · multi-
historian redundancy (client orchestration, build last).

**Won't unblock from the client (GATED):** streaming **process-sample** writes
(`AddS2`) — server cache only ingests from IOServer/AppServer pipelines; confirm your
ingestion model rather than chasing this. Non-analog tag creation likely needs a
distinct server path.

## Cross-cutting realities (apply to every non-DONE row)

- **Inner payloads stay proprietary** even under gRPC — the `bytes` fields carry
  native VTQ / CTagMetadata / event-value formats. These are **version-sensitive**;
  pin to the server version probed at connect and fail closed on mismatch.
- **Validation needs a live Historian** — now available locally, which is what makes
  the CAPTURE-tier items practical.
- **Support tradeoff** — you take on maintenance across Historian versions in exchange
  for shedding the stock SDK's bugs (mixed-mode marshaling, WCF quirks, global state)
  for the surface you cover.

## Bottom line

A modern-.NET HCAL replacement is **feasible and ~60–70% done** for a typical
read+browse+config+event-read workload. The remaining high-value surface is mostly
**BOUNDED/CAPTURE** (incremental, well-understood), with only store-and-forward,
revision-edit, and redundancy being genuine **HARD** subsystems — and one true wall
(**GATED** process-sample writes) that no client can remove.