dohertj2/histsdk
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs/plans: import 2023 R2 gRPC analysis + HCAL reimpl roadmap

Browse Source 
Version-control the planning docs alongside the code they describe:
- grpc-transport.md     — 2023 R2 gRPC transport analysis (sanitized source path)
- hcal-capability-matrix.md — HistorianAccess surface x gRPC ops x histsdk status x feasibility tiers
- hcal-roadmap.md       — ordered build plan M0-M4 + cross-cutting workstreams
- histevents.md         — how a HistorianEvent reaches the DB (client->wire->server)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Joseph Doherty
2026-06-19 14:28:34 -04:00
parent 1e9a87fce9
commit a530ae0f10
4 changed files with 901 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/plans/grpc-transport.md
+265
View File
@@ -0,0 +1,265 @@
# AVEVA Historian SDK 2023 R2 — gRPC Transport Analysis
**Scope:** Documents the new gRPC transport that 2023 R2 adds to the Historian
Client Access Layer (HCAL). Kept deliberately **separate** from the main
`histsdk` reverse-engineering docs — this is 2023 R2 evidence, not the 2020/WCF
protocol the production SDK currently targets.
**Source:** the 2023 R2 `HistorianSDK` installer (**not installed**).
`SDKSetup.msi` was laid out with `msiexec /a` (administrative extract, no
registration) into a local `msi-extract` staging dir, then the managed
assemblies were decompiled with `ilspycmd`.
**Assembly versions analysed:** `2023.1219.4004.5`
(`Archestra.Grpc.Contract.dll`, `Archestra.Historian.GrpcClient.dll`,
`aahClientManaged.dll`).
---
## 1. Headline finding
The 2023 R2 gRPC transport is a **transport swap, not a protocol redesign.**
Every gRPC request/response wraps the **same opaque native binary buffers** that
the 2020/WCF-MDAS path already carries — `OpenConnection3` v6 buffer, NTLM/SSPI
`ValCl` tokens, `DataQueryRequest`, `GetNextQueryResultBuffer` row buffers, the
`Status` err blob, etc. — inside protobuf `bytes` fields.
Concrete proof, from `Archestra.Historian.GrpcClient`:
```csharp
// History/OpenConnection — same byte[] openParameters the WCF path built
OpenConnectionRequest request = new OpenConnectionRequest {
BtConnectionRequest = ByteString.CopyFrom(openParameters)
};
// Retrieval/StartQuery — same queryType + DataQueryRequest bytes + handle
StartQueryRequest request = new StartQueryRequest {
UiHandle = handle,
UiQueryRequestType = queryRequestType,
BtRequestBuffer = ByteString.CopyFrom(requestBuffer)
};
```
**Implication for the `histsdk` project:** all of the hard-won payload
serializers (`HistorianOpen2Protocol`, `HistorianDataQueryProtocol`,
`HistorianEventRowProtocol`, the SSPI `ValCl` token framing, the EnsT2
`CTagMetadata` layout) transfer **unchanged**. Only the envelope around them
changes: protobuf-over-gRPC instead of binary-SOAP-over-`application/x-mdas`.
The WCF `[MessageParameter(Name=…)]` guessing that dominated the 2020 work is
gone — field names and numbers are explicit in the protobuf contract.
---
## 2. Transport stack
From `GrpcClientBase.InitializeBase(target, portNumber, securedConnection, certificateName, trusted)`:
| Aspect | Value / behaviour |
|---|---|
| Library | `Grpc.Net.Client` + **`Grpc.Net.Client.Web`** (`GrpcWebHandler`) |
| Mode | **gRPC-Web**, `GrpcWebMode.GrpcWeb` (binary `application/grpc-web`, **not** `-text`) |
| HTTP version | **HTTP/1.1** (`GrpcWebHandler.HttpVersion = new Version(1,1)`) — *not* HTTP/2 |
| Address | `http://{target}:{port}` insecure, or `https://{certificateName}:{port}` secure |
| Inner handler | `HttpClientHandler` with custom `ServerCertificateCustomValidationCallback` |
| Compression | gzip on by default; request header `grpc-internal-encoding-request: gzip`; custom `CustomCompressionProvider` / `CustomGZipStream` used for bandwidth accounting |
| Default timeout | 60 s per call (`m_timeoutInSeconds = 60`, sent as gRPC deadline) |
| Interceptor | `ClientInterceptor` (logging hook, currently a no-op `LogCall`) |
Because it is **gRPC-Web over HTTP/1.1**, the transport is proxy/firewall
friendly and does not require HTTP/2 negotiation — note `HistorianConnectionArgs.ProxyServer`
(e.g. `http://host:9480`) in the public API.
### Port
- **Default port `32565`** — `HistorianConnectionArgs.TcpPort`, *"the TCP port
of the Historian Client Access Point."* (Note this differs from the 2020 WCF
port `32568` the production SDK uses.)
- All services reach the **same host:port**; gRPC multiplexes by service path
(`/HistoryService/OpenConnection`, `/RetrievalService/StartQuery`, …).
### Channel topology
Five service stubs grouped into four wrapper clients, each constructing its own
`GrpcChannel` to the same endpoint:
| Wrapper (`GrpcClientBase` subclass) | gRPC service stub(s) |
|---|---|
| `GrpcHistoryClient` | `HistoryService` + `TransactionService` (one channel) |
| `GrpcRetrievalClient` | `RetrievalService` |
| `GrpcStatusClient` | `StatusService` |
| `GrpcStorageClient` | `StorageService` |
---
## 3. Authentication model (unchanged in substance)
Auth is **still the native session handshake**, carried over gRPC instead of
WCF. There is **no per-call bearer/auth token in gRPC metadata** — the only
metadata sent is the gzip-encoding hint. Methods pass `m_metadata` (gzip) or
`null`; neither carries credentials. The server keys the session off the
`handle` GUID established by the handshake, exactly as the 2020 path does.
Handshake operations (same byte payloads as 2020):
- `HistoryService.GetInterfaceVersion` → version probe.
- `StorageService.ValidateClientCredential { string Handle; bytes InBuff }`
→ `{ Status; bytes OutBuff }`. **`InBuff`/`OutBuff` carry the NTLM/SSPI
tokens** — same multi-round Negotiate exchange, same field names the 2020
`ildasm` revealed (`inBuff`/`outBuff`), now first-class protobuf fields.
- `HistoryService.ExchangeKey { string StrHandle; bytes BtInput }`
→ `{ Status; bytes BtOutput }` (key-exchange / cert path).
- `HistoryService.OpenConnection { bytes BtConnectionRequest }`
→ `{ Status; bytes BtConnectionResponse }` — same `OpenConnection3` v6
request buffer in, same 42-byte session blob out.
Public-API security knobs (`aahClientManaged.xml`):
- `HistorianConnectionArgs.ConnectionMode` — *"whether GRPC connection to the
Historian Server. **Default is true** (GRPC)."* This is the master switch
selecting gRPC vs legacy.
- `HistorianSecurityMode`: `None`, `Disabled`, `TransportWindows`
(Windows creds), `TransportCertificate` (server cert).
- `AllowUnTrustedConnection` → maps to the `trusted` arg; when false the client
bypasses X509 chain validation (`ValidateServerCertificate` returns true
early). Equivalent to the production SDK's `AllowUntrustedServerCertificate`.
- `AuthenticationMode` default `HistorianNative`; `CertificateInfo.CertificateName`
supplies the `https://{certificateName}:{port}` SNI/host identity.
---
## 4. gRPC service surface (full RPC list)
All methods are unary (`MethodType 0`). Names map 1:1 onto the 2020 WCF
operations the production SDK already understands.
### HistoryService (`/HistoryService/…`)
`GetInterfaceVersion`, `ExchangeKey`, `OpenConnection`, `CloseConnection`,
`UpdateClientStatus`, `RegisterTags`, `EnsureTags`, `AddStreamValues`,
`AddTagExtendedPropertyGroups`, `AddTagExtendedProperties`, `StartJob`,
`GetJobStatus`, `DeleteTagExtendedProperties`, `DeleteTags`,
`AddTagLocalizedProperties`, `DeleteTagLocalizedProperties`
### RetrievalService (`/RetrievalService/…`)
`GetRetrievalInterfaceVersion`, `StartQuery`, `GetNextQueryResultBuffer`,
`EndQuery`, `GetShardTagidsByTagnameAndSource`, `GetTagInfosFromName`,
`GetTagExtendedPropertiesFromName`, `ExecuteSqlCommand`, `StartEventQuery`,
`GetNextEventQueryResultBuffer`, `EndEventQuery`, `StartTagQuery`, `QueryTag`,
`EndTagQuery`, `GetTagLocalizedPropertiesFromName`
### StatusService (`/StatusService/…`)
`GetStatusInterfaceVersion`, `GetSystemParameter`, `SendInfo`, `RequestInfo`,
`DeleteInfo`, `GetHistorianInfo`, `StartProcess`, `StopProcess`, `PingServer`,
`PingPipe`, `ConfigureAutoStartProcess`, `GetHistorianConsoleStatus`,
`GetRuntimeParameter`, `GetSystemTimeZoneName`, `SetHistorianConsoleStatus`,
`CanUpdateAreaHierarchy`, `UpdateAreaHierarchy`, `UpdateObjectHierarchy`
### StorageService (`/StorageService/…`)
`GetInterfaceVersion`, `OpenStorageConnection`, `OpenStorageConnection2`,
`CloseStorageConnection`, `Ping`, `AddTags`, `RegisterTags`, `AddStreamValues`,
`AddStreamValues2`, `GetTagIds`, `GetTags`, `FlushMetadata`, `FlushData`,
`LoadBlocks`, `GetSnapshots`, `StartQuerySnapshot`, `NextQuerySnapshot`,
`EndSnapshot`, `Stop`, `ClearTagidPairs`, `AddTagidPairs`, `GetSFParameter`,
`SetSFParameter`, `SendSnapshotBegin`, `SendSnapshotEnd`, `SendSnapshot`,
`DeleteSnapshot`, `ClearShardTagids`, `AddShardTagids`, `SplitUnknownShards`,
`GetRemainingSnapshotsSize`, `DeleteTags`, `OpenStorageConnection2`,
`ValidateClientCredential`, `GetInfo`
### TransactionService (`/TransactionService/…`)
`ForwardSnapshot`, `ForwardSnapshotBegin`, `ForwardSnapshotEnd`,
`GetTransactionInterfaceVersion`, `AddNonStreamValuesBegin`,
`AddNonStreamValues`, `AddNonStreamValuesEnd`
> A separate `ArchestrA.CloudHistorian.Contract` assembly defines a parallel
> cloud-ingest contract (`AddHistorianValues`, `CreateTags`, `EnqueueTagDataPacket`,
> `Enqueue…`, etc.) used by `aahCloudConfigurator` / `online.wonderware.com`.
> Out of scope here; noted for completeness.
---
## 5. Representative message shapes (protobuf field numbers)
The universal result wrapper and the auth/query messages — note how thin they
are; the real structure lives inside the `bytes` fields.
```proto
// Common result wrapper (ArchestrA.Grpc.Contract.RequestStatus)
message Status {
bool bSuccess = 1; // success flag (replaces WCF return-bool)
bytes btError = 2; // native error buffer (same type/code blob as WCF err)
}
// HistoryService
message OpenConnectionRequest { bytes btConnectionRequest = 1; } // OpenConnection3 v6 buffer
message OpenConnectionResponse { Status status = 1; bytes btConnectionResponse = 2; } // 42-byte session blob
message ExchangeKeyRequest { string strHandle = 1; bytes btInput = 2; }
// StorageService — Negotiate/NTLM handshake
message ValidateClientCredentialRequest { string handle = 1; bytes inBuff = 2; }
message ValidateClientCredentialResponse { Status status = 1; bytes outBuff = 2; }
// RetrievalService
message StartQueryRequest {
uint32 uiHandle = 1;
uint32 uiQueryRequestType = 2; // RetrievalMode → QueryType, same mapping as 2020
bytes btRequestBuffer = 3; // DataQueryRequest bytes, byte-identical to WCF
}
```
Across the contract the recurring pattern is `{ Status status; bytes <payload> }`
for responses and `{ [string handle][uint …] bytes <payload> }` for requests.
**The full canonical IDL has been recovered.** All six `.proto` files were
rendered from the embedded `FileDescriptor`s and protoc-validated — see
`../out/proto/*.proto`, the portable `../out/archestra_grpc.fileset.pb`
(`FileDescriptorSet`), and the per-message field dump `../out/grpc-contract-dump.md`.
`../out/README.md` explains the contract quirks (global proto package, cross-file
name collisions, all-unary RPCs) and the 2020→gRPC read-path mapping. Regenerate
with the `protodump/` tool.
---
## 6. What this means for histsdk (if a gRPC transport is ever added)
This is **not** a request to implement anything — recording the path:
1. Add a transport enum value (e.g. `RemoteGrpc`) alongside `LocalPipe` /
`RemoteTcpIntegrated` / `RemoteTcpCertificate`.
2. Reference `Grpc.Net.Client` + `Grpc.Net.Client.Web`; build a
`GrpcChannel.ForAddress("http(s)://host:32565", { HttpHandler =
GrpcWebHandler(GrpcWeb, HttpClientHandler), … })` with HTTP/1.1.
3. Reuse **every existing payload serializer unchanged** — feed the same byte
buffers into the protobuf `bytes` fields instead of MDAS bodies. The
orchestrator call order (`GetV → ValCl×N → Open2 → Retr.GetV →
IsOriginalAllowed → StartQuery → GetNextQueryResultBuffer…`) is identical.
4. Auth: still the SSPI/Negotiate token loop via `ValidateClientCredential`,
carried in `inBuff`/`outBuff`. No per-call gRPC auth metadata needed.
5. Biggest win: **no WCF `[MessageParameter]` reverse-engineering** — the
protobuf field numbers are authoritative and stable.
Caveat: this is the **2023 R2** server contract. The production SDK targets a
2020-era server; whether that server exposes the gRPC HCAP endpoint at all is a
server-version question, not a client one. Treat this as forward-looking.
---
## 7. Artifacts (all under the separate analysis folder, none committed to histsdk)
```
histsdk-2023r2-analysis/
msi-extract/ # msiexec /a layout of SDKSetup.msi
bin/ # copied key assemblies + aahClientManaged.xml
decompiled/
Archestra.Grpc.Contract/ # full protobuf contract (services + messages)
Archestra.Historian.GrpcClient/ # transport wrappers (channel/auth/calls)
ArchestrA.CloudHistorian.Contract/ # cloud-ingest contract (out of scope)
protodump/ # .NET 10 tool: descriptor graph -> .proto / dump
out/
proto/*.proto # recovered, protoc-validated IDL (6 files)
archestra_grpc.fileset.pb # portable FileDescriptorSet (grpcurl/buf/protoc)
grpc-contract-dump.md # per-message field dump + service tables
README.md # artifact guide + contract quirks + read-path map
docs/grpc-transport.md # this file
```
gRPC redist proof in the installer:
`Redist/HistorianSDK 2023 R2/x64/{GRPCCore,GRPCNetClient,HistorianGRPCClient,HistorianGRPCContract,Protobuf}.msm`
plus shipped `Grpc.Net.Client*.dll`, `Grpc.Core.Api.dll`, `Google.Protobuf.dll`.
docs/plans/hcal-capability-matrix.md
+166
View File
@@ -0,0 +1,166 @@
# HCAL → modern-.NET reimplementation — capability matrix
Feasibility map for a clean managed-.NET client that replaces the AVEVA Historian
SDK (`aahClientManaged` / HCAL). Grounded in: the real `ArchestrA.HistorianAccess`
public surface (`aahClientManaged.xml`), the recovered **2023 R2 gRPC contract**, the
existing **histsdk** reimplementation, and the event/storage analysis in
[`histevents.md`](histevents.md).
## Legend
**Status (histsdk today)** — ✅ implemented + live-verified · 🟗 partial · ⬜ not yet
**Feasibility tier**
| Tier | Meaning | Effort |
|---|---|---|
| **DONE** | already working in histsdk | 0 |
| **TRIVIAL** | gRPC op known, payload already decoded or empty | XS (hrs) |
| **CAPTURE** | one instrument-and-capture of a native payload, then serialize + golden-byte test | S (days) |
| **BOUNDED** | gRPC op exists; decode one proprietary `bytes` payload | S–M |
| **HARD** | whole subsystem to reimplement | L (weeks) |
| **GATED** | blocked server-side — client effort doesn't unblock it | n/a |
Effort = incremental work on top of histsdk's existing infrastructure (auth chain,
transport, frame/byte primitives, test harness). All non-DONE items assume the
**gRPC transport** as the foundation (clean protobuf envelope; only the inner byte
blob needs RE).
---
## 1. Connection & session
| Capability | HCAL API | 2023 R2 gRPC op | histsdk | Tier | Notes |
|---|---|---|---|---|---|
| Probe / version | `TestConnection`, GetV | `*Service.GetInterfaceVersion` | ✅ | DONE | |
| Open connection (Process) | `OpenConnection` | `History.OpenConnection` (+ `ExchangeKey` auth) | ✅ | DONE | full auth chain works |
| Open connection (Event) | `OpenConnection` (Event type) | `History.OpenConnection` event mode | 🟗 | TRIVIAL | read path already opens it; flag = ConnectionType.Event |
| Close connection | `CloseConnection` | `History.CloseConnection` | ✅ | DONE | |
| Connection status | `GetConnectionStatus` | `Status.GetHistorianConsoleStatus` | ✅ | DONE | |
| Open/close **storage** connection | `OpenStorageConnection`, `CloseStorageConnection` | `Storage.OpenStorageConnection2` | ⬜ | BOUNDED | needed for any data-write path; storage-engine session |
## 2. Reads — process data
| Capability | HCAL API | 2023 R2 gRPC op | histsdk | Tier | Notes |
|---|---|---|---|---|---|
| Raw / full history | `CreateHistoryQuery` → Start/MoveNext/End | `Retrieval.StartQuery`→`GetNextQueryResultBuffer`→`EndQuery` | ✅ | DONE | row buffer parsed |
| Aggregate (interp/avg/min/max/…) | `CreateHistoryQuery` (RetrievalMode) | same | ✅ | DONE | all 15 RetrievalModes mapped |
| At-time / value-at | (interp window) | same | ✅ | DONE | |
| Analog summary | `CreateAnalogSummaryQuery` | `Retrieval.StartQuery` (summary mode) | 🟗 | BOUNDED | mode variant of existing query |
| State summary | `CreateStateSummaryQuery` | `Retrieval.StartQuery` (state mode) | ⬜ | BOUNDED | extra row layout to decode |
| Block read | `ReadBlocks` | `Storage.LoadBlocks` | ⬜ | BOUNDED | low-level; rarely needed |
## 3. Reads — events
| Capability | HCAL API | 2023 R2 gRPC op | histsdk | Tier | Notes |
|---|---|---|---|---|---|
| Event query | `CreateEventQuery` → Start/MoveNext/End | `Retrieval.StartEventQuery`→`GetNextEventQueryResultBuffer`→`EndEventQuery` | ✅ | DONE | rows + typed property bag parsed; CM_EVENT registration done |
| Event filters | `EventQuery.AddEventFilter` / `AddEventFilterCondition` | filter bytes in StartEventQuery request | ⬜ | BOUNDED | encode filter predicate into request buffer |
## 4. Browse & metadata
| Capability | HCAL API | 2023 R2 gRPC op | histsdk | Tier | Notes |
|---|---|---|---|---|---|
| Tag name browse | `CreateTagQuery` → `GetTagNames` | `Retrieval.StartTagQuery`/`QueryTag` (or LikeTagnames) | ✅ | DONE | wildcard works |
| Tag metadata | `GetTagInfoByName`, `TagQuery.GetTagInfo` | `Retrieval.GetTagInfosFromName` | ✅ | DONE | |
| Extended properties (read) | `GetTagExtendedPropertiesByName` | `Retrieval.GetTagExtendedPropertiesFromName` | ⬜ | BOUNDED | TEP buffer decode |
| Localized properties (read) | `GetTagLocalizedPropertiesByName` | `Retrieval.GetTagLocalizedPropertiesFromName` | ⬜ | BOUNDED | |
| SQL passthrough | `ExecuteSqlCommand` | `Retrieval.ExecuteSqlCommand` | ⬜ | TRIVIAL | thin string-in / status-out |
## 5. Tag configuration (writes)
| Capability | HCAL API | 2023 R2 gRPC op | histsdk | Tier | Notes |
|---|---|---|---|---|---|
| Create analog tag | `AddTag` | `History.EnsureTags` (EnsT2) | ✅ | DONE | Float/Double/Int2/Int4/UInt2/UInt4 + scaling |
| Create string/discrete tag | `AddTag` | `History.EnsureTags` | ⬜ | GATED/BOUNDED | native AddTag rejects these types server-side; needs different metadata path |
| Delete tag(s) | `DeleteTags` | `History.DeleteTags` | ✅ | DONE | |
| Rename tag(s) | `RenameTags` | (History op) | ⬜ | BOUNDED | `AllowRenameTags` param already probed |
| Add/Delete extended properties | `AddTagExtendedProperties`, `DeleteTagExtendedPropertiesByName` | `History.AddTagExtendedProperties` / `DeleteTagExtendedProperties` | ⬜ | BOUNDED | gRPC op + TEP serialize |
| Add/Delete localized properties | `AddTagLocalizedProperties`, `DeleteTagLocalizedPropertiesByName` | `History.AddTagLocalizedProperties` / `DeleteTagLocalizedProperties` | ⬜ | BOUNDED | |
## 6. Data writes — values
| Capability | HCAL API | 2023 R2 gRPC op | histsdk | Tier | Notes |
|---|---|---|---|---|---|
| Stream process values | `AddStreamedValue(HistorianDataValue)` | `Storage.AddStreamValues` | ⬜ | **GATED** | runtime cache only ingests from IOServer/AppServer pipelines (`129 Tag not found in cache`). Not a client bug |
| Stream **events** | `AddStreamedValue(HistorianEvent)` | `Storage.AddStreamValues` (event VTQ) | ⬜ | **CAPTURE** | full path mapped; need `CCommonArchestraEventValue::PackToVtq` blob bytes. See histevents.md |
| Non-streamed / historical insert | `AddNonStreamedValue`, `SendNonStreamedValues` | `Transaction.AddNonStreamValues(Begin/End)` | ⬜ | BOUNDED | explicit original-data insert via Transaction svc; verify ingest permission on target |
| Versioned streamed value | `AddVersionedStreamedValue` | `Storage.AddStreamValues2` | ⬜ | CAPTURE | revision flag on the VTQ |
## 7. Revisions / edits (modify stored data)
| Capability | HCAL API | 2023 R2 gRPC op | histsdk | Tier | Notes |
|---|---|---|---|---|---|
| Insert/update/delete revision values | `AddRevisionValue(s)`, `AddRevisionValuesBegin/End` | (storage-engine / transaction path) | ⬜ | HARD | prior RE: revision-write needs the non-WCF **storage-engine pipe** (`STransactPipeClient2`), not the WCF/gRPC surface |
| Event update/delete (revise) | `HistorianEvent.Update/.Delete` | `UpdateEventStatus` (+ revised VTQ) | ⬜ | CAPTURE | RevisionVersion + Update/Delete flags in the event VTQ |
## 8. Status & system info
| Capability | HCAL API | 2023 R2 gRPC op | histsdk | Tier | Notes |
|---|---|---|---|---|---|
| System parameter | `GetSystemParameter` | `Status.GetSystemParameter` | ✅ | DONE | |
| Runtime parameter | `GetRuntimeParameter` | `Status.GetRuntimeParameter` | ⬜ | TRIVIAL | same shape as GetSystemParameter |
| Historian info | `GetHistorianInfo` | `Status.GetHistorianInfo` | 🟗 | BOUNDED | GETHI buffer; partially decoded (incl. EventStorageMode @ offset 514) |
| Server timezone | `GetSystemTimeZoneInfo` | `Status.GetSystemTimeZoneName` | ⬜ | TRIVIAL | |
| Historization status | `GetHistorizationStatus` | `Status` op | ⬜ | BOUNDED | |
| Store-and-forward status | `GetStoreForwardStatus` | (push events / pull GETHI) | 🟗 | HARD | currently synthesized; real read needs duplex push or a decoded pull endpoint — see store-forward plan |
## 9. Store-and-forward (offline buffering)
| Capability | HCAL API | 2023 R2 gRPC op | histsdk | Tier | Notes |
|---|---|---|---|---|---|
| SF buffering + replay | (implicit on write conns) | `Storage`/`Transaction` `*Snapshot` + `Forward*Snapshot` | ⬜ | HARD | full subsystem: local cache format, snapshot framing, recovery log, forward-on-reconnect. Pragmatic alt: a simpler local queue, not bit-faithful SF |
| Event SF | (event conn) | `Forward**Event**SnapshotBegin/…/End` | ⬜ | HARD | dedicated event-snapshot SF stream |
| SF parameters | Get/Set SFP | `Storage.GetSFParameter`/`SetSFParameter` | ⬜ | BOUNDED | |
## 10. Redundancy / multi-historian
| Capability | HCAL API | 2023 R2 gRPC op | histsdk | Tier | Notes |
|---|---|---|---|---|---|
| Tiered/redundant access, failover | `MultiHistorianAccess.*` (OpenConnectionToAll, AddSecondaries, partner watchdog, ReSyncTags) | N×single-historian sessions + client logic | ⬜ | HARD | mostly client-side orchestration over §1–§6; build last |
| Replication config | (server `aahReplication`) | — | ⬜ | GATED | server-side concern |
---
## Roll-up & recommended cut line
**Phase 0 — already DONE (✅):** probe · open/close · raw+aggregate+at-time reads ·
event reads · tag browse · tag metadata · system parameter · connection status ·
create/delete analog tag. This is a usable modern client **today**.
**Phase 1 — TRIVIAL/BOUNDED, high value (S–M each):** ExecuteSqlCommand ·
runtime parameter · server timezone · extended/localized property read · event
filters · summary/state-summary queries · rename tags · ext/localized property
writes · GetHistorianInfo. Each is "gRPC op exists, decode one buffer, golden-byte
test." Knocks out most of the remaining read/config surface.
**Phase 2 — CAPTURE (one native capture each, S):** **event sending** (the headline
gap — fully mapped, one `PackToVtq` capture away) · versioned/non-streamed value
writes. Now feasible locally since the Historian is installed.
**Defer / simplify (HARD):** store-and-forward (do a pragmatic local queue instead of
bit-faithful SF) · revision/edit writes (separate storage-engine pipe) · multi-
historian redundancy (client orchestration, build last).
**Won't unblock from the client (GATED):** streaming **process-sample** writes
(`AddS2`) — server cache only ingests from IOServer/AppServer pipelines; confirm your
ingestion model rather than chasing this. Non-analog tag creation likely needs a
distinct server path.
## Cross-cutting realities (apply to every non-DONE row)
- **Inner payloads stay proprietary** even under gRPC — the `bytes` fields carry
native VTQ / CTagMetadata / event-value formats. These are **version-sensitive**;
pin to the server version probed at connect and fail closed on mismatch.
- **Validation needs a live Historian** — now available locally, which is what makes
the CAPTURE-tier items practical.
- **Support tradeoff** — you take on maintenance across Historian versions in exchange
for shedding the stock SDK's bugs (mixed-mode marshaling, WCF quirks, global state)
for the surface you cover.
## Bottom line
A modern-.NET HCAL replacement is **feasible and ~60–70% done** for a typical
read+browse+config+event-read workload. The remaining high-value surface is mostly
**BOUNDED/CAPTURE** (incremental, well-understood), with only store-and-forward,
revision-edit, and redundancy being genuine **HARD** subsystems — and one true wall
(**GATED** process-sample writes) that no client can remove.
docs/plans/hcal-roadmap.md
+171
View File
@@ -0,0 +1,171 @@
# HCAL modern-.NET client — implementation roadmap
Ordered, actionable plan to grow **histsdk** from "reads + basic config" into a broad
HCAL replacement, built on the **2023 R2 gRPC transport**. Derived from
[`hcal-capability-matrix.md`](hcal-capability-matrix.md); event details in
[`histevents.md`](histevents.md).
> Move to the repo's `docs/plans/` when execution starts. Each work item lands as: a
> protocol serializer/parser + golden-byte unit test + an env-gated live integration
> test against the local Historian.
## Guiding principles
1. **gRPC-first.** New ops go on the `RemoteGrpc` transport (clean protobuf envelope);
the inner `bytes` blob is the only thing to RE. Keep WCF as the legacy/Windows path.
2. **Two tests per op, always.** A golden-byte test (deterministic, no server) **and** a
gated live test (`HISTORIAN_GRPC_HOST` / `HISTORIAN_HOST`). No op is "done" without both.
3. **Version-pin, fail closed.** Read server version at connect; gate every byte
serializer on it; throw `ProtocolEvidenceMissingException` on mismatch — never
best-effort parse.
4. **Capture once, encode forever.** For CAPTURE-tier items, instrument one native call,
save a sanitized fixture under `fixtures/protocol/`, then implement against the fixture.
5. **Ship per milestone.** Each milestone is independently releasable.
Effort: **S** ≈ days · **M** ≈ ~1 week · **L** ≈ weeks. Estimates are incremental on
histsdk's existing infra (auth chain, transport, frame primitives, test harness).
---
## Milestone 0 — Foundation: full gRPC parity for the DONE surface (M)
*Goal: everything already working over WCF also works over `RemoteGrpc`, so the whole
read/browse/status surface is Windows-free and the gRPC stack is the default path.*
| ID | Work | gRPC op | Files | Verify | Effort |
|---|---|---|---|---|---|
| R0.1 | Route browse over gRPC | `Retrieval.StartTagQuery`/`QueryTag` or `GetTagInfosFromName` | `Grpc/HistorianGrpcReadOrchestrator` (+ new `…GrpcBrowseClient`), `Historian2020ProtocolDialect` | browse tags live over gRPC | S |
| R0.2 | Route tag metadata over gRPC | `Retrieval.GetTagInfosFromName` | dialect + grpc client | metadata matches WCF result | S |
| R0.3 | Route status/system-param over gRPC | `Status.GetSystemParameter`, `Status.GetHistorianConsoleStatus` | new `Grpc/HistorianGrpcStatusClient` | system param + conn status live | S |
| R0.4 | Probe over gRPC | `*.GetInterfaceVersion` | grpc clients | `ProbeAsync` Windows-free | XS |
| R0.5 | **Capture harness for gRPC payloads** | n/a | reuse `instrument-wcf-*` tooling (same byte blobs) + add a `grpc-call-dump` helper | dump any request/response `bytes` to a fixture | S |
| R0.6 | **Version gate** | server version at connect | `HistorianClientOptions`, orchestrators | mismatched version → throws | S |
**Acceptance:** the entire Phase-0 capability set runs end-to-end over `RemoteGrpc`
(incl. Linux), no WCF on the path. 188+ unit tests green; live gRPC integration suite green.
---
## Milestone 1 — Cheap surface completion (TRIVIAL/BOUNDED) (M–L total)
*Goal: knock out the remaining read/config surface. Order = ascending payload difficulty.*
### 1a. Trivial (XS–S each, no new payload format)
| ID | Capability | gRPC op | Notes |
|---|---|---|---|
| R1.1 | `ExecuteSqlCommandAsync` | `Retrieval.ExecuteSqlCommand` | string in → `iRetValue` + status; thin |
| R1.2 | `GetRuntimeParameterAsync` | `Status.GetRuntimeParameter` | mirror `GetSystemParameter` |
| R1.3 | `GetServerTimeZoneAsync` | `Status.GetSystemTimeZoneName` | string out |
### 1b. Bounded (decode one `bytes` payload; S–M each)
| ID | Capability | gRPC op | Payload to decode | Depends |
|---|---|---|---|---|
| R1.4 | `GetHistorianInfoAsync` | `Status.GetHistorianInfo` | GETHI buffer (partly decoded; incl. `EventStorageMode`@514) | R0.5 |
| R1.5 | Extended-property **read** | `Retrieval.GetTagExtendedPropertiesFromName` | TEP result buffer | R0.5 |
| R1.6 | Localized-property **read** | `Retrieval.GetTagLocalizedPropertiesFromName` | localized buffer | R0.5 |
| R1.7 | Event **filters** | filter bytes in `Retrieval.StartEventQuery` | filter predicate encoding (name/op/value) | R0.5 |
| R1.8 | Analog-summary query | `Retrieval.StartQuery` (summary mode) | summary row layout | — |
| R1.9 | State-summary query | `Retrieval.StartQuery` (state mode) | state-summary row layout | — |
### 1c. Bounded config writes (S–M each)
| ID | Capability | gRPC op | Payload | Notes |
|---|---|---|---|---|
| R1.10 | `RenameTagsAsync` | History rename op | rename request buffer | `AllowRenameTags` already probed |
| R1.11 | Extended-property **write** | `History.AddTagExtendedProperties` (+ groups) / `DeleteTagExtendedProperties` | TEP serialize | mirror analog CTagMetadata discipline |
| R1.12 | Localized-property **write** | `History.AddTagLocalizedProperties` / `DeleteTagLocalizedProperties` | localized serialize | |
| R1.13 | Non-analog tag create (string/discrete) | `History.EnsureTags` | distinct CTagMetadata variant | ⚠ native AddTag rejected some types — confirm server path first; may be GATED |
**Acceptance:** read + browse + metadata + system/status + property R/W + summaries +
event-filtered reads + rename all live-verified over gRPC.
---
## Milestone 2 — Event sending (CAPTURE) (S–M) ← headline gap
*Goal: `SendEventAsync(HistorianEvent)`. Path fully mapped in histevents.md; one capture away.*
| ID | Work | Detail |
|---|---|---|
| R2.1 | Capture the event value blob | Instrument `CCommonArchestraEventValue::PackToVtq` (or dump the VTQ value bytes) on a live `AddStreamedValue(HistorianEvent)`; save sanitized fixture |
| R2.2 | `HistorianEventWriteProtocol` | Serialize header (`ReceivedTime, EventType, EventTime, Id, RevisionVersion, IsUpdate/IsDelete, Namespace`) + typed property bag — **inverse of `HistorianEventRowProtocol`** (reuse typemarkers `0x02/0x10/0x18/0x31/0x43/…`) |
| R2.3 | Event write orchestrator | Open **Event** connection (write mode) → register CM_EVENT (already have) → `Storage.AddStreamValues` with the event VTQ |
| R2.4 | Public API | `HistorianClient.SendEventAsync(HistorianEvent)` (+ `HistorianEvent` model: Type, EventTime, property bag) |
| R2.5 | Round-trip test | Send an event → read it back via `StartEventQuery` / `v_AlarmEventHistory2`; golden-byte on R2.2 |
**Acceptance:** an event sent from histsdk appears in the historian and is read back with
matching Type + properties. **Now practical** — Historian is installed locally.
---
## Milestone 3 — Historical / non-streamed value writes (BOUNDED) (M)
*Goal: insert original historical VTQs (backfill), the path that is NOT the gated cache push.*
| ID | Work | gRPC op |
|---|---|---|
| R3.1 | Decode non-streamed VTQ packet | `Transaction.AddNonStreamValuesBegin/AddNonStreamValues/End` |
| R3.2 | `AddHistoricalValuesAsync` | batched begin→values→end |
| R3.3 | Ingest-permission validation | confirm the target accepts original-data insert (distinct from `AddS2` cache wall) |
**Acceptance:** historical points inserted and read back. Document clearly where this
differs from (gated) streaming sample writes.
---
## Milestone 4 — HARD subsystems (deferred / optional) (L each)
Only if the use case demands them. Each is a real subsystem, not an op.
| ID | Capability | Approach | Risk |
|---|---|---|---|
| R4.1 | Store-and-forward | **Pragmatic local queue** (durable outbox + replay on reconnect) rather than bit-faithful SF cache + `Forward*Snapshot`. Faithful SF = decode SF cache format + snapshot framing + recovery log | high; consider "good enough" |
| R4.2 | Revision / edit writes | `AddRevisionValue(s)` go via the **non-WCF storage-engine pipe** (`STransactPipeClient2`) — separate transport RE | high |
| R4.3 | Real store-forward **status** | duplex push (`SetStoreForwardEvent`) or a decoded pull endpoint — see store-forward plan | medium |
| R4.4 | Multi-historian / redundancy | client-side orchestration over N single-historian sessions (failover, ReSyncTags, partner watchdog) — build last | medium |
---
## Won't-do from the client (GATED)
- **Streaming process-sample writes** (`AddStreamedValue(HistorianDataValue)` / `AddS2`):
runtime cache only ingests from configured IOServer/AppServer pipelines. Confirm your
ingestion architecture instead of pursuing this.
---
## Cross-cutting workstreams (run alongside all milestones)
- **CW-1 Capture tooling** (enables R0.5, R1.x, R2.1): one reusable "call op → dump
request/response `bytes` → sanitized fixture" path. Highest leverage — do first.
- **CW-2 Version compatibility:** matrix of tested Historian versions; serializers keyed
by version; CI gate.
- **CW-3 Cross-platform CI:** run the gRPC suite on Linux/macOS (transport is portable;
explicit-cred auth path).
- **CW-4 Fixtures discipline:** every new op ships a `fixtures/protocol/<op>/` golden file;
sanitize hostnames/tags/GUIDs before commit.
- **CW-5 Public API shape:** keep the modern surface (async, `IAsyncEnumerable`,
cancellation, options record, DI-friendly) consistent as the surface grows.
---
## Sequencing (critical path)
```
CW-1 capture tooling ─┐
M0 gRPC parity ───────┼─→ M1 cheap surface ─→ M2 event send ─→ M3 historical writes ─→ (M4 optional)
R0.6 version gate ────┘
```
Recommended first sprint: **CW-1 + M0 (R0.1–R0.6)** → a fully Windows-free, version-safe
gRPC client at today's capability. Second sprint: **M1a + M2** (cheap wins + the headline
event-send). M3/M4 as demand dictates.
## One-glance status
| Milestone | Tier | Effort | Value | When |
|---|---|---|---|---|
| M0 gRPC parity + capture tooling | foundation | M | unblocks everything, Windows-free | **now** |
| M1 cheap surface | TRIVIAL/BOUNDED | M–L | most remaining read/config | next |
| M2 event send | CAPTURE | S–M | headline write capability | next |
| M3 historical writes | BOUNDED | M | backfill | on demand |
| M4 SF / revisions / redundancy | HARD | L×N | parity completeness | defer |
docs/plans/histevents.md
+299
View File
@@ -0,0 +1,299 @@
# How a HistorianEvent reaches the Historian DB files
Living analysis doc. Traces an event end-to-end: client API → wire → server
storage backend (SQL **Database** vs history **Blocks** `.dat`) → read-back.
Evidence base: 2023 R2 `aahClientManaged.dll` (decompiled `ArchestrA.HistorianAccess`,
`HistorianEvent`, `HistorianEventPropertyType`), native `aahStorage.exe` (string
analysis), the recovered gRPC + CloudHistorian contracts, and the histsdk read-side
reverse-engineering (CM_EVENT registration + event-row parser).
Status legend: ✅ proven (from binary) · 🔶 strong inference · ❓ open.
---
## TL;DR
An event is **not a distinct wire message**. The client turns each `HistorianEvent`
into a `HistorianDataValue` of type `Event` against the built-in **`CM_EVENT`** tag,
marshals it into a native VTQ, and **streams it like any tag value** on a dedicated
*Event* connection. Events are batched into an opaque serialized **event data packet**
and delivered (with their own store-and-forward "event snapshot" path). On the server
they are persisted into **one of two backends, chosen by a server-configured
`EventStorageMode`**: a SQL **Database**, or the history **Blocks** (`.dat`) files.
```
HistorianEvent (Type + typed property bag)
└─(AddStreamedValue)→ HistorianDataValue{Type=Event, TagKey=CM_EVENT, EventTime, Q=192}
└─ HistorianEvent.PackToVtq → CCommonArchestraEventValue::PackToVtq (native value blob)
└─ HISTORIAN_VALUE2 (44B; blob ptr @+33) → HistorianClient.AddHistorianValue → queue
└─ flush → EnqueueEventDataPacket{ byte[] SerializedBytes } (batched VTQs)
└─ SERVER: aahEventStorage.exe (InSQLEventSystem)
per-client event-tag pipeline → recovery-log WAL → backend:
• Blocks → elastic snapshot → frozen → history .dat (Circular/Permanent)
• Database → ArchestrAEvents.EventStorage.Contract assembly → SQL (A2ALMDB)
→ EventReplication (redundant historians)
└─ (offline) → store-and-forward → ForwardEventSnapshotBegin/…/End on reconnect
read-back: Retr.StartEventQuery / SQL provider views (Events, v_AlarmEventHistory2, v_EventSnapshot)
```
---
## 1. The `HistorianEvent` object ✅
Decompiled `ArchestrA.HistorianEvent` — a structured header plus a **typed property bag**:
- **Header fields:** `ID`/`Id` (Guid), `Type`/`EventType` (string, e.g. `"Alarm.Set"`,
`"User.Write"`), `EventTime` (DateTime), `ReceivedTime`, `Severity` (ushort),
`Priority` (ushort), `IsAlarm`, `IsSilenced`, `System`, `Source`, `Source_Name`,
`Area`, `Namespace`, `DisplayText`.
- **Revision fields:** `RevisionVersion` (ushort), `Delete` (bool), `Update` (bool) —
events are revisable (see §4 UpdateEventStatus).
- **Property bag:** `AddProperty(name, value, HistorianEventPropertyType, …)` with typed
overloads. `HistorianEventPropertyType` (alphabetical enum):
`Blob, Boolean, Byte, Date, DateTime, Decimal, Double, Duration, Float, Guid, Hex,
Int, Integer, Long, Short, String, Time, UnsignedByte, UnsignedInt, UnsignedLong,
UnsignedShort, Undefined`.
These map onto the wire property-bag the histsdk **read** parser already decodes
(`HistorianEventRowProtocol`): typemarkers `0x02` Boolean, `0x10` Guid, `0x18` FILETIME,
`0x31` Int32, `0x43` UTF-16 string, … — i.e. the write enum and the read typemarkers are
two views of the same typed-value format. The event-send serialization is the inverse of
that read parser.
---
## 2. Client send path — an event becomes a streamed VTQ ✅
From decompiled `ArchestrA.HistorianAccess` (line refs into the decompile):
1. **Open an Event connection.** `HistorianConnectionArgs.ConnectionType =
HistorianConnectionType.Event`, `ReadOnly = false` (sample `Step10.SendEvents`).
2. **Default event tag.** `CreateDefaultEventTag()` (`:3006`) registers tag `CM_EVENT` /
"AnE Event" / `TagDataType = Event` and stores `eventTagHandle`. Same CM_EVENT
registration histsdk reverse-engineered (RTag2 + EnsT2; tag id
`353b8145-5df0-4d46-a253-871aef49b321`).
3. **Wrap as VTQ.** `AddStreamedValue(HistorianEvent)` (`:3123`):
```csharp
historianDataValue.objValue = historianEvent; // header + property bag
historianDataValue.DataValueType = HistorianDataType.Event;
historianDataValue.TagKey = eventTagHandle; // CM_EVENT
historianDataValue.StartDateTime = historianEvent.EventTime;
historianDataValue.OpcQuality = 192;
return AddStreamedValue((ConnectionIndex)1, historianDataValue, false, out error); // 1=Event
```
4. **Marshal + queue.** The private `AddStreamedValue` (`:3173`):
- builds a 44-byte native `HISTORIAN_VALUE2` (`InitBlockUnaligned(…,0,44)`),
- `HistorianAccessUtil.ConvertManagedStructToUnmanagedStruct(value, &HV2, bVersioned…)`
— its `case HistorianDataType.Event` (`HistorianAccessUtil:89`) calls
**`HistorianEvent.PackToVtq(out byte[])`** to produce the event value blob, whose pointer
is placed at `HISTORIAN_VALUE2+33` (freed after send; offset 33 is the value-union pointer
used for Event/String types),
- `HistorianClient.AddHistorianValue(client, &HV2, &err)` (`:3209`) queues the VTQ into
the native delivery buffer and returns immediately.
So an event uses the **same streaming machinery as a process value**; only `DataValueType`
(`Event`) and the target tag (`CM_EVENT`) differ.
### 2a. Event value serialization — `HistorianEvent.PackToVtq` ✅/🔶
`HistorianEvent.PackToVtq` (`HistorianEvent:1392`) populates a native
**`CCommonArchestraEventStruct`** then hands it to the **native** packer
`CCommonArchestraEventValue::PackToVtq(…, 192, 192, vtq)` (Q=192), associated with the
built-in `EVENT_TAGID` / `EVENT_TAGNAME` (`CTagMetadata.CommonArchestraEvent`). The actual
byte layout is produced in C++ — **not visible in managed code** — so pinning exact write
bytes needs a wire/IL capture, exactly as the read side did. But the **field set + order**
the managed code writes into the struct is now known:
```
SetReceivedTime (uint64 FILETIME, from UniqueTime.GetUniqueFileTime — unique/monotonic)
SetEventType (wchar* string, e.g. "Alarm.Set")
SetEventTime (uint64 FILETIME, from EventTime)
SetId (GUID)
SetRevisionVersion (uint16)
SetIsUpdate (bool) ← revision flags
SetIsDelete (bool)
Namespace (string, trimmed, non-printable-validated)
…then the typed property bag: Dictionary<string, Tuple<HistorianEventPropertyType, object>>
```
This matches `HistorianEventRowProtocol` on read: the property bag is name→(type,value) with
the same typed-value encoding (typemarkers `0x02/0x10/0x18/0x31/0x43/…`). So a managed
event-send serializer is tractable: emit the header struct fields above, then the typed
property bag in the read parser's format. The remaining unknown is only the exact native
framing offsets — best obtained by capturing one `PackToVtq` output, then golden-byte testing.
---
## 3. Event transport / delivery pipeline ✅ (CloudHistorian + gRPC contracts)
Events have a **dedicated, batched** connection + delivery pipeline, distinct from tag data
but structurally parallel:
| Stage | Event op | Tag-data analogue |
|---|---|---|
| Open connection | `OpenEventConnection2 { byte[] ClientInfo } → { byte[] ServerInfo }` | OpenConnection |
| Send batch | `EnqueueEventDataPacket { byte[] SerializedBytes }` | `EnqueueTagDataPacket { byte[] SerializedBytes }` |
| Store-and-forward | `ForwardEventSnapshotBegin / ForwardEventSnapshot / ForwardEventSnapshotEnd` | `ForwardSnapshot…` |
| Revise | `UpdateEventStatus` | (revision write) |
Key point: the **event data packet is an opaque serialized byte buffer** (`SerializedBytes`,
DataMember `d`) — the queued event VTQs batched together, exactly the same envelope shape as
the tag data packet. On-prem this is what the storage-streaming op (`AddStreamValues`)
carries; in the cloud variant it is `EnqueueEventDataPacket`.
Validation surfaced via error codes: `InvalidAlarmEventPropertyLength=212`,
`AlarmEventPropertyHasNonPrintableChar=214`, `AlarmEventPropertyHasInvalidSpecialChar=215`,
`AlarmEventPropertyNameIsAReservedName=216` — the server validates alarm/event property
names + values on ingest.
Offline → events spool to the **store-and-forward** cache and replay as **event snapshots**
(`ForwardEventSnapshot*`) on reconnect — a separate SF stream from tag-data snapshots.
---
## 4. Revisions / updates ✅
`HistorianEvent.Update` / `.Delete` / `.RevisionVersion` + the contract's
`UpdateEventStatus` op mean events are not write-once: an event can be re-sent to update or
delete a previously stored event (e.g. alarm acknowledge/clear), bumping `RevisionVersion`.
---
## 5. The storage-backend switch — `EventStorageMode` ✅
The client reads the server's event-storage backend from `HISTORIAN_INFO` **byte offset 514**
(`HistorianAccess` `:5715`):
```csharp
EventStorageMode = (info[514] == -1) ? Unsupported
: (info[514] == 0) ? Database // SQL Server
: Blocks; // history .dat blocks
```
`HistorianEventStorageMode ∈ { Database, Blocks, Unsupported }`. The destination is a
**server** decision; the client streams the same VTQ regardless.
---
## 6. Server side — where it lands ✅ (confirmed on the live local install)
The server-side event component is **`aahEventStorage.exe`** (service `InSQLEventSystem`,
"AVEVA Historian Event System"; plus `aahEventSvc.exe`), at
`…\Wonderware\Historian\x64\aahEventStorage.exe`. Its string table maps the full pipeline:
```
event packets / forwarded snapshots → per-client "event tag pipeline" → batch enqueue
→ Event Storage Recovery Log (WAL; "enqueuing N events to log",
path SystemParameter EventStorageLogPath = C:\Historian\Data\Logs\EventStorage)
→ persist to the active backend:
Block Storage ("Enabled Block Storage for events") → history .dat blocks
Database ("storing N events in database") → SQL via loaded managed
assembly ArchestrAEvents.EventStorage.Contract.EventStorageDatabaseConnection
(e.g. ";Initial Catalog=A2ALMDB;Integrated Security=true;Encrypt=True;…")
→ also fed to EventReplication (aahReplication.exe) for redundant historians
```
So persistence is **pluggable** (a loadable connection assembly) and dual-mode, guarded by a
recovery log. Which backend is live depends on configuration (the `EventStorageMode` of §5).
### This historian = **Block storage** (verified)
- `C:\Historian\Data\Circular` holds **527 `.dat` history blocks** (`Permanent` empty); the
EventStorage recovery log dir exists. `aahEventStorage` logs `"Enabled Block Storage for
events"`.
- SDK-shape alarm/events are present and retrievable: `Runtime.dbo.v_AlarmEventHistory2`
returns 224 rows over the last 30 days.
- `A2ALMDB` (the System-Platform alarm DB the connection string references) is **not present**
here — that path is only used when integrated with AVEVA System Platform alarming. Absent
it, ArchestrA events land in **blocks**, exactly as `aahStorage.exe` advertises (`"Stores
ArchestrA Event Data"`, snapshot→block).
### The SQL surface is **provider-backed views, not physical tables** ✅
In `Runtime`, the rich event objects are **views with NULL `OBJECT_DEFINITION`** — i.e. the
historian's OLE DB History provider exposes them as virtual/extension tables that read the
block store, *not* stored T-SQL:
- `Events`, `v_EventHistory`, `v_EventSnapshot`, `v_EventStringSnapshot`, **`v_AlarmEventHistory2`**
(columns: `EventStampUTC`, `AlarmState`, `TagName`, `Description`, `Area`, `Type`, `Value`,
`Priority`, `Category`, `Provider`, `Operator`, `DomainName`, `UserFullName`, `MilliSec`, …)
— these are the read-back of the SDK alarm/event property bag.
So `SELECT … FROM Events` (and `v_AlarmEventHistory2`) is **the provider reading the block
store**, which is why the handoff could query events even though they live in `.dat` blocks.
### Database-mode physical store
When events ARE stored in SQL (Database mode / A2ALMDB integration), the writer is the loaded
`ArchestrAEvents.EventStorage.Contract` connection assembly doing batched inserts ("storing N
events in database", "creating event storage database connection role"). The exact table
schema there is the A2ALMDB alarm schema (not present on this box to dump).
### Read-back ✅
Uniform regardless of backend: `Retr.StartEventQuery` → `GetNextEventQueryResultBuffer`
(provider) surfaces events from wherever they were stored — so histsdk `ReadEventsAsync` is
mode-agnostic. Engine filter note: `"EventTime filtering can only be specified through
StartDateTime and EndDateTime"`.
## 6b. Two different "event" subsystems — don't conflate ✅
| | Classic event **detectors** | ArchestrA **alarms/events** (the SDK path) |
|---|---|---|
| What | server-side detectors watching tag conditions | client-streamed `HistorianEvent` (alarms, user events) |
| Config/store | `Runtime.dbo._EventTag` (TagName, DetectorTypeKey, DetectorString, Action*, ScanRate, Edge, Priority) | CM_EVENT / CommonArchestraEvent tag |
| History | **physical** `BASE TABLE Runtime.dbo.EventHistory` (`EventLogKey, TagName, DateTime, DetectDateTime, Edge`); 30 rows | block store (or A2ALMDB), surfaced via `v_AlarmEventHistory2` / `v_EventSnapshot` |
| Source | evaluated by the server | sent via `AddStreamedValue(HistorianEvent)` |
`AddStreamedValue(HistorianEvent)` feeds the **right column** (ArchestrA alarms/events) — it is
**not** the classic `EventHistory` detector log.
---
## 7. Relationship to histsdk
- histsdk implements event **reads** only (`ReadEventsAsync` via `StartEventQuery`); its
CM_EVENT EnsT2/RTag2 dance is read-subscription registration.
- Event **writing** is unimplemented but viable. Chain to replicate: Event-type connection →
register CM_EVENT (done) → serialize `HistorianEvent` (header + typed property bag) into the
event-VTQ value blob (inverse of `HistorianEventRowProtocol`) → batch into an event data
packet → stream via `AddStreamValues` (2023 R2 gRPC: `StorageService.AddStreamValues`).
---
## Open threads
- 🔶 Event value blob: field set/order known (§2a); **exact native framing** still needs one
`CCommonArchestraEventValue::PackToVtq` output capture + golden-byte test (mirror the read-side
`HistorianEventRowProtocol` reverse-engineering). Now feasible locally — the live historian is
installed, so the same instrument-and-capture approach used for reads applies.
- ❓ `EnqueueEventDataPacket.SerializedBytes` packet framing (header + N event VTQs batched).
- ✅ Database-mode store: server writer is `aahEventStorage.exe` loading the managed
`ArchestrAEvents.EventStorage.Contract` connection assembly; SQL retrieval surface is the
provider-backed `Events` / `v_AlarmEventHistory2` / `v_EventSnapshot` views (NULL T-SQL def).
This box runs **Block storage** (A2ALMDB absent). A2ALMDB physical schema still un-dumped (needs
a System-Platform-integrated box).
- ✅ `ArchestraEvent` vs `CommonArchestraEvent`: send path packs **CommonArchestraEvent** via
`EVENT_TAGID`; both are event-tag schemas in `CTagMetadata` (the server stores either).
- ❓ `UpdateEventStatus` wire payload for `Update`/`Delete` revisions.
- ❓ `EventStorage` recovery-log (`C:\Historian\Data\Logs\EventStorage`) on-disk format (WAL).
- 🔶 Decompile `ArchestrAEvents.EventStorage.Contract.dll` (managed) for the exact DB insert
contract/schema — locate it (not under `…\Wonderware\Historian`; check GAC / Framework\Bin).
---
## Changelog
- Rev 4 (live local install): confirmed server side. `aahEventStorage.exe` (`InSQLEventSystem`)
is the event store engine — per-client event-tag pipeline → recovery-log WAL → Block storage
OR SQL (loadable `ArchestrAEvents.EventStorage.Contract` assembly) → EventReplication. This box
uses **Block storage** (527 `.dat` in `C:\Historian\Data\Circular`; A2ALMDB absent). SQL
`Events`/`v_AlarmEventHistory2`/`v_EventSnapshot` are **provider-backed views over the blocks**
(NULL `OBJECT_DEFINITION`), not physical tables — `v_AlarmEventHistory2` (224 rows/30d) is the
SDK-event read surface. Distinguished the classic event-detector subsystem (`_EventTag` →
physical `EventHistory`) from the ArchestrA alarm/event path (the SDK's target).
- Rev 3: event value serialization pinned to native `CCommonArchestraEventValue::PackToVtq`
via managed `HistorianEvent.PackToVtq`; documented the `CCommonArchestraEventStruct` field
set/order (ReceivedTime, EventType, EventTime, Id, RevisionVersion, IsUpdate/IsDelete,
Namespace, typed property bag) and the path to a managed send serializer.
- Rev 2: HistorianEvent structure + HistorianEventPropertyType enum; client marshaling
(HISTORIAN_VALUE2 / ConvertManagedStructToUnmanagedStruct / AddHistorianValue); dedicated
event pipeline (OpenEventConnection2 / EnqueueEventDataPacket / ForwardEventSnapshot /
store-and-forward); revisions (Update/Delete/UpdateEventStatus); Blocks-mode clarified
(events = generic VTQ snapshots, no event-specific block code).
- Rev 1: client send path, EventStorageMode switch, Blocks/Database backends, read-back.