From a530ae0f1097626a95012b46076c5a0211e979a1 Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Fri, 19 Jun 2026 14:28:34 -0400 Subject: [PATCH] docs/plans: import 2023 R2 gRPC analysis + HCAL reimpl roadmap MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Version-control the planning docs alongside the code they describe: - grpc-transport.md — 2023 R2 gRPC transport analysis (sanitized source path) - hcal-capability-matrix.md — HistorianAccess surface x gRPC ops x histsdk status x feasibility tiers - hcal-roadmap.md — ordered build plan M0-M4 + cross-cutting workstreams - histevents.md — how a HistorianEvent reaches the DB (client->wire->server) Co-Authored-By: Claude Opus 4.8 (1M context) --- docs/plans/grpc-transport.md | 265 ++++++++++++++++++++++++ docs/plans/hcal-capability-matrix.md | 166 +++++++++++++++ docs/plans/hcal-roadmap.md | 171 +++++++++++++++ docs/plans/histevents.md | 299 +++++++++++++++++++++++++++ 4 files changed, 901 insertions(+) create mode 100644 docs/plans/grpc-transport.md create mode 100644 docs/plans/hcal-capability-matrix.md create mode 100644 docs/plans/hcal-roadmap.md create mode 100644 docs/plans/histevents.md diff --git a/docs/plans/grpc-transport.md b/docs/plans/grpc-transport.md new file mode 100644 index 0000000..b81bcbc --- /dev/null +++ b/docs/plans/grpc-transport.md @@ -0,0 +1,265 @@ +# AVEVA Historian SDK 2023 R2 — gRPC Transport Analysis + +**Scope:** Documents the new gRPC transport that 2023 R2 adds to the Historian +Client Access Layer (HCAL). Kept deliberately **separate** from the main +`histsdk` reverse-engineering docs — this is 2023 R2 evidence, not the 2020/WCF +protocol the production SDK currently targets. + +**Source:** the 2023 R2 `HistorianSDK` installer (**not installed**). +`SDKSetup.msi` was laid out with `msiexec /a` (administrative extract, no +registration) into a local `msi-extract` staging dir, then the managed +assemblies were decompiled with `ilspycmd`. + +**Assembly versions analysed:** `2023.1219.4004.5` +(`Archestra.Grpc.Contract.dll`, `Archestra.Historian.GrpcClient.dll`, +`aahClientManaged.dll`). + +--- + +## 1. Headline finding + +The 2023 R2 gRPC transport is a **transport swap, not a protocol redesign.** +Every gRPC request/response wraps the **same opaque native binary buffers** that +the 2020/WCF-MDAS path already carries — `OpenConnection3` v6 buffer, NTLM/SSPI +`ValCl` tokens, `DataQueryRequest`, `GetNextQueryResultBuffer` row buffers, the +`Status` err blob, etc. — inside protobuf `bytes` fields. + +Concrete proof, from `Archestra.Historian.GrpcClient`: + +```csharp +// History/OpenConnection — same byte[] openParameters the WCF path built +OpenConnectionRequest request = new OpenConnectionRequest { + BtConnectionRequest = ByteString.CopyFrom(openParameters) +}; +// Retrieval/StartQuery — same queryType + DataQueryRequest bytes + handle +StartQueryRequest request = new StartQueryRequest { + UiHandle = handle, + UiQueryRequestType = queryRequestType, + BtRequestBuffer = ByteString.CopyFrom(requestBuffer) +}; +``` + +**Implication for the `histsdk` project:** all of the hard-won payload +serializers (`HistorianOpen2Protocol`, `HistorianDataQueryProtocol`, +`HistorianEventRowProtocol`, the SSPI `ValCl` token framing, the EnsT2 +`CTagMetadata` layout) transfer **unchanged**. Only the envelope around them +changes: protobuf-over-gRPC instead of binary-SOAP-over-`application/x-mdas`. +The WCF `[MessageParameter(Name=…)]` guessing that dominated the 2020 work is +gone — field names and numbers are explicit in the protobuf contract. + +--- + +## 2. Transport stack + +From `GrpcClientBase.InitializeBase(target, portNumber, securedConnection, certificateName, trusted)`: + +| Aspect | Value / behaviour | +|---|---| +| Library | `Grpc.Net.Client` + **`Grpc.Net.Client.Web`** (`GrpcWebHandler`) | +| Mode | **gRPC-Web**, `GrpcWebMode.GrpcWeb` (binary `application/grpc-web`, **not** `-text`) | +| HTTP version | **HTTP/1.1** (`GrpcWebHandler.HttpVersion = new Version(1,1)`) — *not* HTTP/2 | +| Address | `http://{target}:{port}` insecure, or `https://{certificateName}:{port}` secure | +| Inner handler | `HttpClientHandler` with custom `ServerCertificateCustomValidationCallback` | +| Compression | gzip on by default; request header `grpc-internal-encoding-request: gzip`; custom `CustomCompressionProvider` / `CustomGZipStream` used for bandwidth accounting | +| Default timeout | 60 s per call (`m_timeoutInSeconds = 60`, sent as gRPC deadline) | +| Interceptor | `ClientInterceptor` (logging hook, currently a no-op `LogCall`) | + +Because it is **gRPC-Web over HTTP/1.1**, the transport is proxy/firewall +friendly and does not require HTTP/2 negotiation — note `HistorianConnectionArgs.ProxyServer` +(e.g. `http://host:9480`) in the public API. + +### Port + +- **Default port `32565`** — `HistorianConnectionArgs.TcpPort`, *"the TCP port + of the Historian Client Access Point."* (Note this differs from the 2020 WCF + port `32568` the production SDK uses.) +- All services reach the **same host:port**; gRPC multiplexes by service path + (`/HistoryService/OpenConnection`, `/RetrievalService/StartQuery`, …). + +### Channel topology + +Five service stubs grouped into four wrapper clients, each constructing its own +`GrpcChannel` to the same endpoint: + +| Wrapper (`GrpcClientBase` subclass) | gRPC service stub(s) | +|---|---| +| `GrpcHistoryClient` | `HistoryService` + `TransactionService` (one channel) | +| `GrpcRetrievalClient` | `RetrievalService` | +| `GrpcStatusClient` | `StatusService` | +| `GrpcStorageClient` | `StorageService` | + +--- + +## 3. Authentication model (unchanged in substance) + +Auth is **still the native session handshake**, carried over gRPC instead of +WCF. There is **no per-call bearer/auth token in gRPC metadata** — the only +metadata sent is the gzip-encoding hint. Methods pass `m_metadata` (gzip) or +`null`; neither carries credentials. The server keys the session off the +`handle` GUID established by the handshake, exactly as the 2020 path does. + +Handshake operations (same byte payloads as 2020): + +- `HistoryService.GetInterfaceVersion` → version probe. +- `StorageService.ValidateClientCredential { string Handle; bytes InBuff }` + → `{ Status; bytes OutBuff }`. **`InBuff`/`OutBuff` carry the NTLM/SSPI + tokens** — same multi-round Negotiate exchange, same field names the 2020 + `ildasm` revealed (`inBuff`/`outBuff`), now first-class protobuf fields. +- `HistoryService.ExchangeKey { string StrHandle; bytes BtInput }` + → `{ Status; bytes BtOutput }` (key-exchange / cert path). +- `HistoryService.OpenConnection { bytes BtConnectionRequest }` + → `{ Status; bytes BtConnectionResponse }` — same `OpenConnection3` v6 + request buffer in, same 42-byte session blob out. + +Public-API security knobs (`aahClientManaged.xml`): + +- `HistorianConnectionArgs.ConnectionMode` — *"whether GRPC connection to the + Historian Server. **Default is true** (GRPC)."* This is the master switch + selecting gRPC vs legacy. +- `HistorianSecurityMode`: `None`, `Disabled`, `TransportWindows` + (Windows creds), `TransportCertificate` (server cert). +- `AllowUnTrustedConnection` → maps to the `trusted` arg; when false the client + bypasses X509 chain validation (`ValidateServerCertificate` returns true + early). Equivalent to the production SDK's `AllowUntrustedServerCertificate`. +- `AuthenticationMode` default `HistorianNative`; `CertificateInfo.CertificateName` + supplies the `https://{certificateName}:{port}` SNI/host identity. + +--- + +## 4. gRPC service surface (full RPC list) + +All methods are unary (`MethodType 0`). Names map 1:1 onto the 2020 WCF +operations the production SDK already understands. + +### HistoryService (`/HistoryService/…`) +`GetInterfaceVersion`, `ExchangeKey`, `OpenConnection`, `CloseConnection`, +`UpdateClientStatus`, `RegisterTags`, `EnsureTags`, `AddStreamValues`, +`AddTagExtendedPropertyGroups`, `AddTagExtendedProperties`, `StartJob`, +`GetJobStatus`, `DeleteTagExtendedProperties`, `DeleteTags`, +`AddTagLocalizedProperties`, `DeleteTagLocalizedProperties` + +### RetrievalService (`/RetrievalService/…`) +`GetRetrievalInterfaceVersion`, `StartQuery`, `GetNextQueryResultBuffer`, +`EndQuery`, `GetShardTagidsByTagnameAndSource`, `GetTagInfosFromName`, +`GetTagExtendedPropertiesFromName`, `ExecuteSqlCommand`, `StartEventQuery`, +`GetNextEventQueryResultBuffer`, `EndEventQuery`, `StartTagQuery`, `QueryTag`, +`EndTagQuery`, `GetTagLocalizedPropertiesFromName` + +### StatusService (`/StatusService/…`) +`GetStatusInterfaceVersion`, `GetSystemParameter`, `SendInfo`, `RequestInfo`, +`DeleteInfo`, `GetHistorianInfo`, `StartProcess`, `StopProcess`, `PingServer`, +`PingPipe`, `ConfigureAutoStartProcess`, `GetHistorianConsoleStatus`, +`GetRuntimeParameter`, `GetSystemTimeZoneName`, `SetHistorianConsoleStatus`, +`CanUpdateAreaHierarchy`, `UpdateAreaHierarchy`, `UpdateObjectHierarchy` + +### StorageService (`/StorageService/…`) +`GetInterfaceVersion`, `OpenStorageConnection`, `OpenStorageConnection2`, +`CloseStorageConnection`, `Ping`, `AddTags`, `RegisterTags`, `AddStreamValues`, +`AddStreamValues2`, `GetTagIds`, `GetTags`, `FlushMetadata`, `FlushData`, +`LoadBlocks`, `GetSnapshots`, `StartQuerySnapshot`, `NextQuerySnapshot`, +`EndSnapshot`, `Stop`, `ClearTagidPairs`, `AddTagidPairs`, `GetSFParameter`, +`SetSFParameter`, `SendSnapshotBegin`, `SendSnapshotEnd`, `SendSnapshot`, +`DeleteSnapshot`, `ClearShardTagids`, `AddShardTagids`, `SplitUnknownShards`, +`GetRemainingSnapshotsSize`, `DeleteTags`, `OpenStorageConnection2`, +`ValidateClientCredential`, `GetInfo` + +### TransactionService (`/TransactionService/…`) +`ForwardSnapshot`, `ForwardSnapshotBegin`, `ForwardSnapshotEnd`, +`GetTransactionInterfaceVersion`, `AddNonStreamValuesBegin`, +`AddNonStreamValues`, `AddNonStreamValuesEnd` + +> A separate `ArchestrA.CloudHistorian.Contract` assembly defines a parallel +> cloud-ingest contract (`AddHistorianValues`, `CreateTags`, `EnqueueTagDataPacket`, +> `Enqueue…`, etc.) used by `aahCloudConfigurator` / `online.wonderware.com`. +> Out of scope here; noted for completeness. + +--- + +## 5. Representative message shapes (protobuf field numbers) + +The universal result wrapper and the auth/query messages — note how thin they +are; the real structure lives inside the `bytes` fields. + +```proto +// Common result wrapper (ArchestrA.Grpc.Contract.RequestStatus) +message Status { + bool bSuccess = 1; // success flag (replaces WCF return-bool) + bytes btError = 2; // native error buffer (same type/code blob as WCF err) +} + +// HistoryService +message OpenConnectionRequest { bytes btConnectionRequest = 1; } // OpenConnection3 v6 buffer +message OpenConnectionResponse { Status status = 1; bytes btConnectionResponse = 2; } // 42-byte session blob +message ExchangeKeyRequest { string strHandle = 1; bytes btInput = 2; } + +// StorageService — Negotiate/NTLM handshake +message ValidateClientCredentialRequest { string handle = 1; bytes inBuff = 2; } +message ValidateClientCredentialResponse { Status status = 1; bytes outBuff = 2; } + +// RetrievalService +message StartQueryRequest { + uint32 uiHandle = 1; + uint32 uiQueryRequestType = 2; // RetrievalMode → QueryType, same mapping as 2020 + bytes btRequestBuffer = 3; // DataQueryRequest bytes, byte-identical to WCF +} +``` + +Across the contract the recurring pattern is `{ Status status; bytes }` +for responses and `{ [string handle][uint …] bytes }` for requests. + +**The full canonical IDL has been recovered.** All six `.proto` files were +rendered from the embedded `FileDescriptor`s and protoc-validated — see +`../out/proto/*.proto`, the portable `../out/archestra_grpc.fileset.pb` +(`FileDescriptorSet`), and the per-message field dump `../out/grpc-contract-dump.md`. +`../out/README.md` explains the contract quirks (global proto package, cross-file +name collisions, all-unary RPCs) and the 2020→gRPC read-path mapping. Regenerate +with the `protodump/` tool. + +--- + +## 6. What this means for histsdk (if a gRPC transport is ever added) + +This is **not** a request to implement anything — recording the path: + +1. Add a transport enum value (e.g. `RemoteGrpc`) alongside `LocalPipe` / + `RemoteTcpIntegrated` / `RemoteTcpCertificate`. +2. Reference `Grpc.Net.Client` + `Grpc.Net.Client.Web`; build a + `GrpcChannel.ForAddress("http(s)://host:32565", { HttpHandler = + GrpcWebHandler(GrpcWeb, HttpClientHandler), … })` with HTTP/1.1. +3. Reuse **every existing payload serializer unchanged** — feed the same byte + buffers into the protobuf `bytes` fields instead of MDAS bodies. The + orchestrator call order (`GetV → ValCl×N → Open2 → Retr.GetV → + IsOriginalAllowed → StartQuery → GetNextQueryResultBuffer…`) is identical. +4. Auth: still the SSPI/Negotiate token loop via `ValidateClientCredential`, + carried in `inBuff`/`outBuff`. No per-call gRPC auth metadata needed. +5. Biggest win: **no WCF `[MessageParameter]` reverse-engineering** — the + protobuf field numbers are authoritative and stable. + +Caveat: this is the **2023 R2** server contract. The production SDK targets a +2020-era server; whether that server exposes the gRPC HCAP endpoint at all is a +server-version question, not a client one. Treat this as forward-looking. + +--- + +## 7. Artifacts (all under the separate analysis folder, none committed to histsdk) + +``` +histsdk-2023r2-analysis/ + msi-extract/ # msiexec /a layout of SDKSetup.msi + bin/ # copied key assemblies + aahClientManaged.xml + decompiled/ + Archestra.Grpc.Contract/ # full protobuf contract (services + messages) + Archestra.Historian.GrpcClient/ # transport wrappers (channel/auth/calls) + ArchestrA.CloudHistorian.Contract/ # cloud-ingest contract (out of scope) + protodump/ # .NET 10 tool: descriptor graph -> .proto / dump + out/ + proto/*.proto # recovered, protoc-validated IDL (6 files) + archestra_grpc.fileset.pb # portable FileDescriptorSet (grpcurl/buf/protoc) + grpc-contract-dump.md # per-message field dump + service tables + README.md # artifact guide + contract quirks + read-path map + docs/grpc-transport.md # this file +``` + +gRPC redist proof in the installer: +`Redist/HistorianSDK 2023 R2/x64/{GRPCCore,GRPCNetClient,HistorianGRPCClient,HistorianGRPCContract,Protobuf}.msm` +plus shipped `Grpc.Net.Client*.dll`, `Grpc.Core.Api.dll`, `Google.Protobuf.dll`. diff --git a/docs/plans/hcal-capability-matrix.md b/docs/plans/hcal-capability-matrix.md new file mode 100644 index 0000000..283ec1f --- /dev/null +++ b/docs/plans/hcal-capability-matrix.md @@ -0,0 +1,166 @@ +# HCAL → modern-.NET reimplementation — capability matrix + +Feasibility map for a clean managed-.NET client that replaces the AVEVA Historian +SDK (`aahClientManaged` / HCAL). Grounded in: the real `ArchestrA.HistorianAccess` +public surface (`aahClientManaged.xml`), the recovered **2023 R2 gRPC contract**, the +existing **histsdk** reimplementation, and the event/storage analysis in +[`histevents.md`](histevents.md). + +## Legend + +**Status (histsdk today)** — ✅ implemented + live-verified · 🟗 partial · ⬜ not yet + +**Feasibility tier** +| Tier | Meaning | Effort | +|---|---|---| +| **DONE** | already working in histsdk | 0 | +| **TRIVIAL** | gRPC op known, payload already decoded or empty | XS (hrs) | +| **CAPTURE** | one instrument-and-capture of a native payload, then serialize + golden-byte test | S (days) | +| **BOUNDED** | gRPC op exists; decode one proprietary `bytes` payload | S–M | +| **HARD** | whole subsystem to reimplement | L (weeks) | +| **GATED** | blocked server-side — client effort doesn't unblock it | n/a | + +Effort = incremental work on top of histsdk's existing infrastructure (auth chain, +transport, frame/byte primitives, test harness). All non-DONE items assume the +**gRPC transport** as the foundation (clean protobuf envelope; only the inner byte +blob needs RE). + +--- + +## 1. Connection & session + +| Capability | HCAL API | 2023 R2 gRPC op | histsdk | Tier | Notes | +|---|---|---|---|---|---| +| Probe / version | `TestConnection`, GetV | `*Service.GetInterfaceVersion` | ✅ | DONE | | +| Open connection (Process) | `OpenConnection` | `History.OpenConnection` (+ `ExchangeKey` auth) | ✅ | DONE | full auth chain works | +| Open connection (Event) | `OpenConnection` (Event type) | `History.OpenConnection` event mode | 🟗 | TRIVIAL | read path already opens it; flag = ConnectionType.Event | +| Close connection | `CloseConnection` | `History.CloseConnection` | ✅ | DONE | | +| Connection status | `GetConnectionStatus` | `Status.GetHistorianConsoleStatus` | ✅ | DONE | | +| Open/close **storage** connection | `OpenStorageConnection`, `CloseStorageConnection` | `Storage.OpenStorageConnection2` | ⬜ | BOUNDED | needed for any data-write path; storage-engine session | + +## 2. Reads — process data + +| Capability | HCAL API | 2023 R2 gRPC op | histsdk | Tier | Notes | +|---|---|---|---|---|---| +| Raw / full history | `CreateHistoryQuery` → Start/MoveNext/End | `Retrieval.StartQuery`→`GetNextQueryResultBuffer`→`EndQuery` | ✅ | DONE | row buffer parsed | +| Aggregate (interp/avg/min/max/…) | `CreateHistoryQuery` (RetrievalMode) | same | ✅ | DONE | all 15 RetrievalModes mapped | +| At-time / value-at | (interp window) | same | ✅ | DONE | | +| Analog summary | `CreateAnalogSummaryQuery` | `Retrieval.StartQuery` (summary mode) | 🟗 | BOUNDED | mode variant of existing query | +| State summary | `CreateStateSummaryQuery` | `Retrieval.StartQuery` (state mode) | ⬜ | BOUNDED | extra row layout to decode | +| Block read | `ReadBlocks` | `Storage.LoadBlocks` | ⬜ | BOUNDED | low-level; rarely needed | + +## 3. Reads — events + +| Capability | HCAL API | 2023 R2 gRPC op | histsdk | Tier | Notes | +|---|---|---|---|---|---| +| Event query | `CreateEventQuery` → Start/MoveNext/End | `Retrieval.StartEventQuery`→`GetNextEventQueryResultBuffer`→`EndEventQuery` | ✅ | DONE | rows + typed property bag parsed; CM_EVENT registration done | +| Event filters | `EventQuery.AddEventFilter` / `AddEventFilterCondition` | filter bytes in StartEventQuery request | ⬜ | BOUNDED | encode filter predicate into request buffer | + +## 4. Browse & metadata + +| Capability | HCAL API | 2023 R2 gRPC op | histsdk | Tier | Notes | +|---|---|---|---|---|---| +| Tag name browse | `CreateTagQuery` → `GetTagNames` | `Retrieval.StartTagQuery`/`QueryTag` (or LikeTagnames) | ✅ | DONE | wildcard works | +| Tag metadata | `GetTagInfoByName`, `TagQuery.GetTagInfo` | `Retrieval.GetTagInfosFromName` | ✅ | DONE | | +| Extended properties (read) | `GetTagExtendedPropertiesByName` | `Retrieval.GetTagExtendedPropertiesFromName` | ⬜ | BOUNDED | TEP buffer decode | +| Localized properties (read) | `GetTagLocalizedPropertiesByName` | `Retrieval.GetTagLocalizedPropertiesFromName` | ⬜ | BOUNDED | | +| SQL passthrough | `ExecuteSqlCommand` | `Retrieval.ExecuteSqlCommand` | ⬜ | TRIVIAL | thin string-in / status-out | + +## 5. Tag configuration (writes) + +| Capability | HCAL API | 2023 R2 gRPC op | histsdk | Tier | Notes | +|---|---|---|---|---|---| +| Create analog tag | `AddTag` | `History.EnsureTags` (EnsT2) | ✅ | DONE | Float/Double/Int2/Int4/UInt2/UInt4 + scaling | +| Create string/discrete tag | `AddTag` | `History.EnsureTags` | ⬜ | GATED/BOUNDED | native AddTag rejects these types server-side; needs different metadata path | +| Delete tag(s) | `DeleteTags` | `History.DeleteTags` | ✅ | DONE | | +| Rename tag(s) | `RenameTags` | (History op) | ⬜ | BOUNDED | `AllowRenameTags` param already probed | +| Add/Delete extended properties | `AddTagExtendedProperties`, `DeleteTagExtendedPropertiesByName` | `History.AddTagExtendedProperties` / `DeleteTagExtendedProperties` | ⬜ | BOUNDED | gRPC op + TEP serialize | +| Add/Delete localized properties | `AddTagLocalizedProperties`, `DeleteTagLocalizedPropertiesByName` | `History.AddTagLocalizedProperties` / `DeleteTagLocalizedProperties` | ⬜ | BOUNDED | | + +## 6. Data writes — values + +| Capability | HCAL API | 2023 R2 gRPC op | histsdk | Tier | Notes | +|---|---|---|---|---|---| +| Stream process values | `AddStreamedValue(HistorianDataValue)` | `Storage.AddStreamValues` | ⬜ | **GATED** | runtime cache only ingests from IOServer/AppServer pipelines (`129 Tag not found in cache`). Not a client bug | +| Stream **events** | `AddStreamedValue(HistorianEvent)` | `Storage.AddStreamValues` (event VTQ) | ⬜ | **CAPTURE** | full path mapped; need `CCommonArchestraEventValue::PackToVtq` blob bytes. See histevents.md | +| Non-streamed / historical insert | `AddNonStreamedValue`, `SendNonStreamedValues` | `Transaction.AddNonStreamValues(Begin/End)` | ⬜ | BOUNDED | explicit original-data insert via Transaction svc; verify ingest permission on target | +| Versioned streamed value | `AddVersionedStreamedValue` | `Storage.AddStreamValues2` | ⬜ | CAPTURE | revision flag on the VTQ | + +## 7. Revisions / edits (modify stored data) + +| Capability | HCAL API | 2023 R2 gRPC op | histsdk | Tier | Notes | +|---|---|---|---|---|---| +| Insert/update/delete revision values | `AddRevisionValue(s)`, `AddRevisionValuesBegin/End` | (storage-engine / transaction path) | ⬜ | HARD | prior RE: revision-write needs the non-WCF **storage-engine pipe** (`STransactPipeClient2`), not the WCF/gRPC surface | +| Event update/delete (revise) | `HistorianEvent.Update/.Delete` | `UpdateEventStatus` (+ revised VTQ) | ⬜ | CAPTURE | RevisionVersion + Update/Delete flags in the event VTQ | + +## 8. Status & system info + +| Capability | HCAL API | 2023 R2 gRPC op | histsdk | Tier | Notes | +|---|---|---|---|---|---| +| System parameter | `GetSystemParameter` | `Status.GetSystemParameter` | ✅ | DONE | | +| Runtime parameter | `GetRuntimeParameter` | `Status.GetRuntimeParameter` | ⬜ | TRIVIAL | same shape as GetSystemParameter | +| Historian info | `GetHistorianInfo` | `Status.GetHistorianInfo` | 🟗 | BOUNDED | GETHI buffer; partially decoded (incl. EventStorageMode @ offset 514) | +| Server timezone | `GetSystemTimeZoneInfo` | `Status.GetSystemTimeZoneName` | ⬜ | TRIVIAL | | +| Historization status | `GetHistorizationStatus` | `Status` op | ⬜ | BOUNDED | | +| Store-and-forward status | `GetStoreForwardStatus` | (push events / pull GETHI) | 🟗 | HARD | currently synthesized; real read needs duplex push or a decoded pull endpoint — see store-forward plan | + +## 9. Store-and-forward (offline buffering) + +| Capability | HCAL API | 2023 R2 gRPC op | histsdk | Tier | Notes | +|---|---|---|---|---|---| +| SF buffering + replay | (implicit on write conns) | `Storage`/`Transaction` `*Snapshot` + `Forward*Snapshot` | ⬜ | HARD | full subsystem: local cache format, snapshot framing, recovery log, forward-on-reconnect. Pragmatic alt: a simpler local queue, not bit-faithful SF | +| Event SF | (event conn) | `Forward**Event**SnapshotBegin/…/End` | ⬜ | HARD | dedicated event-snapshot SF stream | +| SF parameters | Get/Set SFP | `Storage.GetSFParameter`/`SetSFParameter` | ⬜ | BOUNDED | | + +## 10. Redundancy / multi-historian + +| Capability | HCAL API | 2023 R2 gRPC op | histsdk | Tier | Notes | +|---|---|---|---|---|---| +| Tiered/redundant access, failover | `MultiHistorianAccess.*` (OpenConnectionToAll, AddSecondaries, partner watchdog, ReSyncTags) | N×single-historian sessions + client logic | ⬜ | HARD | mostly client-side orchestration over §1–§6; build last | +| Replication config | (server `aahReplication`) | — | ⬜ | GATED | server-side concern | + +--- + +## Roll-up & recommended cut line + +**Phase 0 — already DONE (✅):** probe · open/close · raw+aggregate+at-time reads · +event reads · tag browse · tag metadata · system parameter · connection status · +create/delete analog tag. This is a usable modern client **today**. + +**Phase 1 — TRIVIAL/BOUNDED, high value (S–M each):** ExecuteSqlCommand · +runtime parameter · server timezone · extended/localized property read · event +filters · summary/state-summary queries · rename tags · ext/localized property +writes · GetHistorianInfo. Each is "gRPC op exists, decode one buffer, golden-byte +test." Knocks out most of the remaining read/config surface. + +**Phase 2 — CAPTURE (one native capture each, S):** **event sending** (the headline +gap — fully mapped, one `PackToVtq` capture away) · versioned/non-streamed value +writes. Now feasible locally since the Historian is installed. + +**Defer / simplify (HARD):** store-and-forward (do a pragmatic local queue instead of +bit-faithful SF) · revision/edit writes (separate storage-engine pipe) · multi- +historian redundancy (client orchestration, build last). + +**Won't unblock from the client (GATED):** streaming **process-sample** writes +(`AddS2`) — server cache only ingests from IOServer/AppServer pipelines; confirm your +ingestion model rather than chasing this. Non-analog tag creation likely needs a +distinct server path. + +## Cross-cutting realities (apply to every non-DONE row) + +- **Inner payloads stay proprietary** even under gRPC — the `bytes` fields carry + native VTQ / CTagMetadata / event-value formats. These are **version-sensitive**; + pin to the server version probed at connect and fail closed on mismatch. +- **Validation needs a live Historian** — now available locally, which is what makes + the CAPTURE-tier items practical. +- **Support tradeoff** — you take on maintenance across Historian versions in exchange + for shedding the stock SDK's bugs (mixed-mode marshaling, WCF quirks, global state) + for the surface you cover. + +## Bottom line + +A modern-.NET HCAL replacement is **feasible and ~60–70% done** for a typical +read+browse+config+event-read workload. The remaining high-value surface is mostly +**BOUNDED/CAPTURE** (incremental, well-understood), with only store-and-forward, +revision-edit, and redundancy being genuine **HARD** subsystems — and one true wall +(**GATED** process-sample writes) that no client can remove. diff --git a/docs/plans/hcal-roadmap.md b/docs/plans/hcal-roadmap.md new file mode 100644 index 0000000..ac7ed8d --- /dev/null +++ b/docs/plans/hcal-roadmap.md @@ -0,0 +1,171 @@ +# HCAL modern-.NET client — implementation roadmap + +Ordered, actionable plan to grow **histsdk** from "reads + basic config" into a broad +HCAL replacement, built on the **2023 R2 gRPC transport**. Derived from +[`hcal-capability-matrix.md`](hcal-capability-matrix.md); event details in +[`histevents.md`](histevents.md). + +> Move to the repo's `docs/plans/` when execution starts. Each work item lands as: a +> protocol serializer/parser + golden-byte unit test + an env-gated live integration +> test against the local Historian. + +## Guiding principles + +1. **gRPC-first.** New ops go on the `RemoteGrpc` transport (clean protobuf envelope); + the inner `bytes` blob is the only thing to RE. Keep WCF as the legacy/Windows path. +2. **Two tests per op, always.** A golden-byte test (deterministic, no server) **and** a + gated live test (`HISTORIAN_GRPC_HOST` / `HISTORIAN_HOST`). No op is "done" without both. +3. **Version-pin, fail closed.** Read server version at connect; gate every byte + serializer on it; throw `ProtocolEvidenceMissingException` on mismatch — never + best-effort parse. +4. **Capture once, encode forever.** For CAPTURE-tier items, instrument one native call, + save a sanitized fixture under `fixtures/protocol/`, then implement against the fixture. +5. **Ship per milestone.** Each milestone is independently releasable. + +Effort: **S** ≈ days · **M** ≈ ~1 week · **L** ≈ weeks. Estimates are incremental on +histsdk's existing infra (auth chain, transport, frame primitives, test harness). + +--- + +## Milestone 0 — Foundation: full gRPC parity for the DONE surface (M) + +*Goal: everything already working over WCF also works over `RemoteGrpc`, so the whole +read/browse/status surface is Windows-free and the gRPC stack is the default path.* + +| ID | Work | gRPC op | Files | Verify | Effort | +|---|---|---|---|---|---| +| R0.1 | Route browse over gRPC | `Retrieval.StartTagQuery`/`QueryTag` or `GetTagInfosFromName` | `Grpc/HistorianGrpcReadOrchestrator` (+ new `…GrpcBrowseClient`), `Historian2020ProtocolDialect` | browse tags live over gRPC | S | +| R0.2 | Route tag metadata over gRPC | `Retrieval.GetTagInfosFromName` | dialect + grpc client | metadata matches WCF result | S | +| R0.3 | Route status/system-param over gRPC | `Status.GetSystemParameter`, `Status.GetHistorianConsoleStatus` | new `Grpc/HistorianGrpcStatusClient` | system param + conn status live | S | +| R0.4 | Probe over gRPC | `*.GetInterfaceVersion` | grpc clients | `ProbeAsync` Windows-free | XS | +| R0.5 | **Capture harness for gRPC payloads** | n/a | reuse `instrument-wcf-*` tooling (same byte blobs) + add a `grpc-call-dump` helper | dump any request/response `bytes` to a fixture | S | +| R0.6 | **Version gate** | server version at connect | `HistorianClientOptions`, orchestrators | mismatched version → throws | S | + +**Acceptance:** the entire Phase-0 capability set runs end-to-end over `RemoteGrpc` +(incl. Linux), no WCF on the path. 188+ unit tests green; live gRPC integration suite green. + +--- + +## Milestone 1 — Cheap surface completion (TRIVIAL/BOUNDED) (M–L total) + +*Goal: knock out the remaining read/config surface. Order = ascending payload difficulty.* + +### 1a. Trivial (XS–S each, no new payload format) +| ID | Capability | gRPC op | Notes | +|---|---|---|---| +| R1.1 | `ExecuteSqlCommandAsync` | `Retrieval.ExecuteSqlCommand` | string in → `iRetValue` + status; thin | +| R1.2 | `GetRuntimeParameterAsync` | `Status.GetRuntimeParameter` | mirror `GetSystemParameter` | +| R1.3 | `GetServerTimeZoneAsync` | `Status.GetSystemTimeZoneName` | string out | + +### 1b. Bounded (decode one `bytes` payload; S–M each) +| ID | Capability | gRPC op | Payload to decode | Depends | +|---|---|---|---|---| +| R1.4 | `GetHistorianInfoAsync` | `Status.GetHistorianInfo` | GETHI buffer (partly decoded; incl. `EventStorageMode`@514) | R0.5 | +| R1.5 | Extended-property **read** | `Retrieval.GetTagExtendedPropertiesFromName` | TEP result buffer | R0.5 | +| R1.6 | Localized-property **read** | `Retrieval.GetTagLocalizedPropertiesFromName` | localized buffer | R0.5 | +| R1.7 | Event **filters** | filter bytes in `Retrieval.StartEventQuery` | filter predicate encoding (name/op/value) | R0.5 | +| R1.8 | Analog-summary query | `Retrieval.StartQuery` (summary mode) | summary row layout | — | +| R1.9 | State-summary query | `Retrieval.StartQuery` (state mode) | state-summary row layout | — | + +### 1c. Bounded config writes (S–M each) +| ID | Capability | gRPC op | Payload | Notes | +|---|---|---|---|---| +| R1.10 | `RenameTagsAsync` | History rename op | rename request buffer | `AllowRenameTags` already probed | +| R1.11 | Extended-property **write** | `History.AddTagExtendedProperties` (+ groups) / `DeleteTagExtendedProperties` | TEP serialize | mirror analog CTagMetadata discipline | +| R1.12 | Localized-property **write** | `History.AddTagLocalizedProperties` / `DeleteTagLocalizedProperties` | localized serialize | | +| R1.13 | Non-analog tag create (string/discrete) | `History.EnsureTags` | distinct CTagMetadata variant | ⚠ native AddTag rejected some types — confirm server path first; may be GATED | + +**Acceptance:** read + browse + metadata + system/status + property R/W + summaries + +event-filtered reads + rename all live-verified over gRPC. + +--- + +## Milestone 2 — Event sending (CAPTURE) (S–M) ← headline gap + +*Goal: `SendEventAsync(HistorianEvent)`. Path fully mapped in histevents.md; one capture away.* + +| ID | Work | Detail | +|---|---|---| +| R2.1 | Capture the event value blob | Instrument `CCommonArchestraEventValue::PackToVtq` (or dump the VTQ value bytes) on a live `AddStreamedValue(HistorianEvent)`; save sanitized fixture | +| R2.2 | `HistorianEventWriteProtocol` | Serialize header (`ReceivedTime, EventType, EventTime, Id, RevisionVersion, IsUpdate/IsDelete, Namespace`) + typed property bag — **inverse of `HistorianEventRowProtocol`** (reuse typemarkers `0x02/0x10/0x18/0x31/0x43/…`) | +| R2.3 | Event write orchestrator | Open **Event** connection (write mode) → register CM_EVENT (already have) → `Storage.AddStreamValues` with the event VTQ | +| R2.4 | Public API | `HistorianClient.SendEventAsync(HistorianEvent)` (+ `HistorianEvent` model: Type, EventTime, property bag) | +| R2.5 | Round-trip test | Send an event → read it back via `StartEventQuery` / `v_AlarmEventHistory2`; golden-byte on R2.2 | + +**Acceptance:** an event sent from histsdk appears in the historian and is read back with +matching Type + properties. **Now practical** — Historian is installed locally. + +--- + +## Milestone 3 — Historical / non-streamed value writes (BOUNDED) (M) + +*Goal: insert original historical VTQs (backfill), the path that is NOT the gated cache push.* + +| ID | Work | gRPC op | +|---|---|---| +| R3.1 | Decode non-streamed VTQ packet | `Transaction.AddNonStreamValuesBegin/AddNonStreamValues/End` | +| R3.2 | `AddHistoricalValuesAsync` | batched begin→values→end | +| R3.3 | Ingest-permission validation | confirm the target accepts original-data insert (distinct from `AddS2` cache wall) | + +**Acceptance:** historical points inserted and read back. Document clearly where this +differs from (gated) streaming sample writes. + +--- + +## Milestone 4 — HARD subsystems (deferred / optional) (L each) + +Only if the use case demands them. Each is a real subsystem, not an op. + +| ID | Capability | Approach | Risk | +|---|---|---|---| +| R4.1 | Store-and-forward | **Pragmatic local queue** (durable outbox + replay on reconnect) rather than bit-faithful SF cache + `Forward*Snapshot`. Faithful SF = decode SF cache format + snapshot framing + recovery log | high; consider "good enough" | +| R4.2 | Revision / edit writes | `AddRevisionValue(s)` go via the **non-WCF storage-engine pipe** (`STransactPipeClient2`) — separate transport RE | high | +| R4.3 | Real store-forward **status** | duplex push (`SetStoreForwardEvent`) or a decoded pull endpoint — see store-forward plan | medium | +| R4.4 | Multi-historian / redundancy | client-side orchestration over N single-historian sessions (failover, ReSyncTags, partner watchdog) — build last | medium | + +--- + +## Won't-do from the client (GATED) + +- **Streaming process-sample writes** (`AddStreamedValue(HistorianDataValue)` / `AddS2`): + runtime cache only ingests from configured IOServer/AppServer pipelines. Confirm your + ingestion architecture instead of pursuing this. + +--- + +## Cross-cutting workstreams (run alongside all milestones) + +- **CW-1 Capture tooling** (enables R0.5, R1.x, R2.1): one reusable "call op → dump + request/response `bytes` → sanitized fixture" path. Highest leverage — do first. +- **CW-2 Version compatibility:** matrix of tested Historian versions; serializers keyed + by version; CI gate. +- **CW-3 Cross-platform CI:** run the gRPC suite on Linux/macOS (transport is portable; + explicit-cred auth path). +- **CW-4 Fixtures discipline:** every new op ships a `fixtures/protocol//` golden file; + sanitize hostnames/tags/GUIDs before commit. +- **CW-5 Public API shape:** keep the modern surface (async, `IAsyncEnumerable`, + cancellation, options record, DI-friendly) consistent as the surface grows. + +--- + +## Sequencing (critical path) + +``` +CW-1 capture tooling ─┐ +M0 gRPC parity ───────┼─→ M1 cheap surface ─→ M2 event send ─→ M3 historical writes ─→ (M4 optional) +R0.6 version gate ────┘ +``` + +Recommended first sprint: **CW-1 + M0 (R0.1–R0.6)** → a fully Windows-free, version-safe +gRPC client at today's capability. Second sprint: **M1a + M2** (cheap wins + the headline +event-send). M3/M4 as demand dictates. + +## One-glance status + +| Milestone | Tier | Effort | Value | When | +|---|---|---|---|---| +| M0 gRPC parity + capture tooling | foundation | M | unblocks everything, Windows-free | **now** | +| M1 cheap surface | TRIVIAL/BOUNDED | M–L | most remaining read/config | next | +| M2 event send | CAPTURE | S–M | headline write capability | next | +| M3 historical writes | BOUNDED | M | backfill | on demand | +| M4 SF / revisions / redundancy | HARD | L×N | parity completeness | defer | diff --git a/docs/plans/histevents.md b/docs/plans/histevents.md new file mode 100644 index 0000000..c177111 --- /dev/null +++ b/docs/plans/histevents.md @@ -0,0 +1,299 @@ +# How a HistorianEvent reaches the Historian DB files + +Living analysis doc. Traces an event end-to-end: client API → wire → server +storage backend (SQL **Database** vs history **Blocks** `.dat`) → read-back. + +Evidence base: 2023 R2 `aahClientManaged.dll` (decompiled `ArchestrA.HistorianAccess`, +`HistorianEvent`, `HistorianEventPropertyType`), native `aahStorage.exe` (string +analysis), the recovered gRPC + CloudHistorian contracts, and the histsdk read-side +reverse-engineering (CM_EVENT registration + event-row parser). + +Status legend: ✅ proven (from binary) · 🔶 strong inference · ❓ open. + +--- + +## TL;DR + +An event is **not a distinct wire message**. The client turns each `HistorianEvent` +into a `HistorianDataValue` of type `Event` against the built-in **`CM_EVENT`** tag, +marshals it into a native VTQ, and **streams it like any tag value** on a dedicated +*Event* connection. Events are batched into an opaque serialized **event data packet** +and delivered (with their own store-and-forward "event snapshot" path). On the server +they are persisted into **one of two backends, chosen by a server-configured +`EventStorageMode`**: a SQL **Database**, or the history **Blocks** (`.dat`) files. + +``` +HistorianEvent (Type + typed property bag) + └─(AddStreamedValue)→ HistorianDataValue{Type=Event, TagKey=CM_EVENT, EventTime, Q=192} + └─ HistorianEvent.PackToVtq → CCommonArchestraEventValue::PackToVtq (native value blob) + └─ HISTORIAN_VALUE2 (44B; blob ptr @+33) → HistorianClient.AddHistorianValue → queue + └─ flush → EnqueueEventDataPacket{ byte[] SerializedBytes } (batched VTQs) + └─ SERVER: aahEventStorage.exe (InSQLEventSystem) + per-client event-tag pipeline → recovery-log WAL → backend: + • Blocks → elastic snapshot → frozen → history .dat (Circular/Permanent) + • Database → ArchestrAEvents.EventStorage.Contract assembly → SQL (A2ALMDB) + → EventReplication (redundant historians) + └─ (offline) → store-and-forward → ForwardEventSnapshotBegin/…/End on reconnect + read-back: Retr.StartEventQuery / SQL provider views (Events, v_AlarmEventHistory2, v_EventSnapshot) +``` + +--- + +## 1. The `HistorianEvent` object ✅ + +Decompiled `ArchestrA.HistorianEvent` — a structured header plus a **typed property bag**: + +- **Header fields:** `ID`/`Id` (Guid), `Type`/`EventType` (string, e.g. `"Alarm.Set"`, + `"User.Write"`), `EventTime` (DateTime), `ReceivedTime`, `Severity` (ushort), + `Priority` (ushort), `IsAlarm`, `IsSilenced`, `System`, `Source`, `Source_Name`, + `Area`, `Namespace`, `DisplayText`. +- **Revision fields:** `RevisionVersion` (ushort), `Delete` (bool), `Update` (bool) — + events are revisable (see §4 UpdateEventStatus). +- **Property bag:** `AddProperty(name, value, HistorianEventPropertyType, …)` with typed + overloads. `HistorianEventPropertyType` (alphabetical enum): + `Blob, Boolean, Byte, Date, DateTime, Decimal, Double, Duration, Float, Guid, Hex, + Int, Integer, Long, Short, String, Time, UnsignedByte, UnsignedInt, UnsignedLong, + UnsignedShort, Undefined`. + +These map onto the wire property-bag the histsdk **read** parser already decodes +(`HistorianEventRowProtocol`): typemarkers `0x02` Boolean, `0x10` Guid, `0x18` FILETIME, +`0x31` Int32, `0x43` UTF-16 string, … — i.e. the write enum and the read typemarkers are +two views of the same typed-value format. The event-send serialization is the inverse of +that read parser. + +--- + +## 2. Client send path — an event becomes a streamed VTQ ✅ + +From decompiled `ArchestrA.HistorianAccess` (line refs into the decompile): + +1. **Open an Event connection.** `HistorianConnectionArgs.ConnectionType = + HistorianConnectionType.Event`, `ReadOnly = false` (sample `Step10.SendEvents`). +2. **Default event tag.** `CreateDefaultEventTag()` (`:3006`) registers tag `CM_EVENT` / + "AnE Event" / `TagDataType = Event` and stores `eventTagHandle`. Same CM_EVENT + registration histsdk reverse-engineered (RTag2 + EnsT2; tag id + `353b8145-5df0-4d46-a253-871aef49b321`). +3. **Wrap as VTQ.** `AddStreamedValue(HistorianEvent)` (`:3123`): + ```csharp + historianDataValue.objValue = historianEvent; // header + property bag + historianDataValue.DataValueType = HistorianDataType.Event; + historianDataValue.TagKey = eventTagHandle; // CM_EVENT + historianDataValue.StartDateTime = historianEvent.EventTime; + historianDataValue.OpcQuality = 192; + return AddStreamedValue((ConnectionIndex)1, historianDataValue, false, out error); // 1=Event + ``` +4. **Marshal + queue.** The private `AddStreamedValue` (`:3173`): + - builds a 44-byte native `HISTORIAN_VALUE2` (`InitBlockUnaligned(…,0,44)`), + - `HistorianAccessUtil.ConvertManagedStructToUnmanagedStruct(value, &HV2, bVersioned…)` + — its `case HistorianDataType.Event` (`HistorianAccessUtil:89`) calls + **`HistorianEvent.PackToVtq(out byte[])`** to produce the event value blob, whose pointer + is placed at `HISTORIAN_VALUE2+33` (freed after send; offset 33 is the value-union pointer + used for Event/String types), + - `HistorianClient.AddHistorianValue(client, &HV2, &err)` (`:3209`) queues the VTQ into + the native delivery buffer and returns immediately. + +So an event uses the **same streaming machinery as a process value**; only `DataValueType` +(`Event`) and the target tag (`CM_EVENT`) differ. + +### 2a. Event value serialization — `HistorianEvent.PackToVtq` ✅/🔶 + +`HistorianEvent.PackToVtq` (`HistorianEvent:1392`) populates a native +**`CCommonArchestraEventStruct`** then hands it to the **native** packer +`CCommonArchestraEventValue::PackToVtq(…, 192, 192, vtq)` (Q=192), associated with the +built-in `EVENT_TAGID` / `EVENT_TAGNAME` (`CTagMetadata.CommonArchestraEvent`). The actual +byte layout is produced in C++ — **not visible in managed code** — so pinning exact write +bytes needs a wire/IL capture, exactly as the read side did. But the **field set + order** +the managed code writes into the struct is now known: + +``` +SetReceivedTime (uint64 FILETIME, from UniqueTime.GetUniqueFileTime — unique/monotonic) +SetEventType (wchar* string, e.g. "Alarm.Set") +SetEventTime (uint64 FILETIME, from EventTime) +SetId (GUID) +SetRevisionVersion (uint16) +SetIsUpdate (bool) ← revision flags +SetIsDelete (bool) +Namespace (string, trimmed, non-printable-validated) +…then the typed property bag: Dictionary> +``` + +This matches `HistorianEventRowProtocol` on read: the property bag is name→(type,value) with +the same typed-value encoding (typemarkers `0x02/0x10/0x18/0x31/0x43/…`). So a managed +event-send serializer is tractable: emit the header struct fields above, then the typed +property bag in the read parser's format. The remaining unknown is only the exact native +framing offsets — best obtained by capturing one `PackToVtq` output, then golden-byte testing. + +--- + +## 3. Event transport / delivery pipeline ✅ (CloudHistorian + gRPC contracts) + +Events have a **dedicated, batched** connection + delivery pipeline, distinct from tag data +but structurally parallel: + +| Stage | Event op | Tag-data analogue | +|---|---|---| +| Open connection | `OpenEventConnection2 { byte[] ClientInfo } → { byte[] ServerInfo }` | OpenConnection | +| Send batch | `EnqueueEventDataPacket { byte[] SerializedBytes }` | `EnqueueTagDataPacket { byte[] SerializedBytes }` | +| Store-and-forward | `ForwardEventSnapshotBegin / ForwardEventSnapshot / ForwardEventSnapshotEnd` | `ForwardSnapshot…` | +| Revise | `UpdateEventStatus` | (revision write) | + +Key point: the **event data packet is an opaque serialized byte buffer** (`SerializedBytes`, +DataMember `d`) — the queued event VTQs batched together, exactly the same envelope shape as +the tag data packet. On-prem this is what the storage-streaming op (`AddStreamValues`) +carries; in the cloud variant it is `EnqueueEventDataPacket`. + +Validation surfaced via error codes: `InvalidAlarmEventPropertyLength=212`, +`AlarmEventPropertyHasNonPrintableChar=214`, `AlarmEventPropertyHasInvalidSpecialChar=215`, +`AlarmEventPropertyNameIsAReservedName=216` — the server validates alarm/event property +names + values on ingest. + +Offline → events spool to the **store-and-forward** cache and replay as **event snapshots** +(`ForwardEventSnapshot*`) on reconnect — a separate SF stream from tag-data snapshots. + +--- + +## 4. Revisions / updates ✅ + +`HistorianEvent.Update` / `.Delete` / `.RevisionVersion` + the contract's +`UpdateEventStatus` op mean events are not write-once: an event can be re-sent to update or +delete a previously stored event (e.g. alarm acknowledge/clear), bumping `RevisionVersion`. + +--- + +## 5. The storage-backend switch — `EventStorageMode` ✅ + +The client reads the server's event-storage backend from `HISTORIAN_INFO` **byte offset 514** +(`HistorianAccess` `:5715`): + +```csharp +EventStorageMode = (info[514] == -1) ? Unsupported + : (info[514] == 0) ? Database // SQL Server + : Blocks; // history .dat blocks +``` + +`HistorianEventStorageMode ∈ { Database, Blocks, Unsupported }`. The destination is a +**server** decision; the client streams the same VTQ regardless. + +--- + +## 6. Server side — where it lands ✅ (confirmed on the live local install) + +The server-side event component is **`aahEventStorage.exe`** (service `InSQLEventSystem`, +"AVEVA Historian Event System"; plus `aahEventSvc.exe`), at +`…\Wonderware\Historian\x64\aahEventStorage.exe`. Its string table maps the full pipeline: + +``` +event packets / forwarded snapshots → per-client "event tag pipeline" → batch enqueue + → Event Storage Recovery Log (WAL; "enqueuing N events to log", + path SystemParameter EventStorageLogPath = C:\Historian\Data\Logs\EventStorage) + → persist to the active backend: + Block Storage ("Enabled Block Storage for events") → history .dat blocks + Database ("storing N events in database") → SQL via loaded managed + assembly ArchestrAEvents.EventStorage.Contract.EventStorageDatabaseConnection + (e.g. ";Initial Catalog=A2ALMDB;Integrated Security=true;Encrypt=True;…") + → also fed to EventReplication (aahReplication.exe) for redundant historians +``` + +So persistence is **pluggable** (a loadable connection assembly) and dual-mode, guarded by a +recovery log. Which backend is live depends on configuration (the `EventStorageMode` of §5). + +### This historian = **Block storage** (verified) +- `C:\Historian\Data\Circular` holds **527 `.dat` history blocks** (`Permanent` empty); the + EventStorage recovery log dir exists. `aahEventStorage` logs `"Enabled Block Storage for + events"`. +- SDK-shape alarm/events are present and retrievable: `Runtime.dbo.v_AlarmEventHistory2` + returns 224 rows over the last 30 days. +- `A2ALMDB` (the System-Platform alarm DB the connection string references) is **not present** + here — that path is only used when integrated with AVEVA System Platform alarming. Absent + it, ArchestrA events land in **blocks**, exactly as `aahStorage.exe` advertises (`"Stores + ArchestrA Event Data"`, snapshot→block). + +### The SQL surface is **provider-backed views, not physical tables** ✅ +In `Runtime`, the rich event objects are **views with NULL `OBJECT_DEFINITION`** — i.e. the +historian's OLE DB History provider exposes them as virtual/extension tables that read the +block store, *not* stored T-SQL: +- `Events`, `v_EventHistory`, `v_EventSnapshot`, `v_EventStringSnapshot`, **`v_AlarmEventHistory2`** + (columns: `EventStampUTC`, `AlarmState`, `TagName`, `Description`, `Area`, `Type`, `Value`, + `Priority`, `Category`, `Provider`, `Operator`, `DomainName`, `UserFullName`, `MilliSec`, …) + — these are the read-back of the SDK alarm/event property bag. + +So `SELECT … FROM Events` (and `v_AlarmEventHistory2`) is **the provider reading the block +store**, which is why the handoff could query events even though they live in `.dat` blocks. + +### Database-mode physical store +When events ARE stored in SQL (Database mode / A2ALMDB integration), the writer is the loaded +`ArchestrAEvents.EventStorage.Contract` connection assembly doing batched inserts ("storing N +events in database", "creating event storage database connection role"). The exact table +schema there is the A2ALMDB alarm schema (not present on this box to dump). + +### Read-back ✅ +Uniform regardless of backend: `Retr.StartEventQuery` → `GetNextEventQueryResultBuffer` +(provider) surfaces events from wherever they were stored — so histsdk `ReadEventsAsync` is +mode-agnostic. Engine filter note: `"EventTime filtering can only be specified through +StartDateTime and EndDateTime"`. + +## 6b. Two different "event" subsystems — don't conflate ✅ + +| | Classic event **detectors** | ArchestrA **alarms/events** (the SDK path) | +|---|---|---| +| What | server-side detectors watching tag conditions | client-streamed `HistorianEvent` (alarms, user events) | +| Config/store | `Runtime.dbo._EventTag` (TagName, DetectorTypeKey, DetectorString, Action*, ScanRate, Edge, Priority) | CM_EVENT / CommonArchestraEvent tag | +| History | **physical** `BASE TABLE Runtime.dbo.EventHistory` (`EventLogKey, TagName, DateTime, DetectDateTime, Edge`); 30 rows | block store (or A2ALMDB), surfaced via `v_AlarmEventHistory2` / `v_EventSnapshot` | +| Source | evaluated by the server | sent via `AddStreamedValue(HistorianEvent)` | + +`AddStreamedValue(HistorianEvent)` feeds the **right column** (ArchestrA alarms/events) — it is +**not** the classic `EventHistory` detector log. + +--- + +## 7. Relationship to histsdk + +- histsdk implements event **reads** only (`ReadEventsAsync` via `StartEventQuery`); its + CM_EVENT EnsT2/RTag2 dance is read-subscription registration. +- Event **writing** is unimplemented but viable. Chain to replicate: Event-type connection → + register CM_EVENT (done) → serialize `HistorianEvent` (header + typed property bag) into the + event-VTQ value blob (inverse of `HistorianEventRowProtocol`) → batch into an event data + packet → stream via `AddStreamValues` (2023 R2 gRPC: `StorageService.AddStreamValues`). + +--- + +## Open threads + +- 🔶 Event value blob: field set/order known (§2a); **exact native framing** still needs one + `CCommonArchestraEventValue::PackToVtq` output capture + golden-byte test (mirror the read-side + `HistorianEventRowProtocol` reverse-engineering). Now feasible locally — the live historian is + installed, so the same instrument-and-capture approach used for reads applies. +- ❓ `EnqueueEventDataPacket.SerializedBytes` packet framing (header + N event VTQs batched). +- ✅ Database-mode store: server writer is `aahEventStorage.exe` loading the managed + `ArchestrAEvents.EventStorage.Contract` connection assembly; SQL retrieval surface is the + provider-backed `Events` / `v_AlarmEventHistory2` / `v_EventSnapshot` views (NULL T-SQL def). + This box runs **Block storage** (A2ALMDB absent). A2ALMDB physical schema still un-dumped (needs + a System-Platform-integrated box). +- ✅ `ArchestraEvent` vs `CommonArchestraEvent`: send path packs **CommonArchestraEvent** via + `EVENT_TAGID`; both are event-tag schemas in `CTagMetadata` (the server stores either). +- ❓ `UpdateEventStatus` wire payload for `Update`/`Delete` revisions. +- ❓ `EventStorage` recovery-log (`C:\Historian\Data\Logs\EventStorage`) on-disk format (WAL). +- 🔶 Decompile `ArchestrAEvents.EventStorage.Contract.dll` (managed) for the exact DB insert + contract/schema — locate it (not under `…\Wonderware\Historian`; check GAC / Framework\Bin). + +--- + +## Changelog +- Rev 4 (live local install): confirmed server side. `aahEventStorage.exe` (`InSQLEventSystem`) + is the event store engine — per-client event-tag pipeline → recovery-log WAL → Block storage + OR SQL (loadable `ArchestrAEvents.EventStorage.Contract` assembly) → EventReplication. This box + uses **Block storage** (527 `.dat` in `C:\Historian\Data\Circular`; A2ALMDB absent). SQL + `Events`/`v_AlarmEventHistory2`/`v_EventSnapshot` are **provider-backed views over the blocks** + (NULL `OBJECT_DEFINITION`), not physical tables — `v_AlarmEventHistory2` (224 rows/30d) is the + SDK-event read surface. Distinguished the classic event-detector subsystem (`_EventTag` → + physical `EventHistory`) from the ArchestrA alarm/event path (the SDK's target). +- Rev 3: event value serialization pinned to native `CCommonArchestraEventValue::PackToVtq` + via managed `HistorianEvent.PackToVtq`; documented the `CCommonArchestraEventStruct` field + set/order (ReceivedTime, EventType, EventTime, Id, RevisionVersion, IsUpdate/IsDelete, + Namespace, typed property bag) and the path to a managed send serializer. +- Rev 2: HistorianEvent structure + HistorianEventPropertyType enum; client marshaling + (HISTORIAN_VALUE2 / ConvertManagedStructToUnmanagedStruct / AddHistorianValue); dedicated + event pipeline (OpenEventConnection2 / EnqueueEventDataPacket / ForwardEventSnapshot / + store-and-forward); revisions (Update/Delete/UpdateEventStatus); Blocks-mode clarified + (events = generic VTQ snapshots, no event-specific block code). +- Rev 1: client send path, EventStorageMode switch, Blocks/Database backends, read-back.