Joseph Doherty
a530ae0f10
Version-control the planning docs alongside the code they describe: - grpc-transport.md — 2023 R2 gRPC transport analysis (sanitized source path) - hcal-capability-matrix.md — HistorianAccess surface x gRPC ops x histsdk status x feasibility tiers - hcal-roadmap.md — ordered build plan M0-M4 + cross-cutting workstreams - histevents.md — how a HistorianEvent reaches the DB (client->wire->server) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
300 lines
17 KiB
Markdown
300 lines
17 KiB
Markdown
# How a HistorianEvent reaches the Historian DB files
|
|
|
|
Living analysis doc. Traces an event end-to-end: client API → wire → server
|
|
storage backend (SQL **Database** vs history **Blocks** `.dat`) → read-back.
|
|
|
|
Evidence base: 2023 R2 `aahClientManaged.dll` (decompiled `ArchestrA.HistorianAccess`,
|
|
`HistorianEvent`, `HistorianEventPropertyType`), native `aahStorage.exe` (string
|
|
analysis), the recovered gRPC + CloudHistorian contracts, and the histsdk read-side
|
|
reverse-engineering (CM_EVENT registration + event-row parser).
|
|
|
|
Status legend: ✅ proven (from binary) · 🔶 strong inference · ❓ open.
|
|
|
|
---
|
|
|
|
## TL;DR
|
|
|
|
An event is **not a distinct wire message**. The client turns each `HistorianEvent`
|
|
into a `HistorianDataValue` of type `Event` against the built-in **`CM_EVENT`** tag,
|
|
marshals it into a native VTQ, and **streams it like any tag value** on a dedicated
|
|
*Event* connection. Events are batched into an opaque serialized **event data packet**
|
|
and delivered (with their own store-and-forward "event snapshot" path). On the server
|
|
they are persisted into **one of two backends, chosen by a server-configured
|
|
`EventStorageMode`**: a SQL **Database**, or the history **Blocks** (`.dat`) files.
|
|
|
|
```
|
|
HistorianEvent (Type + typed property bag)
|
|
└─(AddStreamedValue)→ HistorianDataValue{Type=Event, TagKey=CM_EVENT, EventTime, Q=192}
|
|
└─ HistorianEvent.PackToVtq → CCommonArchestraEventValue::PackToVtq (native value blob)
|
|
└─ HISTORIAN_VALUE2 (44B; blob ptr @+33) → HistorianClient.AddHistorianValue → queue
|
|
└─ flush → EnqueueEventDataPacket{ byte[] SerializedBytes } (batched VTQs)
|
|
└─ SERVER: aahEventStorage.exe (InSQLEventSystem)
|
|
per-client event-tag pipeline → recovery-log WAL → backend:
|
|
• Blocks → elastic snapshot → frozen → history .dat (Circular/Permanent)
|
|
• Database → ArchestrAEvents.EventStorage.Contract assembly → SQL (A2ALMDB)
|
|
→ EventReplication (redundant historians)
|
|
└─ (offline) → store-and-forward → ForwardEventSnapshotBegin/…/End on reconnect
|
|
read-back: Retr.StartEventQuery / SQL provider views (Events, v_AlarmEventHistory2, v_EventSnapshot)
|
|
```
|
|
|
|
---
|
|
|
|
## 1. The `HistorianEvent` object ✅
|
|
|
|
Decompiled `ArchestrA.HistorianEvent` — a structured header plus a **typed property bag**:
|
|
|
|
- **Header fields:** `ID`/`Id` (Guid), `Type`/`EventType` (string, e.g. `"Alarm.Set"`,
|
|
`"User.Write"`), `EventTime` (DateTime), `ReceivedTime`, `Severity` (ushort),
|
|
`Priority` (ushort), `IsAlarm`, `IsSilenced`, `System`, `Source`, `Source_Name`,
|
|
`Area`, `Namespace`, `DisplayText`.
|
|
- **Revision fields:** `RevisionVersion` (ushort), `Delete` (bool), `Update` (bool) —
|
|
events are revisable (see §4 UpdateEventStatus).
|
|
- **Property bag:** `AddProperty(name, value, HistorianEventPropertyType, …)` with typed
|
|
overloads. `HistorianEventPropertyType` (alphabetical enum):
|
|
`Blob, Boolean, Byte, Date, DateTime, Decimal, Double, Duration, Float, Guid, Hex,
|
|
Int, Integer, Long, Short, String, Time, UnsignedByte, UnsignedInt, UnsignedLong,
|
|
UnsignedShort, Undefined`.
|
|
|
|
These map onto the wire property-bag the histsdk **read** parser already decodes
|
|
(`HistorianEventRowProtocol`): typemarkers `0x02` Boolean, `0x10` Guid, `0x18` FILETIME,
|
|
`0x31` Int32, `0x43` UTF-16 string, … — i.e. the write enum and the read typemarkers are
|
|
two views of the same typed-value format. The event-send serialization is the inverse of
|
|
that read parser.
|
|
|
|
---
|
|
|
|
## 2. Client send path — an event becomes a streamed VTQ ✅
|
|
|
|
From decompiled `ArchestrA.HistorianAccess` (line refs into the decompile):
|
|
|
|
1. **Open an Event connection.** `HistorianConnectionArgs.ConnectionType =
|
|
HistorianConnectionType.Event`, `ReadOnly = false` (sample `Step10.SendEvents`).
|
|
2. **Default event tag.** `CreateDefaultEventTag()` (`:3006`) registers tag `CM_EVENT` /
|
|
"AnE Event" / `TagDataType = Event` and stores `eventTagHandle`. Same CM_EVENT
|
|
registration histsdk reverse-engineered (RTag2 + EnsT2; tag id
|
|
`353b8145-5df0-4d46-a253-871aef49b321`).
|
|
3. **Wrap as VTQ.** `AddStreamedValue(HistorianEvent)` (`:3123`):
|
|
```csharp
|
|
historianDataValue.objValue = historianEvent; // header + property bag
|
|
historianDataValue.DataValueType = HistorianDataType.Event;
|
|
historianDataValue.TagKey = eventTagHandle; // CM_EVENT
|
|
historianDataValue.StartDateTime = historianEvent.EventTime;
|
|
historianDataValue.OpcQuality = 192;
|
|
return AddStreamedValue((ConnectionIndex)1, historianDataValue, false, out error); // 1=Event
|
|
```
|
|
4. **Marshal + queue.** The private `AddStreamedValue` (`:3173`):
|
|
- builds a 44-byte native `HISTORIAN_VALUE2` (`InitBlockUnaligned(…,0,44)`),
|
|
- `HistorianAccessUtil.ConvertManagedStructToUnmanagedStruct(value, &HV2, bVersioned…)`
|
|
— its `case HistorianDataType.Event` (`HistorianAccessUtil:89`) calls
|
|
**`HistorianEvent.PackToVtq(out byte[])`** to produce the event value blob, whose pointer
|
|
is placed at `HISTORIAN_VALUE2+33` (freed after send; offset 33 is the value-union pointer
|
|
used for Event/String types),
|
|
- `HistorianClient.AddHistorianValue(client, &HV2, &err)` (`:3209`) queues the VTQ into
|
|
the native delivery buffer and returns immediately.
|
|
|
|
So an event uses the **same streaming machinery as a process value**; only `DataValueType`
|
|
(`Event`) and the target tag (`CM_EVENT`) differ.
|
|
|
|
### 2a. Event value serialization — `HistorianEvent.PackToVtq` ✅/🔶
|
|
|
|
`HistorianEvent.PackToVtq` (`HistorianEvent:1392`) populates a native
|
|
**`CCommonArchestraEventStruct`** then hands it to the **native** packer
|
|
`CCommonArchestraEventValue::PackToVtq(…, 192, 192, vtq)` (Q=192), associated with the
|
|
built-in `EVENT_TAGID` / `EVENT_TAGNAME` (`CTagMetadata.CommonArchestraEvent`). The actual
|
|
byte layout is produced in C++ — **not visible in managed code** — so pinning exact write
|
|
bytes needs a wire/IL capture, exactly as the read side did. But the **field set + order**
|
|
the managed code writes into the struct is now known:
|
|
|
|
```
|
|
SetReceivedTime (uint64 FILETIME, from UniqueTime.GetUniqueFileTime — unique/monotonic)
|
|
SetEventType (wchar* string, e.g. "Alarm.Set")
|
|
SetEventTime (uint64 FILETIME, from EventTime)
|
|
SetId (GUID)
|
|
SetRevisionVersion (uint16)
|
|
SetIsUpdate (bool) ← revision flags
|
|
SetIsDelete (bool)
|
|
Namespace (string, trimmed, non-printable-validated)
|
|
…then the typed property bag: Dictionary<string, Tuple<HistorianEventPropertyType, object>>
|
|
```
|
|
|
|
This matches `HistorianEventRowProtocol` on read: the property bag is name→(type,value) with
|
|
the same typed-value encoding (typemarkers `0x02/0x10/0x18/0x31/0x43/…`). So a managed
|
|
event-send serializer is tractable: emit the header struct fields above, then the typed
|
|
property bag in the read parser's format. The remaining unknown is only the exact native
|
|
framing offsets — best obtained by capturing one `PackToVtq` output, then golden-byte testing.
|
|
|
|
---
|
|
|
|
## 3. Event transport / delivery pipeline ✅ (CloudHistorian + gRPC contracts)
|
|
|
|
Events have a **dedicated, batched** connection + delivery pipeline, distinct from tag data
|
|
but structurally parallel:
|
|
|
|
| Stage | Event op | Tag-data analogue |
|
|
|---|---|---|
|
|
| Open connection | `OpenEventConnection2 { byte[] ClientInfo } → { byte[] ServerInfo }` | OpenConnection |
|
|
| Send batch | `EnqueueEventDataPacket { byte[] SerializedBytes }` | `EnqueueTagDataPacket { byte[] SerializedBytes }` |
|
|
| Store-and-forward | `ForwardEventSnapshotBegin / ForwardEventSnapshot / ForwardEventSnapshotEnd` | `ForwardSnapshot…` |
|
|
| Revise | `UpdateEventStatus` | (revision write) |
|
|
|
|
Key point: the **event data packet is an opaque serialized byte buffer** (`SerializedBytes`,
|
|
DataMember `d`) — the queued event VTQs batched together, exactly the same envelope shape as
|
|
the tag data packet. On-prem this is what the storage-streaming op (`AddStreamValues`)
|
|
carries; in the cloud variant it is `EnqueueEventDataPacket`.
|
|
|
|
Validation surfaced via error codes: `InvalidAlarmEventPropertyLength=212`,
|
|
`AlarmEventPropertyHasNonPrintableChar=214`, `AlarmEventPropertyHasInvalidSpecialChar=215`,
|
|
`AlarmEventPropertyNameIsAReservedName=216` — the server validates alarm/event property
|
|
names + values on ingest.
|
|
|
|
Offline → events spool to the **store-and-forward** cache and replay as **event snapshots**
|
|
(`ForwardEventSnapshot*`) on reconnect — a separate SF stream from tag-data snapshots.
|
|
|
|
---
|
|
|
|
## 4. Revisions / updates ✅
|
|
|
|
`HistorianEvent.Update` / `.Delete` / `.RevisionVersion` + the contract's
|
|
`UpdateEventStatus` op mean events are not write-once: an event can be re-sent to update or
|
|
delete a previously stored event (e.g. alarm acknowledge/clear), bumping `RevisionVersion`.
|
|
|
|
---
|
|
|
|
## 5. The storage-backend switch — `EventStorageMode` ✅
|
|
|
|
The client reads the server's event-storage backend from `HISTORIAN_INFO` **byte offset 514**
|
|
(`HistorianAccess` `:5715`):
|
|
|
|
```csharp
|
|
EventStorageMode = (info[514] == -1) ? Unsupported
|
|
: (info[514] == 0) ? Database // SQL Server
|
|
: Blocks; // history .dat blocks
|
|
```
|
|
|
|
`HistorianEventStorageMode ∈ { Database, Blocks, Unsupported }`. The destination is a
|
|
**server** decision; the client streams the same VTQ regardless.
|
|
|
|
---
|
|
|
|
## 6. Server side — where it lands ✅ (confirmed on the live local install)
|
|
|
|
The server-side event component is **`aahEventStorage.exe`** (service `InSQLEventSystem`,
|
|
"AVEVA Historian Event System"; plus `aahEventSvc.exe`), at
|
|
`…\Wonderware\Historian\x64\aahEventStorage.exe`. Its string table maps the full pipeline:
|
|
|
|
```
|
|
event packets / forwarded snapshots → per-client "event tag pipeline" → batch enqueue
|
|
→ Event Storage Recovery Log (WAL; "enqueuing N events to log",
|
|
path SystemParameter EventStorageLogPath = C:\Historian\Data\Logs\EventStorage)
|
|
→ persist to the active backend:
|
|
Block Storage ("Enabled Block Storage for events") → history .dat blocks
|
|
Database ("storing N events in database") → SQL via loaded managed
|
|
assembly ArchestrAEvents.EventStorage.Contract.EventStorageDatabaseConnection
|
|
(e.g. ";Initial Catalog=A2ALMDB;Integrated Security=true;Encrypt=True;…")
|
|
→ also fed to EventReplication (aahReplication.exe) for redundant historians
|
|
```
|
|
|
|
So persistence is **pluggable** (a loadable connection assembly) and dual-mode, guarded by a
|
|
recovery log. Which backend is live depends on configuration (the `EventStorageMode` of §5).
|
|
|
|
### This historian = **Block storage** (verified)
|
|
- `C:\Historian\Data\Circular` holds **527 `.dat` history blocks** (`Permanent` empty); the
|
|
EventStorage recovery log dir exists. `aahEventStorage` logs `"Enabled Block Storage for
|
|
events"`.
|
|
- SDK-shape alarm/events are present and retrievable: `Runtime.dbo.v_AlarmEventHistory2`
|
|
returns 224 rows over the last 30 days.
|
|
- `A2ALMDB` (the System-Platform alarm DB the connection string references) is **not present**
|
|
here — that path is only used when integrated with AVEVA System Platform alarming. Absent
|
|
it, ArchestrA events land in **blocks**, exactly as `aahStorage.exe` advertises (`"Stores
|
|
ArchestrA Event Data"`, snapshot→block).
|
|
|
|
### The SQL surface is **provider-backed views, not physical tables** ✅
|
|
In `Runtime`, the rich event objects are **views with NULL `OBJECT_DEFINITION`** — i.e. the
|
|
historian's OLE DB History provider exposes them as virtual/extension tables that read the
|
|
block store, *not* stored T-SQL:
|
|
- `Events`, `v_EventHistory`, `v_EventSnapshot`, `v_EventStringSnapshot`, **`v_AlarmEventHistory2`**
|
|
(columns: `EventStampUTC`, `AlarmState`, `TagName`, `Description`, `Area`, `Type`, `Value`,
|
|
`Priority`, `Category`, `Provider`, `Operator`, `DomainName`, `UserFullName`, `MilliSec`, …)
|
|
— these are the read-back of the SDK alarm/event property bag.
|
|
|
|
So `SELECT … FROM Events` (and `v_AlarmEventHistory2`) is **the provider reading the block
|
|
store**, which is why the handoff could query events even though they live in `.dat` blocks.
|
|
|
|
### Database-mode physical store
|
|
When events ARE stored in SQL (Database mode / A2ALMDB integration), the writer is the loaded
|
|
`ArchestrAEvents.EventStorage.Contract` connection assembly doing batched inserts ("storing N
|
|
events in database", "creating event storage database connection role"). The exact table
|
|
schema there is the A2ALMDB alarm schema (not present on this box to dump).
|
|
|
|
### Read-back ✅
|
|
Uniform regardless of backend: `Retr.StartEventQuery` → `GetNextEventQueryResultBuffer`
|
|
(provider) surfaces events from wherever they were stored — so histsdk `ReadEventsAsync` is
|
|
mode-agnostic. Engine filter note: `"EventTime filtering can only be specified through
|
|
StartDateTime and EndDateTime"`.
|
|
|
|
## 6b. Two different "event" subsystems — don't conflate ✅
|
|
|
|
| | Classic event **detectors** | ArchestrA **alarms/events** (the SDK path) |
|
|
|---|---|---|
|
|
| What | server-side detectors watching tag conditions | client-streamed `HistorianEvent` (alarms, user events) |
|
|
| Config/store | `Runtime.dbo._EventTag` (TagName, DetectorTypeKey, DetectorString, Action*, ScanRate, Edge, Priority) | CM_EVENT / CommonArchestraEvent tag |
|
|
| History | **physical** `BASE TABLE Runtime.dbo.EventHistory` (`EventLogKey, TagName, DateTime, DetectDateTime, Edge`); 30 rows | block store (or A2ALMDB), surfaced via `v_AlarmEventHistory2` / `v_EventSnapshot` |
|
|
| Source | evaluated by the server | sent via `AddStreamedValue(HistorianEvent)` |
|
|
|
|
`AddStreamedValue(HistorianEvent)` feeds the **right column** (ArchestrA alarms/events) — it is
|
|
**not** the classic `EventHistory` detector log.
|
|
|
|
---
|
|
|
|
## 7. Relationship to histsdk
|
|
|
|
- histsdk implements event **reads** only (`ReadEventsAsync` via `StartEventQuery`); its
|
|
CM_EVENT EnsT2/RTag2 dance is read-subscription registration.
|
|
- Event **writing** is unimplemented but viable. Chain to replicate: Event-type connection →
|
|
register CM_EVENT (done) → serialize `HistorianEvent` (header + typed property bag) into the
|
|
event-VTQ value blob (inverse of `HistorianEventRowProtocol`) → batch into an event data
|
|
packet → stream via `AddStreamValues` (2023 R2 gRPC: `StorageService.AddStreamValues`).
|
|
|
|
---
|
|
|
|
## Open threads
|
|
|
|
- 🔶 Event value blob: field set/order known (§2a); **exact native framing** still needs one
|
|
`CCommonArchestraEventValue::PackToVtq` output capture + golden-byte test (mirror the read-side
|
|
`HistorianEventRowProtocol` reverse-engineering). Now feasible locally — the live historian is
|
|
installed, so the same instrument-and-capture approach used for reads applies.
|
|
- ❓ `EnqueueEventDataPacket.SerializedBytes` packet framing (header + N event VTQs batched).
|
|
- ✅ Database-mode store: server writer is `aahEventStorage.exe` loading the managed
|
|
`ArchestrAEvents.EventStorage.Contract` connection assembly; SQL retrieval surface is the
|
|
provider-backed `Events` / `v_AlarmEventHistory2` / `v_EventSnapshot` views (NULL T-SQL def).
|
|
This box runs **Block storage** (A2ALMDB absent). A2ALMDB physical schema still un-dumped (needs
|
|
a System-Platform-integrated box).
|
|
- ✅ `ArchestraEvent` vs `CommonArchestraEvent`: send path packs **CommonArchestraEvent** via
|
|
`EVENT_TAGID`; both are event-tag schemas in `CTagMetadata` (the server stores either).
|
|
- ❓ `UpdateEventStatus` wire payload for `Update`/`Delete` revisions.
|
|
- ❓ `EventStorage` recovery-log (`C:\Historian\Data\Logs\EventStorage`) on-disk format (WAL).
|
|
- 🔶 Decompile `ArchestrAEvents.EventStorage.Contract.dll` (managed) for the exact DB insert
|
|
contract/schema — locate it (not under `…\Wonderware\Historian`; check GAC / Framework\Bin).
|
|
|
|
---
|
|
|
|
## Changelog
|
|
- Rev 4 (live local install): confirmed server side. `aahEventStorage.exe` (`InSQLEventSystem`)
|
|
is the event store engine — per-client event-tag pipeline → recovery-log WAL → Block storage
|
|
OR SQL (loadable `ArchestrAEvents.EventStorage.Contract` assembly) → EventReplication. This box
|
|
uses **Block storage** (527 `.dat` in `C:\Historian\Data\Circular`; A2ALMDB absent). SQL
|
|
`Events`/`v_AlarmEventHistory2`/`v_EventSnapshot` are **provider-backed views over the blocks**
|
|
(NULL `OBJECT_DEFINITION`), not physical tables — `v_AlarmEventHistory2` (224 rows/30d) is the
|
|
SDK-event read surface. Distinguished the classic event-detector subsystem (`_EventTag` →
|
|
physical `EventHistory`) from the ArchestrA alarm/event path (the SDK's target).
|
|
- Rev 3: event value serialization pinned to native `CCommonArchestraEventValue::PackToVtq`
|
|
via managed `HistorianEvent.PackToVtq`; documented the `CCommonArchestraEventStruct` field
|
|
set/order (ReceivedTime, EventType, EventTime, Id, RevisionVersion, IsUpdate/IsDelete,
|
|
Namespace, typed property bag) and the path to a managed send serializer.
|
|
- Rev 2: HistorianEvent structure + HistorianEventPropertyType enum; client marshaling
|
|
(HISTORIAN_VALUE2 / ConvertManagedStructToUnmanagedStruct / AddHistorianValue); dedicated
|
|
event pipeline (OpenEventConnection2 / EnqueueEventDataPacket / ForwardEventSnapshot /
|
|
store-and-forward); revisions (Update/Delete/UpdateEventStatus); Blocks-mode clarified
|
|
(events = generic VTQ snapshots, no event-specific block code).
|
|
- Rev 1: client send path, EventStorageMode switch, Blocks/Database backends, read-back.
|