histsdk/docs/plans/histevents.md

# How a HistorianEvent reaches the Historian DB files

Living analysis doc. Traces an event end-to-end: client API → wire → server
storage backend (SQL **Database** vs history **Blocks** `.dat`) → read-back.

Evidence base: 2023 R2 `aahClientManaged.dll` (decompiled `ArchestrA.HistorianAccess`,
`HistorianEvent`, `HistorianEventPropertyType`), native `aahStorage.exe` (string
analysis), the recovered gRPC + CloudHistorian contracts, and the histsdk read-side
reverse-engineering (CM_EVENT registration + event-row parser).

Status legend: ✅ proven (from binary) · 🔶 strong inference · ❓ open.

---

## TL;DR

An event is **not a distinct wire message**. The client turns each `HistorianEvent`
into a `HistorianDataValue` of type `Event` against the built-in **`CM_EVENT`** tag,
marshals it into a native VTQ, and **streams it like any tag value** on a dedicated
*Event* connection. Events are batched into an opaque serialized **event data packet**
and delivered (with their own store-and-forward "event snapshot" path). On the server
they are persisted into **one of two backends, chosen by a server-configured
`EventStorageMode`**: a SQL **Database**, or the history **Blocks** (`.dat`) files.

```
HistorianEvent (Type + typed property bag)
  └─(AddStreamedValue)→ HistorianDataValue{Type=Event, TagKey=CM_EVENT, EventTime, Q=192}
       └─ HistorianEvent.PackToVtq → CCommonArchestraEventValue::PackToVtq (native value blob)
            └─ HISTORIAN_VALUE2 (44B; blob ptr @+33) → HistorianClient.AddHistorianValue → queue
                 └─ flush → EnqueueEventDataPacket{ byte[] SerializedBytes } (batched VTQs)
                      └─ SERVER: aahEventStorage.exe (InSQLEventSystem)
                           per-client event-tag pipeline → recovery-log WAL → backend:
                             • Blocks   → elastic snapshot → frozen → history .dat (Circular/Permanent)
                             • Database → ArchestrAEvents.EventStorage.Contract assembly → SQL (A2ALMDB)
                           → EventReplication (redundant historians)
                 └─ (offline) → store-and-forward → ForwardEventSnapshotBegin/…/End on reconnect
  read-back: Retr.StartEventQuery / SQL provider views (Events, v_AlarmEventHistory2, v_EventSnapshot)
```

---

## 1. The `HistorianEvent` object ✅

Decompiled `ArchestrA.HistorianEvent` — a structured header plus a **typed property bag**:

- **Header fields:** `ID`/`Id` (Guid), `Type`/`EventType` (string, e.g. `"Alarm.Set"`,
  `"User.Write"`), `EventTime` (DateTime), `ReceivedTime`, `Severity` (ushort),
  `Priority` (ushort), `IsAlarm`, `IsSilenced`, `System`, `Source`, `Source_Name`,
  `Area`, `Namespace`, `DisplayText`.
- **Revision fields:** `RevisionVersion` (ushort), `Delete` (bool), `Update` (bool) —
  events are revisable (see §4 UpdateEventStatus).
- **Property bag:** `AddProperty(name, value, HistorianEventPropertyType, …)` with typed
  overloads. `HistorianEventPropertyType` (alphabetical enum):
  `Blob, Boolean, Byte, Date, DateTime, Decimal, Double, Duration, Float, Guid, Hex,
  Int, Integer, Long, Short, String, Time, UnsignedByte, UnsignedInt, UnsignedLong,
  UnsignedShort, Undefined`.

These map onto the wire property-bag the histsdk **read** parser already decodes
(`HistorianEventRowProtocol`): typemarkers `0x02` Boolean, `0x10` Guid, `0x18` FILETIME,
`0x31` Int32, `0x43` UTF-16 string, … — i.e. the write enum and the read typemarkers are
two views of the same typed-value format. The event-send serialization is the inverse of
that read parser.

---

## 2. Client send path — an event becomes a streamed VTQ ✅

From decompiled `ArchestrA.HistorianAccess` (line refs into the decompile):

1. **Open an Event connection.** `HistorianConnectionArgs.ConnectionType =
   HistorianConnectionType.Event`, `ReadOnly = false` (sample `Step10.SendEvents`).
2. **Default event tag.** `CreateDefaultEventTag()` (`:3006`) registers tag `CM_EVENT` /
   "AnE Event" / `TagDataType = Event` and stores `eventTagHandle`. Same CM_EVENT
   registration histsdk reverse-engineered (RTag2 + EnsT2; tag id
   `353b8145-5df0-4d46-a253-871aef49b321`).
3. **Wrap as VTQ.** `AddStreamedValue(HistorianEvent)` (`:3123`):
   ```csharp
   historianDataValue.objValue      = historianEvent;          // header + property bag
   historianDataValue.DataValueType = HistorianDataType.Event;
   historianDataValue.TagKey        = eventTagHandle;          // CM_EVENT
   historianDataValue.StartDateTime = historianEvent.EventTime;
   historianDataValue.OpcQuality    = 192;
   return AddStreamedValue((ConnectionIndex)1, historianDataValue, false, out error); // 1=Event
   ```
4. **Marshal + queue.** The private `AddStreamedValue` (`:3173`):
   - builds a 44-byte native `HISTORIAN_VALUE2` (`InitBlockUnaligned(…,0,44)`),
   - `HistorianAccessUtil.ConvertManagedStructToUnmanagedStruct(value, &HV2, bVersioned…)`
     — its `case HistorianDataType.Event` (`HistorianAccessUtil:89`) calls
     **`HistorianEvent.PackToVtq(out byte[])`** to produce the event value blob, whose pointer
     is placed at `HISTORIAN_VALUE2+33` (freed after send; offset 33 is the value-union pointer
     used for Event/String types),
   - `HistorianClient.AddHistorianValue(client, &HV2, &err)` (`:3209`) queues the VTQ into
     the native delivery buffer and returns immediately.

So an event uses the **same streaming machinery as a process value**; only `DataValueType`
(`Event`) and the target tag (`CM_EVENT`) differ.

### 2a. Event value serialization — `HistorianEvent.PackToVtq` ✅/🔶

`HistorianEvent.PackToVtq` (`HistorianEvent:1392`) populates a native
**`CCommonArchestraEventStruct`** then hands it to the **native** packer
`CCommonArchestraEventValue::PackToVtq(…, 192, 192, vtq)` (Q=192), associated with the
built-in `EVENT_TAGID` / `EVENT_TAGNAME` (`CTagMetadata.CommonArchestraEvent`). The actual
byte layout is produced in C++ — **not visible in managed code** — so pinning exact write
bytes needs a wire/IL capture, exactly as the read side did. But the **field set + order**
the managed code writes into the struct is now known:

```
SetReceivedTime  (uint64 FILETIME, from UniqueTime.GetUniqueFileTime — unique/monotonic)
SetEventType     (wchar* string, e.g. "Alarm.Set")
SetEventTime     (uint64 FILETIME, from EventTime)
SetId            (GUID)
SetRevisionVersion (uint16)
SetIsUpdate      (bool)        ← revision flags
SetIsDelete      (bool)
Namespace        (string, trimmed, non-printable-validated)
…then the typed property bag: Dictionary<string, Tuple<HistorianEventPropertyType, object>>
```

This matches `HistorianEventRowProtocol` on read: the property bag is name→(type,value) with
the same typed-value encoding (typemarkers `0x02/0x10/0x18/0x31/0x43/…`). So a managed
event-send serializer is tractable: emit the header struct fields above, then the typed
property bag in the read parser's format. The remaining unknown is only the exact native
framing offsets — best obtained by capturing one `PackToVtq` output, then golden-byte testing.

---

## 3. Event transport / delivery pipeline ✅ (CloudHistorian + gRPC contracts)

Events have a **dedicated, batched** connection + delivery pipeline, distinct from tag data
but structurally parallel:

| Stage | Event op | Tag-data analogue |
|---|---|---|
| Open connection | `OpenEventConnection2 { byte[] ClientInfo } → { byte[] ServerInfo }` | OpenConnection |
| Send batch | `EnqueueEventDataPacket { byte[] SerializedBytes }` | `EnqueueTagDataPacket { byte[] SerializedBytes }` |
| Store-and-forward | `ForwardEventSnapshotBegin / ForwardEventSnapshot / ForwardEventSnapshotEnd` | `ForwardSnapshot…` |
| Revise | `UpdateEventStatus` | (revision write) |

Key point: the **event data packet is an opaque serialized byte buffer** (`SerializedBytes`,
DataMember `d`) — the queued event VTQs batched together, exactly the same envelope shape as
the tag data packet. On-prem this is what the storage-streaming op (`AddStreamValues`)
carries; in the cloud variant it is `EnqueueEventDataPacket`.

Validation surfaced via error codes: `InvalidAlarmEventPropertyLength=212`,
`AlarmEventPropertyHasNonPrintableChar=214`, `AlarmEventPropertyHasInvalidSpecialChar=215`,
`AlarmEventPropertyNameIsAReservedName=216` — the server validates alarm/event property
names + values on ingest.

Offline → events spool to the **store-and-forward** cache and replay as **event snapshots**
(`ForwardEventSnapshot*`) on reconnect — a separate SF stream from tag-data snapshots.

---

## 4. Revisions / updates ✅

`HistorianEvent.Update` / `.Delete` / `.RevisionVersion` + the contract's
`UpdateEventStatus` op mean events are not write-once: an event can be re-sent to update or
delete a previously stored event (e.g. alarm acknowledge/clear), bumping `RevisionVersion`.

---

## 5. The storage-backend switch — `EventStorageMode` ✅

The client reads the server's event-storage backend from `HISTORIAN_INFO` **byte offset 514**
(`HistorianAccess` `:5715`):

```csharp
EventStorageMode = (info[514] == -1) ? Unsupported
                 : (info[514] == 0)  ? Database     // SQL Server
                                     : Blocks;       // history .dat blocks
```

`HistorianEventStorageMode ∈ { Database, Blocks, Unsupported }`. The destination is a
**server** decision; the client streams the same VTQ regardless.

---

## 6. Server side — where it lands ✅ (confirmed on the live local install)

The server-side event component is **`aahEventStorage.exe`** (service `InSQLEventSystem`,
"AVEVA Historian Event System"; plus `aahEventSvc.exe`), at
`…\Wonderware\Historian\x64\aahEventStorage.exe`. Its string table maps the full pipeline:

```
event packets / forwarded snapshots → per-client "event tag pipeline" → batch enqueue
   → Event Storage Recovery Log (WAL; "enqueuing N events to log",
        path SystemParameter EventStorageLogPath = C:\Historian\Data\Logs\EventStorage)
   → persist to the active backend:
        Block Storage  ("Enabled Block Storage for events")  → history .dat blocks
        Database       ("storing N events in database")      → SQL via loaded managed
              assembly ArchestrAEvents.EventStorage.Contract.EventStorageDatabaseConnection
              (e.g. ";Initial Catalog=A2ALMDB;Integrated Security=true;Encrypt=True;…")
   → also fed to EventReplication (aahReplication.exe) for redundant historians
```

So persistence is **pluggable** (a loadable connection assembly) and dual-mode, guarded by a
recovery log. Which backend is live depends on configuration (the `EventStorageMode` of §5).

### This historian = **Block storage** (verified)
- `C:\Historian\Data\Circular` holds **527 `.dat` history blocks** (`Permanent` empty); the
  EventStorage recovery log dir exists. `aahEventStorage` logs `"Enabled Block Storage for
  events"`.
- SDK-shape alarm/events are present and retrievable: `Runtime.dbo.v_AlarmEventHistory2`
  returns 224 rows over the last 30 days.
- `A2ALMDB` (the System-Platform alarm DB the connection string references) is **not present**
  here — that path is only used when integrated with AVEVA System Platform alarming. Absent
  it, ArchestrA events land in **blocks**, exactly as `aahStorage.exe` advertises (`"Stores
  ArchestrA Event Data"`, snapshot→block).

### The SQL surface is **provider-backed views, not physical tables** ✅
In `Runtime`, the rich event objects are **views with NULL `OBJECT_DEFINITION`** — i.e. the
historian's OLE DB History provider exposes them as virtual/extension tables that read the
block store, *not* stored T-SQL:
- `Events`, `v_EventHistory`, `v_EventSnapshot`, `v_EventStringSnapshot`, **`v_AlarmEventHistory2`**
  (columns: `EventStampUTC`, `AlarmState`, `TagName`, `Description`, `Area`, `Type`, `Value`,
  `Priority`, `Category`, `Provider`, `Operator`, `DomainName`, `UserFullName`, `MilliSec`, …)
  — these are the read-back of the SDK alarm/event property bag.

So `SELECT … FROM Events` (and `v_AlarmEventHistory2`) is **the provider reading the block
store**, which is why the handoff could query events even though they live in `.dat` blocks.

### Database-mode physical store
When events ARE stored in SQL (Database mode / A2ALMDB integration), the writer is the loaded
`ArchestrAEvents.EventStorage.Contract` connection assembly doing batched inserts ("storing N
events in database", "creating event storage database connection role"). The exact table
schema there is the A2ALMDB alarm schema (not present on this box to dump).

### Read-back ✅
Uniform regardless of backend: `Retr.StartEventQuery` → `GetNextEventQueryResultBuffer`
(provider) surfaces events from wherever they were stored — so histsdk `ReadEventsAsync` is
mode-agnostic. Engine filter note: `"EventTime filtering can only be specified through
StartDateTime and EndDateTime"`.

## 6b. Two different "event" subsystems — don't conflate ✅

| | Classic event **detectors** | ArchestrA **alarms/events** (the SDK path) |
|---|---|---|
| What | server-side detectors watching tag conditions | client-streamed `HistorianEvent` (alarms, user events) |
| Config/store | `Runtime.dbo._EventTag` (TagName, DetectorTypeKey, DetectorString, Action*, ScanRate, Edge, Priority) | CM_EVENT / CommonArchestraEvent tag |
| History | **physical** `BASE TABLE Runtime.dbo.EventHistory` (`EventLogKey, TagName, DateTime, DetectDateTime, Edge`); 30 rows | block store (or A2ALMDB), surfaced via `v_AlarmEventHistory2` / `v_EventSnapshot` |
| Source | evaluated by the server | sent via `AddStreamedValue(HistorianEvent)` |

`AddStreamedValue(HistorianEvent)` feeds the **right column** (ArchestrA alarms/events) — it is
**not** the classic `EventHistory` detector log.

---

## 7. Relationship to histsdk

- histsdk implements event **reads** only (`ReadEventsAsync` via `StartEventQuery`); its
  CM_EVENT EnsT2/RTag2 dance is read-subscription registration.
- Event **writing** is unimplemented but viable. Chain to replicate: Event-type connection →
  register CM_EVENT (done) → serialize `HistorianEvent` (header + typed property bag) into the
  event-VTQ value blob (inverse of `HistorianEventRowProtocol`) → batch into an event data
  packet → stream via `AddStreamValues` (2023 R2 gRPC: `StorageService.AddStreamValues`).

---

## Event-send wire format ✅ (captured 2026-06-20)

`AddStreamedValue(HistorianEvent)` leaves the client over **WCF as `AddS2`
(`IHistoryServiceContract2.AddStreamValues2(string handle, byte[] pBuf, out errorBuffer)`)** —
**not** the storage-engine pipe. (Captured with the NativeTraceHarness `event-send` scenario +
instrument-wcf-writemessage; two events diffed to separate constant framing from value fields.)
CM_EVENT is a built-in registered tag, so the `129 TagNotFoundInCache` gate that blocks `AddS2`
for user tags doesn't apply. Open2 uses event connection-mode **0x501** (vs 0x402 read / 0x401
write). The `pBuf` is a storage **sample buffer** wrapping the event VTQ:

```
0x00  UInt16  0x534F "OS"
0x02  UInt16  sampleCount = 1
0x04  UInt32  packet length = delivered byte[].Length   (= valueBlob.Length + 11)
0x08  UInt16  valueBlob.Length + 1
0x0A  valueBlob:
  +0x00 GUID   CM_EVENT tag id (353b8145-5df0-4d46-a253-871aef49b321)
  +0x10 Int64  EventTime FILETIME UTC (floored to ms — the VTQ timestamp)
  +0x18 UInt16 OpcQuality = 192
  +0x1A UInt16 192
  +0x1C UInt16 0x118D   (opaque CCommonArchestraEventValue descriptor — constant)
  +0x1E GUID   event Id
  +0x2E Int64  ReceivedTime FILETIME UTC (full 100ns; native uses a unique/monotonic time)
  +0x36 compact-ASCII string  Namespace
  +.... compact-ASCII string  EventType
  +.... UInt16 eventStructVersion = 5
  +.... UInt16 propertyCount
  +.... propertyCount × { compact-ASCII name; value: UInt8 typeMarker, UInt8 len, UInt8 status, len×bytes }
trailing: 1 pad byte so byte[].Length == packet length (UInt32 @0x04). The native relies on
          the MDAS encoder adding this byte (non-deterministic value); the SDK emits 0x00.
```

Property values reuse `HistorianEventRowProtocol`'s typed encoding; only `0x43` (UTF-16 string)
is captured on the write side so far. Implemented in `HistorianEventWriteProtocol` (golden-tested)
and shipped as `HistorianClient.SendEventAsync`. **Server accepts the `AddS2` (success, empty
error buffer); on-box persistence is environment-gated (the native client behaves identically).**

## Open threads

- ⚠️ Event **persistence**: accepted `AddS2` events do not land in `v_AlarmEventHistory2` on the
  local dev box (native client identical) — the event storage/ingestion pipeline isn't active
  here. Needs a Historian with active event storage to verify send→store→read-back end-to-end.
- 🔶 Non-string event property write encodings (bool/int/filetime/guid/double) and revision/
  update/delete event sends (RevisionVersion≠0, IsUpdate/IsDelete) are **not captured**; the SDK
  throws `ProtocolEvidenceMissingException` for them. One capture each to extend.
- ❓ `EnqueueEventDataPacket.SerializedBytes` packet framing (header + N event VTQs batched).
- ✅ Database-mode store: server writer is `aahEventStorage.exe` loading the managed
  `ArchestrAEvents.EventStorage.Contract` connection assembly; SQL retrieval surface is the
  provider-backed `Events` / `v_AlarmEventHistory2` / `v_EventSnapshot` views (NULL T-SQL def).
  This box runs **Block storage** (A2ALMDB absent). A2ALMDB physical schema still un-dumped (needs
  a System-Platform-integrated box).
- ✅ `ArchestraEvent` vs `CommonArchestraEvent`: send path packs **CommonArchestraEvent** via
  `EVENT_TAGID`; both are event-tag schemas in `CTagMetadata` (the server stores either).
- ❓ `UpdateEventStatus` wire payload for `Update`/`Delete` revisions.
- ❓ `EventStorage` recovery-log (`C:\Historian\Data\Logs\EventStorage`) on-disk format (WAL).
- 🔶 Decompile `ArchestrAEvents.EventStorage.Contract.dll` (managed) for the exact DB insert
  contract/schema — locate it (not under `…\Wonderware\Historian`; check GAC / Framework\Bin).

---

## Changelog
- Rev 4 (live local install): confirmed server side. `aahEventStorage.exe` (`InSQLEventSystem`)
  is the event store engine — per-client event-tag pipeline → recovery-log WAL → Block storage
  OR SQL (loadable `ArchestrAEvents.EventStorage.Contract` assembly) → EventReplication. This box
  uses **Block storage** (527 `.dat` in `C:\Historian\Data\Circular`; A2ALMDB absent). SQL
  `Events`/`v_AlarmEventHistory2`/`v_EventSnapshot` are **provider-backed views over the blocks**
  (NULL `OBJECT_DEFINITION`), not physical tables — `v_AlarmEventHistory2` (224 rows/30d) is the
  SDK-event read surface. Distinguished the classic event-detector subsystem (`_EventTag` →
  physical `EventHistory`) from the ArchestrA alarm/event path (the SDK's target).
- Rev 3: event value serialization pinned to native `CCommonArchestraEventValue::PackToVtq`
  via managed `HistorianEvent.PackToVtq`; documented the `CCommonArchestraEventStruct` field
  set/order (ReceivedTime, EventType, EventTime, Id, RevisionVersion, IsUpdate/IsDelete,
  Namespace, typed property bag) and the path to a managed send serializer.
- Rev 2: HistorianEvent structure + HistorianEventPropertyType enum; client marshaling
  (HISTORIAN_VALUE2 / ConvertManagedStructToUnmanagedStruct / AddHistorianValue); dedicated
  event pipeline (OpenEventConnection2 / EnqueueEventDataPacket / ForwardEventSnapshot /
  store-and-forward); revisions (Update/Delete/UpdateEventStatus); Blocks-mode clarified
  (events = generic VTQ snapshots, no event-specific block code).
- Rev 1: client send path, EventStorageMode switch, Blocks/Database backends, read-back.