Files
histsdk/README.md
T
Joseph Doherty b3417c2f6a docs(grpc): record DelTep multiplexed-channel probe as disproven
README transport matrix + grpc-tooling-completion.md §Out-of-scope: the gRPC
multiplexed-channel hypothesis for DeleteTagExtendedProperties was probed live
2026-06-22 and disproven — primes succeed on the shared channel but DelTep is
still rejected (native code=1), property survives. Stays server-blocked on both
transports, not shipped.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B6mcaT2PjRFKcogzp9UkfC
2026-06-22 06:55:05 -04:00

342 lines
20 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# histsdk
Pure managed .NET 10 client for AVEVA Historian's binary WCF protocol. The
production SDK has no dependency on `aahClientManaged.dll`, `aahClient.dll`, or
any other AVEVA native runtime — the wire protocol is reverse-engineered and
re-implemented in C#.
The supported surface (per [`CLAUDE.md`](CLAUDE.md)):
| Operation | Status |
|---|---|
| `ProbeAsync` | live-verified |
| `ReadRawAsync` | live-verified |
| `ReadAggregateAsync` | live-verified across all 16 retrieval modes |
| `ReadAtTimeAsync` | live-verified |
| `ReadEventsAsync` | live-verified (typed event + 31-property property bag) |
| `BrowseTagNamesAsync` | live-verified |
| `GetTagMetadataAsync` | live-verified for 17 distinct native data-type codes |
| `GetConnectionStatusAsync` | synthesized from authenticated probe (matches native semantic) |
| `GetStoreForwardStatusAsync` | gRPC: measured idle-state (live-verified — contacts server, reports `ErrorOccurred` when unreachable; active-SF magnitude is D2-gated). Non-gRPC: synthesized defaults |
| `GetSystemParameterAsync` | live-verified via `Stat/GetSystemParameter` |
| `EnsureTagAsync` | live-verified for analog Float/Double/Int2/Int4/UInt4; `ApplyScaling=true` persists distinct MinRaw/MaxRaw |
| `DeleteTagAsync` | live-verified |
Out of scope: writing samples (`AddS2` is architecturally blocked — the server's
runtime cache only ingests from configured IOServer / Application Server
pipelines), store-forward write, configuration changes, discrete/string tag
creation (native `AddTag` rejects them).
## Quick start
```csharp
using AVEVA.Historian.Client;
using AVEVA.Historian.Client.Models;
await using HistorianClient client = new(new HistorianClientOptions
{
Host = "localhost",
IntegratedSecurity = true,
Transport = HistorianTransport.LocalPipe,
});
DateTime endUtc = DateTime.UtcNow;
DateTime startUtc = endUtc - TimeSpan.FromMinutes(10);
await foreach (HistorianSample sample in client.ReadRawAsync(
"SysTimeSec", startUtc, endUtc, maxValues: 100))
{
Console.WriteLine($"{sample.TimestampUtc:o} {sample.NumericValue} Q={sample.Quality}");
}
```
For a remote Historian over Net.TCP set `Transport = HistorianTransport.RemoteTcpIntegrated`
and `Host` to the server hostname. Both `RemoteTcpIntegrated` (Windows transport
auth) and `RemoteTcpCertificate` (server-cert TLS) are now live-verified for
`ProbeAsync`; `RemoteTcpIntegrated` is additionally live-verified for the full
read / browse / metadata / event / status surface.
## Transport matrix (WCF vs gRPC)
The SDK speaks two wire protocols. **WCF** is the 2020 binary MDAS protocol over
Net.TCP `32568` (transports `LocalPipe`, `RemoteTcpIntegrated`,
`RemoteTcpCertificate`); **gRPC** is the 2023 R2 HCAL transport over HTTP/2
`32565` (transport `RemoteGrpc`). The core read chain reuses the *same* native
Open2 buffers and SSPI tokens on both — on gRPC they simply ride inside protobuf
`bytes` fields — so reads are at parity. The surfaces diverge at the edges.
Legend: ✅ tooled + live-verified · ⚠️ tooled, partial/synthesized ·
🧪 tooled + routed but **sandbox-gated** (mutates server state, not yet run
destructively against a live box) · 🔌 **the gRPC server exposes the RPC
(recovered in `Grpc/Protos/*.proto`) but the SDK doesn't drive it yet** —
untooled/uncaptured, *not* a protocol gap · ⛔ tooled but **server-walled** (the
request rides the RPC but the server faults on an unmet precondition) ·
❌ unavailable on that transport.
| Operation | WCF | gRPC | Notes |
|---|:---:|:---:|---|
| `ProbeAsync` | ✅ | ✅ | |
| `ReadRawAsync` | ✅ | ✅ | |
| `ReadAggregateAsync` | ✅ | ✅ | all 16 retrieval modes |
| `ReadAtTimeAsync` | ✅ | ✅ | |
| `BrowseTagNamesAsync` | ✅ | ✅ | |
| `GetTagMetadataAsync` | ✅ | ✅ | |
| `GetSystemParameterAsync` | ✅ | ✅ | |
| `AddHistoricalValuesAsync` | ❌ | ✅ | historical/backfill writes ride `HistoryService.AddStreamValues`; non-gRPC throws `ProtocolEvidenceMissingException` |
| `GetServerTimeZoneAsync` | ❌ | ✅ | 2020 `GetSystemTimeZoneName` is a client-side stub (empty); WCF throws |
| `GetStoreForwardStatusAsync` | ⚠️ | ✅ | gRPC contacts the server (measured idle-state, reports `ErrorOccurred`); WCF returns synthesized all-false. Active-SF magnitude is D2-gated on both |
| `GetRuntimeParameterAsync` | ✅ | ✅ | tooled + live-verified over gRPC (`StatusService.GetRuntimeParameter`, the 2020 `GETRP` buffers ride unchanged) |
| `GetTagExtendedPropertiesAsync` | ✅ | ✅ | tooled + live-verified over gRPC (`RetrievalService.GetTagExtendedPropertiesFromName`, the `GetTepByNm` buffers ride unchanged). The shared parser now handles the live multi-property response shape (one group per property + a uint16 searchability-flags trailer), fixed 2026-06-22 |
| `ExecuteSqlCommandAsync` | ✅ | ⛔ | gRPC request rides `RetrievalService.ExecuteSqlCommand`, but the server-side `CSrvDbConnection.ExecuteSqlCommand` faults (`IndexOutOfRange`, native err 38) — an unmet DB-connection precondition. A `HistoryService.RegisterTags` prime does **not** clear it (tried live 2026-06-22, both 0x402/0x401). Bounded behind `ProtocolEvidenceMissingException`. Use WCF |
| `ReadEventsAsync` | ✅ | ⚠️ | tooled + routed over gRPC: the full CM_EVENT registration replay (`UpdateClientStatus``RegisterTags``EnsureTags` + discovery probes) runs and `StartEventQuery` succeeds, but `GetNextEventQueryResultBuffer` **long-polls** on no data (it blocks to the deadline rather than returning the synchronous 5-byte code-85 terminal the WCF op gives). The read is **hard-bounded** (≤30s) and throws `ProtocolEvidenceMissingException` on the no-row path rather than assert a false empty. Row-level retrieval is **not yet live-verified** — the dev box holds no events; pending a capture against an event-bearing 2023 R2 server. Use WCF for event reads |
| `SendEventAsync` | ✅ | 🔌 | rides `AddStreamValues` family; no distinct event-send RPC, framing uncaptured over gRPC |
| `EnsureTagAsync` / `DeleteTagAsync` / `RenameTagsAsync` | ✅ | ✅ | live-verified 2026-06-22 over gRPC (`HistoryService.EnsureTags` / `DeleteTags` / `StartJob`, write-enabled 0x401 session, WCF serializers reused) via a self-cleaning sandbox-tag lifecycle. Rename is an async StartJob — transiently rejectable right after create, so callers should retry |
| `AddTagExtendedPropertiesAsync` | ✅ | ✅ | live-verified 2026-06-22 over gRPC (`HistoryService.AddTagExtendedProperties`, write-enabled session); a written prop now round-trips through `GetTagExtendedPropertiesAsync` (the multi-property parser fix above). `DeleteTagExtendedProperties` stays unshipped: probed over gRPC 2026-06-22 (prime `GetTgByNm`+`GetTepByNm` then `DelTep`, all on the one shared channel) — the server still rejects the delete (native code=1) and the property survives, so gRPC's multiplexed channel does **not** lift the WCF per-connection working-set wall |
| `GetConnectionStatusAsync` | ✅ | ✅ | live-verified 2026-06-22 over gRPC — measured from the handshake (`OpenConnection` yields a storage-session GUID ⇒ connected). No dedicated RPC on either transport; store-forward connectivity stays false (D2-gated) |
| `ReadBlocksAsync` | ❌ | ❌ | `StartBlockRetrievalQuery` never captured on either transport — throws `ProtocolEvidenceMissingException` |
In short: **WCF is the broad, mature surface** (every config write, events, SQL,
and all reads), while **gRPC is the narrower *tooled* surface** — but the 2023 R2
gRPC *contract* is actually a **superset** of WCF. The recovered config RPCs carry
the **same opaque `bytes` buffers the existing WCF serializers already emit**,
keyed by the same `strHandle`/`uiHandle` session handle the read path obtains —
confirmed by tooling the read-side config ops (`GetRuntimeParameter`,
`GetTagExtendedProperties`) live: the WCF buffers ride the gRPC RPC unchanged and
the server accepts them. Two caveats surfaced when capturing the rest: `ExecuteSqlCommand`
is **server-walled** (the front-door `CSrvDbConnection` faults on a DB-connection
precondition the managed session doesn't establish — the same *class* of wall as
`OpenStorageConnection`), and `ReadEvents` is now tooled over gRPC (the CM_EVENT
registration state machine is ported and `StartEventQuery` succeeds) but its row
retrieval is not yet live-verified: the gRPC server long-polls
`GetNextEventQueryResultBuffer` on no data instead of returning the WCF code-85
terminal, so on the idle dev box the bounded read throws
`ProtocolEvidenceMissingException` rather than fabricate an empty result —
confirming rows awaits an event-bearing 2023 R2 server. The remaining 🔌 row
(`SendEventAsync`) is a **capture-and-wire** item (route the existing serializer
into a gRPC orchestrator + live-capture), not protocol-discovery — but per
"capture first, never guess wire bytes" it stays untooled until verified live. The
natural production pattern today: `RemoteGrpc` now covers reads,
`AddHistoricalValuesAsync`, the tag-config writes (create/delete/rename/extended
properties, including read-back), and connection status — all live-verified. Use
WCF for SQL (server-walled on gRPC) and event reads/sends (gRPC event rows are
long-poll-blocked pending an event-bearing server).
> A 2023 R2 server reports History interface version 12 (vs. 11 on 2020). The
> connect-time version gate accepts both — they are byte-compatible — so gRPC
> works against a v12 server with the default `VerifyServerInterfaceVersion=true`;
> no opt-out is required.
## Resilience subsystems (M4)
Two **pure client-side** subsystems layered on top of `HistorianClient`. They use
only the public SDK surface — no extra reverse-engineering, no server-side
protocol — and are fully unit-tested without a live server.
### Store-and-forward (`AVEVA.Historian.Client.StoreForward`)
A durable local outbox that buffers writes when the Historian is unreachable and
replays them on reconnect. `HistorianStoreForwardWriter` wraps a `HistorianClient`
(or any `IHistorianWriteSink`) and persists each enqueued write to an
`IHistorianOutboxStore``FileHistorianOutboxStore` (crash-durable, atomic
JSON-per-entry, FIFO by filename sequence, corrupt-file quarantine) or
`InMemoryHistorianOutboxStore`. A background drain loop retries with FIFO
head-of-line ordering, optional `MaxDeliveryAttempts` dead-lettering, and a
`DropOldest`/`Reject` overflow policy.
```csharp
using AVEVA.Historian.Client.StoreForward;
await using HistorianStoreForwardWriter writer = new(
client,
new FileHistorianOutboxStore(@"C:\ProgramData\histsdk\outbox"));
await writer.StartAsync(); // background drain on reconnect
await writer.EnqueueHistoricalValuesAsync("MyTag",
[new HistorianHistoricalValue(DateTime.UtcNow, 42.0)]); // returns immediately, durable
HistorianStoreForwardStatusSnapshot status = await writer.GetStatusAsync();
// status.PendingCount / .Storing / .ErrorOccurred — mirrors the server SF semantics
```
This is a pragmatic outbox, **not** the bit-faithful native SF cache
(`Forward*Snapshot` decode), which stays deferred.
### Multi-historian redundancy (`AVEVA.Historian.Client.Redundancy`)
`HistorianRedundantClient` fronts N members as one logical client. Reads fail over
to the next healthy member in priority order (streaming reads fail over only
*before the first row* — mid-stream failures propagate to avoid duplicated/skipped
rows); writes fan out (`AllMembers`/`PreferredOnly`) under an `All`/`Any`
acknowledgement policy and return a per-member `HistorianRedundantWriteResult`. A
failure-threshold demotion + background watchdog restores recovered members.
```csharp
using AVEVA.Historian.Client.Redundancy;
await using HistorianRedundantClient cluster = new(
[
new HistorianClientMember("primary", primaryClient),
new HistorianClientMember("secondary", secondaryClient),
]);
await cluster.StartAsync(); // health watchdog
await foreach (HistorianSample s in cluster.ReadRawAsync("MyTag", startUtc, endUtc, 100))
{
// served by the first healthy member; transparently fails over on connect
}
HistorianRedundantWriteResult w = await cluster.AddHistoricalValuesAsync("MyTag",
[new HistorianHistoricalValue(DateTime.UtcNow, 42.0)]); // fanned out across members
```
Backing a member's writes with a `HistorianStoreForwardWriter` gives the pragmatic
equivalent of native ReSyncTags: a down member buffers locally and replays on
recovery.
## Build & test
```powershell
dotnet build .\Histsdk.slnx --no-restore
dotnet test .\Histsdk.slnx --no-build --logger "console;verbosity=minimal"
```
Run a single test class:
```powershell
dotnet test .\Histsdk.slnx --no-build --filter "FullyQualifiedName~WcfDataQueryProtocolTests"
```
Live integration tests (`tests/AVEVA.Historian.Client.Tests/HistorianClientIntegrationTests.cs`)
are gated and skip cleanly without these env vars:
```powershell
$env:HISTORIAN_HOST = 'localhost'
$env:HISTORIAN_TEST_TAG = 'SysTimeSec' # any tag the server has data for
# Optional for non-integrated auth:
$env:HISTORIAN_USER, $env:HISTORIAN_PASSWORD
$env:HISTORIAN_TAG_FILTER = 'Sys*' # or any LIKE-pattern
```
The 2023 R2 `RemoteGrpc` transport has its own gated live suite
(`HistorianGrpcIntegrationTests`) covering the full tooled gRPC surface — probe,
raw / aggregate (incl. multiple retrieval modes) / at-time reads, browse,
metadata, system-parameter, server time-zone, and measured store-forward status.
It skips cleanly unless `HISTORIAN_GRPC_HOST` is set:
```powershell
$env:HISTORIAN_GRPC_HOST = 'my-2023r2-host' # gates the gRPC suite
$env:HISTORIAN_TEST_TAG = 'SysTimeSec'
$env:HISTORIAN_USER, $env:HISTORIAN_PASSWORD # required for the authenticated ops
# Optional:
$env:HISTORIAN_GRPC_PORT = '32565' # default 32565
$env:HISTORIAN_GRPC_TLS = 'true' # gRPC over TLS
$env:HISTORIAN_GRPC_DNSID = 'my-2023r2-host' # cert DNS name when connecting by IP
$env:HISTORIAN_GRPC_TIMEOUT = '120' # per-call deadline (s); raise for slow links
$env:HISTORIAN_WRITE_SANDBOX_TAG = 'MyFloatTag' # gates the AddHistoricalValues write test
$env:HISTORIAN_GRPC_WRITE_SANDBOX_TAG = 'SandboxTag' # gates the DESTRUCTIVE tag create/rename/delete lifecycle test
```
The aggregate tests self-calibrate their query window from a real raw sample, so
they pass against an idle 2023 R2 server (no recent collection) as well as a
live-collecting one.
## Repository layout
```
src/AVEVA.Historian.Client/ Production SDK — pure managed .NET 10. No native AVEVA dependency.
tests/ Unit tests (golden-byte / round-trip) + gated live integration tests.
tools/ Reverse-engineering tooling (NOT shipped):
AVEVA.Historian.ReverseEngineering/ .NET 10 CLI: WCF probes, dnlib IL inspection,
IL-rewrite instrumentation (instrument-wcf-{write,read}message,
instrument-openconnection*, instrument-startdataquery, etc.).
AVEVA.Historian.NativeTraceHarness/ .NET Framework harness that loads aahClientManaged.dll for
byte-for-byte parity testing against the native wrapper.
AVEVA.Historian.NetFxWcfProbe/ .NET Framework WCF probe for ruling out .NET 10-only differences.
AVEVA.Historian.ReverseInstrumentation/ Helper assembly injected into IL-rewritten wrapper copies.
AVEVA.Historian.WcfCaptureServer/ Fake server for endpoint experiments.
scripts/ PowerShell + Frida runners + Python decoders for capture analysis.
fixtures/protocol/ Sanitized golden-byte fixtures.
docs/reverse-engineering/ Sanitized handoff evidence; commit-safe summaries only.
```
Native AVEVA binaries (`current/`, `aveva-install-x64/`, `aveva-install-x86/`) are
**gitignored**. Each developer fetches their own copy from the AVEVA installer; we
neither modify nor redistribute them.
## Documentation
Read in order before non-trivial work:
| Doc | Purpose |
|---|---|
| [`AGENTS.md`](AGENTS.md) | Standing project constraints and safety rules. |
| [`CLAUDE.md`](CLAUDE.md) | Build commands, code architecture, the active protocol blocker (now resolved) and SDK surface. |
| [`docs/reverse-engineering/handoff.md`](docs/reverse-engineering/handoff.md) | Decision record + decoded protocol evidence (binding shapes, descriptor layouts, Open2 v6 response, event-row wire format, property-bag value encoding). |
| [`instructions.md`](instructions.md) | Original plan + decision record. |
## Architecture
Three intentionally decoupled subsystems under `src/AVEVA.Historian.Client/`:
- **`HistorianClient` + `HistorianClientOptions`** — public façade. Validates inputs;
delegates reads to `Historian2020ProtocolDialect`; delegates probe / tag metadata /
browse / status helpers to the WCF layer.
- **`Wcf/`** — managed WCF/MDAS layer. Custom `MdasMessageEncoder` wraps SOAP 1.2 +
WS-Addressing 1.0 in AVEVA's `application/x-mdas` framing. Three binding flavors
via `HistorianWcfBindingFactory`: plain MDAS over Net.NamedPipe (local), MDAS +
Windows transport (remote `/Hist-Integrated`), MDAS + certificate (remote
`/HistCert`). Service contracts in `Wcf/Contracts/` mirror the server-side WCF
surface (versioned per native interface — `IHistoryServiceContract`,
`IRetrievalServiceContract2..4`, `IStatusServiceContract2`, etc.).
- **`Protocol/`** — binary frame layer. `Historian2020ProtocolDialect` is the
version-anchored bridge between the public façade and the WCF + frame layers.
Methods without protocol evidence throw `ProtocolEvidenceMissingException` rather
than guessing wire bytes.
- **`Models/`** — public DTOs and enums. `HistorianSample`, `HistorianAggregateSample`,
`HistorianEvent`, `HistorianTagMetadata`, `HistorianDataType`, `RetrievalMode`,
`HistorianTransport`, etc.
Read flow end-to-end (live-verified against `localhost`):
```
Hist.GetV → Hist.ValCl × N → Hist.Open2 → Retr.GetV → Retr.IsOriginalAllowed →
Retr.StartQuery2 → loop Retr.GetNextQueryResultBuffer2 → typed HistorianSample rows
```
Event flow end-to-end (live-verified):
```
Hist.GetV → Hist.ValCl × N → Hist.Open2 → Hist.UpdC3 → 6× Stat.GetSystemParameter →
Hist.RTag2 → Stat.GetSystemParameter(AllowRenameTags) → Trx.GetV → Stat.GetV →
Retr.GetV → Hist.EnsT2(CmEventTagId) → Retr.StartEventQuery →
loop Retr.GetNextEventQueryResultBuffer → typed HistorianEvent rows with
property dictionary → Retr.EndEventQuery → Hist.Close2
```
## Safety
- Production code under `src/` must remain pure managed .NET 10 with no native AVEVA
reference. Reverse-engineering harnesses under `tools/` may reference native binaries.
- Never commit credentials, hostnames, user names, customer tag names, or raw packet
captures. `*.ndjson`, `current/`, `aveva-install-*/`, and `artifacts/` are gitignored
precisely because they accumulate identity-bearing runtime data.
- Methods without protocol evidence throw `ProtocolEvidenceMissingException`. Do not
stub fake behavior — leave them throwing until evidence supports an implementation.
## Status
313 unit + live integration tests pass (`dotnet test --logger "console;verbosity=minimal"`).
Full SDK surface — reads, browse, metadata, status, plus the two write ops
(`EnsureTagAsync` / `DeleteTagAsync`) — verified end-to-end against both a
local Historian (`LocalPipe`) and a remote Historian (`RemoteTcpIntegrated`
over Net.TCP with Windows transport auth). `RemoteTcpCertificate` ProbeAsync
is live-verified; deeper coverage over the cert transport plus the
explicit-credentials path await additional verification.
The 2023 R2 `RemoteGrpc` transport's full tooled surface is live-verified against
a real 2023 R2 server: probe, raw / aggregate (TimeWeightedAverage +
Minimum/MaximumWithTime + BestFit) / at-time reads, browse, metadata,
system-parameter, server time-zone, and measured store-forward status — plus the
gRPC-only `AddHistoricalValuesAsync` backfill write. The remaining gRPC config ops
are exposed by the server but untooled (see the transport matrix above).