histsdk/README.md

# histsdk

Pure managed .NET 10 client for AVEVA Historian's binary WCF protocol. The
production SDK has no dependency on `aahClientManaged.dll`, `aahClient.dll`, or
any other AVEVA native runtime — the wire protocol is reverse-engineered and
re-implemented in C#.

The supported surface (per [`CLAUDE.md`](CLAUDE.md)):

| Operation | Status |
|---|---|
| `ProbeAsync` | live-verified |
| `ReadRawAsync` | live-verified |
| `ReadAggregateAsync` | live-verified across all 16 retrieval modes |
| `ReadAtTimeAsync` | live-verified |
| `ReadEventsAsync` | live-verified (typed event + 31-property property bag) |
| `BrowseTagNamesAsync` | live-verified |
| `GetTagMetadataAsync` | live-verified for 17 distinct native data-type codes |
| `GetConnectionStatusAsync` | synthesized from authenticated probe (matches native semantic) |
| `GetStoreForwardStatusAsync` | gRPC: measured idle-state (live-verified — contacts server, reports `ErrorOccurred` when unreachable; active-SF magnitude is D2-gated). Non-gRPC: synthesized defaults |
| `GetSystemParameterAsync` | live-verified via `Stat/GetSystemParameter` |
| `EnsureTagAsync` | live-verified for analog Float/Double/Int2/Int4/UInt4; `ApplyScaling=true` persists distinct MinRaw/MaxRaw |
| `DeleteTagAsync` | live-verified |

Out of scope: writing samples (`AddS2` is architecturally blocked — the server's
runtime cache only ingests from configured IOServer / Application Server
pipelines), store-forward write, configuration changes, discrete/string tag
creation (native `AddTag` rejects them).

## Quick start

```csharp
using AVEVA.Historian.Client;
using AVEVA.Historian.Client.Models;

await using HistorianClient client = new(new HistorianClientOptions
{
    Host = "localhost",
    IntegratedSecurity = true,
    Transport = HistorianTransport.LocalPipe,
});

DateTime endUtc   = DateTime.UtcNow;
DateTime startUtc = endUtc - TimeSpan.FromMinutes(10);

await foreach (HistorianSample sample in client.ReadRawAsync(
    "SysTimeSec", startUtc, endUtc, maxValues: 100))
{
    Console.WriteLine($"{sample.TimestampUtc:o}  {sample.NumericValue}  Q={sample.Quality}");
}
```

For a remote Historian over Net.TCP set `Transport = HistorianTransport.RemoteTcpIntegrated`
and `Host` to the server hostname. Both `RemoteTcpIntegrated` (Windows transport
auth) and `RemoteTcpCertificate` (server-cert TLS) are now live-verified for
`ProbeAsync`; `RemoteTcpIntegrated` is additionally live-verified for the full
read / browse / metadata / event / status surface.

## Transport matrix (WCF vs gRPC)

The SDK speaks two wire protocols. **WCF** is the 2020 binary MDAS protocol over
Net.TCP `32568` (transports `LocalPipe`, `RemoteTcpIntegrated`,
`RemoteTcpCertificate`); **gRPC** is the 2023 R2 HCAL transport over HTTP/2
`32565` (transport `RemoteGrpc`). The core read chain reuses the *same* native
Open2 buffers and SSPI tokens on both — on gRPC they simply ride inside protobuf
`bytes` fields — so reads are at parity. The surfaces diverge at the edges.

Legend: ✅ tooled + live-verified · ⚠️ tooled, partial/synthesized ·
🔌 **the gRPC server exposes the RPC (recovered in `Grpc/Protos/*.proto`) but the
SDK doesn't drive it yet** — untooled/uncaptured, *not* a protocol gap ·
❌ unavailable on that transport.

| Operation | WCF | gRPC | Notes |
|---|:---:|:---:|---|
| `ProbeAsync` | ✅ | ✅ | |
| `ReadRawAsync` | ✅ | ✅ | |
| `ReadAggregateAsync` | ✅ | ✅ | all 16 retrieval modes |
| `ReadAtTimeAsync` | ✅ | ✅ | |
| `BrowseTagNamesAsync` | ✅ | ✅ | |
| `GetTagMetadataAsync` | ✅ | ✅ | |
| `GetSystemParameterAsync` | ✅ | ✅ | |
| `AddHistoricalValuesAsync` | ❌ | ✅ | historical/backfill writes ride `HistoryService.AddStreamValues`; non-gRPC throws `ProtocolEvidenceMissingException` |
| `GetServerTimeZoneAsync` | ❌ | ✅ | 2020 `GetSystemTimeZoneName` is a client-side stub (empty); WCF throws |
| `GetStoreForwardStatusAsync` | ⚠️ | ✅ | gRPC contacts the server (measured idle-state, reports `ErrorOccurred`); WCF returns synthesized all-false. Active-SF magnitude is D2-gated on both |
| `ReadEventsAsync` | ✅ | 🔌 | gRPC `RetrievalService.StartEventQuery` / `GetNextEventQueryResultBuffer` / `EndEventQuery` recovered (`bytes btRequest` + handle); not tooled over gRPC |
| `SendEventAsync` | ✅ | 🔌 | rides `AddStreamValues` family; no distinct event-send RPC, framing uncaptured over gRPC |
| `EnsureTagAsync` / `DeleteTagAsync` / `RenameTagsAsync` | ✅ | 🔌 | gRPC `HistoryService.EnsureTags` / `DeleteTags` / `StartJob`(+`GetJobStatus`) recovered (`bytes btTagInfos`/`btTagnames`/`btInput` + handle) |
| `GetTagExtendedPropertiesAsync` / `AddTagExtendedPropertiesAsync` | ✅ | 🔌 | gRPC `RetrievalService.GetTagExtendedPropertiesFromName` + `HistoryService.AddTagExtendedProperties`; gRPC also exposes `DeleteTagExtendedProperties` (WCF delete was server-blocked) |
| `ExecuteSqlCommandAsync` | ✅ | 🔌 | gRPC `RetrievalService.ExecuteSqlCommand` (`StrCommand` + `uiOption`, mirrors WCF `ExeC`/`GetR`) |
| `GetRuntimeParameterAsync` | ✅ | 🔌 | gRPC `StatusService.GetRuntimeParameter` (`bytes btRequest` + handle) |
| `GetConnectionStatusAsync` | ✅ | ❌ | synthesized from an authenticated probe — no dedicated RPC on either transport (gRPC `PingServer`/`GetHistorianConsoleStatus` could synthesize it) |
| `ReadBlocksAsync` | ❌ | ❌ | `StartBlockRetrievalQuery` never captured on either transport — throws `ProtocolEvidenceMissingException` |

In short: **WCF is the broad, mature surface** (every config write, events, SQL,
and all reads), while **gRPC is the narrower *tooled* surface** — but the 2023 R2
gRPC *contract* is actually a **superset** of WCF. Every 🔌 row above has a
recovered RPC carrying the **same opaque `bytes` buffers the existing WCF
serializers already emit**, keyed by the same `strHandle`/`uiHandle` session
handle the gRPC read path already obtains. So these are **capture-and-wire** items
(route the existing serializer into a gRPC orchestrator + golden-capture the
framing), **not** protocol-discovery items. We have only *buffer-verified* two
gRPC families live — the read chain and `AddStreamValues` — so per the
"capture first, never guess wire bytes" rule the 🔌 rows stay untooled until each
is captured. The natural production pattern today remains WCF for config/reads and
`RemoteGrpc` reserved for `AddHistoricalValuesAsync`.

> A 2023 R2 server reports History interface version 12 (vs. 11 on 2020). The
> connect-time version gate accepts both — they are byte-compatible — so gRPC
> works against a v12 server with the default `VerifyServerInterfaceVersion=true`;
> no opt-out is required.

## Resilience subsystems (M4)

Two **pure client-side** subsystems layered on top of `HistorianClient`. They use
only the public SDK surface — no extra reverse-engineering, no server-side
protocol — and are fully unit-tested without a live server.

### Store-and-forward (`AVEVA.Historian.Client.StoreForward`)

A durable local outbox that buffers writes when the Historian is unreachable and
replays them on reconnect. `HistorianStoreForwardWriter` wraps a `HistorianClient`
(or any `IHistorianWriteSink`) and persists each enqueued write to an
`IHistorianOutboxStore` — `FileHistorianOutboxStore` (crash-durable, atomic
JSON-per-entry, FIFO by filename sequence, corrupt-file quarantine) or
`InMemoryHistorianOutboxStore`. A background drain loop retries with FIFO
head-of-line ordering, optional `MaxDeliveryAttempts` dead-lettering, and a
`DropOldest`/`Reject` overflow policy.

```csharp
using AVEVA.Historian.Client.StoreForward;

await using HistorianStoreForwardWriter writer = new(
    client,
    new FileHistorianOutboxStore(@"C:\ProgramData\histsdk\outbox"));
await writer.StartAsync();                       // background drain on reconnect

await writer.EnqueueHistoricalValuesAsync("MyTag",
    [new HistorianHistoricalValue(DateTime.UtcNow, 42.0)]);   // returns immediately, durable

HistorianStoreForwardStatusSnapshot status = await writer.GetStatusAsync();
// status.PendingCount / .Storing / .ErrorOccurred — mirrors the server SF semantics
```

This is a pragmatic outbox, **not** the bit-faithful native SF cache
(`Forward*Snapshot` decode), which stays deferred.

### Multi-historian redundancy (`AVEVA.Historian.Client.Redundancy`)

`HistorianRedundantClient` fronts N members as one logical client. Reads fail over
to the next healthy member in priority order (streaming reads fail over only
*before the first row* — mid-stream failures propagate to avoid duplicated/skipped
rows); writes fan out (`AllMembers`/`PreferredOnly`) under an `All`/`Any`
acknowledgement policy and return a per-member `HistorianRedundantWriteResult`. A
failure-threshold demotion + background watchdog restores recovered members.

```csharp
using AVEVA.Historian.Client.Redundancy;

await using HistorianRedundantClient cluster = new(
[
    new HistorianClientMember("primary",   primaryClient),
    new HistorianClientMember("secondary", secondaryClient),
]);
await cluster.StartAsync();                       // health watchdog

await foreach (HistorianSample s in cluster.ReadRawAsync("MyTag", startUtc, endUtc, 100))
{
    // served by the first healthy member; transparently fails over on connect
}

HistorianRedundantWriteResult w = await cluster.AddHistoricalValuesAsync("MyTag",
    [new HistorianHistoricalValue(DateTime.UtcNow, 42.0)]);   // fanned out across members
```

Backing a member's writes with a `HistorianStoreForwardWriter` gives the pragmatic
equivalent of native ReSyncTags: a down member buffers locally and replays on
recovery.

## Build & test

```powershell
dotnet build  .\Histsdk.slnx --no-restore
dotnet test   .\Histsdk.slnx --no-build --logger "console;verbosity=minimal"
```

Run a single test class:

```powershell
dotnet test .\Histsdk.slnx --no-build --filter "FullyQualifiedName~WcfDataQueryProtocolTests"
```

Live integration tests (`tests/AVEVA.Historian.Client.Tests/HistorianClientIntegrationTests.cs`)
are gated and skip cleanly without these env vars:

```powershell
$env:HISTORIAN_HOST       = 'localhost'
$env:HISTORIAN_TEST_TAG   = 'SysTimeSec'      # any tag the server has data for
# Optional for non-integrated auth:
$env:HISTORIAN_USER, $env:HISTORIAN_PASSWORD
$env:HISTORIAN_TAG_FILTER = 'Sys*'             # or any LIKE-pattern
```

The 2023 R2 `RemoteGrpc` transport has its own gated live suite
(`HistorianGrpcIntegrationTests`) covering the full tooled gRPC surface — probe,
raw / aggregate (incl. multiple retrieval modes) / at-time reads, browse,
metadata, system-parameter, server time-zone, and measured store-forward status.
It skips cleanly unless `HISTORIAN_GRPC_HOST` is set:

```powershell
$env:HISTORIAN_GRPC_HOST  = 'my-2023r2-host'  # gates the gRPC suite
$env:HISTORIAN_TEST_TAG   = 'SysTimeSec'
$env:HISTORIAN_USER, $env:HISTORIAN_PASSWORD  # required for the authenticated ops
# Optional:
$env:HISTORIAN_GRPC_PORT     = '32565'        # default 32565
$env:HISTORIAN_GRPC_TLS      = 'true'         # gRPC over TLS
$env:HISTORIAN_GRPC_DNSID    = 'my-2023r2-host'  # cert DNS name when connecting by IP
$env:HISTORIAN_GRPC_TIMEOUT  = '120'          # per-call deadline (s); raise for slow links
$env:HISTORIAN_WRITE_SANDBOX_TAG = 'MyFloatTag'  # gates the AddHistoricalValues write test
```

The aggregate tests self-calibrate their query window from a real raw sample, so
they pass against an idle 2023 R2 server (no recent collection) as well as a
live-collecting one.

## Repository layout

```
src/AVEVA.Historian.Client/    Production SDK — pure managed .NET 10. No native AVEVA dependency.
tests/                          Unit tests (golden-byte / round-trip) + gated live integration tests.
tools/                          Reverse-engineering tooling (NOT shipped):
  AVEVA.Historian.ReverseEngineering/  .NET 10 CLI: WCF probes, dnlib IL inspection,
                                       IL-rewrite instrumentation (instrument-wcf-{write,read}message,
                                       instrument-openconnection*, instrument-startdataquery, etc.).
  AVEVA.Historian.NativeTraceHarness/  .NET Framework harness that loads aahClientManaged.dll for
                                       byte-for-byte parity testing against the native wrapper.
  AVEVA.Historian.NetFxWcfProbe/       .NET Framework WCF probe for ruling out .NET 10-only differences.
  AVEVA.Historian.ReverseInstrumentation/  Helper assembly injected into IL-rewritten wrapper copies.
  AVEVA.Historian.WcfCaptureServer/    Fake server for endpoint experiments.
scripts/                        PowerShell + Frida runners + Python decoders for capture analysis.
fixtures/protocol/              Sanitized golden-byte fixtures.
docs/reverse-engineering/       Sanitized handoff evidence; commit-safe summaries only.
```

Native AVEVA binaries (`current/`, `aveva-install-x64/`, `aveva-install-x86/`) are
**gitignored**. Each developer fetches their own copy from the AVEVA installer; we
neither modify nor redistribute them.

## Documentation

Read in order before non-trivial work:

| Doc | Purpose |
|---|---|
| [`AGENTS.md`](AGENTS.md) | Standing project constraints and safety rules. |
| [`CLAUDE.md`](CLAUDE.md) | Build commands, code architecture, the active protocol blocker (now resolved) and SDK surface. |
| [`docs/reverse-engineering/handoff.md`](docs/reverse-engineering/handoff.md) | Decision record + decoded protocol evidence (binding shapes, descriptor layouts, Open2 v6 response, event-row wire format, property-bag value encoding). |
| [`instructions.md`](instructions.md) | Original plan + decision record. |

## Architecture

Three intentionally decoupled subsystems under `src/AVEVA.Historian.Client/`:

- **`HistorianClient` + `HistorianClientOptions`** — public façade. Validates inputs;
  delegates reads to `Historian2020ProtocolDialect`; delegates probe / tag metadata /
  browse / status helpers to the WCF layer.
- **`Wcf/`** — managed WCF/MDAS layer. Custom `MdasMessageEncoder` wraps SOAP 1.2 +
  WS-Addressing 1.0 in AVEVA's `application/x-mdas` framing. Three binding flavors
  via `HistorianWcfBindingFactory`: plain MDAS over Net.NamedPipe (local), MDAS +
  Windows transport (remote `/Hist-Integrated`), MDAS + certificate (remote
  `/HistCert`). Service contracts in `Wcf/Contracts/` mirror the server-side WCF
  surface (versioned per native interface — `IHistoryServiceContract`,
  `IRetrievalServiceContract2..4`, `IStatusServiceContract2`, etc.).
- **`Protocol/`** — binary frame layer. `Historian2020ProtocolDialect` is the
  version-anchored bridge between the public façade and the WCF + frame layers.
  Methods without protocol evidence throw `ProtocolEvidenceMissingException` rather
  than guessing wire bytes.
- **`Models/`** — public DTOs and enums. `HistorianSample`, `HistorianAggregateSample`,
  `HistorianEvent`, `HistorianTagMetadata`, `HistorianDataType`, `RetrievalMode`,
  `HistorianTransport`, etc.

Read flow end-to-end (live-verified against `localhost`):

```
Hist.GetV → Hist.ValCl × N → Hist.Open2 → Retr.GetV → Retr.IsOriginalAllowed →
Retr.StartQuery2 → loop Retr.GetNextQueryResultBuffer2 → typed HistorianSample rows
```

Event flow end-to-end (live-verified):

```
Hist.GetV → Hist.ValCl × N → Hist.Open2 → Hist.UpdC3 → 6× Stat.GetSystemParameter →
Hist.RTag2 → Stat.GetSystemParameter(AllowRenameTags) → Trx.GetV → Stat.GetV →
Retr.GetV → Hist.EnsT2(CmEventTagId) → Retr.StartEventQuery →
loop Retr.GetNextEventQueryResultBuffer → typed HistorianEvent rows with
property dictionary → Retr.EndEventQuery → Hist.Close2
```

## Safety

- Production code under `src/` must remain pure managed .NET 10 with no native AVEVA
  reference. Reverse-engineering harnesses under `tools/` may reference native binaries.
- Never commit credentials, hostnames, user names, customer tag names, or raw packet
  captures. `*.ndjson`, `current/`, `aveva-install-*/`, and `artifacts/` are gitignored
  precisely because they accumulate identity-bearing runtime data.
- Methods without protocol evidence throw `ProtocolEvidenceMissingException`. Do not
  stub fake behavior — leave them throwing until evidence supports an implementation.

## Status

313 unit + live integration tests pass (`dotnet test --logger "console;verbosity=minimal"`).
Full SDK surface — reads, browse, metadata, status, plus the two write ops
(`EnsureTagAsync` / `DeleteTagAsync`) — verified end-to-end against both a
local Historian (`LocalPipe`) and a remote Historian (`RemoteTcpIntegrated`
over Net.TCP with Windows transport auth). `RemoteTcpCertificate` ProbeAsync
is live-verified; deeper coverage over the cert transport plus the
explicit-credentials path await additional verification.

The 2023 R2 `RemoteGrpc` transport's full tooled surface is live-verified against
a real 2023 R2 server: probe, raw / aggregate (TimeWeightedAverage +
Minimum/MaximumWithTime + BestFit) / at-time reads, browse, metadata,
system-parameter, server time-zone, and measured store-forward status — plus the
gRPC-only `AddHistoricalValuesAsync` backfill write. The remaining gRPC config ops
are exposed by the server but untooled (see the transport matrix above).