histsdk/docs/plans/grpc-transport.md

# AVEVA Historian SDK 2023 R2 — gRPC Transport Analysis

**Scope:** Documents the new gRPC transport that 2023 R2 adds to the Historian
Client Access Layer (HCAL). Kept deliberately **separate** from the main
`histsdk` reverse-engineering docs — this is 2023 R2 evidence, not the 2020/WCF
protocol the production SDK currently targets.

**Source:** the 2023 R2 `HistorianSDK` installer (**not installed**).
`SDKSetup.msi` was laid out with `msiexec /a` (administrative extract, no
registration) into a local `msi-extract` staging dir, then the managed
assemblies were decompiled with `ilspycmd`.

**Assembly versions analysed:** `2023.1219.4004.5`
(`Archestra.Grpc.Contract.dll`, `Archestra.Historian.GrpcClient.dll`,
`aahClientManaged.dll`).

---

## 1. Headline finding

The 2023 R2 gRPC transport is a **transport swap, not a protocol redesign.**
Every gRPC request/response wraps the **same opaque native binary buffers** that
the 2020/WCF-MDAS path already carries — `OpenConnection3` v6 buffer, NTLM/SSPI
`ValCl` tokens, `DataQueryRequest`, `GetNextQueryResultBuffer` row buffers, the
`Status` err blob, etc. — inside protobuf `bytes` fields.

Concrete proof, from `Archestra.Historian.GrpcClient`:

```csharp
// History/OpenConnection — same byte[] openParameters the WCF path built
OpenConnectionRequest request = new OpenConnectionRequest {
    BtConnectionRequest = ByteString.CopyFrom(openParameters)
};
// Retrieval/StartQuery — same queryType + DataQueryRequest bytes + handle
StartQueryRequest request = new StartQueryRequest {
    UiHandle = handle,
    UiQueryRequestType = queryRequestType,
    BtRequestBuffer = ByteString.CopyFrom(requestBuffer)
};
```

**Implication for the `histsdk` project:** all of the hard-won payload
serializers (`HistorianOpen2Protocol`, `HistorianDataQueryProtocol`,
`HistorianEventRowProtocol`, the SSPI `ValCl` token framing, the EnsT2
`CTagMetadata` layout) transfer **unchanged**. Only the envelope around them
changes: protobuf-over-gRPC instead of binary-SOAP-over-`application/x-mdas`.
The WCF `[MessageParameter(Name=…)]` guessing that dominated the 2020 work is
gone — field names and numbers are explicit in the protobuf contract.

---

## 2. Transport stack

From `GrpcClientBase.InitializeBase(target, portNumber, securedConnection, certificateName, trusted)`:

| Aspect | Value / behaviour |
|---|---|
| Library | `Grpc.Net.Client` + **`Grpc.Net.Client.Web`** (`GrpcWebHandler`) |
| Mode | **gRPC-Web**, `GrpcWebMode.GrpcWeb` (binary `application/grpc-web`, **not** `-text`) |
| HTTP version | **HTTP/1.1** (`GrpcWebHandler.HttpVersion = new Version(1,1)`) — *not* HTTP/2 |
| Address | `http://{target}:{port}` insecure, or `https://{certificateName}:{port}` secure |
| Inner handler | `HttpClientHandler` with custom `ServerCertificateCustomValidationCallback` |
| Compression | gzip on by default; request header `grpc-internal-encoding-request: gzip`; custom `CustomCompressionProvider` / `CustomGZipStream` used for bandwidth accounting |
| Default timeout | 60 s per call (`m_timeoutInSeconds = 60`, sent as gRPC deadline) |
| Interceptor | `ClientInterceptor` (logging hook, currently a no-op `LogCall`) |

Because it is **gRPC-Web over HTTP/1.1**, the transport is proxy/firewall
friendly and does not require HTTP/2 negotiation — note `HistorianConnectionArgs.ProxyServer`
(e.g. `http://host:9480`) in the public API.

### Port

- **Default port `32565`** — `HistorianConnectionArgs.TcpPort`, *"the TCP port
  of the Historian Client Access Point."* (Note this differs from the 2020 WCF
  port `32568` the production SDK uses.)
- All services reach the **same host:port**; gRPC multiplexes by service path
  (`/HistoryService/OpenConnection`, `/RetrievalService/StartQuery`, …).

### Channel topology

Five service stubs grouped into four wrapper clients, each constructing its own
`GrpcChannel` to the same endpoint:

| Wrapper (`GrpcClientBase` subclass) | gRPC service stub(s) |
|---|---|
| `GrpcHistoryClient` | `HistoryService` + `TransactionService` (one channel) |
| `GrpcRetrievalClient` | `RetrievalService` |
| `GrpcStatusClient` | `StatusService` |
| `GrpcStorageClient` | `StorageService` |

---

## 3. Authentication model (unchanged in substance)

Auth is **still the native session handshake**, carried over gRPC instead of
WCF. There is **no per-call bearer/auth token in gRPC metadata** — the only
metadata sent is the gzip-encoding hint. Methods pass `m_metadata` (gzip) or
`null`; neither carries credentials. The server keys the session off the
`handle` GUID established by the handshake, exactly as the 2020 path does.

Handshake operations (same byte payloads as 2020):

- `HistoryService.GetInterfaceVersion` → version probe.
- `StorageService.ValidateClientCredential { string Handle; bytes InBuff }`
  → `{ Status; bytes OutBuff }`. **`InBuff`/`OutBuff` carry the NTLM/SSPI
  tokens** — same multi-round Negotiate exchange, same field names the 2020
  `ildasm` revealed (`inBuff`/`outBuff`), now first-class protobuf fields.
- `HistoryService.ExchangeKey { string StrHandle; bytes BtInput }`
  → `{ Status; bytes BtOutput }` (key-exchange / cert path).
- `HistoryService.OpenConnection { bytes BtConnectionRequest }`
  → `{ Status; bytes BtConnectionResponse }` — same `OpenConnection3` v6
  request buffer in, same 42-byte session blob out.

Public-API security knobs (`aahClientManaged.xml`):

- `HistorianConnectionArgs.ConnectionMode` — *"whether GRPC connection to the
  Historian Server. **Default is true** (GRPC)."* This is the master switch
  selecting gRPC vs legacy.
- `HistorianSecurityMode`: `None`, `Disabled`, `TransportWindows`
  (Windows creds), `TransportCertificate` (server cert).
- `AllowUnTrustedConnection` → maps to the `trusted` arg; when false the client
  bypasses X509 chain validation (`ValidateServerCertificate` returns true
  early). Equivalent to the production SDK's `AllowUntrustedServerCertificate`.
- `AuthenticationMode` default `HistorianNative`; `CertificateInfo.CertificateName`
  supplies the `https://{certificateName}:{port}` SNI/host identity.

---

## 4. gRPC service surface (full RPC list)

All methods are unary (`MethodType 0`). Names map 1:1 onto the 2020 WCF
operations the production SDK already understands.

### HistoryService (`/HistoryService/…`)
`GetInterfaceVersion`, `ExchangeKey`, `OpenConnection`, `CloseConnection`,
`UpdateClientStatus`, `RegisterTags`, `EnsureTags`, `AddStreamValues`,
`AddTagExtendedPropertyGroups`, `AddTagExtendedProperties`, `StartJob`,
`GetJobStatus`, `DeleteTagExtendedProperties`, `DeleteTags`,
`AddTagLocalizedProperties`, `DeleteTagLocalizedProperties`

### RetrievalService (`/RetrievalService/…`)
`GetRetrievalInterfaceVersion`, `StartQuery`, `GetNextQueryResultBuffer`,
`EndQuery`, `GetShardTagidsByTagnameAndSource`, `GetTagInfosFromName`,
`GetTagExtendedPropertiesFromName`, `ExecuteSqlCommand`, `StartEventQuery`,
`GetNextEventQueryResultBuffer`, `EndEventQuery`, `StartTagQuery`, `QueryTag`,
`EndTagQuery`, `GetTagLocalizedPropertiesFromName`

### StatusService (`/StatusService/…`)
`GetStatusInterfaceVersion`, `GetSystemParameter`, `SendInfo`, `RequestInfo`,
`DeleteInfo`, `GetHistorianInfo`, `StartProcess`, `StopProcess`, `PingServer`,
`PingPipe`, `ConfigureAutoStartProcess`, `GetHistorianConsoleStatus`,
`GetRuntimeParameter`, `GetSystemTimeZoneName`, `SetHistorianConsoleStatus`,
`CanUpdateAreaHierarchy`, `UpdateAreaHierarchy`, `UpdateObjectHierarchy`

### StorageService (`/StorageService/…`)
`GetInterfaceVersion`, `OpenStorageConnection`, `OpenStorageConnection2`,
`CloseStorageConnection`, `Ping`, `AddTags`, `RegisterTags`, `AddStreamValues`,
`AddStreamValues2`, `GetTagIds`, `GetTags`, `FlushMetadata`, `FlushData`,
`LoadBlocks`, `GetSnapshots`, `StartQuerySnapshot`, `NextQuerySnapshot`,
`EndSnapshot`, `Stop`, `ClearTagidPairs`, `AddTagidPairs`, `GetSFParameter`,
`SetSFParameter`, `SendSnapshotBegin`, `SendSnapshotEnd`, `SendSnapshot`,
`DeleteSnapshot`, `ClearShardTagids`, `AddShardTagids`, `SplitUnknownShards`,
`GetRemainingSnapshotsSize`, `DeleteTags`, `OpenStorageConnection2`,
`ValidateClientCredential`, `GetInfo`

### TransactionService (`/TransactionService/…`)
`ForwardSnapshot`, `ForwardSnapshotBegin`, `ForwardSnapshotEnd`,
`GetTransactionInterfaceVersion`, `AddNonStreamValuesBegin`,
`AddNonStreamValues`, `AddNonStreamValuesEnd`

> A separate `ArchestrA.CloudHistorian.Contract` assembly defines a parallel
> cloud-ingest contract (`AddHistorianValues`, `CreateTags`, `EnqueueTagDataPacket`,
> `Enqueue…`, etc.) used by `aahCloudConfigurator` / `online.wonderware.com`.
> Out of scope here; noted for completeness.

---

## 5. Representative message shapes (protobuf field numbers)

The universal result wrapper and the auth/query messages — note how thin they
are; the real structure lives inside the `bytes` fields.

```proto
// Common result wrapper (ArchestrA.Grpc.Contract.RequestStatus)
message Status {
  bool  bSuccess = 1;   // success flag (replaces WCF return-bool)
  bytes btError  = 2;   // native error buffer (same type/code blob as WCF err)
}

// HistoryService
message OpenConnectionRequest  { bytes btConnectionRequest  = 1; } // OpenConnection3 v6 buffer
message OpenConnectionResponse { Status status = 1; bytes btConnectionResponse = 2; } // 42-byte session blob
message ExchangeKeyRequest     { string strHandle = 1; bytes btInput = 2; }

// StorageService — Negotiate/NTLM handshake
message ValidateClientCredentialRequest  { string handle = 1; bytes inBuff  = 2; }
message ValidateClientCredentialResponse { Status status = 1; bytes outBuff = 2; }

// RetrievalService
message StartQueryRequest {
  uint32 uiHandle           = 1;
  uint32 uiQueryRequestType = 2;   // RetrievalMode → QueryType, same mapping as 2020
  bytes  btRequestBuffer    = 3;   // DataQueryRequest bytes, byte-identical to WCF
}
```

Across the contract the recurring pattern is `{ Status status; bytes <payload> }`
for responses and `{ [string handle][uint …] bytes <payload> }` for requests.

**The full canonical IDL has been recovered.** All six `.proto` files were
rendered from the embedded `FileDescriptor`s and protoc-validated — see
`../out/proto/*.proto`, the portable `../out/archestra_grpc.fileset.pb`
(`FileDescriptorSet`), and the per-message field dump `../out/grpc-contract-dump.md`.
`../out/README.md` explains the contract quirks (global proto package, cross-file
name collisions, all-unary RPCs) and the 2020→gRPC read-path mapping. Regenerate
with the `protodump/` tool.

---

## 6. What this means for histsdk (if a gRPC transport is ever added)

This is **not** a request to implement anything — recording the path:

1. Add a transport enum value (e.g. `RemoteGrpc`) alongside `LocalPipe` /
   `RemoteTcpIntegrated` / `RemoteTcpCertificate`.
2. Reference `Grpc.Net.Client` + `Grpc.Net.Client.Web`; build a
   `GrpcChannel.ForAddress("http(s)://host:32565", { HttpHandler =
   GrpcWebHandler(GrpcWeb, HttpClientHandler), … })` with HTTP/1.1.
3. Reuse **every existing payload serializer unchanged** — feed the same byte
   buffers into the protobuf `bytes` fields instead of MDAS bodies. The
   orchestrator call order (`GetV → ValCl×N → Open2 → Retr.GetV →
   IsOriginalAllowed → StartQuery → GetNextQueryResultBuffer…`) is identical.
4. Auth: still the SSPI/Negotiate token loop via `ValidateClientCredential`,
   carried in `inBuff`/`outBuff`. No per-call gRPC auth metadata needed.
5. Biggest win: **no WCF `[MessageParameter]` reverse-engineering** — the
   protobuf field numbers are authoritative and stable.

Caveat: this is the **2023 R2** server contract. The production SDK targets a
2020-era server; whether that server exposes the gRPC HCAP endpoint at all is a
server-version question, not a client one. Treat this as forward-looking.

---

## 7. Artifacts (all under the separate analysis folder, none committed to histsdk)

```
histsdk-2023r2-analysis/
  msi-extract/                         # msiexec /a layout of SDKSetup.msi
  bin/                                 # copied key assemblies + aahClientManaged.xml
  decompiled/
    Archestra.Grpc.Contract/           # full protobuf contract (services + messages)
    Archestra.Historian.GrpcClient/    # transport wrappers (channel/auth/calls)
    ArchestrA.CloudHistorian.Contract/ # cloud-ingest contract (out of scope)
  protodump/                           # .NET 10 tool: descriptor graph -> .proto / dump
  out/
    proto/*.proto                      # recovered, protoc-validated IDL (6 files)
    archestra_grpc.fileset.pb          # portable FileDescriptorSet (grpcurl/buf/protoc)
    grpc-contract-dump.md              # per-message field dump + service tables
    README.md                          # artifact guide + contract quirks + read-path map
  docs/grpc-transport.md               # this file
```

gRPC redist proof in the installer:
`Redist/HistorianSDK 2023 R2/x64/{GRPCCore,GRPCNetClient,HistorianGRPCClient,HistorianGRPCContract,Protobuf}.msm`
plus shipped `Grpc.Net.Client*.dll`, `Grpc.Core.Api.dll`, `Google.Protobuf.dll`.