histsdk/docs/plans/store-forward-cache-reverse-engineering.md

# Store/Forward Cache Reverse-Engineering Plan

Last updated: 2026-06-21

> **2026-06-21 R4.3 re-scope — read this first.** The original plan below
> (2026-05-04) was written against the 2020 Net.TCP/WCF transport, before the
> 2023 R2 gRPC transport existed. Its single biggest open risk — *"is SF state
> readable via a one-shot pull, or only via a duplex push contract we'd have to
> add?"* (Q1/Q2 + §3 Step 3 + Risk 4) — is now **answered: pull, no duplex**.
> The recovered gRPC `StorageService` contract exposes SF state as plain
> request/response RPCs. The current R4.3 scope and recommended path are in
> §9 ("2026-06-21 gRPC re-scope"); the 2020-WCF body below is retained as
> background, not the recommended route.

Original last-updated: 2026-05-04

This document plans the reverse-engineering effort needed to replace the
synthesized `GetStoreForwardStatusAsync` in
`src/AVEVA.Historian.Client/Wcf/HistorianWcfStatusClient.cs` (lines 101-117)
with a real, evidence-backed implementation. It is a *plan*, not the work
itself. No code changes; no captures collected.

Read this together with:

- `docs/reverse-engineering/handoff.md` — read/event protocol decoding state
- `src/AVEVA.Historian.Client/Wcf/Contracts/IStorageServiceContract.cs` — the
  WCF contract that already declares the SF parameter ops
- `src/AVEVA.Historian.Client/Models/HistorianStoreForwardStatus.cs` — the
  output model the implementation must populate

## 1. Goal

"SF support works" means, end-to-end:

1. **Primary deliverable.** `client.GetStoreForwardStatusAsync()` against a
   live local Historian returns a `HistorianStoreForwardStatus` whose
   `Pending`, `Storing`, `DataStored`, `ErrorOccurred`, `Error`, `ServerName`,
   and `ConnectionKind` fields reflect actual server-reported state, not the
   synthesized defaults at
   `HistorianWcfStatusClient.cs:107-117`.
2. **Secondary deliverable.** The SDK can also answer the higher-level
   "is SF currently buffering?" question accurately when the runtime DB is
   *down*, not just when it is up. That is the case the real native client
   handles correctly and where the synthesized default (`Storing = false`,
   `ErrorOccurred = false`) is silently wrong today.
3. **Non-goals.** Writing into SF, replaying SF buffers, configuring SF
   parameters, redundant-partner SF aggregation
   (`HistorianStoreForwardStatus.AddPartnerStoreForwardStatus`,
   token `0x060060B8`). Read-only matches the project mission in
   `CLAUDE.md`.

The success bar is parity with the native wrapper's
`ArchestrA.HistorianAccess.GetStoreForwardStatus`
(MD token `0x06006186` in `current/aahClientManaged.dll`),
not a superset.

## 2. Architecture Investigation (open questions, in priority order)

Answer these before writing any production code. Each has a discovery action
in §3.

### Q1. Is SF status read from a local in-process struct, a separate WCF endpoint, or a Named Pipe IPC?

Current evidence: **all three are plausible, but the wrapper actually uses
"in-process struct kept current by server-pushed WCF events"**. Specifically:

- `ArchestrA.HistorianAccess.GetStoreForwardStatus`
  (token `0x06006187`, the private 2-arg overload) does *not* call WCF.
  It calls `mdas_GetStorageStatus` (a `calli` against the
  `INSQL_MDAS_ERROR (IntPtr handle, uint, HISTORIAN_STORAGE_STATUS*)` C
  signature in `current/aahClient.dll` exports) and then maps the result
  through `HistorianAccessUtil.ConvertUnmanagedSFStorageStatusToManagedStorageStatus`
  (token `0x060060E4`).
- Mutators like `CConfigStatusClient.SetMdasStoreForwardEvent`
  (token `0x060029DC`) and `aahClientCommon.CStatus.SetStoreForwardEvent`
  (token `0x06002A04`) are wired to the WCF callback
  `IStatusServiceContract2.SetStoreForwardEvent`
  (`StatusServiceContract.IStatusServiceContract2.SetStoreForwardEvent`,
  token `0x06005F57`). The server *pushes* SF state changes; the client
  caches them.
- Confirm: read the IL of token `0x06006187` and verify the only system call
  is `mdas_GetStorageStatus`. The first 200 instructions confirm this:
  `GetClient(ConnectionIndex)` → `calli` against the
  `INSQL_MDAS_ERROR(IntPtr,uint,HISTORIAN_STORAGE_STATUS*)` signature →
  `ConvertUnmanagedSFStorageStatusToManagedStorageStatus`.

Implication: **the SDK cannot ship a synchronous probe that calls one WCF
operation and gets the answer**. It must subscribe to the same status-event
stream the native wrapper subscribes to, or call a status query that returns
the cached snapshot from the server.

### Q2. Is there a single-shot WCF query that returns the same snapshot?

Likely yes. Hypothesis: `IStatusServiceContract2.GetHistorianInfo`
(`GETHI`, see `IStatusServiceContract2.cs:24-30`) returns a multi-key status
blob whose schema includes SF state. Alternative: a status-only key passed to
`GetSystemParameter` (already plumbed via `HistorianWcfStatusClient.GetSystemParameterAsync`).
Both are testable without writing protocol code by sending probe payloads
and observing the response shape.

### Q3. Does SF have its own sidecar process / pipe / WCF endpoint we are missing?

Strong evidence the answer is yes when SF is *enabled*:

- `aahClientCommon.CSFConnection.GetSFPipeName` (token `0x06004B72`),
  `GetSFPath` (`0x06004B71`), `IsConnected` (`0x06004B73`), `IsEnabled`
  (`0x06004B6F`) — there is a separately-named SF Named Pipe distinct
  from the main MDAS pipe.
- `aahClientCommon.CSFConnection.StartStoreforward` (token `0x06004BC6`).
- `IStorageServiceContract` already declares `GetStoreForwardParameter`
  / `SetStoreForwardParameter` (`GetSFP`/`SetSFP`,
  see `IStorageServiceContract.cs:81-85`) and `Storage` is a separate
  WCF service slot in `HistorianWcfServiceNames.cs:15`.
- `CWcfConfig.ConfigurePipeProxy<IStorageServiceContract>` (token
  `0x06004B1C`) and `CWcfConfig.ConfigureTcpProxy<IStorageServiceContract>`
  (token `0x06004B1B`) confirm the storage proxy supports both transports —
  same dual-transport pattern the History/Retrieval proxies use.
- `CStorageEngineConsoleClient.GetPipeNameStr` (token `0x06000E2D`) /
  `GetFullPipeNameStr` (token `0x06000E2E`) wraps the storage-engine
  console pipe via `STransactPipeClient2` (a *non-WCF* binary pipe
  protocol).

Open: **is the SF sidecar even running on the dev host this SDK is being
tested against?** `handoff.md` does not record an SF process being
observed. `aveva-install-x64/` and `aveva-install-x86/` ship only DLLs
(no `aahStoreForwardClient.exe` / `aahSFClient.exe` / similar). The SF
sidecar is part of the Historian *server* install, not the client
redistributable. So:

- On the developer machine, SF is reachable only because the local
  Historian server is installed.
- A pure-client install (the deployment target this SDK ships into) may
  *never* have SF.

This shapes the success criteria: when SF is not configured, a correct
implementation returns `Pending = false`, `ErrorOccurred = false`,
`DataStored = false`, `Storing = false` — i.e. the same shape the
synthesized defaults produce today. The interesting case is *when SF is
configured and active*.

### Q4. Is SF state authoritative on the Historian server or on a per-client basis?

Native wrapper reads it from `HistorianClient*` (the per-connection C++
object). This means it is *connection-scoped* server-pushed state. We
do not need to enumerate cluster-wide SF state — the server reports
"my SF buffer for this client's writes" only. This matches our read-only
mission: we are not a writer, so the only SF state of interest is the
server-side cache for *other* writers, which the server can report to
us as a passive observer.

### Q5. Does any SF probe require Admin?

`CSFConnection.GetSFPipeName` returns a kernel object name. Reading
from it requires the pipe ACL to permit the caller. If the SF pipe is
ACL'd to `LocalSystem` only, the SDK cannot read it without
impersonation — and the SDK runs as the calling process. This is a
hard limit, not a bug.

## 3. Discovery Workstreams

Run these in parallel. None require a live server beyond what the
existing test rig already has.

### Workstream A — Static IL inspection (parallel-safe, read-only)

Owner action items, in order:

1. Dump full IL of token `0x06006187`
   (`HistorianAccess.GetStoreForwardStatus(ConnectionIndex,out)`):
   ```powershell
   dotnet run --no-build --project tools\AVEVA.Historian.ReverseEngineering -- `
     dnlib-method current\aahClientManaged.dll HistorianAccess.GetStoreForwardStatus --instructions
   ```
   Save under `docs/reverse-engineering/historianaccess-getstoreforwardstatus-il-latest.txt`.
   Confirm the `calli` target signature
   `INSQL_MDAS_ERROR(IntPtr,uint,HISTORIAN_STORAGE_STATUS*)` and that
   the only WCF entry-points it touches are zero.
2. Dump IL of `HistorianAccessUtil.ConvertUnmanagedSFStorageStatusToManagedStorageStatus`
   (token `0x060060E4`). This is the unmanaged→managed mapping; it
   tells us which fields of `HISTORIAN_STORAGE_STATUS` populate which
   fields of `HistorianStoreForwardStatus`. We will need the same
   mapping in reverse on the wire response.
3. Inventory every method that *writes* into the local SF status
   struct:
   ```
   methods current\aahClientManaged.dll SetStoreForward
   methods current\aahClientManaged.dll SetMdasStoreForward
   ```
   The known set as of writing:
   `CConfigStatusClient.SetMdasStoreForwardEvent` (`0x060029DC`),
   `aahClientCommon.CStatus.SetStoreForwardEvent` (`0x06002A04`),
   `CStatusConnectionDirect.SetStoreForwardEvent` (`0x06004DF8`),
   `CStatusConnectionWCF.SetStoreForwardEvent` (`0x06004E4E`),
   `CClientCommon.SetStoreForwardEventOnServer` (`0x06002EC0`).
   The `WCF` variant is the one whose IL maps onto
   `IStatusServiceContract2.SetStoreForwardEvent`
   (token `0x06005F57`) — read its IL and document the request/response
   shape.
4. Dump IL of `IStatusServiceContract2.SetStoreForwardEvent`
   (`0x06005F57`) parameter types. The `[OperationContract]`
   declaration in the wrapper assembly already encodes the wire shape;
   this gives us the bytes the server pushes us.

### Workstream B — Install inventory (parallel-safe)

1. Inventory `aveva-install-x64\` and `aveva-install-x86\` for any
   binary whose name contains `Store`, `Forward`, `SF`, `Cache`,
   `Spool`. As of this checkout: **none**, only DLLs. Confirm.
2. Inventory the deployed Historian server (out-of-band; not in this
   repo) for `aahStoreForwardClient.exe`,
   `aahStoreForwardServer.exe`, `aahSFCache.exe`, or any service
   registered with `Description` matching `*Forward*`. Capture the
   service name, account identity, and pipe ACLs (`accesschk -wuvc`).
3. Walk the registry: `HKLM\SOFTWARE\ArchestrA\Historian` and any
   sub-key matching `*StoreForward*`, recording paths and pipe names.
   Sanitize before committing.

### Workstream C — WCF probe (parallel-safe)

Use the existing `wcf-probe` and `wcf-status` subcommands of
`tools\AVEVA.Historian.ReverseEngineering`:

1. `wcf-probe $env:HISTORIAN_HOST 32568` — confirm `Storage/GetV` is
   reachable. (It is the third service slot in
   `HistorianWcfServiceNames`.) Document the returned interface
   version.
2. `wcf-status $env:HISTORIAN_HOST 32568 <param-name>` — sweep
   plausible SF parameter names (`SF.Status`, `StoreForward.State`,
   `SFCacheBytes`, etc.) through `GetSystemParameter` and record what
   the server accepts. Cheap, read-only, no session needed beyond the
   already-decoded auth chain.
3. Probe `GetHistorianInfo` (`GETHI`,
   `IStatusServiceContract2.cs:24`) with the byte request shape used
   by the native wrapper. The request bytes are visible if we run
   `instrument-wcf-readquery`-style instrumentation against
   `CConfigStatusClient.SetMdasStoreForwardEvent`'s upstream caller —
   see Workstream D.

### Workstream D — Native capture (sequential after A and C)

Two captures are needed:

1. **Native call to `mdas_GetStorageStatus`.** Run
   `tools\AVEVA.Historian.NativeTraceHarness` with a new scenario
   `--scenario sfstatus` (to be added) that invokes
   `HistorianAccess.GetStoreForwardStatus()` and dumps the
   `HISTORIAN_STORAGE_STATUS` C struct memory before the managed
   conversion runs. This pins the binary layout of the struct
   (offsets, field widths, endianness) without us guessing.
2. **WCF push of SF events.** Configure the local Historian to enter
   SF mode (stop the runtime DB writer; let the writer's queue
   trigger SF) and capture the WCF traffic with the existing
   `instrument-wcf-readquery` sibling — i.e. add an
   `instrument-wcf-setstoreforwardevent` subcommand that
   IL-rewrites `aahClientManaged.dll` to log the bytes the server
   sends to `IStatusServiceContract2.SetStoreForwardEvent`. Save
   the rewrite under `docs/reverse-engineering/dnlib-write-copy/`,
   never `current/`.

Workstream D is the only step that needs an actively-storing SF
sidecar. Plan: stop the Historian Runtime DB SQL service, write a
single test point via the wrapper's writer harness, and capture the
SF event push, then restart Runtime DB and capture the
"end-of-SF / data drained" push.

### Workstream E — On-disk cache (only if Workstream D fails)

If the WCF push protocol turns out to be impractical to reproduce
(e.g. requires duplex contract, callback channel, or a server-side
session-bind we cannot match from our managed client), fall back to
inspecting the on-disk SF cache directly. Steps:

1. Resolve `CSFConnection.GetSFPath` IL to find the cache directory
   convention (likely `%ProgramData%\ArchestrA\Historian\Cache\` or
   similar — to be confirmed, **never assume the path**).
2. Inventory file types: `.sfdata`, `.sfindex`, `.cache` — whatever
   the directory contains.
3. Decode the file header. The presence/size of `.sfdata` files is
   sufficient to populate `DataStored` and `Pending`; we do not
   need to decode the value payload.

This fallback is only for `DataStored` / `Pending`. `Storing` and
`Error` fundamentally require a live server-state read.

## 4. Concrete Reverse-Engineering Steps (execution order)

Mirrors the read/event decoding workflow that succeeded for raw
queries.

### Step 1 — Find native methods that touch SF

Already done; baseline evidence is recorded in §2 Q1/Q3 above. Key
tokens to reference:

- `0x06006186`, `0x06006187` — public/private
  `HistorianAccess.GetStoreForwardStatus`
- `0x060060E4` —
  `HistorianAccessUtil.ConvertUnmanagedSFStorageStatusToManagedStorageStatus`
- `0x060029DC` — `CConfigStatusClient.SetMdasStoreForwardEvent`
- `0x06002A04` — `aahClientCommon.CStatus.SetStoreForwardEvent`
- `0x06002DFF` — `aahClientCommon.CClientCommon.IsInStoreForward`
- `0x06002E18` — `aahClientCommon.CClientCommon.SetStoreForwardParams`
- `0x06002EC0` — `CClientCommon.SetStoreForwardEventOnServer`
- `0x06004BC6` — `aahClientCommon.CSFConnection.StartStoreforward`
- `0x06004B6F`..`0x06004B73` — CSFConnection getters (path, pipe,
  enabled, connected)
- `0x06004DF8`, `0x06004E4E` — direct vs WCF status connections
- `0x06005F57` — `IStatusServiceContract2.SetStoreForwardEvent` MD ref
- `0x06006193` — `HistorianAccess.IsBothConnectionRequested` (used by
  the public arity-0 GetStoreForwardStatus to decide whether to fan
  out to a redundant partner)

### Step 2 — Decode `HISTORIAN_STORAGE_STATUS` layout

Run Workstream A.2 (decode token `0x060060E4`) and Workstream D.1
(native struct memory dump). Together they pin the field layout.

The managed struct fields we already know we need to populate
(from `HistorianStoreForwardStatus.cs`):
`ServerName`, `Pending`, `ErrorOccurred`, `Error`, `DataStored`,
`Storing`, `ConnectionKind`. The native struct will have ≥7
fields plus padding. Express the mapping as a comment table in
the implementation.

### Step 3 — Decide the wire model

Two possible implementations:

1. **Push-mode (native parity).** SDK opens an authenticated WCF
   session that the server treats as a status subscriber, listens
   for `IStatusServiceContract2.SetStoreForwardEvent` callbacks,
   maintains a local cache, and `GetStoreForwardStatusAsync`
   returns from the cache. This requires WCF duplex
   (`CallbackContract`) which is not currently exercised
   anywhere in `src/AVEVA.Historian.Client/Wcf/`.
2. **Pull-mode (probe).** SDK calls `GetHistorianInfo` (`GETHI`)
   or a discovered `Storage`-service equivalent and maps the
   one-shot response. No subscription state required.

Pull-mode is strongly preferred: it matches the SDK's existing
WCF style, avoids duplex contracts, and the existing code path
in `HistorianWcfStatusClient.GetSystemParameter` is the right
shape. Only fall back to push-mode if Workstream C.3 proves the
server has no pull endpoint that returns SF state.

### Step 4 — Implement the managed contract method

Once Step 3 picks pull-mode, implement against the WCF contract
(likely a new `[OperationContract]` on `IStatusServiceContract2`
or a method on `IStorageServiceContract`). Follow the existing
parameter-naming discipline from the resolved
`ValidateClientCredential` blocker:
**use `[MessageParameter(Name = "...")]` to match exact server
element names — do not let WCF derive them from C# parameter
names.** See `handoff.md` "Active Blocker" entry for the
2026-05-04 fix.

### Step 5 — Add golden-byte fixtures

Add a request and response fixture under
`fixtures/protocol/store-forward-status/`:

- `request-get-storage-status.bin` — bytes the SDK sends.
- `response-get-storage-status-running-normal.bin` — server
  not in SF.
- `response-get-storage-status-active-sf.bin` — server actively
  storing.
- `response-get-storage-status-error.bin` — server's SF errored.

Capture sources: the same instrumented native wrapper runs that
populate Workstream D. Sanitize hostnames, GUIDs, and timestamps
before committing.

### Step 6 — Replace the synthesized stub

Replace `SynthesizeStoreForwardStatus` (lines 107-117 of
`HistorianWcfStatusClient.cs`) with a real implementation. Keep
the synthesized fallback for the case where the storage service
returns a "no SF configured" sentinel — that is *not* an error
condition, it is the normal state for client-only deployments.

Add a unit test class `WcfStoreForwardStatusProtocolTests` next
to the existing `WcfDataQueryProtocolTests` etc., with golden-byte
parse tests using the fixtures from Step 5.

Update the operation status table in `README.md:20` from
"synthesized defaults (no SF sidecar to probe)" to
"live-verified" once the integration test passes.

## 5. Risks and Gotchas

1. **SF may not be present on the test host.** The dev Historian
   probably has SF disabled by default; turning it on means
   stopping Runtime DB SQL services, which is invasive. Plan to do
   capture work on a dedicated sacrificial Historian VM, not the
   shared dev box.
2. **SF sidecar may require Admin or LocalSystem to query.** Any
   pipe-direct fallback (Workstream E) will fail under standard
   user accounts. Document the privilege requirement explicitly
   in the SDK XML doc comments on `GetStoreForwardStatusAsync`.
3. **State is volatile.** Probes that take >100 ms can race
   against the server's own SF state machine. Capture *both*
   request and response in the same instrumented run; do not
   try to correlate two captures.
4. **Push-mode would force a duplex WCF contract.** None of the
   existing decoded operations use duplex. Adding it widens the
   managed WCF surface significantly and risks .NET-WCF
   compatibility issues we have not yet hit. Pull-mode first.
5. **The wrapper's `IsBothConnectionRequested` (token `0x06006193`)
   path indicates a "primary + partner" topology.** Out of scope
   for this pass per §1, but if the server returns partner data
   in the same response we must skip-decode (not throw on)
   unknown trailing bytes.
6. **`Open2`-only sessions never receive SF events.** `handoff.md`
   "Active Blocker" notes the wrapper's full chain
   (`OpenConnection3` after the `ValCl` rounds) is the path that
   produces a session the server treats as a real client. SF
   probes must run from inside that chain — re-using
   `HistorianWcfAuthChainHelper.OpenAuthenticatedConnection`,
   the same call site already used by `GetSystemParameter` at
   `HistorianWcfStatusClient.cs:42`.
7. **`HISTORIAN_STORAGE_STATUS` field order is not contractual.**
   The struct is C++ inside the closed source. If AVEVA reorders
   fields between Historian versions, our decoder breaks. Pin the
   decoder to the Historian server version observed at session
   open (already exposed via `IRetrievalServiceContractN`) and
   reject mismatched versions explicitly with
   `ProtocolEvidenceMissingException`. Do not silently best-effort
   parse.
8. **Sanitization.** Pipe names, registry paths, and SF cache
   directory paths can leak hostnames and account names. Run the
   `rg` sanitizer (handoff.md "Next Pickup Steps") after every
   doc edit.

## 6. Success Criteria

A real implementation is "done" when all of the following hold:

1. `client.GetStoreForwardStatusAsync()` returns
   `Pending = true` and `Storing = true` while the local
   Historian's SF cache is actively buffering writes (verifiable
   by stopping the Runtime DB and writing a value).
2. Returns `Pending = false` and `Storing = false` within
   ≤ 5 seconds after the Runtime DB recovers and SF drains.
3. Returns `ErrorOccurred = true` and a non-null, actionable
   `Error` message when the SF cache itself fails (disk full,
   pipe closed, etc.).
4. Returns the synthesized "no SF" shape (all-false) without
   throwing on a Historian where SF is not configured.
5. Two new golden-byte unit tests pass (active-SF and idle-SF
   responses).
6. `ProtocolGuardrailTests` no longer needs to exempt
   `GetStoreForwardStatusAsync` from any "must throw
   `ProtocolEvidenceMissingException`" rule — the method is now
   evidence-backed.
7. Live integration test
   `HistorianClientIntegrationTests.GetStoreForwardStatusAsync_ReturnsServerState`
   (to be added) passes when `HISTORIAN_HOST` is set, skips
   cleanly otherwise.
8. `README.md:20` operation status table is updated from
   "synthesized defaults" to "live-verified".

## 7. Open Questions for the Implementer

Resolve these before writing production code:

1. Does the server expose a *pull* endpoint that returns the full
   `HISTORIAN_STORAGE_STATUS` snapshot, or only push events?
   (Workstream C.3 answers this.)
2. What is the binary layout of `HISTORIAN_STORAGE_STATUS`?
   (Workstream A.2 + D.1.)
3. What is the `[OperationContract]` shape on
   `IStatusServiceContract2.SetStoreForwardEvent`? Specifically:
   parameter count, byte-buffer parameters, and exact
   `MessageParameter` names? (Workstream A.4.)
4. Is the `Storage` service slot at
   `net.pipe://<host>/Storage` and `net.tcp://<host>:32568/Storage`
   reachable on a non-Historian-server install? Or does it 404
   when only the client redistributable is present? (Workstream
   B + C.1.)
5. Does the SF status snapshot include partner / redundant SF
   state inline, or is it returned from a separate call?
   (Workstream A.1, look for branches under
   `IsBothConnectionRequested`.)
6. Does the SF status read require `OpenConnection3` to have
   succeeded, or is `Open2` enough? (Trial: try the discovered
   pull endpoint after `Open2` only, before doing
   `OpenConnection3`. If it works, the implementation is much
   simpler.)
7. What happens when SF is *disabled* by configuration vs
   *enabled but idle*? Both should map to `Pending=false,
   Storing=false`, but the underlying server response may be a
   sentinel error vs an all-zeros struct. The implementation must
   distinguish "no SF" (return defaults silently) from "SF errored"
   (return `ErrorOccurred = true`).

## 8. Out of Scope

Explicitly not part of this plan:

- SF write-back (the project mission is read-only;
  `IStorageServiceContract.AddStreamValues` etc. stay
  unimplemented).
- Setting SF parameters
  (`IStorageServiceContract.SetStoreForwardParameter`).
- Redundant-partner SF aggregation
  (`HistorianStoreForwardStatus.AddPartnerStoreForwardStatus`).
- Reverse-engineering the on-disk SF cache file format beyond
  presence / file count (Workstream E is a fallback, not a
  primary deliverable).
- Anything in the
  `aahClientCommon.CSFConnection.StartStoreforward` /
  `SetStorageStopped` / `SetTagSynchronized` write surface.

## 9. 2026-06-21 gRPC re-scope (current R4.3 plan)

This supersedes the recommended *route* in §2/§3/§4. The deliverable
(§1) and success criteria (§6) are unchanged. What changed is the
transport and the resolved architecture risk.

### 9.1 What the recovered gRPC contract already gives us

The 2023 R2 contract under `src/AVEVA.Historian.Client/Grpc/Protos/`
exposes SF state through **first-class pull RPCs** on `StorageService`
(`StorageService.proto`) — no duplex/callback contract, no native
`HISTORIAN_STORAGE_STATUS` C-struct decode:

- `GetSFParameter(uint32 Handle, string ParameterName)
  → (Status status, string ParamaterValue)` — the direct analogue of the
  already-shipped `GetSystemParameter`/`GetRuntimeParameter` string-keyed
  pulls. This is the primary SF-state lever: a name→value read.
- `GetRemainingSnapshotsSize(uint32 Handle)
  → (Status status, uint64 SnapshotSize)` — the pending-buffer magnitude
  in one call. Non-zero ⇒ data is queued (`Pending`/`DataStored=true`);
  zero ⇒ drained. The cleanest single signal for the idle-vs-active split.
- `GetInfo(string Request) → (Status status, bytes info)` — generic
  server info blob; a fallback if a named SF key lives here instead of in
  `GetSFParameter`.
- `OpenStorageConnectionResponse.ServerStatus` (field 5) and the
  `GetSnapshots`/`StartQuerySnapshot` family — secondary signals.

`SetSFParameter` exists too but is **out of scope** (read-only mission, §8).

The `TransactionService.ForwardSnapshot{,Begin,End}` RPCs are the SF
cache *replay/transfer* path (write-side), **not** a status read — also
out of scope here; they belong to the deferred bit-faithful SF cache work,
not to `GetStoreForwardStatusAsync`.

### 9.2 Plumbing that already exists (reuse, don't rebuild)

- `HistorianGrpcHandshake.OpenSession` — authenticated gRPC session
  (`ValidateClientCredential` NTLM loop + Open2) yielding `ClientHandle`
  (uint) + storage-session GUID. Live-verified against the 2023 R2 box.
- `HistorianGrpcStorageConnectionProbe` — already constructs a
  `StorageService.StorageServiceClient`, primes `GetInterfaceVersion`, and
  calls `OpenStorageConnection`/`CloseStorageConnection`. The SF-status
  probe is a near-clone that swaps the `OpenStorageConnection` body for
  `GetSFParameter`/`GetRemainingSnapshotsSize` calls.
- `HistorianGrpcChannelFactory` / `HistorianGrpcConnection` — channel,
  metadata, deadlines.

### 9.3 The one open risk that survives: which `Handle`?

`GetSFParameter`/`GetRemainingSnapshotsSize` both take `uint32 Handle`.
Unknown: do they accept the **session `ClientHandle`** (from
`OpenSession`, which is cheap and unblocked), or do they require the
**storage console `Handle`** returned by `OpenStorageConnection` — which
is the D2 wall (`OpenStorageConnection` routes to the
`\\.\pipe\aahStorageEngine\console` session and is the same storage-engine
pipe that blocks revision writes)? See
[[project_roadmap_exhausted_2020wcf]] and `HistorianGrpcStorageConnectionProbe`
header.

- **Best case:** these read-only status RPCs accept the session
  `ClientHandle` (status reads shouldn't need a console writer session).
  Then R4.3-over-gRPC is unblocked end-to-end and is a small, shippable
  feature.
- **Worst case:** they require the `OpenStorageConnection` `Handle` ⇒
  R4.3 inherits the D2 storage-engine-pipe wall and stays blocked on the
  same root cause as R4.2. Either way the probe answers it in one run.

### 9.4 Discovery steps (execution order)

1. **Add `grpc-sf-status-probe` to `tools/AVEVA.Historian.ReverseEngineering`**
   (mirror `HistorianGrpcStorageConnectionProbe`). Against the live 2023 R2
   server it:
   - opens an authenticated session, gets `ClientHandle`;
   - calls `GetRemainingSnapshotsSize(ClientHandle)` and reports
     `status.bSuccess` + `SnapshotSize` + any error buffer;
   - sweeps `GetSFParameter(ClientHandle, name)` over a candidate
     name list (`Status`, `Storing`, `Pending`, `DataStored`,
     `SF.Status`, `StoreForwardStatus`, `Forward`, `CacheSize`,
     `ErrorOccurred`, plus any names surfaced by Workstream A's IL of
     `ConvertUnmanagedSFStorageStatusToManagedStorageStatus`);
   - records which names the server accepts and the returned values.
   - If every call fails with an auth/handle-shaped error, retry once
     with the `OpenStorageConnection` `Handle` to disambiguate §9.3.
2. **Idle baseline first** — run against the server with SF *not* active.
   Establishes the "no SF / drained" response shape (expected:
   `SnapshotSize=0`, parameter reads succeed-with-defaults or
   return a "not configured" sentinel). This alone may be enough to ship
   an honest idle-state implementation that is strictly better than
   today's hardcoded all-false synthesis (it would be *measured* false).
3. **Active-SF capture** — only if step 2 proves the read works and we
   need the active-state fixtures. Force SF on the sacrificial Historian
   VM (stop Runtime DB writer; let the queue spill to SF), re-run the
   probe, capture the non-zero/`Storing=true` response. This is the one
   invasive step and the gate on full success criteria §6.1–6.3.
4. **Map + implement** — add `GrpcGetStoreForwardStatus` to the gRPC
   read orchestrator, map the probed fields onto
   `HistorianStoreForwardStatus`, route `GetStoreForwardStatusAsync`
   to it when `Transport == RemoteGrpc` (keep the synthesized fallback
   for non-gRPC transports and for the "no SF configured" sentinel).
   Add golden-byte fixtures (idle + active) and
   `WcfStoreForwardStatusProtocolTests`-style parse tests. Gate the live
   integration test on `HISTORIAN_GRPC_HOST`.

### 9.5 Effort / feasibility summary

- **Risk collapsed:** pull-vs-push (the old plan's worst risk) is settled
  — it's a pull. No duplex WCF/gRPC callback contract.
- **No native struct decode:** `GetSFParameter` returns a *string*; we
  skip the `HISTORIAN_STORAGE_STATUS` C-layout RE entirely (Workstream
  A.2 / D.1 become "nice-to-have for field names", not blocking).
- **Reuses shipped plumbing:** session open + `StorageServiceClient` +
  channel already exist and are live-verified.
- **Remaining unknowns are empirical, one probe-run each:** (a) the
  accepted parameter-name vocabulary, (b) which `Handle` the status RPCs
  want (§9.3 — the only thing that could re-block it), (c) the
  active-SF response shape (needs the invasive force-SF step).
- **Net:** Step 1–2 are low-risk and could land a *measured* idle-state
  `GetStoreForwardStatusAsync` over gRPC quickly. Steps 3–4 (full
  success criteria) still need the sacrificial-VM force-SF capture and
  are gated on §9.3 not landing on the D2 wall.

### 9.6 Out of scope (unchanged from §8, restated for gRPC)

`SetSFParameter`, `ForwardSnapshot*` (SF replay/transfer), the on-disk
cache file format, and redundant-partner SF aggregation all remain out of
scope. R4.3 is read-only status, gRPC-first.

### 9.7 Idle-baseline run — RESULTS (2026-06-21)

Built `HistorianGrpcStoreForwardStatusProbe` + the `grpc-sf-status-probe`
CLI command and ran it against the **live 2023 R2 server** with the
historian in its **idle / not-actively-storing** state (storage interface
v4, authenticated session opened OK). Tested both read-only (`0x402`) and
write-enabled (`0x401`) sessions. Findings, with the §9.3 handle question
**resolved**:

1. **Direct `StorageService` SF pull RPCs are D2-gated — confirmed the
   §9.3 worst-case branch.**
   - `GetRemainingSnapshotsSize(session.ClientHandle)` →
     `bSuccess=false`, error buffer `04 84 00 00 00` (= status `0x84` /
     **132 `OperationNotEnabled`**). **Identical under `0x401` and
     `0x402`** — so it is NOT the read/write connection-mode gate; the
     History-session `ClientHandle` is simply not a valid handle for this
     op's handle-space.
   - `GetSFParameter(session.ClientHandle, <name>)` → server-side
     `RpcException(Unknown, "Exception was thrown by handler")` for **all
     16** candidate names, both session modes.
   - These two ops need the **`OpenStorageConnection` console handle**,
     and `OpenStorageConnection` itself fails with the storage-engine
     console error (`84 55 00 00 00 01 02 00 09 15 00`
     + ASCII `"OpenStorageConnection"`) — the **D2 storage-engine-pipe
     wall**, the same root cause that blocks R4.2 revision writes. We
     cannot obtain the console handle, so these two SF RPCs are
     unreachable from a pure managed client. See
     [[project_roadmap_exhausted_2020wcf]].

2. **One reachable session-handle lever found:**
   `StatusService.GetHistorianConsoleStatus(strHandle)` **SUCCEEDS** with
   the session string handle (uppercase Open2 GUID) — no console handle
   needed — and returns `uiConsoleStatus = 3` at idle. This is the only
   SF-adjacent signal reachable from the managed client. **Its enum
   semantics are unknown** (3 = presumably "running/normal"); whether it
   shifts when SF is actively storing is the open question.

3. `StatusService.GetHistorianInfo(strHandle, btRequest)` → `bSuccess=
   false` for every `btRequest` candidate (empty / `u32(0)` / ascii+utf16
   `"StoreForward"`); its request framing is not yet known. Lower-yield
   than `GetHistorianConsoleStatus`; revisit only if needed.

**Net idle-baseline conclusion.** R4.3's clean direct route
(`GetSFParameter` / `GetRemainingSnapshotsSize`) is **blocked behind the
D2 storage-engine console pipe**, exactly like R4.2 — a pure managed
client cannot open the console session those ops require. The *only*
reachable SF-adjacent signal is `GetHistorianConsoleStatus` → a status
uint. Two paths forward:

- **(a) Ship a measured idle-state only. — SHIPPED + LIVE-VERIFIED 2026-06-21.**
  `HistorianGrpcStatusClient.GetStoreForwardStatusAsync` opens a session,
  calls `GetHistorianConsoleStatus`, and returns
  `HistorianStoreForwardStatus` all-false but *measured*: it actually
  contacts the server and reports `ErrorOccurred=true` (with the underlying
  error) when the server is unreachable / the console-status call fails —
  strictly better than the blind hardcoded synthesis, which never contacts
  the server. Routed via `Historian2020ProtocolDialect.GetStoreForwardStatusAsync`
  when `Transport == RemoteGrpc` (non-gRPC keeps the synthesized fallback).
  Gated live test `HistorianGrpcIntegrationTests.GetStoreForwardStatusAsync_OverGrpc_ReturnsMeasuredIdleState`
  passes against the real 2023 R2 server. `Storing`/`Pending`/`DataStored`
  magnitude is intentionally NOT surfaced — it lives behind the D2 wall (see
  path (b)).
- **(b) Full success criteria (§6) stay blocked** on the D2 console-pipe
  wall. Decoding the active-SF `uiConsoleStatus` value and any
  `GetSystemParameter` SF keys still needs the invasive force-SF capture
  on a sacrificial Historian — and even then `Storing`/`DataStored`
  magnitude is only available via the D2-gated `GetRemainingSnapshotsSize`.

Probe code: `src/AVEVA.Historian.Client/Grpc/HistorianGrpcStoreForwardStatusProbe.cs`,
CLI `grpc-sf-status-probe <host> [port] [--tls] [--dnsid <n>] [--write-session]`.
Writes nothing; releases any console session immediately.