histsdk/docs/plans/store-forward-cache-reverse-engineering.md

Store/Forward Cache Reverse-Engineering Plan

Last updated: 2026-05-04

This document plans the reverse-engineering effort needed to replace the synthesized GetStoreForwardStatusAsync in src/AVEVA.Historian.Client/Wcf/HistorianWcfStatusClient.cs (lines 101-117) with a real, evidence-backed implementation. It is a plan, not the work itself. No code changes; no captures collected.

Read this together with:

• docs/reverse-engineering/handoff.md — read/event protocol decoding state
• src/AVEVA.Historian.Client/Wcf/Contracts/IStorageServiceContract.cs — the WCF contract that already declares the SF parameter ops
• src/AVEVA.Historian.Client/Models/HistorianStoreForwardStatus.cs — the output model the implementation must populate

1. Goal

"SF support works" means, end-to-end:

1. Primary deliverable. client.GetStoreForwardStatusAsync() against a live local Historian returns a HistorianStoreForwardStatus whose Pending, Storing, DataStored, ErrorOccurred, Error, ServerName, and ConnectionKind fields reflect actual server-reported state, not the synthesized defaults at HistorianWcfStatusClient.cs:107-117.
2. Secondary deliverable. The SDK can also answer the higher-level "is SF currently buffering?" question accurately when the runtime DB is down, not just when it is up. That is the case the real native client handles correctly and where the synthesized default (Storing = false, ErrorOccurred = false) is silently wrong today.
3. Non-goals. Writing into SF, replaying SF buffers, configuring SF parameters, redundant-partner SF aggregation (HistorianStoreForwardStatus.AddPartnerStoreForwardStatus, token 0x060060B8). Read-only matches the project mission in CLAUDE.md.

The success bar is parity with the native wrapper's ArchestrA.HistorianAccess.GetStoreForwardStatus (MD token 0x06006186 in current/aahClientManaged.dll), not a superset.

2. Architecture Investigation (open questions, in priority order)

Answer these before writing any production code. Each has a discovery action in §3.

Q1. Is SF status read from a local in-process struct, a separate WCF endpoint, or a Named Pipe IPC?

Current evidence: all three are plausible, but the wrapper actually uses "in-process struct kept current by server-pushed WCF events". Specifically:

• ArchestrA.HistorianAccess.GetStoreForwardStatus (token 0x06006187, the private 2-arg overload) does not call WCF. It calls mdas_GetStorageStatus (a calli against the INSQL_MDAS_ERROR (IntPtr handle, uint, HISTORIAN_STORAGE_STATUS*) C signature in current/aahClient.dll exports) and then maps the result through HistorianAccessUtil.ConvertUnmanagedSFStorageStatusToManagedStorageStatus (token 0x060060E4).
• Mutators like CConfigStatusClient.SetMdasStoreForwardEvent (token 0x060029DC) and aahClientCommon.CStatus.SetStoreForwardEvent (token 0x06002A04) are wired to the WCF callback IStatusServiceContract2.SetStoreForwardEvent (StatusServiceContract.IStatusServiceContract2.SetStoreForwardEvent, token 0x06005F57). The server pushes SF state changes; the client caches them.
• Confirm: read the IL of token 0x06006187 and verify the only system call is mdas_GetStorageStatus. The first 200 instructions confirm this: GetClient(ConnectionIndex) → calli against the INSQL_MDAS_ERROR(IntPtr,uint,HISTORIAN_STORAGE_STATUS*) signature → ConvertUnmanagedSFStorageStatusToManagedStorageStatus.

Implication: the SDK cannot ship a synchronous probe that calls one WCF operation and gets the answer. It must subscribe to the same status-event stream the native wrapper subscribes to, or call a status query that returns the cached snapshot from the server.

Q2. Is there a single-shot WCF query that returns the same snapshot?

Likely yes. Hypothesis: IStatusServiceContract2.GetHistorianInfo (GETHI, see IStatusServiceContract2.cs:24-30) returns a multi-key status blob whose schema includes SF state. Alternative: a status-only key passed to GetSystemParameter (already plumbed via HistorianWcfStatusClient.GetSystemParameterAsync). Both are testable without writing protocol code by sending probe payloads and observing the response shape.

Q3. Does SF have its own sidecar process / pipe / WCF endpoint we are missing?

Strong evidence the answer is yes when SF is enabled:

• aahClientCommon.CSFConnection.GetSFPipeName (token 0x06004B72), GetSFPath (0x06004B71), IsConnected (0x06004B73), IsEnabled (0x06004B6F) — there is a separately-named SF Named Pipe distinct from the main MDAS pipe.
• aahClientCommon.CSFConnection.StartStoreforward (token 0x06004BC6).
• IStorageServiceContract already declares GetStoreForwardParameter / SetStoreForwardParameter (GetSFP/SetSFP, see IStorageServiceContract.cs:81-85) and Storage is a separate WCF service slot in HistorianWcfServiceNames.cs:15.
• CWcfConfig.ConfigurePipeProxy<IStorageServiceContract> (token 0x06004B1C) and CWcfConfig.ConfigureTcpProxy<IStorageServiceContract> (token 0x06004B1B) confirm the storage proxy supports both transports — same dual-transport pattern the History/Retrieval proxies use.
• CStorageEngineConsoleClient.GetPipeNameStr (token 0x06000E2D) / GetFullPipeNameStr (token 0x06000E2E) wraps the storage-engine console pipe via STransactPipeClient2 (a non-WCF binary pipe protocol).

Open: is the SF sidecar even running on the dev host this SDK is being tested against? handoff.md does not record an SF process being observed. aveva-install-x64/ and aveva-install-x86/ ship only DLLs (no aahStoreForwardClient.exe / aahSFClient.exe / similar). The SF sidecar is part of the Historian server install, not the client redistributable. So:

• On the developer machine, SF is reachable only because the local Historian server is installed.
• A pure-client install (the deployment target this SDK ships into) may never have SF.

This shapes the success criteria: when SF is not configured, a correct implementation returns Pending = false, ErrorOccurred = false, DataStored = false, Storing = false — i.e. the same shape the synthesized defaults produce today. The interesting case is when SF is configured and active.

Q4. Is SF state authoritative on the Historian server or on a per-client basis?

Native wrapper reads it from HistorianClient* (the per-connection C++ object). This means it is connection-scoped server-pushed state. We do not need to enumerate cluster-wide SF state — the server reports "my SF buffer for this client's writes" only. This matches our read-only mission: we are not a writer, so the only SF state of interest is the server-side cache for other writers, which the server can report to us as a passive observer.

Q5. Does any SF probe require Admin?

CSFConnection.GetSFPipeName returns a kernel object name. Reading from it requires the pipe ACL to permit the caller. If the SF pipe is ACL'd to LocalSystem only, the SDK cannot read it without impersonation — and the SDK runs as the calling process. This is a hard limit, not a bug.

3. Discovery Workstreams

Run these in parallel. None require a live server beyond what the existing test rig already has.

Workstream A — Static IL inspection (parallel-safe, read-only)

Owner action items, in order:

1. Dump full IL of token 0x06006187 (HistorianAccess.GetStoreForwardStatus(ConnectionIndex,out)):

   dotnet run --no-build --project tools\AVEVA.Historian.ReverseEngineering -- `
     dnlib-method current\aahClientManaged.dll HistorianAccess.GetStoreForwardStatus --instructions
   

   Save under docs/reverse-engineering/historianaccess-getstoreforwardstatus-il-latest.txt. Confirm the calli target signature INSQL_MDAS_ERROR(IntPtr,uint,HISTORIAN_STORAGE_STATUS*) and that the only WCF entry-points it touches are zero.
2. Dump IL of HistorianAccessUtil.ConvertUnmanagedSFStorageStatusToManagedStorageStatus (token 0x060060E4). This is the unmanaged→managed mapping; it tells us which fields of HISTORIAN_STORAGE_STATUS populate which fields of HistorianStoreForwardStatus. We will need the same mapping in reverse on the wire response.
3. Inventory every method that writes into the local SF status struct:

   methods current\aahClientManaged.dll SetStoreForward
   methods current\aahClientManaged.dll SetMdasStoreForward
   

   The known set as of writing: CConfigStatusClient.SetMdasStoreForwardEvent (0x060029DC), aahClientCommon.CStatus.SetStoreForwardEvent (0x06002A04), CStatusConnectionDirect.SetStoreForwardEvent (0x06004DF8), CStatusConnectionWCF.SetStoreForwardEvent (0x06004E4E), CClientCommon.SetStoreForwardEventOnServer (0x06002EC0). The WCF variant is the one whose IL maps onto IStatusServiceContract2.SetStoreForwardEvent (token 0x06005F57) — read its IL and document the request/response shape.
4. Dump IL of IStatusServiceContract2.SetStoreForwardEvent (0x06005F57) parameter types. The [OperationContract] declaration in the wrapper assembly already encodes the wire shape; this gives us the bytes the server pushes us.

Workstream B — Install inventory (parallel-safe)

1. Inventory aveva-install-x64\ and aveva-install-x86\ for any binary whose name contains Store, Forward, SF, Cache, Spool. As of this checkout: none, only DLLs. Confirm.
2. Inventory the deployed Historian server (out-of-band; not in this repo) for aahStoreForwardClient.exe, aahStoreForwardServer.exe, aahSFCache.exe, or any service registered with Description matching *Forward*. Capture the service name, account identity, and pipe ACLs (accesschk -wuvc).
3. Walk the registry: HKLM\SOFTWARE\ArchestrA\Historian and any sub-key matching *StoreForward*, recording paths and pipe names. Sanitize before committing.

Workstream C — WCF probe (parallel-safe)

Use the existing wcf-probe and wcf-status subcommands of tools\AVEVA.Historian.ReverseEngineering:

1. wcf-probe $env:HISTORIAN_HOST 32568 — confirm Storage/GetV is reachable. (It is the third service slot in HistorianWcfServiceNames.) Document the returned interface version.
2. wcf-status $env:HISTORIAN_HOST 32568 <param-name> — sweep plausible SF parameter names (SF.Status, StoreForward.State, SFCacheBytes, etc.) through GetSystemParameter and record what the server accepts. Cheap, read-only, no session needed beyond the already-decoded auth chain.
3. Probe GetHistorianInfo (GETHI, IStatusServiceContract2.cs:24) with the byte request shape used by the native wrapper. The request bytes are visible if we run instrument-wcf-readquery-style instrumentation against CConfigStatusClient.SetMdasStoreForwardEvent's upstream caller — see Workstream D.

Workstream D — Native capture (sequential after A and C)

Two captures are needed:

1. Native call to mdas_GetStorageStatus. Run tools\AVEVA.Historian.NativeTraceHarness with a new scenario --scenario sfstatus (to be added) that invokes HistorianAccess.GetStoreForwardStatus() and dumps the HISTORIAN_STORAGE_STATUS C struct memory before the managed conversion runs. This pins the binary layout of the struct (offsets, field widths, endianness) without us guessing.
2. WCF push of SF events. Configure the local Historian to enter SF mode (stop the runtime DB writer; let the writer's queue trigger SF) and capture the WCF traffic with the existing instrument-wcf-readquery sibling — i.e. add an instrument-wcf-setstoreforwardevent subcommand that IL-rewrites aahClientManaged.dll to log the bytes the server sends to IStatusServiceContract2.SetStoreForwardEvent. Save the rewrite under docs/reverse-engineering/dnlib-write-copy/, never current/.

Workstream D is the only step that needs an actively-storing SF sidecar. Plan: stop the Historian Runtime DB SQL service, write a single test point via the wrapper's writer harness, and capture the SF event push, then restart Runtime DB and capture the "end-of-SF / data drained" push.

Workstream E — On-disk cache (only if Workstream D fails)

If the WCF push protocol turns out to be impractical to reproduce (e.g. requires duplex contract, callback channel, or a server-side session-bind we cannot match from our managed client), fall back to inspecting the on-disk SF cache directly. Steps:

1. Resolve CSFConnection.GetSFPath IL to find the cache directory convention (likely %ProgramData%\ArchestrA\Historian\Cache\ or similar — to be confirmed, never assume the path).
2. Inventory file types: .sfdata, .sfindex, .cache — whatever the directory contains.
3. Decode the file header. The presence/size of .sfdata files is sufficient to populate DataStored and Pending; we do not need to decode the value payload.

This fallback is only for DataStored / Pending. Storing and Error fundamentally require a live server-state read.

4. Concrete Reverse-Engineering Steps (execution order)

Mirrors the read/event decoding workflow that succeeded for raw queries.

Step 1 — Find native methods that touch SF

Already done; baseline evidence is recorded in §2 Q1/Q3 above. Key tokens to reference:

• 0x06006186, 0x06006187 — public/private HistorianAccess.GetStoreForwardStatus
• 0x060060E4 — HistorianAccessUtil.ConvertUnmanagedSFStorageStatusToManagedStorageStatus
• 0x060029DC — CConfigStatusClient.SetMdasStoreForwardEvent
• 0x06002A04 — aahClientCommon.CStatus.SetStoreForwardEvent
• 0x06002DFF — aahClientCommon.CClientCommon.IsInStoreForward
• 0x06002E18 — aahClientCommon.CClientCommon.SetStoreForwardParams
• 0x06002EC0 — CClientCommon.SetStoreForwardEventOnServer
• 0x06004BC6 — aahClientCommon.CSFConnection.StartStoreforward
• 0x06004B6F..0x06004B73 — CSFConnection getters (path, pipe, enabled, connected)
• 0x06004DF8, 0x06004E4E — direct vs WCF status connections
• 0x06005F57 — IStatusServiceContract2.SetStoreForwardEvent MD ref
• 0x06006193 — HistorianAccess.IsBothConnectionRequested (used by the public arity-0 GetStoreForwardStatus to decide whether to fan out to a redundant partner)

Step 2 — Decode HISTORIAN_STORAGE_STATUS layout

Run Workstream A.2 (decode token 0x060060E4) and Workstream D.1 (native struct memory dump). Together they pin the field layout.

The managed struct fields we already know we need to populate (from HistorianStoreForwardStatus.cs): ServerName, Pending, ErrorOccurred, Error, DataStored, Storing, ConnectionKind. The native struct will have ≥7 fields plus padding. Express the mapping as a comment table in the implementation.

Step 3 — Decide the wire model

Two possible implementations:

1. Push-mode (native parity). SDK opens an authenticated WCF session that the server treats as a status subscriber, listens for IStatusServiceContract2.SetStoreForwardEvent callbacks, maintains a local cache, and GetStoreForwardStatusAsync returns from the cache. This requires WCF duplex (CallbackContract) which is not currently exercised anywhere in src/AVEVA.Historian.Client/Wcf/.
2. Pull-mode (probe). SDK calls GetHistorianInfo (GETHI) or a discovered Storage-service equivalent and maps the one-shot response. No subscription state required.

Pull-mode is strongly preferred: it matches the SDK's existing WCF style, avoids duplex contracts, and the existing code path in HistorianWcfStatusClient.GetSystemParameter is the right shape. Only fall back to push-mode if Workstream C.3 proves the server has no pull endpoint that returns SF state.

Step 4 — Implement the managed contract method

Once Step 3 picks pull-mode, implement against the WCF contract (likely a new [OperationContract] on IStatusServiceContract2 or a method on IStorageServiceContract). Follow the existing parameter-naming discipline from the resolved ValidateClientCredential blocker: use [MessageParameter(Name = "...")] to match exact server element names — do not let WCF derive them from C# parameter names. See handoff.md "Active Blocker" entry for the 2026-05-04 fix.

Step 5 — Add golden-byte fixtures

Add a request and response fixture under fixtures/protocol/store-forward-status/:

• request-get-storage-status.bin — bytes the SDK sends.
• response-get-storage-status-running-normal.bin — server not in SF.
• response-get-storage-status-active-sf.bin — server actively storing.
• response-get-storage-status-error.bin — server's SF errored.

Capture sources: the same instrumented native wrapper runs that populate Workstream D. Sanitize hostnames, GUIDs, and timestamps before committing.

Step 6 — Replace the synthesized stub

Replace SynthesizeStoreForwardStatus (lines 107-117 of HistorianWcfStatusClient.cs) with a real implementation. Keep the synthesized fallback for the case where the storage service returns a "no SF configured" sentinel — that is not an error condition, it is the normal state for client-only deployments.

Add a unit test class WcfStoreForwardStatusProtocolTests next to the existing WcfDataQueryProtocolTests etc., with golden-byte parse tests using the fixtures from Step 5.

Update the operation status table in README.md:20 from "synthesized defaults (no SF sidecar to probe)" to "live-verified" once the integration test passes.

5. Risks and Gotchas

1. SF may not be present on the test host. The dev Historian probably has SF disabled by default; turning it on means stopping Runtime DB SQL services, which is invasive. Plan to do capture work on a dedicated sacrificial Historian VM, not the shared dev box.
2. SF sidecar may require Admin or LocalSystem to query. Any pipe-direct fallback (Workstream E) will fail under standard user accounts. Document the privilege requirement explicitly in the SDK XML doc comments on GetStoreForwardStatusAsync.
3. State is volatile. Probes that take >100 ms can race against the server's own SF state machine. Capture both request and response in the same instrumented run; do not try to correlate two captures.
4. Push-mode would force a duplex WCF contract. None of the existing decoded operations use duplex. Adding it widens the managed WCF surface significantly and risks .NET-WCF compatibility issues we have not yet hit. Pull-mode first.
5. The wrapper's IsBothConnectionRequested (token 0x06006193) path indicates a "primary + partner" topology. Out of scope for this pass per §1, but if the server returns partner data in the same response we must skip-decode (not throw on) unknown trailing bytes.
6. Open2-only sessions never receive SF events. handoff.md "Active Blocker" notes the wrapper's full chain (OpenConnection3 after the ValCl rounds) is the path that produces a session the server treats as a real client. SF probes must run from inside that chain — re-using HistorianWcfAuthChainHelper.OpenAuthenticatedConnection, the same call site already used by GetSystemParameter at HistorianWcfStatusClient.cs:42.
7. HISTORIAN_STORAGE_STATUS field order is not contractual. The struct is C++ inside the closed source. If AVEVA reorders fields between Historian versions, our decoder breaks. Pin the decoder to the Historian server version observed at session open (already exposed via IRetrievalServiceContractN) and reject mismatched versions explicitly with ProtocolEvidenceMissingException. Do not silently best-effort parse.
8. Sanitization. Pipe names, registry paths, and SF cache directory paths can leak hostnames and account names. Run the rg sanitizer (handoff.md "Next Pickup Steps") after every doc edit.

6. Success Criteria

A real implementation is "done" when all of the following hold:

1. client.GetStoreForwardStatusAsync() returns Pending = true and Storing = true while the local Historian's SF cache is actively buffering writes (verifiable by stopping the Runtime DB and writing a value).
2. Returns Pending = false and Storing = false within ≤ 5 seconds after the Runtime DB recovers and SF drains.
3. Returns ErrorOccurred = true and a non-null, actionable Error message when the SF cache itself fails (disk full, pipe closed, etc.).
4. Returns the synthesized "no SF" shape (all-false) without throwing on a Historian where SF is not configured.
5. Two new golden-byte unit tests pass (active-SF and idle-SF responses).
6. ProtocolGuardrailTests no longer needs to exempt GetStoreForwardStatusAsync from any "must throw ProtocolEvidenceMissingException" rule — the method is now evidence-backed.
7. Live integration test HistorianClientIntegrationTests.GetStoreForwardStatusAsync_ReturnsServerState (to be added) passes when HISTORIAN_HOST is set, skips cleanly otherwise.
8. README.md:20 operation status table is updated from "synthesized defaults" to "live-verified".

7. Open Questions for the Implementer

Resolve these before writing production code:

1. Does the server expose a pull endpoint that returns the full HISTORIAN_STORAGE_STATUS snapshot, or only push events? (Workstream C.3 answers this.)
2. What is the binary layout of HISTORIAN_STORAGE_STATUS? (Workstream A.2 + D.1.)
3. What is the [OperationContract] shape on IStatusServiceContract2.SetStoreForwardEvent? Specifically: parameter count, byte-buffer parameters, and exact MessageParameter names? (Workstream A.4.)
4. Is the Storage service slot at net.pipe://<host>/Storage and net.tcp://<host>:32568/Storage reachable on a non-Historian-server install? Or does it 404 when only the client redistributable is present? (Workstream B + C.1.)
5. Does the SF status snapshot include partner / redundant SF state inline, or is it returned from a separate call? (Workstream A.1, look for branches under IsBothConnectionRequested.)
6. Does the SF status read require OpenConnection3 to have succeeded, or is Open2 enough? (Trial: try the discovered pull endpoint after Open2 only, before doing OpenConnection3. If it works, the implementation is much simpler.)
7. What happens when SF is disabled by configuration vs enabled but idle? Both should map to Pending=false, Storing=false, but the underlying server response may be a sentinel error vs an all-zeros struct. The implementation must distinguish "no SF" (return defaults silently) from "SF errored" (return ErrorOccurred = true).

8. Out of Scope

Explicitly not part of this plan:

• SF write-back (the project mission is read-only; IStorageServiceContract.AddStreamValues etc. stay unimplemented).
• Setting SF parameters (IStorageServiceContract.SetStoreForwardParameter).
• Redundant-partner SF aggregation (HistorianStoreForwardStatus.AddPartnerStoreForwardStatus).
• Reverse-engineering the on-disk SF cache file format beyond presence / file count (Workstream E is a fallback, not a primary deliverable).
• Anything in the aahClientCommon.CSFConnection.StartStoreforward / SetStorageStopped / SetTagSynchronized write surface.