Add HistorianGrpcStoreForwardStatusProbe and the `grpc-sf-status-probe` CLI
command. The idle-baseline run against the live 2023 R2 server resolves the
plan's §9.3 handle question: the direct StorageService SF pull RPCs
(GetSFParameter / GetRemainingSnapshotsSize) require the OpenStorageConnection
console handle and are D2-gated (err 132, identical under read-only and
write-enabled sessions), while StatusService.GetHistorianConsoleStatus IS
reachable on the session string handle (=3 at idle).
Records the gRPC re-scope and the idle-baseline findings in
docs/plans/store-forward-cache-reverse-engineering.md §9. The probe writes
nothing and releases any console session immediately.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B6mcaT2PjRFKcogzp9UkfC
Extends the IL-rewrite to log out (byref) byte[] params at method exit (ldarg + ldind.ref), not
just byte[] inputs. This captured GetTagInfosFromName's response, which located the per-tag GUID at
offset 8 = exactly where ParseTagInfoRecord reads "typeId" — proving the SDK already parses the GUID
AddStreamValues needs.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B6mcaT2PjRFKcogzp9UkfC
Drove the native 2023 R2 client through a committed non-streamed (historical backfill) write to a
sandbox tag, with the IL-rewritten managed gRPC client dumping every byte[] payload. Read the value
back over gRPC = end-to-end validated.
Key discovery: the M3 historical write does NOT use AddNonStreamValues/TransactionService (the
roadmap's assumption from the static decompile). The native client routes it over
HistoryService.AddStreamValues with an "ON" storage-sample buffer (structurally the AddS2 "OS"
family), plus EnsureTags for registration:
AddStreamValues.values (56B): "ON"(0x4E4F) + u16 count=1 + u32 totalLen + u16 payloadLen +
16B tag GUID + FILETIME(sample) + u16 quality=192 + u32 type/desc + FILETIME(received) +
8B double value.
EnsureTags.tagInfos (144B): the analog CTagMetadata the SDK's EnsureTagAsync already builds
(0x4E marker ... fe 00 trailer).
Tooling that produced it:
- instrument-grpc-nonstream now instruments EVERY byte[]-input method on every Grpc*Client
(45 methods) so the real wire path surfaces regardless of assumptions.
- harness pre-loads the instrumented GrpcClient by identity (LoadFrom context reuses an
already-loaded assembly before sibling-probing, so the rewrite wins over the bin original);
capture-write sequence fixed to Begin -> Add -> SendValues -> End (End-before-Send = err 160
InvalidBatchId); GetTagInfoByName(cache:false) + resync wait resolves the server key; cache
gate (D2's 129) does NOT block the primed 2023 R2 client.
Buffers captured to gitignored artifacts/. Next: build the "ON" AddStreamValues serializer in
src/ (adapt the existing AddS2 "OS" serializer) + EnsureTags + ship AddHistoricalValuesAsync.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B6mcaT2PjRFKcogzp9UkfC
Adds the dnlib instrument command + harness wiring to capture the two non-streamed-write
buffers from the native 2023 R2 client:
- `instrument-grpc-nonstream <GrpcClient.dll> [out]` injects CaptureLogger.LogByteArray at the
entry of GrpcHistoryClient.RegisterTags (byte[] tagInfos) and AddNonStreamValues (byte[] inBuff),
writing the rewrite to docs/reverse-engineering/dnlib-write-copy/grpc2023 (gitignored — derived
AVEVA binary). dnlib preserves the AVEVA public-key identity so aahClientManaged still binds the
rewritten copy under the LoadFrom context (no SN re-verification).
- harness `--grpc-rewrite <dir>` probes that dir first, so the instrumented GrpcClient.dll +
ReverseInstrumentation.dll load ahead of the originals. load-check confirms the rewritten
strong-named copy binds (HistorianConnectionMode.Historian=2; GrpcHistoryClient RegisterTags +
AddNonStreamValues present).
Next: capture-write scenario (open write-enabled -> sandbox tag -> read-prime -> AddNonStreamedValue),
which dumps tagInfos + inBuff to the capture NDJSON. Prod write — confirm before running.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B6mcaT2PjRFKcogzp9UkfC
Live-probed StorageService.OpenStorageConnection against the 2023 R2 server over a
write-enabled (0x401) session. Every attempt — sweeping ConnectionMode (0x401/0x402/0x1),
StorageSessionId-in (Open2-GUID / empty), and FreeDiskSpace — returns the IDENTICAL native
error type=4 code=85 ("session not registered"), so it's a structural refusal, not a bad
field value.
Decode (two corroborating facts):
- Error 85 is the same code the event read returns before RegisterTags2 (see
HistorianWcfEventOrchestrator) — a generic "session not registered for this op".
- The 2023 R2 decompile shows OpenStorageConnection lives on a SEPARATE GrpcStorageClient
(the storage engine's SF/snapshot channel, own port + service identity); HistorianAccess
drives non-streamed writes through the native C++ HistorianClient, never this op.
So the roadmap's mapped "missing console session" step was wrong. The real non-streamed-write
precondition is the front-door HistoryService.RegisterTags (RTag2-family) for the target tag —
which is exactly why the R3.1 batch failed at AddNonStreamValues (no tag registered ->
StoreNonStreamValues had no route). Matches the original 2020-WCF D2 hypothesis.
Remaining (both need a native gRPC capture; do not guess bytes): the regular-tag RegisterTags
btTagInfos (only CM_EVENT's tag-GUID form is known) and the AddNonStreamValues btInput.
- HistorianGrpcStorageConnectionProbe + grpc-open-storage-connection CLI (opens nothing
persistent; CloseStorageConnection on success)
- corrected revision-write-path.md §R3.1 follow-up + hcal-roadmap R3.1/R3.2 rows
- gated regression test pinning the error-85 refusal
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B6mcaT2PjRFKcogzp9UkfC
Empirically decoded the AddNonStreamValues btInput framing against the live 2023
R2 server (grpc-nonstream-decode command + ProbeNonStreamedBuffersAsync driver).
Every transaction rolled back (bCommit=false) — no data written.
Finding: the btInput is assembled native-C++-side (not in any decompile), so 6
evidence-based framings (44-54B, packed HISTORIAN_VALUE2 variants) were probed.
All 6 returned the IDENTICAL server error while an empty buffer returned a
different InvalidParameter — so non-empty buffers pass parameter validation into
CHistStorageConnection::StoreNonStreamValues, which routes to the
\.\pipe\aahStorageEngine\console pipe server-side. Identical-across-framings =>
the blocker is NOT the btInput layout but a missing storage-engine console
session / tag-registration precondition for the connection.
Next step (untested): StorageService.OpenStorageConnection + tag registration
(RegisterTags/AddTagidPairs/AddShardTagids) before AddNonStreamValues, then
commit + read-back on a sandbox tag. Documented in revision-write-path.md
(R3.1 decode section); raw artifact gitignored.
272 unit tests pass.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B6mcaT2PjRFKcogzp9UkfC
The D2 storage-engine-pipe wall is WCF-transport-specific. On the 2023 R2 gRPC
front door, TransactionService is a first-class service AND the gateway to the
storage engine, so the Open2 storage-session GUID (uppercase) is accepted
directly as strHandle with no legacy pipe.
Live-verified against the real 2023 R2 server over a write-enabled (0x401) gRPC
session: AddNonStreamValuesBegin returns a real strTransactionId, and
AddNonStreamValuesEnd(bCommit=false) discards it cleanly (no data written). On
2020 WCF the same op returns UnknownClient(51) for every handle + priming chain.
- HistorianGrpcRevisionProbe + grpc-revision-probe CLI command + gated test
NonStreamedWriteTransaction_OverGrpc_BeginsAndDiscards (live pass).
- HistorianGrpcHandshake.OpenSession gains an optional connectionMode param
(default read-only 0x402; pass 0x401 for write-enabled) — non-breaking.
- Docs: revision-write-path.md "the wall is gone" section; roadmap M3 section,
R3.1-R3.3 rows, one-glance table, and status note updated honestly.
Not yet shipped: the AddNonStreamValues btInput VTQ buffer is uncaptured (never
guess wire bytes), so no value-commit is implemented. Scope is non-streamed
ORIGINAL backfill; revision EDITS (R4.2) remain pipe-only even on gRPC.
272 unit tests pass; sanitization scan clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B6mcaT2PjRFKcogzp9UkfC
Pushed on recovering the summary query params. Findings:
- Added `enum-dump` to the RE CLI (dumps a managed enum's literal members).
Confirmed INSQL_QUERYTYPE / HISTORIAN_SUMMARYTYPE are value__-only in the
managed metadata — their named members are native-side constants, so they
can't be recovered statically. Same for CColumnNameMap.LoadColumnNameMap
(column->bit built from native string/const data, not IL ldstr/ldc).
- Live-probed StartQuery2 against SysTimeSec sweeping QueryType/SummaryType/
ColumnSelectorFlags. The server ACCEPTS SummaryType 1/2/4/5 (returns a valid
version-9 buffer) but yields 0 rows; column flags don't change that;
QueryType 15/16 don't exist. So a summary query is NOT Full+SummaryType+
flags — the config lives in the AutoSummaryParameters trailer (currently
zeroed) and/or a native summary QueryType.
Conclusion recorded in the plan: the request shape needs a NATIVE capture
(instrument-wcf-writemessage on a real summary query), not managed-metadata
recovery or blind probing. Decode targets remain located. No guessed code in
src/; probe scaffolding removed. 208 tests green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds the highest-leverage reverse-engineering primitive from the roadmap: one
path to turn a live operation buffer into a committable golden fixture. Unblocks
every capture-tier item (R0.5, R1.x, R2.1).
- ProtocolCaptureSanitizer: redacts identity-bearing values (host, tag, user,
machine) from a native buffer in BOTH ASCII and UTF-16LE, overwriting in place
with an 'X' fill so length and every field offset are preserved (keeps the
fixture useful for byte-layout RE). ASCII-letter matching is case-insensitive;
secrets < 3 chars are skipped to avoid collision corruption. AssertNoSecretsRemain
is a fail-closed safety net that refuses to emit if any value survives.
- ProtocolFixtureWriter: serializes a capture to fixtures/protocol/<op>/<name>.json
with sanitized hex, length, SHA-256 of the sanitized bytes, and a scrub report.
Timestamps are passed in (deterministic / testable).
- capture-tag-info CLI command: captures a live GetTagInfoFromName response and
writes the fixture. The same native bytes ride inside 2023 R2 gRPC
GetTagInfosFromName, so the fixture is transport-agnostic.
- 11 unit tests for the sanitizer/writer (test project now references the RE tool).
- First real fixture: get-tag-info/analog-*.json — a 98-byte Int4 CTagMetadata
buffer captured live from the local Historian 2020 server, tag name redacted,
verified to contain no identity (descriptor 03 c3 00 31 = Int4, as documented).
180 non-live unit tests green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Full read-only SDK (src/AVEVA.Historian.Client) implementing the CLAUDE.md required
surface against AVEVA Historian's binary WCF protocol — no native AVEVA runtime
dependency. All operations live-verified against a local Historian:
- ProbeAsync, ReadRawAsync, ReadAggregateAsync, ReadAtTimeAsync, ReadEventsAsync
- BrowseTagNamesAsync, GetTagMetadataAsync (17 native data-type codes mapped)
- GetConnectionStatusAsync, GetStoreForwardStatusAsync, GetSystemParameterAsync
- 108/108 unit + integration tests pass
Includes the reverse-engineering toolkit (tools/AVEVA.Historian.ReverseEngineering)
used to decode the protocol: WCF probes, IL inspection via dnlib, and IL-rewrite
instrumentation (instrument-wcf-{write,read}message etc.) plus the .NET Framework
trace harness (tools/AVEVA.Historian.NativeTraceHarness) for parity testing.
Sanitized handoff evidence under docs/reverse-engineering/. Native AVEVA binaries
(current/, aveva-install-x64/, aveva-install-x86/) are gitignored — fetch separately
from the AVEVA installer.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Joseph Doherty