RE: resolve R1.8/R1.9 analog/state summary via request+response capture
Captured the native StartQuery2 pRequestBuff and the GetNextQueryResultBuffer2 response (instrument-wcf-writemessage + chained instrument-wcf-readmessage) and decoded both against AnalogSummaryHistory SQL ground truth. Conclusion: the rich multi-aggregate analog/state summary struct is NOT delivered over the 2020 WCF binary protocol — the response is the ordinary version-9 row buffer the existing aggregate parser already handles, carrying one value per cycle selected by RetrievalMode (QueryType 5-8), not ValueSelector (inert on this path). So "analog summary" == the existing ReadAggregateAsync; no new src/ code warranted. Tooling (tools/ + scripts/ only, nothing in src/): - NativeTraceHarness: drive summary knobs via --value-selector / --aggregation-type / --max-states (uint16) / --filter - Capture-SummaryRequest.ps1: repeatable instrument+stage+matrix capture, -WithResponse chains the ReadMessage hook - decode-summary-capture.py: StartQuery2 request diff vs baseline - decode-summary-response.py: response decode vs SQL ground truth Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -1,7 +1,59 @@
|
|||||||
# R1.8 / R1.9 — Analog-summary & State-summary queries (implementation plan)
|
# R1.8 / R1.9 — Analog-summary & State-summary queries (implementation plan)
|
||||||
|
|
||||||
**Status (2026-06-20): scoped + de-risked. Reachable on 2020 WCF. Decode targets located in
|
**Status (2026-06-21): RESOLVED by request + response capture. Conclusion: the rich
|
||||||
the native dll. Ready to implement; not yet started in `src/` (no guessed code shipped).**
|
multi-aggregate analog/state summary struct is NOT delivered over the 2020 WCF binary protocol.
|
||||||
|
The per-cycle aggregate values it would expose are ALREADY shipped via `ReadAggregateAsync`
|
||||||
|
(RetrievalMode → QueryType 5–8). No new `src/` code is warranted for R1.8/R1.9 on 2020 WCF.**
|
||||||
|
|
||||||
|
## RESOLVED — what the response capture proved (2026-06-21)
|
||||||
|
|
||||||
|
The request side was recovered first (table further down), then the `GetNextQueryResultBuffer2`
|
||||||
|
**response** was captured (`instrument-wcf-readmessage`, both hooks chained) and decoded against
|
||||||
|
`AnalogSummaryHistory` SQL ground truth for `SysTimeSec` over a 6 h window / 1 h cycle. Findings:
|
||||||
|
|
||||||
|
1. **The response is the ordinary version-9 row buffer** — same layout the existing raw/aggregate
|
||||||
|
parser (`TryParseGetNextQueryResultBufferAggregateRows`) already handles: `uint16 version=9`,
|
||||||
|
`uint32 rowCount`, then per-row `tagKey + nameLen + name + ValueCount + cycleEnd FILETIME +
|
||||||
|
quality + OpcQuality + Value(double) + PercentGood(double) + trailer(cycleStart FILETIME …)`.
|
||||||
|
The captured 7-row buffer decoded with `Value=31.0`, `PercentGood=100.0`, `ValueCount=1`,
|
||||||
|
`OpcQuality=192` — matching the SQL row exactly.
|
||||||
|
|
||||||
|
2. **There is NO rich `CAnalogSummaryValue` struct on the wire.** Each row carries a *single*
|
||||||
|
value, not Min+Max+First+Last+Avg+Integral together. The all-aggregates-in-one-row shape that
|
||||||
|
`CAnalogSummaryValue` / `AnalogSummaryHistory` represents is the **SQL/OLEDB provider's** shape,
|
||||||
|
not the binary `StartQuery2` retrieval's.
|
||||||
|
|
||||||
|
3. **The single value is selected by `RetrievalMode` (QueryType), not by `ValueSelector`.** Proven
|
||||||
|
against the same constant tag where only the *kind* of aggregate distinguishes the result:
|
||||||
|
- `RetrievalMode=Integral` (QueryType 8) → `Value = 111600.0` (= SQL `Integral`) ✓
|
||||||
|
- `RetrievalMode=TimeWeightedAverage` (QueryType 5) → `Value = 31.0` (= SQL `Average`) ✓
|
||||||
|
- `Cyclic` (QueryType 0) **+ `ValueSelector=Integral`** → `Value = 31.0` (selector **ignored**;
|
||||||
|
the request byte `ValueSelector@0x59=0x04` was confirmed sent, yet the cyclic value came back).
|
||||||
|
|
||||||
|
So `ValueSelector` / `AggregationType` / `MaxStates` are **inert on the WCF retrieval path** —
|
||||||
|
they configure the SQL provider's summary tables, not this binary query.
|
||||||
|
|
||||||
|
4. **Resolution unit is correct in the SDK.** The wire `Resolution` is 100 ns ticks (= ms × 10000).
|
||||||
|
`SerializeFullHistoryRequest` writes `TimeSpan.Ticks`, which the golden test
|
||||||
|
`SerializerMatchesInstrumentedNativeTimeWeightedAverageRequest` already verifies byte-for-byte
|
||||||
|
against native (`FromMinutes(1)` → `600000000`). No bug.
|
||||||
|
|
||||||
|
**Therefore:** "analog summary" over 2020 WCF == the existing aggregate read. To get Min, Max,
|
||||||
|
Average and Integral for a cycle you issue the corresponding `RetrievalMode` queries
|
||||||
|
(`MinimumWithTime` / `MaximumWithTime` / `TimeWeightedAverage` / `Integral`), each returning that
|
||||||
|
one aggregate per cycle — all already implemented, mapped (QueryType 5–8) and golden-tested in
|
||||||
|
`ReadAggregateAsync`. **R1.8/R1.9 need no new protocol code on this server.** A genuine
|
||||||
|
all-aggregates-at-once summary would require the gRPC front door or the SQL provider, neither of
|
||||||
|
which is the 2020 WCF binary path.
|
||||||
|
|
||||||
|
Capture/decode tooling is committed and repeatable: `scripts/Capture-SummaryRequest.ps1`
|
||||||
|
(`-WithResponse` chains ReadMessage), `scripts/decode-summary-capture.py` (request diff),
|
||||||
|
`scripts/decode-summary-response.py <config>` (response decode vs SQL ground truth). Raw captures
|
||||||
|
live under `artifacts/reverse-engineering/instrumented-wcf-writemessage-summary/` (gitignored).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
_Original scoping notes below remain for context. They led to the capture; the conclusion above supersedes their "ready to implement" framing._
|
||||||
|
|
||||||
Unlike the M1 *read* items gated by the [string-handle wall](../reverse-engineering/wcf-string-handle-wall.md),
|
Unlike the M1 *read* items gated by the [string-handle wall](../reverse-engineering/wcf-string-handle-wall.md),
|
||||||
summary queries ride the **proven `uint`-handle `StartQuery2`** path — the same call the working
|
summary queries ride the **proven `uint`-handle `StartQuery2`** path — the same call the working
|
||||||
@@ -34,43 +86,61 @@ Found via `methods … Summary` + `dnlib-method`:
|
|||||||
| `CTypeMetadata.IsAnalogSummary` / `IsStateSummary` | `0x060001A4/A5` | server-side type gating |
|
| `CTypeMetadata.IsAnalogSummary` / `IsStateSummary` | `0x060001A4/A5` | server-side type gating |
|
||||||
| `INSQL_QUERYTYPE` / `HISTORIAN_SUMMARYTYPE` | enums `0200013F` / `02000191` | the `QueryType` / `SummaryType` values to send |
|
| `INSQL_QUERYTYPE` / `HISTORIAN_SUMMARYTYPE` | enums `0200013F` / `02000191` | the `QueryType` / `SummaryType` values to send |
|
||||||
|
|
||||||
## Empirical probe results (2026-06-20, live `SysTimeSec` over `StartQuery2`)
|
## Native request capture (2026-06-21) — request shape RECOVERED
|
||||||
|
|
||||||
Swept `QueryType`/`SummaryType`/`ColumnSelectorFlags` against the live 2020 server:
|
The earlier blind probing (sweeping `SummaryType`/`ColumnSelectorFlags` over the managed
|
||||||
|
serializer) was the wrong lever: it returned 0-row buffers because the managed `SummaryType`
|
||||||
|
field is **not** how the native client encodes a summary. A real capture settled it.
|
||||||
|
|
||||||
- `QueryType=2 (Full)`, `SummaryType ∈ {0,3,6}` → normal 109-byte version-9 data buffer.
|
**Capture pipeline (now repeatable):** `scripts/Capture-SummaryRequest.ps1` IL-rewrites a copy
|
||||||
- `QueryType=2`, `SummaryType ∈ {1,2,4,5}` → **valid version-9 buffer with 0 rows** (`09 00 00 00 00 00`).
|
of `aahClientManaged.dll` (`instrument-wcf-writemessage`), stages it alongside the strong-named
|
||||||
The server **accepts** these summary types but yields no rows.
|
`ReverseInstrumentation` logger, then drives the `NativeTraceHarness` history scenario through a
|
||||||
- The 0-row result is **unchanged** by `ColumnSelectorFlags` (tried default, all-bits
|
candidate matrix while logging every outgoing MDAS body. `scripts/decode-summary-capture.py`
|
||||||
`0xFFFF…FFFF`, high-dword, low-48). So column flags are *not* the unlock.
|
extracts the `Retr/StartQuery2` `pRequestBuff` from each and diffs the summary candidates against
|
||||||
- `QueryType ∈ {15,16}` → `GetNext` blocks/times out (no such INSQL_QUERYTYPE ordinal).
|
a tag-matched `baseline-full`. The harness now exposes `--value-selector` / `--aggregation-type`
|
||||||
|
/ `--max-states` / `--filter` so the native `HistoryQueryArgs` summary knobs can be driven.
|
||||||
|
|
||||||
**Conclusion:** a summary query is *not* `Full + SummaryType + column flags`. The summary
|
**There is no separate "summary" QueryType or `SummaryType` field.** A summary is an ordinary
|
||||||
configuration lives elsewhere in the request — almost certainly the **`AutoSummaryParameters`
|
`StartQuery2` request (`QueryType` = the chosen `RetrievalMode`, e.g. `Cyclic`=0) with three
|
||||||
trailer** (`SerializeFullHistoryRequest` currently writes it all-zero via
|
things set: the **ValueSelector** byte, the **AggregationType** byte, a non-zero **Resolution**
|
||||||
`WriteAutoSummaryParameters`) and/or a native summary `QueryType`. Both are **native-side
|
(which fills the previously-zeroed `AutoSummaryParameters` trailer), and — for state summary —
|
||||||
constants** (`HISTORIAN_SUMMARYTYPE` / `INSQL_QUERYTYPE` are `value__`-only in managed metadata;
|
the **MaxStates** field. The server then returns analog- vs state-summary rows based on the tag
|
||||||
`CColumnNameMap.LoadColumnNameMap` builds column→bit via native string/const data, not IL
|
type plus these fields. Offsets below are **into the StartQuery2 `pRequestBuff`** (229-byte
|
||||||
`ldstr`/`ldc`). So they cannot be recovered from managed metadata, and blind probing of the
|
`SysTimeSec` baseline; verified byte-for-byte against the native client):
|
||||||
obvious fields returns empty.
|
|
||||||
|
|
||||||
**Therefore the right next step is a native request capture, not more probing:** drive the native
|
| Offset | Field | Type | Evidence |
|
||||||
client (NativeTraceHarness / a real summary query) and capture the `pRequestBuff` bytes via the
|
|---|---|---|---|
|
||||||
existing `instrument-wcf-writemessage` pipeline — the same method that produced every other proven
|
| `0x01` | QueryType | uint32 LE | Full→`02`, Cyclic→`00` (matches the verified `RetrievalMode`→`QueryType` map) |
|
||||||
request shape (reads, events, EnsT2). Diff that buffer against a normal Full request to read off
|
| `0x1D` | Resolution | float64 LE | `36e9` ticks → `00 00 00 D0 88 C3 20 42` = `0x4220C388D0000000` (1 h). Zero for non-summary reads |
|
||||||
the exact `QueryType` + `SummaryType` + `AutoSummaryParameters` layout, then implement against it.
|
| `0x32` | Timezone | len-prefixed UTF-16 | `"UTC"` |
|
||||||
|
| `0x49` | Filter | len-prefixed UTF-16 | `"NoFilter"` default; driven by `--filter` |
|
||||||
|
| `0x59` | **ValueSelector** | byte | baseline `01` (Auto); `--value-selector Minimum`→`06`, `Maximum`→`07`, `Average`→`08` — exact `HistorianValueSelector` values |
|
||||||
|
| `0x5B` | **AggregationType** | byte | baseline `03`; `--aggregation-type Average`→`02` — exact `HistorianAggregationType` values |
|
||||||
|
| `~0x5F` | ColumnSelectorFlags | bytes | `FF 82 07 00 82 81` — matches the `0x0000_8182_0007_82FF` reads already send; **unchanged** by summary |
|
||||||
|
| `0x6B` | Tag name | len-prefixed UTF-16 | `count, "SysTimeSec"` |
|
||||||
|
| after tag | **MaxStates** | uint16 LE | the `01`-default byte after the tag block; `--max-states 10`→`0A` (state summary, R1.9) |
|
||||||
|
| `~0xAA` | **AutoSummaryParameters** | block | zero for plain reads; `80 1E 08 6B 47 01` when Resolution set (identical across analog *and* state) — the resolution-derived cycle block |
|
||||||
|
|
||||||
## Open questions (nail these next, in order)
|
State summary (R1.9) is the **same request** with `MaxStates` > 0 (the analog `ValueSelector`/
|
||||||
|
`AggregationType` bytes stay at their `01`/`03` defaults); the analog-vs-state distinction on the
|
||||||
|
wire is which of those fields is non-default, plus the tag type. Note `MaxStates` is a **UInt16**
|
||||||
|
on `HistoryQueryArgs` (passing UInt32 throws) — the harness casts accordingly.
|
||||||
|
|
||||||
1. **Request params.** Recover the exact `QueryType` + `SummaryType` (+ whether `ColumnSelectorFlags`
|
Raw captures live under `artifacts/reverse-engineering/instrumented-wcf-writemessage-summary/`
|
||||||
must change) for analog vs state summary. Source: decompile `INSQL_QUERYTYPE`/`HISTORIAN_SUMMARYTYPE`
|
(gitignored). Re-run with `scripts/Capture-SummaryRequest.ps1` (analog: `SysTimeSec`; state:
|
||||||
enum members and `Select{Analog,State}SummaryColumns` (the `GetColumnFlag` column-name `ldstr`
|
`-TagName SysPulse`, the local discrete tag).
|
||||||
operands → the flag set). The standard QueryType map (Cyclic=0 … EndBound=14) is already verified;
|
|
||||||
summary is expected to be a `SummaryType`≠0 with an existing base `QueryType`, **not** a new mode
|
## Open questions (only the row layout remains)
|
||||||
ordinal — confirm.
|
|
||||||
2. **Row layout.** Capture a real `GetNextQueryResultBuffer2` buffer for an analog summary of a
|
1. ~~**Request params.**~~ **DONE** — see the table above. ValueSelector @ `0x59`,
|
||||||
data-bearing tag (`SysTimeSec`) over a multi-hour window with an interval, then decode against the
|
AggregationType @ `0x5B`, Resolution @ `0x1D` (→ AutoSummaryParameters @ `~0xAA`),
|
||||||
`CAnalogSummaryValue` field set. Likely each row = StartDateTime FILETIME + the 8 typed fields.
|
MaxStates after the tag block. No new QueryType/SummaryType ordinal involved.
|
||||||
|
2. **Row layout (next concrete step).** Capture the `GetNextQueryResultBuffer2` *response* for an
|
||||||
|
analog summary of `SysTimeSec` over a multi-hour window with a 1 h resolution — instrument
|
||||||
|
`ReadMessage` (`instrument-wcf-readmessage`, symmetric to the WriteMessage capture already
|
||||||
|
wired here) and decode against the `CAnalogSummaryValue` field set
|
||||||
|
(StartDateTime + Min/Max/First/Last/ValueCount/TimeGood/Integral/IntegralOfSquares). The
|
||||||
|
request side is no longer a blocker.
|
||||||
|
|
||||||
## Implementation steps (per the project's two-tests discipline)
|
## Implementation steps (per the project's two-tests discipline)
|
||||||
|
|
||||||
@@ -84,9 +154,13 @@ the exact `QueryType` + `SummaryType` + `AutoSummaryParameters` layout, then imp
|
|||||||
`HistorianStateSummary` (per-state contained/partial/entry-count). Reuse `RunQuery` plumbing.
|
`HistorianStateSummary` (per-state contained/partial/entry-count). Reuse `RunQuery` plumbing.
|
||||||
5. Golden-byte test on the parser + gated live test on `localhost` (assert non-empty, fields sane).
|
5. Golden-byte test on the parser + gated live test on `localhost` (assert non-empty, fields sane).
|
||||||
|
|
||||||
## Why stop here this session
|
## State of play
|
||||||
|
|
||||||
`UnpackFromValueBuffer` is reader-call-based (no literal offset table), so a correct parser needs a
|
The **request side is fully recovered** from real bytes (table above) — the managed
|
||||||
**captured real buffer** to decode against — that's the next concrete action, not a guess. Per
|
`HistorianDataQueryRequest` builder can now set `ValueSelector`/`AggregationType`/`Resolution`
|
||||||
project rule ("never guess wire bytes; leave throwing until evidence supports it") no summary code
|
(+ `MaxStates` for state) against ground truth rather than guesses. What remains is the
|
||||||
was added to `src/` yet. Everything above is the evidence needed to implement directly.
|
**response row layout**: `CAnalogSummaryValue.UnpackFromValueBuffer` is reader-call-based (no
|
||||||
|
literal offset table), so the parser needs a captured real *response* buffer to decode against
|
||||||
|
(step 2 in Open questions — `instrument-wcf-readmessage`, already wired alongside the WriteMessage
|
||||||
|
capture). Per project rule ("never guess wire bytes; leave throwing until evidence supports it")
|
||||||
|
no summary code is in `src/` yet — that lands once the response fixture exists.
|
||||||
|
|||||||
@@ -0,0 +1,150 @@
|
|||||||
|
<#
|
||||||
|
.SYNOPSIS
|
||||||
|
Captures the native AVEVA client's StartQuery2 request bytes for analog/state
|
||||||
|
summary queries (HCAL roadmap R1.8/R1.9) so the managed SDK's summary request
|
||||||
|
shape can be decoded against ground truth instead of guessed.
|
||||||
|
|
||||||
|
.DESCRIPTION
|
||||||
|
Drives the .NET-Framework NativeTraceHarness against the live Historian with an
|
||||||
|
IL-rewritten copy of aahClientManaged.dll whose ClientMessageEncoder.WriteMessage
|
||||||
|
is instrumented to log every outgoing MDAS body (the same pipeline that produced
|
||||||
|
every other proven request shape). For each candidate HistoryQueryArgs config it
|
||||||
|
writes a per-config NDJSON capture under
|
||||||
|
artifacts/reverse-engineering/instrumented-wcf-writemessage-summary/ (gitignored).
|
||||||
|
|
||||||
|
The default matrix is:
|
||||||
|
- baseline-full : RetrievalMode=Full (the known-good non-summary request)
|
||||||
|
- analog-avg : RetrievalMode=Cyclic + ValueSelector=Average + Resolution
|
||||||
|
- analog-min : RetrievalMode=Cyclic + ValueSelector=Minimum + Resolution
|
||||||
|
- analog-agg-avg : RetrievalMode=Cyclic + AggregationType=Average + Resolution
|
||||||
|
- state-summary : RetrievalMode=Cyclic + MaxStates>0 + Resolution
|
||||||
|
|
||||||
|
Diff any candidate against baseline-full (scripts/decode-summary-capture.py) to read
|
||||||
|
off the exact QueryType / SummaryType / AutoSummaryParameters bytes the native client
|
||||||
|
sets for a summary, then implement the managed request against that.
|
||||||
|
|
||||||
|
.NOTES
|
||||||
|
Artifacts are diagnostic. Sanitize before copying anything into docs/ — never commit
|
||||||
|
raw capture NDJSON, credentials, hostnames, or customer tag names.
|
||||||
|
#>
|
||||||
|
[CmdletBinding()]
|
||||||
|
param(
|
||||||
|
[string]$ServerName = "localhost",
|
||||||
|
[int]$TcpPort = 32568,
|
||||||
|
# SysTimeSec is the local data-bearing system tag (OtOpcUaParityTest_001.Counter is stale/empty).
|
||||||
|
[string]$TagName = "SysTimeSec",
|
||||||
|
[int]$LookbackMinutes = 240,
|
||||||
|
[int]$MaxRows = 4,
|
||||||
|
# 1-hour summary cycle in 100ns ticks (1h = 36,000,000,000 ticks).
|
||||||
|
[uint64]$ResolutionTicks = 36000000000,
|
||||||
|
[string]$Configuration = "Debug",
|
||||||
|
# Restrict the run to a single named config from the matrix (default: run all).
|
||||||
|
[string]$OnlyConfig = "",
|
||||||
|
# Also instrument ReadMessage so each capture includes the incoming WCF response bodies
|
||||||
|
# (the GetNextQueryResultBuffer2 pResultBuff summary rows). Decoded by decode-summary-response.py.
|
||||||
|
[switch]$WithResponse
|
||||||
|
)
|
||||||
|
|
||||||
|
$ErrorActionPreference = "Stop"
|
||||||
|
$repoRoot = Split-Path -Parent $PSScriptRoot
|
||||||
|
Set-Location $repoRoot
|
||||||
|
|
||||||
|
$reProj = Join-Path $repoRoot "tools\AVEVA.Historian.ReverseEngineering\AVEVA.Historian.ReverseEngineering.csproj"
|
||||||
|
$harnessProj = Join-Path $repoRoot "tools\AVEVA.Historian.NativeTraceHarness\AVEVA.Historian.NativeTraceHarness.csproj"
|
||||||
|
$instrProj = Join-Path $repoRoot "tools\AVEVA.Historian.ReverseInstrumentation\AVEVA.Historian.ReverseInstrumentation.csproj"
|
||||||
|
|
||||||
|
$captureDir = Join-Path $repoRoot "artifacts\reverse-engineering\instrumented-wcf-writemessage-summary"
|
||||||
|
$currentCopy = Join-Path $captureDir "current-copy"
|
||||||
|
$instrDll = Join-Path $captureDir "aahClientManaged.dll"
|
||||||
|
|
||||||
|
Write-Host "== Building tooling ($Configuration) ==" -ForegroundColor Cyan
|
||||||
|
dotnet build $reProj -c $Configuration --nologo -v q | Out-Null
|
||||||
|
dotnet build $instrProj -c $Configuration --nologo -v q | Out-Null
|
||||||
|
dotnet build $harnessProj -c $Configuration --nologo -v q | Out-Null
|
||||||
|
|
||||||
|
$instrSourceDll = Get-ChildItem -Recurse (Join-Path $repoRoot "tools\AVEVA.Historian.ReverseInstrumentation\bin\$Configuration") `
|
||||||
|
-Filter "AVEVA.Historian.ReverseInstrumentation.dll" | Select-Object -First 1 -ExpandProperty FullName
|
||||||
|
if (-not $instrSourceDll) { throw "ReverseInstrumentation.dll not found under bin\$Configuration." }
|
||||||
|
|
||||||
|
Write-Host "== Instrumenting WriteMessage$(if ($WithResponse) { ' + ReadMessage' }) ==" -ForegroundColor Cyan
|
||||||
|
New-Item -ItemType Directory -Force -Path $captureDir | Out-Null
|
||||||
|
if ($WithResponse) {
|
||||||
|
# Chain via a distinct intermediate file (reading+writing the same path drops the second
|
||||||
|
# hook on the mixed-mode native image). Final dll carries both hooks with distinct Phase
|
||||||
|
# strings: WCF.WriteMessage.Body and WCF.ReadMessage.Body.
|
||||||
|
$writeOnly = Join-Path $captureDir "aahClientManaged.write.dll"
|
||||||
|
dotnet run --no-build -c $Configuration --project $reProj -- `
|
||||||
|
instrument-wcf-writemessage (Join-Path $repoRoot "current\aahClientManaged.dll") $writeOnly | Out-Null
|
||||||
|
dotnet run --no-build -c $Configuration --project $reProj -- `
|
||||||
|
instrument-wcf-readmessage $writeOnly $instrDll | Out-Null
|
||||||
|
} else {
|
||||||
|
dotnet run --no-build -c $Configuration --project $reProj -- `
|
||||||
|
instrument-wcf-writemessage (Join-Path $repoRoot "current\aahClientManaged.dll") $instrDll | Out-Null
|
||||||
|
}
|
||||||
|
|
||||||
|
Write-Host "== Staging current-copy ==" -ForegroundColor Cyan
|
||||||
|
# Mirror current/ into current-copy, then overwrite the managed dll with the instrumented
|
||||||
|
# build and drop the strong-named logger assembly alongside it so the injected call binds.
|
||||||
|
robocopy (Join-Path $repoRoot "current") $currentCopy /MIR /NJH /NJS /NDL /NP /NC /NS | Out-Null
|
||||||
|
Copy-Item -Force $instrDll (Join-Path $currentCopy "aahClientManaged.dll")
|
||||||
|
Copy-Item -Force $instrSourceDll (Join-Path $currentCopy "AVEVA.Historian.ReverseInstrumentation.dll")
|
||||||
|
|
||||||
|
# Candidate matrix: name + harness arg list. Summary configs all use Cyclic + a resolution;
|
||||||
|
# the differentiator is which summary knob is set.
|
||||||
|
$matrix = @(
|
||||||
|
@{ Name = "baseline-full"; Args = @("--retrieval-mode", "Full") },
|
||||||
|
@{ Name = "analog-avg"; Args = @("--retrieval-mode", "Cyclic", "--value-selector", "Average", "--resolution-ticks", "$ResolutionTicks") },
|
||||||
|
@{ Name = "analog-min"; Args = @("--retrieval-mode", "Cyclic", "--value-selector", "Minimum", "--resolution-ticks", "$ResolutionTicks") },
|
||||||
|
@{ Name = "analog-max"; Args = @("--retrieval-mode", "Cyclic", "--value-selector", "Maximum", "--resolution-ticks", "$ResolutionTicks") },
|
||||||
|
@{ Name = "analog-integral"; Args = @("--retrieval-mode", "Cyclic", "--value-selector", "Integral", "--resolution-ticks", "$ResolutionTicks") },
|
||||||
|
@{ Name = "mode-integral"; Args = @("--retrieval-mode", "Integral", "--resolution-ticks", "$ResolutionTicks") },
|
||||||
|
@{ Name = "mode-twavg"; Args = @("--retrieval-mode", "TimeWeightedAverage", "--resolution-ticks", "$ResolutionTicks") },
|
||||||
|
@{ Name = "analog-agg-avg"; Args = @("--retrieval-mode", "Cyclic", "--aggregation-type", "Average", "--resolution-ticks", "$ResolutionTicks") },
|
||||||
|
@{ Name = "state-summary"; Args = @("--retrieval-mode", "Cyclic", "--max-states", "10", "--resolution-ticks", "$ResolutionTicks") }
|
||||||
|
)
|
||||||
|
|
||||||
|
if ($OnlyConfig) { $matrix = $matrix | Where-Object { $_.Name -eq $OnlyConfig } }
|
||||||
|
if (-not $matrix) { throw "No matrix entry named '$OnlyConfig'." }
|
||||||
|
|
||||||
|
$harnessDll = Join-Path $currentCopy "aahClientManaged.dll"
|
||||||
|
$summary = @()
|
||||||
|
|
||||||
|
foreach ($cfg in $matrix) {
|
||||||
|
$name = $cfg.Name
|
||||||
|
$capturePath = Join-Path $captureDir "summary-capture-$name-latest.ndjson"
|
||||||
|
if (Test-Path $capturePath) { Remove-Item -Force $capturePath }
|
||||||
|
$env:AVEVA_HISTORIAN_RE_CAPTURE = $capturePath
|
||||||
|
|
||||||
|
Write-Host "== Capturing: $name ==" -ForegroundColor Green
|
||||||
|
$harnessArgs = @(
|
||||||
|
"--scenario", "history",
|
||||||
|
"--server-name", $ServerName,
|
||||||
|
"--tcp-port", "$TcpPort",
|
||||||
|
"--tag", $TagName,
|
||||||
|
"--lookback-minutes", "$LookbackMinutes",
|
||||||
|
"--max-rows", "$MaxRows",
|
||||||
|
"--current-dir", $currentCopy,
|
||||||
|
"--managed-dll-path", $harnessDll
|
||||||
|
) + $cfg.Args
|
||||||
|
|
||||||
|
# Don't let a single config that errors (e.g. state summary on an analog tag) abort the
|
||||||
|
# whole matrix, and don't treat dotnet's stderr noise as a terminating error.
|
||||||
|
try {
|
||||||
|
$prevEap = $ErrorActionPreference
|
||||||
|
$ErrorActionPreference = "Continue"
|
||||||
|
& dotnet run --no-build -c $Configuration --project $harnessProj -- @harnessArgs 2>&1 | Out-Null
|
||||||
|
} catch {
|
||||||
|
Write-Host " (config '$name' raised: $($_.Exception.Message))" -ForegroundColor Yellow
|
||||||
|
} finally {
|
||||||
|
$ErrorActionPreference = $prevEap
|
||||||
|
}
|
||||||
|
$recCount = if (Test-Path $capturePath) { (Get-Content $capturePath | Where-Object { $_.Trim() }).Count } else { 0 }
|
||||||
|
Write-Host " -> $recCount records -> $capturePath"
|
||||||
|
$summary += [pscustomobject]@{ Config = $name; Records = $recCount; Capture = $capturePath }
|
||||||
|
}
|
||||||
|
|
||||||
|
Remove-Item Env:\AVEVA_HISTORIAN_RE_CAPTURE -ErrorAction SilentlyContinue
|
||||||
|
|
||||||
|
Write-Host "`n== Capture summary ==" -ForegroundColor Cyan
|
||||||
|
$summary | Format-Table -AutoSize
|
||||||
|
Write-Host "Decode with: python scripts\decode-summary-capture.py" -ForegroundColor Cyan
|
||||||
@@ -0,0 +1,115 @@
|
|||||||
|
"""Decoder for the analog/state summary request capture (HCAL roadmap R1.8/R1.9).
|
||||||
|
|
||||||
|
Reads the per-config NDJSON captures produced by scripts/Capture-SummaryRequest.ps1
|
||||||
|
under artifacts/reverse-engineering/instrumented-wcf-writemessage-summary/, extracts
|
||||||
|
the Retr/StartQuery2 `pRequestBuff` payload from each, hex-dumps it, and diffs every
|
||||||
|
summary candidate against the baseline-full request so the differing bytes (the native
|
||||||
|
QueryType / SummaryType / AutoSummaryParameters fields) stand out.
|
||||||
|
|
||||||
|
Output is diagnostic. The only printed strings are the SDK-chosen system tag name and
|
||||||
|
protocol field markers — sanitize before copying any of it into docs/.
|
||||||
|
"""
|
||||||
|
import base64
|
||||||
|
import json
|
||||||
|
import sys
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
REPO_ROOT = Path(__file__).resolve().parent.parent
|
||||||
|
CAPTURE_DIR = REPO_ROOT / "artifacts" / "reverse-engineering" / "instrumented-wcf-writemessage-summary"
|
||||||
|
|
||||||
|
ACTION = b"aa/Retr/StartQuery2"
|
||||||
|
PARAM = b"pRequestBuff"
|
||||||
|
|
||||||
|
|
||||||
|
def extract_request_buffer(records):
|
||||||
|
"""Return the pRequestBuff bytes from the first StartQuery2 write record, or None."""
|
||||||
|
for rec in records:
|
||||||
|
if rec.get("Phase") != "WCF.WriteMessage.Body":
|
||||||
|
continue
|
||||||
|
body = base64.b64decode(rec["Base64"])
|
||||||
|
if ACTION not in body:
|
||||||
|
continue
|
||||||
|
i = body.find(PARAM)
|
||||||
|
if i < 0:
|
||||||
|
continue
|
||||||
|
i += len(PARAM)
|
||||||
|
marker = body[i]
|
||||||
|
# MDAS length markers (same scheme as the write decoder).
|
||||||
|
if marker == 0x9E:
|
||||||
|
length = body[i + 1]
|
||||||
|
return body[i + 2:i + 2 + length]
|
||||||
|
if marker == 0x9F:
|
||||||
|
length = int.from_bytes(body[i + 1:i + 3], "little")
|
||||||
|
return body[i + 3:i + 3 + length]
|
||||||
|
if marker == 0xA0:
|
||||||
|
length = int.from_bytes(body[i + 1:i + 3], "little")
|
||||||
|
return body[i + 3:i + 3 + length + 1]
|
||||||
|
return None
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def hexdump(payload, diff_against=None):
|
||||||
|
for off in range(0, len(payload), 16):
|
||||||
|
chunk = payload[off:off + 16]
|
||||||
|
cells = []
|
||||||
|
for j, c in enumerate(chunk):
|
||||||
|
mark = ""
|
||||||
|
if diff_against is not None:
|
||||||
|
k = off + j
|
||||||
|
if k >= len(diff_against) or diff_against[k] != c:
|
||||||
|
mark = "*"
|
||||||
|
cells.append(f"{c:02X}{mark}")
|
||||||
|
hp = " ".join(cells)
|
||||||
|
ap = "".join(chr(c) if 32 <= c < 127 else "." for c in chunk)
|
||||||
|
print(f" {off:04X} {hp:<56} |{ap}|")
|
||||||
|
|
||||||
|
|
||||||
|
def load(path):
|
||||||
|
with path.open(encoding="utf-8-sig") as fh:
|
||||||
|
return [json.loads(line) for line in fh if line.strip()]
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> int:
|
||||||
|
if not CAPTURE_DIR.exists():
|
||||||
|
print(f"Capture dir not found: {CAPTURE_DIR}")
|
||||||
|
print("Run scripts/Capture-SummaryRequest.ps1 first.")
|
||||||
|
return 1
|
||||||
|
|
||||||
|
captures = sorted(CAPTURE_DIR.glob("summary-capture-*-latest.ndjson"))
|
||||||
|
if not captures:
|
||||||
|
print(f"No capture files in {CAPTURE_DIR}")
|
||||||
|
return 1
|
||||||
|
|
||||||
|
buffers = {}
|
||||||
|
for path in captures:
|
||||||
|
name = path.stem.replace("summary-capture-", "").replace("-latest", "")
|
||||||
|
records = load(path)
|
||||||
|
buf = extract_request_buffer(records)
|
||||||
|
buffers[name] = buf
|
||||||
|
status = f"{len(buf)} bytes" if buf else "<no StartQuery2 request found>"
|
||||||
|
print(f"{name:<18} records={len(records):>3} pRequestBuff={status}")
|
||||||
|
|
||||||
|
baseline = buffers.get("baseline-full")
|
||||||
|
print()
|
||||||
|
if not baseline:
|
||||||
|
print("No baseline-full request buffer captured; cannot diff. Dumping each raw.")
|
||||||
|
for name, buf in buffers.items():
|
||||||
|
if buf:
|
||||||
|
print(f"\n== {name} pRequestBuff ({len(buf)} bytes) ==")
|
||||||
|
hexdump(buf)
|
||||||
|
return 0
|
||||||
|
|
||||||
|
print(f"== baseline-full pRequestBuff ({len(baseline)} bytes) ==")
|
||||||
|
hexdump(baseline)
|
||||||
|
|
||||||
|
for name, buf in buffers.items():
|
||||||
|
if name == "baseline-full" or not buf:
|
||||||
|
continue
|
||||||
|
print(f"\n== {name} pRequestBuff ({len(buf)} bytes) — '*' marks bytes differing from baseline ==")
|
||||||
|
hexdump(buf, diff_against=baseline)
|
||||||
|
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
sys.exit(main())
|
||||||
@@ -0,0 +1,123 @@
|
|||||||
|
"""Decode the GetNextQueryResultBuffer2 *response* for an analog summary (HCAL R1.8).
|
||||||
|
|
||||||
|
Reads the both-hooks capture produced by
|
||||||
|
scripts/Capture-SummaryRequest.ps1 -OnlyConfig analog-avg -WithResponse
|
||||||
|
finds the ReadMessage record carrying GetNextQueryResultBuffer2Response, extracts the
|
||||||
|
`pResultBuff` payload, hex-dumps it, and annotates every 8-byte window that decodes to a
|
||||||
|
known ground-truth value (the AnalogSummaryHistory row for SysTimeSec) so the field offsets
|
||||||
|
of CAnalogSummaryValue can be read off directly.
|
||||||
|
|
||||||
|
Output is diagnostic; the only printed strings are the SDK-chosen system tag name and field
|
||||||
|
markers. Sanitize before copying into docs/.
|
||||||
|
"""
|
||||||
|
import base64
|
||||||
|
import json
|
||||||
|
import struct
|
||||||
|
import sys
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
REPO_ROOT = Path(__file__).resolve().parent.parent
|
||||||
|
# Config name (analog-avg / analog-min / analog-max / …) selectable via argv[1].
|
||||||
|
CONFIG = sys.argv[1] if len(sys.argv) > 1 else "analog-avg"
|
||||||
|
CAPTURE = (REPO_ROOT / "artifacts" / "reverse-engineering"
|
||||||
|
/ "instrumented-wcf-writemessage-summary" / f"summary-capture-{CONFIG}-latest.ndjson")
|
||||||
|
|
||||||
|
RESP = b"GetNextQueryResultBuffer2Response"
|
||||||
|
PARAM = b"pResultBuff"
|
||||||
|
|
||||||
|
# Ground-truth values from AnalogSummaryHistory(SysTimeSec, 1h cycle) — used to label offsets.
|
||||||
|
KNOWN_DOUBLES = {
|
||||||
|
31.0: "31.0 (First/Last/Average)",
|
||||||
|
100.0: "100.0 (PercentGood)",
|
||||||
|
0.031: "0.031 (Integral)",
|
||||||
|
111600.0: "111600.0 (Integral, full-cycle)",
|
||||||
|
1.0: "1.0 (ValueCount as double?)",
|
||||||
|
}
|
||||||
|
KNOWN_U32 = {
|
||||||
|
1: "ValueCount=1",
|
||||||
|
192: "OPCQuality=192",
|
||||||
|
100: "PercentGood=100",
|
||||||
|
9: "version=9",
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def extract_param(body, param):
|
||||||
|
i = body.find(param)
|
||||||
|
if i < 0:
|
||||||
|
return None
|
||||||
|
i += len(param)
|
||||||
|
marker = body[i]
|
||||||
|
if marker == 0x9E:
|
||||||
|
length = body[i + 1]
|
||||||
|
return body[i + 2:i + 2 + length]
|
||||||
|
if marker == 0x9F:
|
||||||
|
length = int.from_bytes(body[i + 1:i + 3], "little")
|
||||||
|
return body[i + 3:i + 3 + length]
|
||||||
|
if marker == 0xA0:
|
||||||
|
length = int.from_bytes(body[i + 1:i + 3], "little")
|
||||||
|
return body[i + 3:i + 3 + length + 1]
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> int:
|
||||||
|
if not CAPTURE.exists():
|
||||||
|
print(f"Capture not found: {CAPTURE}")
|
||||||
|
print("Run: scripts/Capture-SummaryRequest.ps1 -OnlyConfig analog-avg -WithResponse")
|
||||||
|
return 1
|
||||||
|
|
||||||
|
with CAPTURE.open(encoding="utf-8-sig") as fh:
|
||||||
|
records = [json.loads(line) for line in fh if line.strip()]
|
||||||
|
|
||||||
|
payload = None
|
||||||
|
for rec in records:
|
||||||
|
if rec.get("Phase") != "WCF.ReadMessage.Body":
|
||||||
|
continue
|
||||||
|
body = base64.b64decode(rec["Base64"])
|
||||||
|
if RESP not in body:
|
||||||
|
continue
|
||||||
|
payload = extract_param(body, PARAM)
|
||||||
|
break
|
||||||
|
|
||||||
|
if payload is None:
|
||||||
|
print("No GetNextQueryResultBuffer2Response / pResultBuff found in capture.")
|
||||||
|
return 2
|
||||||
|
|
||||||
|
print(f"pResultBuff: {len(payload)} bytes")
|
||||||
|
if len(payload) >= 6:
|
||||||
|
version = int.from_bytes(payload[0:2], "little")
|
||||||
|
row_count = int.from_bytes(payload[2:6], "little")
|
||||||
|
print(f" header: version={version} rowCount={row_count}")
|
||||||
|
print()
|
||||||
|
|
||||||
|
# Annotated hex dump.
|
||||||
|
for off in range(0, len(payload), 16):
|
||||||
|
chunk = payload[off:off + 16]
|
||||||
|
hp = " ".join(f"{c:02X}" for c in chunk)
|
||||||
|
ap = "".join(chr(c) if 32 <= c < 127 else "." for c in chunk)
|
||||||
|
print(f" {off:04X} {hp:<48} |{ap}|")
|
||||||
|
|
||||||
|
# Scan every 8-byte window for known doubles, and every 4-byte window for known u32s.
|
||||||
|
print("\n== Known-value hits (offset -> field) ==")
|
||||||
|
for off in range(0, len(payload) - 7):
|
||||||
|
val = struct.unpack_from("<d", payload, off)[0]
|
||||||
|
for known, label in KNOWN_DOUBLES.items():
|
||||||
|
if val == known or (known != 0 and abs(val - known) < 1e-9 * max(1.0, abs(known))):
|
||||||
|
print(f" 0x{off:04X} double {val!r:>14} -> {label}")
|
||||||
|
for off in range(0, len(payload) - 3):
|
||||||
|
val = int.from_bytes(payload[off:off + 4], "little")
|
||||||
|
if val in KNOWN_U32:
|
||||||
|
print(f" 0x{off:04X} uint32 {val:>14} -> {KNOWN_U32[val]}")
|
||||||
|
|
||||||
|
# FILETIME windows (plausible 2026 timestamps: 0x01DC.. high dword).
|
||||||
|
print("\n== Plausible FILETIME windows (Int64, year ~2020-2030) ==")
|
||||||
|
for off in range(0, len(payload) - 7):
|
||||||
|
ft = int.from_bytes(payload[off:off + 8], "little")
|
||||||
|
# FILETIME for 2020-01-01 ~= 0x01D5BF.. ; 2030 ~= 0x01E5.. — gate by high word.
|
||||||
|
if 0x01D5_0000_0000_0000 <= ft <= 0x01E6_0000_0000_0000:
|
||||||
|
print(f" 0x{off:04X} filetime 0x{ft:016X}")
|
||||||
|
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
sys.exit(main())
|
||||||
@@ -36,6 +36,13 @@ internal static class Program
|
|||||||
string runtimeMethodPointerFilters = GetArg(args, "--runtime-method-pointer-filters")
|
string runtimeMethodPointerFilters = GetArg(args, "--runtime-method-pointer-filters")
|
||||||
?? "StartDataQuery;StartQuery;GetNextRow;StartEventQuery";
|
?? "StartDataQuery;StartQuery;GetNextRow;StartEventQuery";
|
||||||
ulong resolutionTicks = ulong.TryParse(GetArg(args, "--resolution-ticks"), out ulong parsedResolutionTicks) ? parsedResolutionTicks : 0;
|
ulong resolutionTicks = ulong.TryParse(GetArg(args, "--resolution-ticks"), out ulong parsedResolutionTicks) ? parsedResolutionTicks : 0;
|
||||||
|
// Summary-query knobs on HistoryQueryArgs (R1.8/R1.9 capture). Left null/0 = not set,
|
||||||
|
// so a normal Full read is unaffected. ValueSelector/AggregationType/MaxStates/Filter
|
||||||
|
// are the native properties that turn a Cyclic/Full query into an analog/state summary.
|
||||||
|
string? valueSelectorName = GetArg(args, "--value-selector");
|
||||||
|
string? aggregationTypeName = GetArg(args, "--aggregation-type");
|
||||||
|
uint maxStates = uint.TryParse(GetArg(args, "--max-states"), out uint parsedMaxStates) ? parsedMaxStates : 0;
|
||||||
|
string? historyFilter = GetArg(args, "--filter");
|
||||||
DateTime endUtc = TryParseUtc(GetArg(args, "--end-utc")) ?? DateTime.UtcNow;
|
DateTime endUtc = TryParseUtc(GetArg(args, "--end-utc")) ?? DateTime.UtcNow;
|
||||||
DateTime startUtc = TryParseUtc(GetArg(args, "--start-utc")) ?? endUtc.AddMinutes(-lookbackMinutes);
|
DateTime startUtc = TryParseUtc(GetArg(args, "--start-utc")) ?? endUtc.AddMinutes(-lookbackMinutes);
|
||||||
|
|
||||||
@@ -789,6 +796,26 @@ internal static class Program
|
|||||||
{
|
{
|
||||||
SetProperty(queryArgs, "Resolution", resolutionTicks);
|
SetProperty(queryArgs, "Resolution", resolutionTicks);
|
||||||
}
|
}
|
||||||
|
// Summary knobs — only set when explicitly supplied so plain reads are untouched.
|
||||||
|
if (valueSelectorName is not null)
|
||||||
|
{
|
||||||
|
Type valueSelectorType = GetType(assembly, "ArchestrA.HistorianValueSelector");
|
||||||
|
SetProperty(queryArgs, "ValueSelector", Enum.Parse(valueSelectorType, valueSelectorName, ignoreCase: true));
|
||||||
|
}
|
||||||
|
if (aggregationTypeName is not null)
|
||||||
|
{
|
||||||
|
Type aggregationType = GetType(assembly, "ArchestrA.HistorianAggregationType");
|
||||||
|
SetProperty(queryArgs, "AggregationType", Enum.Parse(aggregationType, aggregationTypeName, ignoreCase: true));
|
||||||
|
}
|
||||||
|
if (maxStates > 0)
|
||||||
|
{
|
||||||
|
// HistoryQueryArgs.MaxStates is a UInt16 on the native wrapper.
|
||||||
|
SetProperty(queryArgs, "MaxStates", checked((ushort)maxStates));
|
||||||
|
}
|
||||||
|
if (historyFilter is not null)
|
||||||
|
{
|
||||||
|
SetProperty(queryArgs, "Filter", historyFilter);
|
||||||
|
}
|
||||||
snapshots["QueryArgsBeforeStart"] = SnapshotObject(queryArgs);
|
snapshots["QueryArgsBeforeStart"] = SnapshotObject(queryArgs);
|
||||||
|
|
||||||
startError = Activator.CreateInstance(errorType)!;
|
startError = Activator.CreateInstance(errorType)!;
|
||||||
@@ -890,6 +917,10 @@ internal static class Program
|
|||||||
LookbackMinutes = lookbackMinutes,
|
LookbackMinutes = lookbackMinutes,
|
||||||
RetrievalMode = retrievalModeName,
|
RetrievalMode = retrievalModeName,
|
||||||
ResolutionTicks = resolutionTicks,
|
ResolutionTicks = resolutionTicks,
|
||||||
|
ValueSelector = valueSelectorName,
|
||||||
|
AggregationType = aggregationTypeName,
|
||||||
|
MaxStates = maxStates,
|
||||||
|
HistoryFilter = historyFilter,
|
||||||
StartUtc = startUtc.ToString("O"),
|
StartUtc = startUtc.ToString("O"),
|
||||||
EndUtc = endUtc.ToString("O"),
|
EndUtc = endUtc.ToString("O"),
|
||||||
OpenSuccess = openSuccess,
|
OpenSuccess = openSuccess,
|
||||||
|
|||||||
Reference in New Issue
Block a user