RE: resolve R1.8/R1.9 analog/state summary via request+response capture
Captured the native StartQuery2 pRequestBuff and the GetNextQueryResultBuffer2 response (instrument-wcf-writemessage + chained instrument-wcf-readmessage) and decoded both against AnalogSummaryHistory SQL ground truth. Conclusion: the rich multi-aggregate analog/state summary struct is NOT delivered over the 2020 WCF binary protocol — the response is the ordinary version-9 row buffer the existing aggregate parser already handles, carrying one value per cycle selected by RetrievalMode (QueryType 5-8), not ValueSelector (inert on this path). So "analog summary" == the existing ReadAggregateAsync; no new src/ code warranted. Tooling (tools/ + scripts/ only, nothing in src/): - NativeTraceHarness: drive summary knobs via --value-selector / --aggregation-type / --max-states (uint16) / --filter - Capture-SummaryRequest.ps1: repeatable instrument+stage+matrix capture, -WithResponse chains the ReadMessage hook - decode-summary-capture.py: StartQuery2 request diff vs baseline - decode-summary-response.py: response decode vs SQL ground truth Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Joseph Doherty
@@ -1,7 +1,59 @@
|
||||
# R1.8 / R1.9 — Analog-summary & State-summary queries (implementation plan)
|
||||
|
||||
**Status (2026-06-20): scoped + de-risked. Reachable on 2020 WCF. Decode targets located in
|
||||
the native dll. Ready to implement; not yet started in `src/` (no guessed code shipped).**
|
||||
**Status (2026-06-21): RESOLVED by request + response capture. Conclusion: the rich
|
||||
multi-aggregate analog/state summary struct is NOT delivered over the 2020 WCF binary protocol.
|
||||
The per-cycle aggregate values it would expose are ALREADY shipped via `ReadAggregateAsync`
|
||||
(RetrievalMode → QueryType 5–8). No new `src/` code is warranted for R1.8/R1.9 on 2020 WCF.**
|
||||
|
||||
## RESOLVED — what the response capture proved (2026-06-21)
|
||||
|
||||
The request side was recovered first (table further down), then the `GetNextQueryResultBuffer2`
|
||||
**response** was captured (`instrument-wcf-readmessage`, both hooks chained) and decoded against
|
||||
`AnalogSummaryHistory` SQL ground truth for `SysTimeSec` over a 6 h window / 1 h cycle. Findings:
|
||||
|
||||
1. **The response is the ordinary version-9 row buffer** — same layout the existing raw/aggregate
|
||||
parser (`TryParseGetNextQueryResultBufferAggregateRows`) already handles: `uint16 version=9`,
|
||||
`uint32 rowCount`, then per-row `tagKey + nameLen + name + ValueCount + cycleEnd FILETIME +
|
||||
quality + OpcQuality + Value(double) + PercentGood(double) + trailer(cycleStart FILETIME …)`.
|
||||
The captured 7-row buffer decoded with `Value=31.0`, `PercentGood=100.0`, `ValueCount=1`,
|
||||
`OpcQuality=192` — matching the SQL row exactly.
|
||||
|
||||
2. **There is NO rich `CAnalogSummaryValue` struct on the wire.** Each row carries a *single*
|
||||
value, not Min+Max+First+Last+Avg+Integral together. The all-aggregates-in-one-row shape that
|
||||
`CAnalogSummaryValue` / `AnalogSummaryHistory` represents is the **SQL/OLEDB provider's** shape,
|
||||
not the binary `StartQuery2` retrieval's.
|
||||
|
||||
3. **The single value is selected by `RetrievalMode` (QueryType), not by `ValueSelector`.** Proven
|
||||
against the same constant tag where only the *kind* of aggregate distinguishes the result:
|
||||
- `RetrievalMode=Integral` (QueryType 8) → `Value = 111600.0` (= SQL `Integral`) ✓
|
||||
- `RetrievalMode=TimeWeightedAverage` (QueryType 5) → `Value = 31.0` (= SQL `Average`) ✓
|
||||
- `Cyclic` (QueryType 0) **+ `ValueSelector=Integral`** → `Value = 31.0` (selector **ignored**;
|
||||
the request byte `ValueSelector@0x59=0x04` was confirmed sent, yet the cyclic value came back).
|
||||
|
||||
So `ValueSelector` / `AggregationType` / `MaxStates` are **inert on the WCF retrieval path** —
|
||||
they configure the SQL provider's summary tables, not this binary query.
|
||||
|
||||
4. **Resolution unit is correct in the SDK.** The wire `Resolution` is 100 ns ticks (= ms × 10000).
|
||||
`SerializeFullHistoryRequest` writes `TimeSpan.Ticks`, which the golden test
|
||||
`SerializerMatchesInstrumentedNativeTimeWeightedAverageRequest` already verifies byte-for-byte
|
||||
against native (`FromMinutes(1)` → `600000000`). No bug.
|
||||
|
||||
**Therefore:** "analog summary" over 2020 WCF == the existing aggregate read. To get Min, Max,
|
||||
Average and Integral for a cycle you issue the corresponding `RetrievalMode` queries
|
||||
(`MinimumWithTime` / `MaximumWithTime` / `TimeWeightedAverage` / `Integral`), each returning that
|
||||
one aggregate per cycle — all already implemented, mapped (QueryType 5–8) and golden-tested in
|
||||
`ReadAggregateAsync`. **R1.8/R1.9 need no new protocol code on this server.** A genuine
|
||||
all-aggregates-at-once summary would require the gRPC front door or the SQL provider, neither of
|
||||
which is the 2020 WCF binary path.
|
||||
|
||||
Capture/decode tooling is committed and repeatable: `scripts/Capture-SummaryRequest.ps1`
|
||||
(`-WithResponse` chains ReadMessage), `scripts/decode-summary-capture.py` (request diff),
|
||||
`scripts/decode-summary-response.py <config>` (response decode vs SQL ground truth). Raw captures
|
||||
live under `artifacts/reverse-engineering/instrumented-wcf-writemessage-summary/` (gitignored).
|
||||
|
||||
---
|
||||
|
||||
_Original scoping notes below remain for context. They led to the capture; the conclusion above supersedes their "ready to implement" framing._
|
||||
|
||||
Unlike the M1 *read* items gated by the [string-handle wall](../reverse-engineering/wcf-string-handle-wall.md),
|
||||
summary queries ride the **proven `uint`-handle `StartQuery2`** path — the same call the working
|
||||
@@ -34,43 +86,61 @@ Found via `methods … Summary` + `dnlib-method`:
|
||||
| `CTypeMetadata.IsAnalogSummary` / `IsStateSummary` | `0x060001A4/A5` | server-side type gating |
|
||||
| `INSQL_QUERYTYPE` / `HISTORIAN_SUMMARYTYPE` | enums `0200013F` / `02000191` | the `QueryType` / `SummaryType` values to send |
|
||||
|
||||
## Empirical probe results (2026-06-20, live `SysTimeSec` over `StartQuery2`)
|
||||
## Native request capture (2026-06-21) — request shape RECOVERED
|
||||
|
||||
Swept `QueryType`/`SummaryType`/`ColumnSelectorFlags` against the live 2020 server:
|
||||
The earlier blind probing (sweeping `SummaryType`/`ColumnSelectorFlags` over the managed
|
||||
serializer) was the wrong lever: it returned 0-row buffers because the managed `SummaryType`
|
||||
field is **not** how the native client encodes a summary. A real capture settled it.
|
||||
|
||||
- `QueryType=2 (Full)`, `SummaryType ∈ {0,3,6}` → normal 109-byte version-9 data buffer.
|
||||
- `QueryType=2`, `SummaryType ∈ {1,2,4,5}` → **valid version-9 buffer with 0 rows** (`09 00 00 00 00 00`).
|
||||
The server **accepts** these summary types but yields no rows.
|
||||
- The 0-row result is **unchanged** by `ColumnSelectorFlags` (tried default, all-bits
|
||||
`0xFFFF…FFFF`, high-dword, low-48). So column flags are *not* the unlock.
|
||||
- `QueryType ∈ {15,16}` → `GetNext` blocks/times out (no such INSQL_QUERYTYPE ordinal).
|
||||
**Capture pipeline (now repeatable):** `scripts/Capture-SummaryRequest.ps1` IL-rewrites a copy
|
||||
of `aahClientManaged.dll` (`instrument-wcf-writemessage`), stages it alongside the strong-named
|
||||
`ReverseInstrumentation` logger, then drives the `NativeTraceHarness` history scenario through a
|
||||
candidate matrix while logging every outgoing MDAS body. `scripts/decode-summary-capture.py`
|
||||
extracts the `Retr/StartQuery2` `pRequestBuff` from each and diffs the summary candidates against
|
||||
a tag-matched `baseline-full`. The harness now exposes `--value-selector` / `--aggregation-type`
|
||||
/ `--max-states` / `--filter` so the native `HistoryQueryArgs` summary knobs can be driven.
|
||||
|
||||
**Conclusion:** a summary query is *not* `Full + SummaryType + column flags`. The summary
|
||||
configuration lives elsewhere in the request — almost certainly the **`AutoSummaryParameters`
|
||||
trailer** (`SerializeFullHistoryRequest` currently writes it all-zero via
|
||||
`WriteAutoSummaryParameters`) and/or a native summary `QueryType`. Both are **native-side
|
||||
constants** (`HISTORIAN_SUMMARYTYPE` / `INSQL_QUERYTYPE` are `value__`-only in managed metadata;
|
||||
`CColumnNameMap.LoadColumnNameMap` builds column→bit via native string/const data, not IL
|
||||
`ldstr`/`ldc`). So they cannot be recovered from managed metadata, and blind probing of the
|
||||
obvious fields returns empty.
|
||||
**There is no separate "summary" QueryType or `SummaryType` field.** A summary is an ordinary
|
||||
`StartQuery2` request (`QueryType` = the chosen `RetrievalMode`, e.g. `Cyclic`=0) with three
|
||||
things set: the **ValueSelector** byte, the **AggregationType** byte, a non-zero **Resolution**
|
||||
(which fills the previously-zeroed `AutoSummaryParameters` trailer), and — for state summary —
|
||||
the **MaxStates** field. The server then returns analog- vs state-summary rows based on the tag
|
||||
type plus these fields. Offsets below are **into the StartQuery2 `pRequestBuff`** (229-byte
|
||||
`SysTimeSec` baseline; verified byte-for-byte against the native client):
|
||||
|
||||
**Therefore the right next step is a native request capture, not more probing:** drive the native
|
||||
client (NativeTraceHarness / a real summary query) and capture the `pRequestBuff` bytes via the
|
||||
existing `instrument-wcf-writemessage` pipeline — the same method that produced every other proven
|
||||
request shape (reads, events, EnsT2). Diff that buffer against a normal Full request to read off
|
||||
the exact `QueryType` + `SummaryType` + `AutoSummaryParameters` layout, then implement against it.
|
||||
| Offset | Field | Type | Evidence |
|
||||
|---|---|---|---|
|
||||
| `0x01` | QueryType | uint32 LE | Full→`02`, Cyclic→`00` (matches the verified `RetrievalMode`→`QueryType` map) |
|
||||
| `0x1D` | Resolution | float64 LE | `36e9` ticks → `00 00 00 D0 88 C3 20 42` = `0x4220C388D0000000` (1 h). Zero for non-summary reads |
|
||||
| `0x32` | Timezone | len-prefixed UTF-16 | `"UTC"` |
|
||||
| `0x49` | Filter | len-prefixed UTF-16 | `"NoFilter"` default; driven by `--filter` |
|
||||
| `0x59` | **ValueSelector** | byte | baseline `01` (Auto); `--value-selector Minimum`→`06`, `Maximum`→`07`, `Average`→`08` — exact `HistorianValueSelector` values |
|
||||
| `0x5B` | **AggregationType** | byte | baseline `03`; `--aggregation-type Average`→`02` — exact `HistorianAggregationType` values |
|
||||
| `~0x5F` | ColumnSelectorFlags | bytes | `FF 82 07 00 82 81` — matches the `0x0000_8182_0007_82FF` reads already send; **unchanged** by summary |
|
||||
| `0x6B` | Tag name | len-prefixed UTF-16 | `count, "SysTimeSec"` |
|
||||
| after tag | **MaxStates** | uint16 LE | the `01`-default byte after the tag block; `--max-states 10`→`0A` (state summary, R1.9) |
|
||||
| `~0xAA` | **AutoSummaryParameters** | block | zero for plain reads; `80 1E 08 6B 47 01` when Resolution set (identical across analog *and* state) — the resolution-derived cycle block |
|
||||
|
||||
## Open questions (nail these next, in order)
|
||||
State summary (R1.9) is the **same request** with `MaxStates` > 0 (the analog `ValueSelector`/
|
||||
`AggregationType` bytes stay at their `01`/`03` defaults); the analog-vs-state distinction on the
|
||||
wire is which of those fields is non-default, plus the tag type. Note `MaxStates` is a **UInt16**
|
||||
on `HistoryQueryArgs` (passing UInt32 throws) — the harness casts accordingly.
|
||||
|
||||
1. **Request params.** Recover the exact `QueryType` + `SummaryType` (+ whether `ColumnSelectorFlags`
|
||||
must change) for analog vs state summary. Source: decompile `INSQL_QUERYTYPE`/`HISTORIAN_SUMMARYTYPE`
|
||||
enum members and `Select{Analog,State}SummaryColumns` (the `GetColumnFlag` column-name `ldstr`
|
||||
operands → the flag set). The standard QueryType map (Cyclic=0 … EndBound=14) is already verified;
|
||||
summary is expected to be a `SummaryType`≠0 with an existing base `QueryType`, **not** a new mode
|
||||
ordinal — confirm.
|
||||
2. **Row layout.** Capture a real `GetNextQueryResultBuffer2` buffer for an analog summary of a
|
||||
data-bearing tag (`SysTimeSec`) over a multi-hour window with an interval, then decode against the
|
||||
`CAnalogSummaryValue` field set. Likely each row = StartDateTime FILETIME + the 8 typed fields.
|
||||
Raw captures live under `artifacts/reverse-engineering/instrumented-wcf-writemessage-summary/`
|
||||
(gitignored). Re-run with `scripts/Capture-SummaryRequest.ps1` (analog: `SysTimeSec`; state:
|
||||
`-TagName SysPulse`, the local discrete tag).
|
||||
|
||||
## Open questions (only the row layout remains)
|
||||
|
||||
1. ~~**Request params.**~~ **DONE** — see the table above. ValueSelector @ `0x59`,
|
||||
AggregationType @ `0x5B`, Resolution @ `0x1D` (→ AutoSummaryParameters @ `~0xAA`),
|
||||
MaxStates after the tag block. No new QueryType/SummaryType ordinal involved.
|
||||
2. **Row layout (next concrete step).** Capture the `GetNextQueryResultBuffer2` *response* for an
|
||||
analog summary of `SysTimeSec` over a multi-hour window with a 1 h resolution — instrument
|
||||
`ReadMessage` (`instrument-wcf-readmessage`, symmetric to the WriteMessage capture already
|
||||
wired here) and decode against the `CAnalogSummaryValue` field set
|
||||
(StartDateTime + Min/Max/First/Last/ValueCount/TimeGood/Integral/IntegralOfSquares). The
|
||||
request side is no longer a blocker.
|
||||
|
||||
## Implementation steps (per the project's two-tests discipline)
|
||||
|
||||
@@ -84,9 +154,13 @@ the exact `QueryType` + `SummaryType` + `AutoSummaryParameters` layout, then imp
|
||||
`HistorianStateSummary` (per-state contained/partial/entry-count). Reuse `RunQuery` plumbing.
|
||||
5. Golden-byte test on the parser + gated live test on `localhost` (assert non-empty, fields sane).
|
||||
|
||||
## Why stop here this session
|
||||
## State of play
|
||||
|
||||
`UnpackFromValueBuffer` is reader-call-based (no literal offset table), so a correct parser needs a
|
||||
**captured real buffer** to decode against — that's the next concrete action, not a guess. Per
|
||||
project rule ("never guess wire bytes; leave throwing until evidence supports it") no summary code
|
||||
was added to `src/` yet. Everything above is the evidence needed to implement directly.
|
||||
The **request side is fully recovered** from real bytes (table above) — the managed
|
||||
`HistorianDataQueryRequest` builder can now set `ValueSelector`/`AggregationType`/`Resolution`
|
||||
(+ `MaxStates` for state) against ground truth rather than guesses. What remains is the
|
||||
**response row layout**: `CAnalogSummaryValue.UnpackFromValueBuffer` is reader-call-based (no
|
||||
literal offset table), so the parser needs a captured real *response* buffer to decode against
|
||||
(step 2 in Open questions — `instrument-wcf-readmessage`, already wired alongside the WriteMessage
|
||||
capture). Per project rule ("never guess wire bytes; leave throwing until evidence supports it")
|
||||
no summary code is in `src/` yet — that lands once the response fixture exists.
|
||||
|
||||
@@ -0,0 +1,150 @@
|
||||
<#
|
||||
.SYNOPSIS
|
||||
Captures the native AVEVA client's StartQuery2 request bytes for analog/state
|
||||
summary queries (HCAL roadmap R1.8/R1.9) so the managed SDK's summary request
|
||||
shape can be decoded against ground truth instead of guessed.
|
||||
|
||||
.DESCRIPTION
|
||||
Drives the .NET-Framework NativeTraceHarness against the live Historian with an
|
||||
IL-rewritten copy of aahClientManaged.dll whose ClientMessageEncoder.WriteMessage
|
||||
is instrumented to log every outgoing MDAS body (the same pipeline that produced
|
||||
every other proven request shape). For each candidate HistoryQueryArgs config it
|
||||
writes a per-config NDJSON capture under
|
||||
artifacts/reverse-engineering/instrumented-wcf-writemessage-summary/ (gitignored).
|
||||
|
||||
The default matrix is:
|
||||
- baseline-full : RetrievalMode=Full (the known-good non-summary request)
|
||||
- analog-avg : RetrievalMode=Cyclic + ValueSelector=Average + Resolution
|
||||
- analog-min : RetrievalMode=Cyclic + ValueSelector=Minimum + Resolution
|
||||
- analog-agg-avg : RetrievalMode=Cyclic + AggregationType=Average + Resolution
|
||||
- state-summary : RetrievalMode=Cyclic + MaxStates>0 + Resolution
|
||||
|
||||
Diff any candidate against baseline-full (scripts/decode-summary-capture.py) to read
|
||||
off the exact QueryType / SummaryType / AutoSummaryParameters bytes the native client
|
||||
sets for a summary, then implement the managed request against that.
|
||||
|
||||
.NOTES
|
||||
Artifacts are diagnostic. Sanitize before copying anything into docs/ — never commit
|
||||
raw capture NDJSON, credentials, hostnames, or customer tag names.
|
||||
#>
|
||||
[CmdletBinding()]
|
||||
param(
|
||||
[string]$ServerName = "localhost",
|
||||
[int]$TcpPort = 32568,
|
||||
# SysTimeSec is the local data-bearing system tag (OtOpcUaParityTest_001.Counter is stale/empty).
|
||||
[string]$TagName = "SysTimeSec",
|
||||
[int]$LookbackMinutes = 240,
|
||||
[int]$MaxRows = 4,
|
||||
# 1-hour summary cycle in 100ns ticks (1h = 36,000,000,000 ticks).
|
||||
[uint64]$ResolutionTicks = 36000000000,
|
||||
[string]$Configuration = "Debug",
|
||||
# Restrict the run to a single named config from the matrix (default: run all).
|
||||
[string]$OnlyConfig = "",
|
||||
# Also instrument ReadMessage so each capture includes the incoming WCF response bodies
|
||||
# (the GetNextQueryResultBuffer2 pResultBuff summary rows). Decoded by decode-summary-response.py.
|
||||
[switch]$WithResponse
|
||||
)
|
||||
|
||||
$ErrorActionPreference = "Stop"
|
||||
$repoRoot = Split-Path -Parent $PSScriptRoot
|
||||
Set-Location $repoRoot
|
||||
|
||||
$reProj = Join-Path $repoRoot "tools\AVEVA.Historian.ReverseEngineering\AVEVA.Historian.ReverseEngineering.csproj"
|
||||
$harnessProj = Join-Path $repoRoot "tools\AVEVA.Historian.NativeTraceHarness\AVEVA.Historian.NativeTraceHarness.csproj"
|
||||
$instrProj = Join-Path $repoRoot "tools\AVEVA.Historian.ReverseInstrumentation\AVEVA.Historian.ReverseInstrumentation.csproj"
|
||||
|
||||
$captureDir = Join-Path $repoRoot "artifacts\reverse-engineering\instrumented-wcf-writemessage-summary"
|
||||
$currentCopy = Join-Path $captureDir "current-copy"
|
||||
$instrDll = Join-Path $captureDir "aahClientManaged.dll"
|
||||
|
||||
Write-Host "== Building tooling ($Configuration) ==" -ForegroundColor Cyan
|
||||
dotnet build $reProj -c $Configuration --nologo -v q | Out-Null
|
||||
dotnet build $instrProj -c $Configuration --nologo -v q | Out-Null
|
||||
dotnet build $harnessProj -c $Configuration --nologo -v q | Out-Null
|
||||
|
||||
$instrSourceDll = Get-ChildItem -Recurse (Join-Path $repoRoot "tools\AVEVA.Historian.ReverseInstrumentation\bin\$Configuration") `
|
||||
-Filter "AVEVA.Historian.ReverseInstrumentation.dll" | Select-Object -First 1 -ExpandProperty FullName
|
||||
if (-not $instrSourceDll) { throw "ReverseInstrumentation.dll not found under bin\$Configuration." }
|
||||
|
||||
Write-Host "== Instrumenting WriteMessage$(if ($WithResponse) { ' + ReadMessage' }) ==" -ForegroundColor Cyan
|
||||
New-Item -ItemType Directory -Force -Path $captureDir | Out-Null
|
||||
if ($WithResponse) {
|
||||
# Chain via a distinct intermediate file (reading+writing the same path drops the second
|
||||
# hook on the mixed-mode native image). Final dll carries both hooks with distinct Phase
|
||||
# strings: WCF.WriteMessage.Body and WCF.ReadMessage.Body.
|
||||
$writeOnly = Join-Path $captureDir "aahClientManaged.write.dll"
|
||||
dotnet run --no-build -c $Configuration --project $reProj -- `
|
||||
instrument-wcf-writemessage (Join-Path $repoRoot "current\aahClientManaged.dll") $writeOnly | Out-Null
|
||||
dotnet run --no-build -c $Configuration --project $reProj -- `
|
||||
instrument-wcf-readmessage $writeOnly $instrDll | Out-Null
|
||||
} else {
|
||||
dotnet run --no-build -c $Configuration --project $reProj -- `
|
||||
instrument-wcf-writemessage (Join-Path $repoRoot "current\aahClientManaged.dll") $instrDll | Out-Null
|
||||
}
|
||||
|
||||
Write-Host "== Staging current-copy ==" -ForegroundColor Cyan
|
||||
# Mirror current/ into current-copy, then overwrite the managed dll with the instrumented
|
||||
# build and drop the strong-named logger assembly alongside it so the injected call binds.
|
||||
robocopy (Join-Path $repoRoot "current") $currentCopy /MIR /NJH /NJS /NDL /NP /NC /NS | Out-Null
|
||||
Copy-Item -Force $instrDll (Join-Path $currentCopy "aahClientManaged.dll")
|
||||
Copy-Item -Force $instrSourceDll (Join-Path $currentCopy "AVEVA.Historian.ReverseInstrumentation.dll")
|
||||
|
||||
# Candidate matrix: name + harness arg list. Summary configs all use Cyclic + a resolution;
|
||||
# the differentiator is which summary knob is set.
|
||||
$matrix = @(
|
||||
@{ Name = "baseline-full"; Args = @("--retrieval-mode", "Full") },
|
||||
@{ Name = "analog-avg"; Args = @("--retrieval-mode", "Cyclic", "--value-selector", "Average", "--resolution-ticks", "$ResolutionTicks") },
|
||||
@{ Name = "analog-min"; Args = @("--retrieval-mode", "Cyclic", "--value-selector", "Minimum", "--resolution-ticks", "$ResolutionTicks") },
|
||||
@{ Name = "analog-max"; Args = @("--retrieval-mode", "Cyclic", "--value-selector", "Maximum", "--resolution-ticks", "$ResolutionTicks") },
|
||||
@{ Name = "analog-integral"; Args = @("--retrieval-mode", "Cyclic", "--value-selector", "Integral", "--resolution-ticks", "$ResolutionTicks") },
|
||||
@{ Name = "mode-integral"; Args = @("--retrieval-mode", "Integral", "--resolution-ticks", "$ResolutionTicks") },
|
||||
@{ Name = "mode-twavg"; Args = @("--retrieval-mode", "TimeWeightedAverage", "--resolution-ticks", "$ResolutionTicks") },
|
||||
@{ Name = "analog-agg-avg"; Args = @("--retrieval-mode", "Cyclic", "--aggregation-type", "Average", "--resolution-ticks", "$ResolutionTicks") },
|
||||
@{ Name = "state-summary"; Args = @("--retrieval-mode", "Cyclic", "--max-states", "10", "--resolution-ticks", "$ResolutionTicks") }
|
||||
)
|
||||
|
||||
if ($OnlyConfig) { $matrix = $matrix | Where-Object { $_.Name -eq $OnlyConfig } }
|
||||
if (-not $matrix) { throw "No matrix entry named '$OnlyConfig'." }
|
||||
|
||||
$harnessDll = Join-Path $currentCopy "aahClientManaged.dll"
|
||||
$summary = @()
|
||||
|
||||
foreach ($cfg in $matrix) {
|
||||
$name = $cfg.Name
|
||||
$capturePath = Join-Path $captureDir "summary-capture-$name-latest.ndjson"
|
||||
if (Test-Path $capturePath) { Remove-Item -Force $capturePath }
|
||||
$env:AVEVA_HISTORIAN_RE_CAPTURE = $capturePath
|
||||
|
||||
Write-Host "== Capturing: $name ==" -ForegroundColor Green
|
||||
$harnessArgs = @(
|
||||
"--scenario", "history",
|
||||
"--server-name", $ServerName,
|
||||
"--tcp-port", "$TcpPort",
|
||||
"--tag", $TagName,
|
||||
"--lookback-minutes", "$LookbackMinutes",
|
||||
"--max-rows", "$MaxRows",
|
||||
"--current-dir", $currentCopy,
|
||||
"--managed-dll-path", $harnessDll
|
||||
) + $cfg.Args
|
||||
|
||||
# Don't let a single config that errors (e.g. state summary on an analog tag) abort the
|
||||
# whole matrix, and don't treat dotnet's stderr noise as a terminating error.
|
||||
try {
|
||||
$prevEap = $ErrorActionPreference
|
||||
$ErrorActionPreference = "Continue"
|
||||
& dotnet run --no-build -c $Configuration --project $harnessProj -- @harnessArgs 2>&1 | Out-Null
|
||||
} catch {
|
||||
Write-Host " (config '$name' raised: $($_.Exception.Message))" -ForegroundColor Yellow
|
||||
} finally {
|
||||
$ErrorActionPreference = $prevEap
|
||||
}
|
||||
$recCount = if (Test-Path $capturePath) { (Get-Content $capturePath | Where-Object { $_.Trim() }).Count } else { 0 }
|
||||
Write-Host " -> $recCount records -> $capturePath"
|
||||
$summary += [pscustomobject]@{ Config = $name; Records = $recCount; Capture = $capturePath }
|
||||
}
|
||||
|
||||
Remove-Item Env:\AVEVA_HISTORIAN_RE_CAPTURE -ErrorAction SilentlyContinue
|
||||
|
||||
Write-Host "`n== Capture summary ==" -ForegroundColor Cyan
|
||||
$summary | Format-Table -AutoSize
|
||||
Write-Host "Decode with: python scripts\decode-summary-capture.py" -ForegroundColor Cyan
|
||||
@@ -0,0 +1,115 @@
|
||||
"""Decoder for the analog/state summary request capture (HCAL roadmap R1.8/R1.9).
|
||||
|
||||
Reads the per-config NDJSON captures produced by scripts/Capture-SummaryRequest.ps1
|
||||
under artifacts/reverse-engineering/instrumented-wcf-writemessage-summary/, extracts
|
||||
the Retr/StartQuery2 `pRequestBuff` payload from each, hex-dumps it, and diffs every
|
||||
summary candidate against the baseline-full request so the differing bytes (the native
|
||||
QueryType / SummaryType / AutoSummaryParameters fields) stand out.
|
||||
|
||||
Output is diagnostic. The only printed strings are the SDK-chosen system tag name and
|
||||
protocol field markers — sanitize before copying any of it into docs/.
|
||||
"""
|
||||
import base64
|
||||
import json
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
REPO_ROOT = Path(__file__).resolve().parent.parent
|
||||
CAPTURE_DIR = REPO_ROOT / "artifacts" / "reverse-engineering" / "instrumented-wcf-writemessage-summary"
|
||||
|
||||
ACTION = b"aa/Retr/StartQuery2"
|
||||
PARAM = b"pRequestBuff"
|
||||
|
||||
|
||||
def extract_request_buffer(records):
|
||||
"""Return the pRequestBuff bytes from the first StartQuery2 write record, or None."""
|
||||
for rec in records:
|
||||
if rec.get("Phase") != "WCF.WriteMessage.Body":
|
||||
continue
|
||||
body = base64.b64decode(rec["Base64"])
|
||||
if ACTION not in body:
|
||||
continue
|
||||
i = body.find(PARAM)
|
||||
if i < 0:
|
||||
continue
|
||||
i += len(PARAM)
|
||||
marker = body[i]
|
||||
# MDAS length markers (same scheme as the write decoder).
|
||||
if marker == 0x9E:
|
||||
length = body[i + 1]
|
||||
return body[i + 2:i + 2 + length]
|
||||
if marker == 0x9F:
|
||||
length = int.from_bytes(body[i + 1:i + 3], "little")
|
||||
return body[i + 3:i + 3 + length]
|
||||
if marker == 0xA0:
|
||||
length = int.from_bytes(body[i + 1:i + 3], "little")
|
||||
return body[i + 3:i + 3 + length + 1]
|
||||
return None
|
||||
return None
|
||||
|
||||
|
||||
def hexdump(payload, diff_against=None):
|
||||
for off in range(0, len(payload), 16):
|
||||
chunk = payload[off:off + 16]
|
||||
cells = []
|
||||
for j, c in enumerate(chunk):
|
||||
mark = ""
|
||||
if diff_against is not None:
|
||||
k = off + j
|
||||
if k >= len(diff_against) or diff_against[k] != c:
|
||||
mark = "*"
|
||||
cells.append(f"{c:02X}{mark}")
|
||||
hp = " ".join(cells)
|
||||
ap = "".join(chr(c) if 32 <= c < 127 else "." for c in chunk)
|
||||
print(f" {off:04X} {hp:<56} |{ap}|")
|
||||
|
||||
|
||||
def load(path):
|
||||
with path.open(encoding="utf-8-sig") as fh:
|
||||
return [json.loads(line) for line in fh if line.strip()]
|
||||
|
||||
|
||||
def main() -> int:
|
||||
if not CAPTURE_DIR.exists():
|
||||
print(f"Capture dir not found: {CAPTURE_DIR}")
|
||||
print("Run scripts/Capture-SummaryRequest.ps1 first.")
|
||||
return 1
|
||||
|
||||
captures = sorted(CAPTURE_DIR.glob("summary-capture-*-latest.ndjson"))
|
||||
if not captures:
|
||||
print(f"No capture files in {CAPTURE_DIR}")
|
||||
return 1
|
||||
|
||||
buffers = {}
|
||||
for path in captures:
|
||||
name = path.stem.replace("summary-capture-", "").replace("-latest", "")
|
||||
records = load(path)
|
||||
buf = extract_request_buffer(records)
|
||||
buffers[name] = buf
|
||||
status = f"{len(buf)} bytes" if buf else "<no StartQuery2 request found>"
|
||||
print(f"{name:<18} records={len(records):>3} pRequestBuff={status}")
|
||||
|
||||
baseline = buffers.get("baseline-full")
|
||||
print()
|
||||
if not baseline:
|
||||
print("No baseline-full request buffer captured; cannot diff. Dumping each raw.")
|
||||
for name, buf in buffers.items():
|
||||
if buf:
|
||||
print(f"\n== {name} pRequestBuff ({len(buf)} bytes) ==")
|
||||
hexdump(buf)
|
||||
return 0
|
||||
|
||||
print(f"== baseline-full pRequestBuff ({len(baseline)} bytes) ==")
|
||||
hexdump(baseline)
|
||||
|
||||
for name, buf in buffers.items():
|
||||
if name == "baseline-full" or not buf:
|
||||
continue
|
||||
print(f"\n== {name} pRequestBuff ({len(buf)} bytes) — '*' marks bytes differing from baseline ==")
|
||||
hexdump(buf, diff_against=baseline)
|
||||
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,123 @@
|
||||
"""Decode the GetNextQueryResultBuffer2 *response* for an analog summary (HCAL R1.8).
|
||||
|
||||
Reads the both-hooks capture produced by
|
||||
scripts/Capture-SummaryRequest.ps1 -OnlyConfig analog-avg -WithResponse
|
||||
finds the ReadMessage record carrying GetNextQueryResultBuffer2Response, extracts the
|
||||
`pResultBuff` payload, hex-dumps it, and annotates every 8-byte window that decodes to a
|
||||
known ground-truth value (the AnalogSummaryHistory row for SysTimeSec) so the field offsets
|
||||
of CAnalogSummaryValue can be read off directly.
|
||||
|
||||
Output is diagnostic; the only printed strings are the SDK-chosen system tag name and field
|
||||
markers. Sanitize before copying into docs/.
|
||||
"""
|
||||
import base64
|
||||
import json
|
||||
import struct
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
REPO_ROOT = Path(__file__).resolve().parent.parent
|
||||
# Config name (analog-avg / analog-min / analog-max / …) selectable via argv[1].
|
||||
CONFIG = sys.argv[1] if len(sys.argv) > 1 else "analog-avg"
|
||||
CAPTURE = (REPO_ROOT / "artifacts" / "reverse-engineering"
|
||||
/ "instrumented-wcf-writemessage-summary" / f"summary-capture-{CONFIG}-latest.ndjson")
|
||||
|
||||
RESP = b"GetNextQueryResultBuffer2Response"
|
||||
PARAM = b"pResultBuff"
|
||||
|
||||
# Ground-truth values from AnalogSummaryHistory(SysTimeSec, 1h cycle) — used to label offsets.
|
||||
KNOWN_DOUBLES = {
|
||||
31.0: "31.0 (First/Last/Average)",
|
||||
100.0: "100.0 (PercentGood)",
|
||||
0.031: "0.031 (Integral)",
|
||||
111600.0: "111600.0 (Integral, full-cycle)",
|
||||
1.0: "1.0 (ValueCount as double?)",
|
||||
}
|
||||
KNOWN_U32 = {
|
||||
1: "ValueCount=1",
|
||||
192: "OPCQuality=192",
|
||||
100: "PercentGood=100",
|
||||
9: "version=9",
|
||||
}
|
||||
|
||||
|
||||
def extract_param(body, param):
|
||||
i = body.find(param)
|
||||
if i < 0:
|
||||
return None
|
||||
i += len(param)
|
||||
marker = body[i]
|
||||
if marker == 0x9E:
|
||||
length = body[i + 1]
|
||||
return body[i + 2:i + 2 + length]
|
||||
if marker == 0x9F:
|
||||
length = int.from_bytes(body[i + 1:i + 3], "little")
|
||||
return body[i + 3:i + 3 + length]
|
||||
if marker == 0xA0:
|
||||
length = int.from_bytes(body[i + 1:i + 3], "little")
|
||||
return body[i + 3:i + 3 + length + 1]
|
||||
return None
|
||||
|
||||
|
||||
def main() -> int:
|
||||
if not CAPTURE.exists():
|
||||
print(f"Capture not found: {CAPTURE}")
|
||||
print("Run: scripts/Capture-SummaryRequest.ps1 -OnlyConfig analog-avg -WithResponse")
|
||||
return 1
|
||||
|
||||
with CAPTURE.open(encoding="utf-8-sig") as fh:
|
||||
records = [json.loads(line) for line in fh if line.strip()]
|
||||
|
||||
payload = None
|
||||
for rec in records:
|
||||
if rec.get("Phase") != "WCF.ReadMessage.Body":
|
||||
continue
|
||||
body = base64.b64decode(rec["Base64"])
|
||||
if RESP not in body:
|
||||
continue
|
||||
payload = extract_param(body, PARAM)
|
||||
break
|
||||
|
||||
if payload is None:
|
||||
print("No GetNextQueryResultBuffer2Response / pResultBuff found in capture.")
|
||||
return 2
|
||||
|
||||
print(f"pResultBuff: {len(payload)} bytes")
|
||||
if len(payload) >= 6:
|
||||
version = int.from_bytes(payload[0:2], "little")
|
||||
row_count = int.from_bytes(payload[2:6], "little")
|
||||
print(f" header: version={version} rowCount={row_count}")
|
||||
print()
|
||||
|
||||
# Annotated hex dump.
|
||||
for off in range(0, len(payload), 16):
|
||||
chunk = payload[off:off + 16]
|
||||
hp = " ".join(f"{c:02X}" for c in chunk)
|
||||
ap = "".join(chr(c) if 32 <= c < 127 else "." for c in chunk)
|
||||
print(f" {off:04X} {hp:<48} |{ap}|")
|
||||
|
||||
# Scan every 8-byte window for known doubles, and every 4-byte window for known u32s.
|
||||
print("\n== Known-value hits (offset -> field) ==")
|
||||
for off in range(0, len(payload) - 7):
|
||||
val = struct.unpack_from("<d", payload, off)[0]
|
||||
for known, label in KNOWN_DOUBLES.items():
|
||||
if val == known or (known != 0 and abs(val - known) < 1e-9 * max(1.0, abs(known))):
|
||||
print(f" 0x{off:04X} double {val!r:>14} -> {label}")
|
||||
for off in range(0, len(payload) - 3):
|
||||
val = int.from_bytes(payload[off:off + 4], "little")
|
||||
if val in KNOWN_U32:
|
||||
print(f" 0x{off:04X} uint32 {val:>14} -> {KNOWN_U32[val]}")
|
||||
|
||||
# FILETIME windows (plausible 2026 timestamps: 0x01DC.. high dword).
|
||||
print("\n== Plausible FILETIME windows (Int64, year ~2020-2030) ==")
|
||||
for off in range(0, len(payload) - 7):
|
||||
ft = int.from_bytes(payload[off:off + 8], "little")
|
||||
# FILETIME for 2020-01-01 ~= 0x01D5BF.. ; 2030 ~= 0x01E5.. — gate by high word.
|
||||
if 0x01D5_0000_0000_0000 <= ft <= 0x01E6_0000_0000_0000:
|
||||
print(f" 0x{off:04X} filetime 0x{ft:016X}")
|
||||
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -36,6 +36,13 @@ internal static class Program
|
||||
string runtimeMethodPointerFilters = GetArg(args, "--runtime-method-pointer-filters")
|
||||
?? "StartDataQuery;StartQuery;GetNextRow;StartEventQuery";
|
||||
ulong resolutionTicks = ulong.TryParse(GetArg(args, "--resolution-ticks"), out ulong parsedResolutionTicks) ? parsedResolutionTicks : 0;
|
||||
// Summary-query knobs on HistoryQueryArgs (R1.8/R1.9 capture). Left null/0 = not set,
|
||||
// so a normal Full read is unaffected. ValueSelector/AggregationType/MaxStates/Filter
|
||||
// are the native properties that turn a Cyclic/Full query into an analog/state summary.
|
||||
string? valueSelectorName = GetArg(args, "--value-selector");
|
||||
string? aggregationTypeName = GetArg(args, "--aggregation-type");
|
||||
uint maxStates = uint.TryParse(GetArg(args, "--max-states"), out uint parsedMaxStates) ? parsedMaxStates : 0;
|
||||
string? historyFilter = GetArg(args, "--filter");
|
||||
DateTime endUtc = TryParseUtc(GetArg(args, "--end-utc")) ?? DateTime.UtcNow;
|
||||
DateTime startUtc = TryParseUtc(GetArg(args, "--start-utc")) ?? endUtc.AddMinutes(-lookbackMinutes);
|
||||
|
||||
@@ -789,6 +796,26 @@ internal static class Program
|
||||
{
|
||||
SetProperty(queryArgs, "Resolution", resolutionTicks);
|
||||
}
|
||||
// Summary knobs — only set when explicitly supplied so plain reads are untouched.
|
||||
if (valueSelectorName is not null)
|
||||
{
|
||||
Type valueSelectorType = GetType(assembly, "ArchestrA.HistorianValueSelector");
|
||||
SetProperty(queryArgs, "ValueSelector", Enum.Parse(valueSelectorType, valueSelectorName, ignoreCase: true));
|
||||
}
|
||||
if (aggregationTypeName is not null)
|
||||
{
|
||||
Type aggregationType = GetType(assembly, "ArchestrA.HistorianAggregationType");
|
||||
SetProperty(queryArgs, "AggregationType", Enum.Parse(aggregationType, aggregationTypeName, ignoreCase: true));
|
||||
}
|
||||
if (maxStates > 0)
|
||||
{
|
||||
// HistoryQueryArgs.MaxStates is a UInt16 on the native wrapper.
|
||||
SetProperty(queryArgs, "MaxStates", checked((ushort)maxStates));
|
||||
}
|
||||
if (historyFilter is not null)
|
||||
{
|
||||
SetProperty(queryArgs, "Filter", historyFilter);
|
||||
}
|
||||
snapshots["QueryArgsBeforeStart"] = SnapshotObject(queryArgs);
|
||||
|
||||
startError = Activator.CreateInstance(errorType)!;
|
||||
@@ -890,6 +917,10 @@ internal static class Program
|
||||
LookbackMinutes = lookbackMinutes,
|
||||
RetrievalMode = retrievalModeName,
|
||||
ResolutionTicks = resolutionTicks,
|
||||
ValueSelector = valueSelectorName,
|
||||
AggregationType = aggregationTypeName,
|
||||
MaxStates = maxStates,
|
||||
HistoryFilter = historyFilter,
|
||||
StartUtc = startUtc.ToString("O"),
|
||||
EndUtc = endUtc.ToString("O"),
|
||||
OpenSuccess = openSuccess,
|
||||
|
||||
Reference in New Issue
Block a user