diff --git a/docs/plans/r1.8-r1.9-summary-queries.md b/docs/plans/r1.8-r1.9-summary-queries.md index 92b03ba..45afe19 100644 --- a/docs/plans/r1.8-r1.9-summary-queries.md +++ b/docs/plans/r1.8-r1.9-summary-queries.md @@ -1,7 +1,59 @@ # R1.8 / R1.9 — Analog-summary & State-summary queries (implementation plan) -**Status (2026-06-20): scoped + de-risked. Reachable on 2020 WCF. Decode targets located in -the native dll. Ready to implement; not yet started in `src/` (no guessed code shipped).** +**Status (2026-06-21): RESOLVED by request + response capture. Conclusion: the rich +multi-aggregate analog/state summary struct is NOT delivered over the 2020 WCF binary protocol. +The per-cycle aggregate values it would expose are ALREADY shipped via `ReadAggregateAsync` +(RetrievalMode → QueryType 5–8). No new `src/` code is warranted for R1.8/R1.9 on 2020 WCF.** + +## RESOLVED — what the response capture proved (2026-06-21) + +The request side was recovered first (table further down), then the `GetNextQueryResultBuffer2` +**response** was captured (`instrument-wcf-readmessage`, both hooks chained) and decoded against +`AnalogSummaryHistory` SQL ground truth for `SysTimeSec` over a 6 h window / 1 h cycle. Findings: + +1. **The response is the ordinary version-9 row buffer** — same layout the existing raw/aggregate + parser (`TryParseGetNextQueryResultBufferAggregateRows`) already handles: `uint16 version=9`, + `uint32 rowCount`, then per-row `tagKey + nameLen + name + ValueCount + cycleEnd FILETIME + + quality + OpcQuality + Value(double) + PercentGood(double) + trailer(cycleStart FILETIME …)`. + The captured 7-row buffer decoded with `Value=31.0`, `PercentGood=100.0`, `ValueCount=1`, + `OpcQuality=192` — matching the SQL row exactly. + +2. **There is NO rich `CAnalogSummaryValue` struct on the wire.** Each row carries a *single* + value, not Min+Max+First+Last+Avg+Integral together. The all-aggregates-in-one-row shape that + `CAnalogSummaryValue` / `AnalogSummaryHistory` represents is the **SQL/OLEDB provider's** shape, + not the binary `StartQuery2` retrieval's. + +3. **The single value is selected by `RetrievalMode` (QueryType), not by `ValueSelector`.** Proven + against the same constant tag where only the *kind* of aggregate distinguishes the result: + - `RetrievalMode=Integral` (QueryType 8) → `Value = 111600.0` (= SQL `Integral`) ✓ + - `RetrievalMode=TimeWeightedAverage` (QueryType 5) → `Value = 31.0` (= SQL `Average`) ✓ + - `Cyclic` (QueryType 0) **+ `ValueSelector=Integral`** → `Value = 31.0` (selector **ignored**; + the request byte `ValueSelector@0x59=0x04` was confirmed sent, yet the cyclic value came back). + + So `ValueSelector` / `AggregationType` / `MaxStates` are **inert on the WCF retrieval path** — + they configure the SQL provider's summary tables, not this binary query. + +4. **Resolution unit is correct in the SDK.** The wire `Resolution` is 100 ns ticks (= ms × 10000). + `SerializeFullHistoryRequest` writes `TimeSpan.Ticks`, which the golden test + `SerializerMatchesInstrumentedNativeTimeWeightedAverageRequest` already verifies byte-for-byte + against native (`FromMinutes(1)` → `600000000`). No bug. + +**Therefore:** "analog summary" over 2020 WCF == the existing aggregate read. To get Min, Max, +Average and Integral for a cycle you issue the corresponding `RetrievalMode` queries +(`MinimumWithTime` / `MaximumWithTime` / `TimeWeightedAverage` / `Integral`), each returning that +one aggregate per cycle — all already implemented, mapped (QueryType 5–8) and golden-tested in +`ReadAggregateAsync`. **R1.8/R1.9 need no new protocol code on this server.** A genuine +all-aggregates-at-once summary would require the gRPC front door or the SQL provider, neither of +which is the 2020 WCF binary path. + +Capture/decode tooling is committed and repeatable: `scripts/Capture-SummaryRequest.ps1` +(`-WithResponse` chains ReadMessage), `scripts/decode-summary-capture.py` (request diff), +`scripts/decode-summary-response.py ` (response decode vs SQL ground truth). Raw captures +live under `artifacts/reverse-engineering/instrumented-wcf-writemessage-summary/` (gitignored). + +--- + +_Original scoping notes below remain for context. They led to the capture; the conclusion above supersedes their "ready to implement" framing._ Unlike the M1 *read* items gated by the [string-handle wall](../reverse-engineering/wcf-string-handle-wall.md), summary queries ride the **proven `uint`-handle `StartQuery2`** path — the same call the working @@ -34,43 +86,61 @@ Found via `methods … Summary` + `dnlib-method`: | `CTypeMetadata.IsAnalogSummary` / `IsStateSummary` | `0x060001A4/A5` | server-side type gating | | `INSQL_QUERYTYPE` / `HISTORIAN_SUMMARYTYPE` | enums `0200013F` / `02000191` | the `QueryType` / `SummaryType` values to send | -## Empirical probe results (2026-06-20, live `SysTimeSec` over `StartQuery2`) +## Native request capture (2026-06-21) — request shape RECOVERED -Swept `QueryType`/`SummaryType`/`ColumnSelectorFlags` against the live 2020 server: +The earlier blind probing (sweeping `SummaryType`/`ColumnSelectorFlags` over the managed +serializer) was the wrong lever: it returned 0-row buffers because the managed `SummaryType` +field is **not** how the native client encodes a summary. A real capture settled it. -- `QueryType=2 (Full)`, `SummaryType ∈ {0,3,6}` → normal 109-byte version-9 data buffer. -- `QueryType=2`, `SummaryType ∈ {1,2,4,5}` → **valid version-9 buffer with 0 rows** (`09 00 00 00 00 00`). - The server **accepts** these summary types but yields no rows. -- The 0-row result is **unchanged** by `ColumnSelectorFlags` (tried default, all-bits - `0xFFFF…FFFF`, high-dword, low-48). So column flags are *not* the unlock. -- `QueryType ∈ {15,16}` → `GetNext` blocks/times out (no such INSQL_QUERYTYPE ordinal). +**Capture pipeline (now repeatable):** `scripts/Capture-SummaryRequest.ps1` IL-rewrites a copy +of `aahClientManaged.dll` (`instrument-wcf-writemessage`), stages it alongside the strong-named +`ReverseInstrumentation` logger, then drives the `NativeTraceHarness` history scenario through a +candidate matrix while logging every outgoing MDAS body. `scripts/decode-summary-capture.py` +extracts the `Retr/StartQuery2` `pRequestBuff` from each and diffs the summary candidates against +a tag-matched `baseline-full`. The harness now exposes `--value-selector` / `--aggregation-type` +/ `--max-states` / `--filter` so the native `HistoryQueryArgs` summary knobs can be driven. -**Conclusion:** a summary query is *not* `Full + SummaryType + column flags`. The summary -configuration lives elsewhere in the request — almost certainly the **`AutoSummaryParameters` -trailer** (`SerializeFullHistoryRequest` currently writes it all-zero via -`WriteAutoSummaryParameters`) and/or a native summary `QueryType`. Both are **native-side -constants** (`HISTORIAN_SUMMARYTYPE` / `INSQL_QUERYTYPE` are `value__`-only in managed metadata; -`CColumnNameMap.LoadColumnNameMap` builds column→bit via native string/const data, not IL -`ldstr`/`ldc`). So they cannot be recovered from managed metadata, and blind probing of the -obvious fields returns empty. +**There is no separate "summary" QueryType or `SummaryType` field.** A summary is an ordinary +`StartQuery2` request (`QueryType` = the chosen `RetrievalMode`, e.g. `Cyclic`=0) with three +things set: the **ValueSelector** byte, the **AggregationType** byte, a non-zero **Resolution** +(which fills the previously-zeroed `AutoSummaryParameters` trailer), and — for state summary — +the **MaxStates** field. The server then returns analog- vs state-summary rows based on the tag +type plus these fields. Offsets below are **into the StartQuery2 `pRequestBuff`** (229-byte +`SysTimeSec` baseline; verified byte-for-byte against the native client): -**Therefore the right next step is a native request capture, not more probing:** drive the native -client (NativeTraceHarness / a real summary query) and capture the `pRequestBuff` bytes via the -existing `instrument-wcf-writemessage` pipeline — the same method that produced every other proven -request shape (reads, events, EnsT2). Diff that buffer against a normal Full request to read off -the exact `QueryType` + `SummaryType` + `AutoSummaryParameters` layout, then implement against it. +| Offset | Field | Type | Evidence | +|---|---|---|---| +| `0x01` | QueryType | uint32 LE | Full→`02`, Cyclic→`00` (matches the verified `RetrievalMode`→`QueryType` map) | +| `0x1D` | Resolution | float64 LE | `36e9` ticks → `00 00 00 D0 88 C3 20 42` = `0x4220C388D0000000` (1 h). Zero for non-summary reads | +| `0x32` | Timezone | len-prefixed UTF-16 | `"UTC"` | +| `0x49` | Filter | len-prefixed UTF-16 | `"NoFilter"` default; driven by `--filter` | +| `0x59` | **ValueSelector** | byte | baseline `01` (Auto); `--value-selector Minimum`→`06`, `Maximum`→`07`, `Average`→`08` — exact `HistorianValueSelector` values | +| `0x5B` | **AggregationType** | byte | baseline `03`; `--aggregation-type Average`→`02` — exact `HistorianAggregationType` values | +| `~0x5F` | ColumnSelectorFlags | bytes | `FF 82 07 00 82 81` — matches the `0x0000_8182_0007_82FF` reads already send; **unchanged** by summary | +| `0x6B` | Tag name | len-prefixed UTF-16 | `count, "SysTimeSec"` | +| after tag | **MaxStates** | uint16 LE | the `01`-default byte after the tag block; `--max-states 10`→`0A` (state summary, R1.9) | +| `~0xAA` | **AutoSummaryParameters** | block | zero for plain reads; `80 1E 08 6B 47 01` when Resolution set (identical across analog *and* state) — the resolution-derived cycle block | -## Open questions (nail these next, in order) +State summary (R1.9) is the **same request** with `MaxStates` > 0 (the analog `ValueSelector`/ +`AggregationType` bytes stay at their `01`/`03` defaults); the analog-vs-state distinction on the +wire is which of those fields is non-default, plus the tag type. Note `MaxStates` is a **UInt16** +on `HistoryQueryArgs` (passing UInt32 throws) — the harness casts accordingly. -1. **Request params.** Recover the exact `QueryType` + `SummaryType` (+ whether `ColumnSelectorFlags` - must change) for analog vs state summary. Source: decompile `INSQL_QUERYTYPE`/`HISTORIAN_SUMMARYTYPE` - enum members and `Select{Analog,State}SummaryColumns` (the `GetColumnFlag` column-name `ldstr` - operands → the flag set). The standard QueryType map (Cyclic=0 … EndBound=14) is already verified; - summary is expected to be a `SummaryType`≠0 with an existing base `QueryType`, **not** a new mode - ordinal — confirm. -2. **Row layout.** Capture a real `GetNextQueryResultBuffer2` buffer for an analog summary of a - data-bearing tag (`SysTimeSec`) over a multi-hour window with an interval, then decode against the - `CAnalogSummaryValue` field set. Likely each row = StartDateTime FILETIME + the 8 typed fields. +Raw captures live under `artifacts/reverse-engineering/instrumented-wcf-writemessage-summary/` +(gitignored). Re-run with `scripts/Capture-SummaryRequest.ps1` (analog: `SysTimeSec`; state: +`-TagName SysPulse`, the local discrete tag). + +## Open questions (only the row layout remains) + +1. ~~**Request params.**~~ **DONE** — see the table above. ValueSelector @ `0x59`, + AggregationType @ `0x5B`, Resolution @ `0x1D` (→ AutoSummaryParameters @ `~0xAA`), + MaxStates after the tag block. No new QueryType/SummaryType ordinal involved. +2. **Row layout (next concrete step).** Capture the `GetNextQueryResultBuffer2` *response* for an + analog summary of `SysTimeSec` over a multi-hour window with a 1 h resolution — instrument + `ReadMessage` (`instrument-wcf-readmessage`, symmetric to the WriteMessage capture already + wired here) and decode against the `CAnalogSummaryValue` field set + (StartDateTime + Min/Max/First/Last/ValueCount/TimeGood/Integral/IntegralOfSquares). The + request side is no longer a blocker. ## Implementation steps (per the project's two-tests discipline) @@ -84,9 +154,13 @@ the exact `QueryType` + `SummaryType` + `AutoSummaryParameters` layout, then imp `HistorianStateSummary` (per-state contained/partial/entry-count). Reuse `RunQuery` plumbing. 5. Golden-byte test on the parser + gated live test on `localhost` (assert non-empty, fields sane). -## Why stop here this session +## State of play -`UnpackFromValueBuffer` is reader-call-based (no literal offset table), so a correct parser needs a -**captured real buffer** to decode against — that's the next concrete action, not a guess. Per -project rule ("never guess wire bytes; leave throwing until evidence supports it") no summary code -was added to `src/` yet. Everything above is the evidence needed to implement directly. +The **request side is fully recovered** from real bytes (table above) — the managed +`HistorianDataQueryRequest` builder can now set `ValueSelector`/`AggregationType`/`Resolution` +(+ `MaxStates` for state) against ground truth rather than guesses. What remains is the +**response row layout**: `CAnalogSummaryValue.UnpackFromValueBuffer` is reader-call-based (no +literal offset table), so the parser needs a captured real *response* buffer to decode against +(step 2 in Open questions — `instrument-wcf-readmessage`, already wired alongside the WriteMessage +capture). Per project rule ("never guess wire bytes; leave throwing until evidence supports it") +no summary code is in `src/` yet — that lands once the response fixture exists. diff --git a/scripts/Capture-SummaryRequest.ps1 b/scripts/Capture-SummaryRequest.ps1 new file mode 100644 index 0000000..9a5c6bf --- /dev/null +++ b/scripts/Capture-SummaryRequest.ps1 @@ -0,0 +1,150 @@ +<# +.SYNOPSIS + Captures the native AVEVA client's StartQuery2 request bytes for analog/state + summary queries (HCAL roadmap R1.8/R1.9) so the managed SDK's summary request + shape can be decoded against ground truth instead of guessed. + +.DESCRIPTION + Drives the .NET-Framework NativeTraceHarness against the live Historian with an + IL-rewritten copy of aahClientManaged.dll whose ClientMessageEncoder.WriteMessage + is instrumented to log every outgoing MDAS body (the same pipeline that produced + every other proven request shape). For each candidate HistoryQueryArgs config it + writes a per-config NDJSON capture under + artifacts/reverse-engineering/instrumented-wcf-writemessage-summary/ (gitignored). + + The default matrix is: + - baseline-full : RetrievalMode=Full (the known-good non-summary request) + - analog-avg : RetrievalMode=Cyclic + ValueSelector=Average + Resolution + - analog-min : RetrievalMode=Cyclic + ValueSelector=Minimum + Resolution + - analog-agg-avg : RetrievalMode=Cyclic + AggregationType=Average + Resolution + - state-summary : RetrievalMode=Cyclic + MaxStates>0 + Resolution + + Diff any candidate against baseline-full (scripts/decode-summary-capture.py) to read + off the exact QueryType / SummaryType / AutoSummaryParameters bytes the native client + sets for a summary, then implement the managed request against that. + +.NOTES + Artifacts are diagnostic. Sanitize before copying anything into docs/ — never commit + raw capture NDJSON, credentials, hostnames, or customer tag names. +#> +[CmdletBinding()] +param( + [string]$ServerName = "localhost", + [int]$TcpPort = 32568, + # SysTimeSec is the local data-bearing system tag (OtOpcUaParityTest_001.Counter is stale/empty). + [string]$TagName = "SysTimeSec", + [int]$LookbackMinutes = 240, + [int]$MaxRows = 4, + # 1-hour summary cycle in 100ns ticks (1h = 36,000,000,000 ticks). + [uint64]$ResolutionTicks = 36000000000, + [string]$Configuration = "Debug", + # Restrict the run to a single named config from the matrix (default: run all). + [string]$OnlyConfig = "", + # Also instrument ReadMessage so each capture includes the incoming WCF response bodies + # (the GetNextQueryResultBuffer2 pResultBuff summary rows). Decoded by decode-summary-response.py. + [switch]$WithResponse +) + +$ErrorActionPreference = "Stop" +$repoRoot = Split-Path -Parent $PSScriptRoot +Set-Location $repoRoot + +$reProj = Join-Path $repoRoot "tools\AVEVA.Historian.ReverseEngineering\AVEVA.Historian.ReverseEngineering.csproj" +$harnessProj = Join-Path $repoRoot "tools\AVEVA.Historian.NativeTraceHarness\AVEVA.Historian.NativeTraceHarness.csproj" +$instrProj = Join-Path $repoRoot "tools\AVEVA.Historian.ReverseInstrumentation\AVEVA.Historian.ReverseInstrumentation.csproj" + +$captureDir = Join-Path $repoRoot "artifacts\reverse-engineering\instrumented-wcf-writemessage-summary" +$currentCopy = Join-Path $captureDir "current-copy" +$instrDll = Join-Path $captureDir "aahClientManaged.dll" + +Write-Host "== Building tooling ($Configuration) ==" -ForegroundColor Cyan +dotnet build $reProj -c $Configuration --nologo -v q | Out-Null +dotnet build $instrProj -c $Configuration --nologo -v q | Out-Null +dotnet build $harnessProj -c $Configuration --nologo -v q | Out-Null + +$instrSourceDll = Get-ChildItem -Recurse (Join-Path $repoRoot "tools\AVEVA.Historian.ReverseInstrumentation\bin\$Configuration") ` + -Filter "AVEVA.Historian.ReverseInstrumentation.dll" | Select-Object -First 1 -ExpandProperty FullName +if (-not $instrSourceDll) { throw "ReverseInstrumentation.dll not found under bin\$Configuration." } + +Write-Host "== Instrumenting WriteMessage$(if ($WithResponse) { ' + ReadMessage' }) ==" -ForegroundColor Cyan +New-Item -ItemType Directory -Force -Path $captureDir | Out-Null +if ($WithResponse) { + # Chain via a distinct intermediate file (reading+writing the same path drops the second + # hook on the mixed-mode native image). Final dll carries both hooks with distinct Phase + # strings: WCF.WriteMessage.Body and WCF.ReadMessage.Body. + $writeOnly = Join-Path $captureDir "aahClientManaged.write.dll" + dotnet run --no-build -c $Configuration --project $reProj -- ` + instrument-wcf-writemessage (Join-Path $repoRoot "current\aahClientManaged.dll") $writeOnly | Out-Null + dotnet run --no-build -c $Configuration --project $reProj -- ` + instrument-wcf-readmessage $writeOnly $instrDll | Out-Null +} else { + dotnet run --no-build -c $Configuration --project $reProj -- ` + instrument-wcf-writemessage (Join-Path $repoRoot "current\aahClientManaged.dll") $instrDll | Out-Null +} + +Write-Host "== Staging current-copy ==" -ForegroundColor Cyan +# Mirror current/ into current-copy, then overwrite the managed dll with the instrumented +# build and drop the strong-named logger assembly alongside it so the injected call binds. +robocopy (Join-Path $repoRoot "current") $currentCopy /MIR /NJH /NJS /NDL /NP /NC /NS | Out-Null +Copy-Item -Force $instrDll (Join-Path $currentCopy "aahClientManaged.dll") +Copy-Item -Force $instrSourceDll (Join-Path $currentCopy "AVEVA.Historian.ReverseInstrumentation.dll") + +# Candidate matrix: name + harness arg list. Summary configs all use Cyclic + a resolution; +# the differentiator is which summary knob is set. +$matrix = @( + @{ Name = "baseline-full"; Args = @("--retrieval-mode", "Full") }, + @{ Name = "analog-avg"; Args = @("--retrieval-mode", "Cyclic", "--value-selector", "Average", "--resolution-ticks", "$ResolutionTicks") }, + @{ Name = "analog-min"; Args = @("--retrieval-mode", "Cyclic", "--value-selector", "Minimum", "--resolution-ticks", "$ResolutionTicks") }, + @{ Name = "analog-max"; Args = @("--retrieval-mode", "Cyclic", "--value-selector", "Maximum", "--resolution-ticks", "$ResolutionTicks") }, + @{ Name = "analog-integral"; Args = @("--retrieval-mode", "Cyclic", "--value-selector", "Integral", "--resolution-ticks", "$ResolutionTicks") }, + @{ Name = "mode-integral"; Args = @("--retrieval-mode", "Integral", "--resolution-ticks", "$ResolutionTicks") }, + @{ Name = "mode-twavg"; Args = @("--retrieval-mode", "TimeWeightedAverage", "--resolution-ticks", "$ResolutionTicks") }, + @{ Name = "analog-agg-avg"; Args = @("--retrieval-mode", "Cyclic", "--aggregation-type", "Average", "--resolution-ticks", "$ResolutionTicks") }, + @{ Name = "state-summary"; Args = @("--retrieval-mode", "Cyclic", "--max-states", "10", "--resolution-ticks", "$ResolutionTicks") } +) + +if ($OnlyConfig) { $matrix = $matrix | Where-Object { $_.Name -eq $OnlyConfig } } +if (-not $matrix) { throw "No matrix entry named '$OnlyConfig'." } + +$harnessDll = Join-Path $currentCopy "aahClientManaged.dll" +$summary = @() + +foreach ($cfg in $matrix) { + $name = $cfg.Name + $capturePath = Join-Path $captureDir "summary-capture-$name-latest.ndjson" + if (Test-Path $capturePath) { Remove-Item -Force $capturePath } + $env:AVEVA_HISTORIAN_RE_CAPTURE = $capturePath + + Write-Host "== Capturing: $name ==" -ForegroundColor Green + $harnessArgs = @( + "--scenario", "history", + "--server-name", $ServerName, + "--tcp-port", "$TcpPort", + "--tag", $TagName, + "--lookback-minutes", "$LookbackMinutes", + "--max-rows", "$MaxRows", + "--current-dir", $currentCopy, + "--managed-dll-path", $harnessDll + ) + $cfg.Args + + # Don't let a single config that errors (e.g. state summary on an analog tag) abort the + # whole matrix, and don't treat dotnet's stderr noise as a terminating error. + try { + $prevEap = $ErrorActionPreference + $ErrorActionPreference = "Continue" + & dotnet run --no-build -c $Configuration --project $harnessProj -- @harnessArgs 2>&1 | Out-Null + } catch { + Write-Host " (config '$name' raised: $($_.Exception.Message))" -ForegroundColor Yellow + } finally { + $ErrorActionPreference = $prevEap + } + $recCount = if (Test-Path $capturePath) { (Get-Content $capturePath | Where-Object { $_.Trim() }).Count } else { 0 } + Write-Host " -> $recCount records -> $capturePath" + $summary += [pscustomobject]@{ Config = $name; Records = $recCount; Capture = $capturePath } +} + +Remove-Item Env:\AVEVA_HISTORIAN_RE_CAPTURE -ErrorAction SilentlyContinue + +Write-Host "`n== Capture summary ==" -ForegroundColor Cyan +$summary | Format-Table -AutoSize +Write-Host "Decode with: python scripts\decode-summary-capture.py" -ForegroundColor Cyan diff --git a/scripts/decode-summary-capture.py b/scripts/decode-summary-capture.py new file mode 100644 index 0000000..8be08bb --- /dev/null +++ b/scripts/decode-summary-capture.py @@ -0,0 +1,115 @@ +"""Decoder for the analog/state summary request capture (HCAL roadmap R1.8/R1.9). + +Reads the per-config NDJSON captures produced by scripts/Capture-SummaryRequest.ps1 +under artifacts/reverse-engineering/instrumented-wcf-writemessage-summary/, extracts +the Retr/StartQuery2 `pRequestBuff` payload from each, hex-dumps it, and diffs every +summary candidate against the baseline-full request so the differing bytes (the native +QueryType / SummaryType / AutoSummaryParameters fields) stand out. + +Output is diagnostic. The only printed strings are the SDK-chosen system tag name and +protocol field markers — sanitize before copying any of it into docs/. +""" +import base64 +import json +import sys +from pathlib import Path + +REPO_ROOT = Path(__file__).resolve().parent.parent +CAPTURE_DIR = REPO_ROOT / "artifacts" / "reverse-engineering" / "instrumented-wcf-writemessage-summary" + +ACTION = b"aa/Retr/StartQuery2" +PARAM = b"pRequestBuff" + + +def extract_request_buffer(records): + """Return the pRequestBuff bytes from the first StartQuery2 write record, or None.""" + for rec in records: + if rec.get("Phase") != "WCF.WriteMessage.Body": + continue + body = base64.b64decode(rec["Base64"]) + if ACTION not in body: + continue + i = body.find(PARAM) + if i < 0: + continue + i += len(PARAM) + marker = body[i] + # MDAS length markers (same scheme as the write decoder). + if marker == 0x9E: + length = body[i + 1] + return body[i + 2:i + 2 + length] + if marker == 0x9F: + length = int.from_bytes(body[i + 1:i + 3], "little") + return body[i + 3:i + 3 + length] + if marker == 0xA0: + length = int.from_bytes(body[i + 1:i + 3], "little") + return body[i + 3:i + 3 + length + 1] + return None + return None + + +def hexdump(payload, diff_against=None): + for off in range(0, len(payload), 16): + chunk = payload[off:off + 16] + cells = [] + for j, c in enumerate(chunk): + mark = "" + if diff_against is not None: + k = off + j + if k >= len(diff_against) or diff_against[k] != c: + mark = "*" + cells.append(f"{c:02X}{mark}") + hp = " ".join(cells) + ap = "".join(chr(c) if 32 <= c < 127 else "." for c in chunk) + print(f" {off:04X} {hp:<56} |{ap}|") + + +def load(path): + with path.open(encoding="utf-8-sig") as fh: + return [json.loads(line) for line in fh if line.strip()] + + +def main() -> int: + if not CAPTURE_DIR.exists(): + print(f"Capture dir not found: {CAPTURE_DIR}") + print("Run scripts/Capture-SummaryRequest.ps1 first.") + return 1 + + captures = sorted(CAPTURE_DIR.glob("summary-capture-*-latest.ndjson")) + if not captures: + print(f"No capture files in {CAPTURE_DIR}") + return 1 + + buffers = {} + for path in captures: + name = path.stem.replace("summary-capture-", "").replace("-latest", "") + records = load(path) + buf = extract_request_buffer(records) + buffers[name] = buf + status = f"{len(buf)} bytes" if buf else "" + print(f"{name:<18} records={len(records):>3} pRequestBuff={status}") + + baseline = buffers.get("baseline-full") + print() + if not baseline: + print("No baseline-full request buffer captured; cannot diff. Dumping each raw.") + for name, buf in buffers.items(): + if buf: + print(f"\n== {name} pRequestBuff ({len(buf)} bytes) ==") + hexdump(buf) + return 0 + + print(f"== baseline-full pRequestBuff ({len(baseline)} bytes) ==") + hexdump(baseline) + + for name, buf in buffers.items(): + if name == "baseline-full" or not buf: + continue + print(f"\n== {name} pRequestBuff ({len(buf)} bytes) — '*' marks bytes differing from baseline ==") + hexdump(buf, diff_against=baseline) + + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/scripts/decode-summary-response.py b/scripts/decode-summary-response.py new file mode 100644 index 0000000..f45fce9 --- /dev/null +++ b/scripts/decode-summary-response.py @@ -0,0 +1,123 @@ +"""Decode the GetNextQueryResultBuffer2 *response* for an analog summary (HCAL R1.8). + +Reads the both-hooks capture produced by + scripts/Capture-SummaryRequest.ps1 -OnlyConfig analog-avg -WithResponse +finds the ReadMessage record carrying GetNextQueryResultBuffer2Response, extracts the +`pResultBuff` payload, hex-dumps it, and annotates every 8-byte window that decodes to a +known ground-truth value (the AnalogSummaryHistory row for SysTimeSec) so the field offsets +of CAnalogSummaryValue can be read off directly. + +Output is diagnostic; the only printed strings are the SDK-chosen system tag name and field +markers. Sanitize before copying into docs/. +""" +import base64 +import json +import struct +import sys +from pathlib import Path + +REPO_ROOT = Path(__file__).resolve().parent.parent +# Config name (analog-avg / analog-min / analog-max / …) selectable via argv[1]. +CONFIG = sys.argv[1] if len(sys.argv) > 1 else "analog-avg" +CAPTURE = (REPO_ROOT / "artifacts" / "reverse-engineering" + / "instrumented-wcf-writemessage-summary" / f"summary-capture-{CONFIG}-latest.ndjson") + +RESP = b"GetNextQueryResultBuffer2Response" +PARAM = b"pResultBuff" + +# Ground-truth values from AnalogSummaryHistory(SysTimeSec, 1h cycle) — used to label offsets. +KNOWN_DOUBLES = { + 31.0: "31.0 (First/Last/Average)", + 100.0: "100.0 (PercentGood)", + 0.031: "0.031 (Integral)", + 111600.0: "111600.0 (Integral, full-cycle)", + 1.0: "1.0 (ValueCount as double?)", +} +KNOWN_U32 = { + 1: "ValueCount=1", + 192: "OPCQuality=192", + 100: "PercentGood=100", + 9: "version=9", +} + + +def extract_param(body, param): + i = body.find(param) + if i < 0: + return None + i += len(param) + marker = body[i] + if marker == 0x9E: + length = body[i + 1] + return body[i + 2:i + 2 + length] + if marker == 0x9F: + length = int.from_bytes(body[i + 1:i + 3], "little") + return body[i + 3:i + 3 + length] + if marker == 0xA0: + length = int.from_bytes(body[i + 1:i + 3], "little") + return body[i + 3:i + 3 + length + 1] + return None + + +def main() -> int: + if not CAPTURE.exists(): + print(f"Capture not found: {CAPTURE}") + print("Run: scripts/Capture-SummaryRequest.ps1 -OnlyConfig analog-avg -WithResponse") + return 1 + + with CAPTURE.open(encoding="utf-8-sig") as fh: + records = [json.loads(line) for line in fh if line.strip()] + + payload = None + for rec in records: + if rec.get("Phase") != "WCF.ReadMessage.Body": + continue + body = base64.b64decode(rec["Base64"]) + if RESP not in body: + continue + payload = extract_param(body, PARAM) + break + + if payload is None: + print("No GetNextQueryResultBuffer2Response / pResultBuff found in capture.") + return 2 + + print(f"pResultBuff: {len(payload)} bytes") + if len(payload) >= 6: + version = int.from_bytes(payload[0:2], "little") + row_count = int.from_bytes(payload[2:6], "little") + print(f" header: version={version} rowCount={row_count}") + print() + + # Annotated hex dump. + for off in range(0, len(payload), 16): + chunk = payload[off:off + 16] + hp = " ".join(f"{c:02X}" for c in chunk) + ap = "".join(chr(c) if 32 <= c < 127 else "." for c in chunk) + print(f" {off:04X} {hp:<48} |{ap}|") + + # Scan every 8-byte window for known doubles, and every 4-byte window for known u32s. + print("\n== Known-value hits (offset -> field) ==") + for off in range(0, len(payload) - 7): + val = struct.unpack_from("14} -> {label}") + for off in range(0, len(payload) - 3): + val = int.from_bytes(payload[off:off + 4], "little") + if val in KNOWN_U32: + print(f" 0x{off:04X} uint32 {val:>14} -> {KNOWN_U32[val]}") + + # FILETIME windows (plausible 2026 timestamps: 0x01DC.. high dword). + print("\n== Plausible FILETIME windows (Int64, year ~2020-2030) ==") + for off in range(0, len(payload) - 7): + ft = int.from_bytes(payload[off:off + 8], "little") + # FILETIME for 2020-01-01 ~= 0x01D5BF.. ; 2030 ~= 0x01E5.. — gate by high word. + if 0x01D5_0000_0000_0000 <= ft <= 0x01E6_0000_0000_0000: + print(f" 0x{off:04X} filetime 0x{ft:016X}") + + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/tools/AVEVA.Historian.NativeTraceHarness/Program.cs b/tools/AVEVA.Historian.NativeTraceHarness/Program.cs index c29be6d..abb80eb 100644 --- a/tools/AVEVA.Historian.NativeTraceHarness/Program.cs +++ b/tools/AVEVA.Historian.NativeTraceHarness/Program.cs @@ -36,6 +36,13 @@ internal static class Program string runtimeMethodPointerFilters = GetArg(args, "--runtime-method-pointer-filters") ?? "StartDataQuery;StartQuery;GetNextRow;StartEventQuery"; ulong resolutionTicks = ulong.TryParse(GetArg(args, "--resolution-ticks"), out ulong parsedResolutionTicks) ? parsedResolutionTicks : 0; + // Summary-query knobs on HistoryQueryArgs (R1.8/R1.9 capture). Left null/0 = not set, + // so a normal Full read is unaffected. ValueSelector/AggregationType/MaxStates/Filter + // are the native properties that turn a Cyclic/Full query into an analog/state summary. + string? valueSelectorName = GetArg(args, "--value-selector"); + string? aggregationTypeName = GetArg(args, "--aggregation-type"); + uint maxStates = uint.TryParse(GetArg(args, "--max-states"), out uint parsedMaxStates) ? parsedMaxStates : 0; + string? historyFilter = GetArg(args, "--filter"); DateTime endUtc = TryParseUtc(GetArg(args, "--end-utc")) ?? DateTime.UtcNow; DateTime startUtc = TryParseUtc(GetArg(args, "--start-utc")) ?? endUtc.AddMinutes(-lookbackMinutes); @@ -789,6 +796,26 @@ internal static class Program { SetProperty(queryArgs, "Resolution", resolutionTicks); } + // Summary knobs — only set when explicitly supplied so plain reads are untouched. + if (valueSelectorName is not null) + { + Type valueSelectorType = GetType(assembly, "ArchestrA.HistorianValueSelector"); + SetProperty(queryArgs, "ValueSelector", Enum.Parse(valueSelectorType, valueSelectorName, ignoreCase: true)); + } + if (aggregationTypeName is not null) + { + Type aggregationType = GetType(assembly, "ArchestrA.HistorianAggregationType"); + SetProperty(queryArgs, "AggregationType", Enum.Parse(aggregationType, aggregationTypeName, ignoreCase: true)); + } + if (maxStates > 0) + { + // HistoryQueryArgs.MaxStates is a UInt16 on the native wrapper. + SetProperty(queryArgs, "MaxStates", checked((ushort)maxStates)); + } + if (historyFilter is not null) + { + SetProperty(queryArgs, "Filter", historyFilter); + } snapshots["QueryArgsBeforeStart"] = SnapshotObject(queryArgs); startError = Activator.CreateInstance(errorType)!; @@ -890,6 +917,10 @@ internal static class Program LookbackMinutes = lookbackMinutes, RetrievalMode = retrievalModeName, ResolutionTicks = resolutionTicks, + ValueSelector = valueSelectorName, + AggregationType = aggregationTypeName, + MaxStates = maxStates, + HistoryFilter = historyFilter, StartUtc = startUtc.ToString("O"), EndUtc = endUtc.ToString("O"), OpenSuccess = openSuccess,