M2: implement SendEventAsync — event-send rides WCF AddS2, not the storage pipe
Roadmap Milestone 2 (event sending). Capture disproved the assumption that event delivery uses the non-WCF storage-engine pipe (which would block it like revision writes): a native AddStreamedValue(HistorianEvent) leaves over WCF as AddS2 (IHistoryServiceContract2.AddStreamValues2). CM_EVENT is a built-in registered tag, so the 129 TagNotFoundInCache gate that blocks AddS2 for user tags does not apply. - R2.1: NativeTraceHarness "event-send" scenario + Capture-EventSend.ps1; two captures diffed to separate constant framing from value-dependent fields. - R2.2: HistorianEventWriteProtocol serializes the AddS2 pBuf (storage sample buffer wrapping the event VTQ) — golden-byte tested. Decoded "OS" sig + length fields + CM_EVENT tag id + EventTime/ReceivedTime FILETIMEs + Opc 192 + 0x118D descriptor + event Id + Namespace + EventType + version 5 + typed property bag. - R2.3/R2.4: HistorianWcfEventOrchestrator.SendEventAsync (Open2 event-mode 0x501 -> reuse CM_EVENT RTag2/EnsT2 -> AddStreamValues2) + HistorianClient.SendEventAsync. - R2.5: gated live test; server accepts the AddS2 (success, empty error buffer). Server requires delivered byte[].Length == declared packet length (uint32@0x04); the native relies on the MDAS encoder adding a pad byte, so the SDK emits an explicit trailing 0x00 (else AddS2 rejects with "CValuStream buffer size vs packet length mismatch"). Original events only (RevisionVersion=0) with string properties; other property types + revision/update/delete throw ProtocolEvidenceMissingException. Caveat (documented): accepted events are not persisted on the local dev box; the native client behaves identically (event ingestion pipeline inactive) — not an SDK gap. 212 unit tests pass; 16/16 event tests pass live. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01B6mcaT2PjRFKcogzp9UkfC
This commit is contained in:
Joseph Doherty
@@ -0,0 +1,110 @@
|
||||
<#
|
||||
.SYNOPSIS
|
||||
Captures the native AVEVA client's event-SEND wire traffic (HCAL roadmap R2.1) to
|
||||
determine whether AddStreamedValue(HistorianEvent) rides the WCF MDAS path (capturable
|
||||
+ implementable as a pure-managed-WCF SDK op) or the storage-engine shared-memory pipe
|
||||
(like revision writes — which would block M2 as a WCF SDK).
|
||||
|
||||
.DESCRIPTION
|
||||
Drives the .NET-Framework NativeTraceHarness's `event-send` scenario against the live
|
||||
Historian with an IL-rewritten copy of aahClientManaged.dll whose
|
||||
ClientMessageEncoder.WriteMessage AND ReadMessage are instrumented to log every MDAS
|
||||
body (the same pipeline that produced every other proven request/response shape). The
|
||||
harness opens an Event connection (ReadOnly=false), builds a clearly-marked test
|
||||
HistorianEvent, calls AddStreamedValue(HistorianEvent), then CloseStorageConnection to
|
||||
flush the queued event onto the wire.
|
||||
|
||||
Decode with scripts/decode-event-send-capture.py: if a StartStorage/AddStreamValues/
|
||||
EnqueueEventDataPacket body appears on WCF.WriteMessage.Body, M2 is viable over WCF and
|
||||
the body carries the PackToVtq event value blob to decode (R2.2). If NOTHING event-shaped
|
||||
appears on the WCF path even though the native AddStreamedValue returned success, the
|
||||
delivery used the storage-engine pipe and M2 is architecturally blocked over WCF — the
|
||||
same conclusion as the revision-write path (docs/plans/revision-write-path.md).
|
||||
|
||||
.NOTES
|
||||
Writes a real (clearly-marked) test event into the historian's event history. Artifacts
|
||||
are diagnostic and gitignored. Sanitize before copying anything into docs/ — never commit
|
||||
raw capture NDJSON, credentials, hostnames, or customer tag names.
|
||||
#>
|
||||
[CmdletBinding()]
|
||||
param(
|
||||
[string]$ServerName = "localhost",
|
||||
[int]$TcpPort = 32568,
|
||||
[string]$EventType = "User.Write",
|
||||
[int]$FlushSeconds = 6,
|
||||
[string]$Configuration = "Debug"
|
||||
)
|
||||
|
||||
$ErrorActionPreference = "Stop"
|
||||
$repoRoot = Split-Path -Parent $PSScriptRoot
|
||||
Set-Location $repoRoot
|
||||
|
||||
$reProj = Join-Path $repoRoot "tools\AVEVA.Historian.ReverseEngineering\AVEVA.Historian.ReverseEngineering.csproj"
|
||||
$harnessProj = Join-Path $repoRoot "tools\AVEVA.Historian.NativeTraceHarness\AVEVA.Historian.NativeTraceHarness.csproj"
|
||||
$instrProj = Join-Path $repoRoot "tools\AVEVA.Historian.ReverseInstrumentation\AVEVA.Historian.ReverseInstrumentation.csproj"
|
||||
|
||||
$captureDir = Join-Path $repoRoot "artifacts\reverse-engineering\instrumented-wcf-event-send"
|
||||
$currentCopy = Join-Path $captureDir "current-copy"
|
||||
$instrDll = Join-Path $captureDir "aahClientManaged.dll"
|
||||
$capturePath = Join-Path $captureDir "event-send-capture-latest.ndjson"
|
||||
|
||||
Write-Host "== Building tooling ($Configuration) ==" -ForegroundColor Cyan
|
||||
dotnet build $reProj -c $Configuration --nologo -v q | Out-Null
|
||||
dotnet build $instrProj -c $Configuration --nologo -v q | Out-Null
|
||||
dotnet build $harnessProj -c $Configuration --nologo -v q | Out-Null
|
||||
|
||||
$instrSourceDll = Get-ChildItem -Recurse (Join-Path $repoRoot "tools\AVEVA.Historian.ReverseInstrumentation\bin\$Configuration") `
|
||||
-Filter "AVEVA.Historian.ReverseInstrumentation.dll" | Select-Object -First 1 -ExpandProperty FullName
|
||||
if (-not $instrSourceDll) { throw "ReverseInstrumentation.dll not found under bin\$Configuration." }
|
||||
|
||||
Write-Host "== Instrumenting WriteMessage + ReadMessage ==" -ForegroundColor Cyan
|
||||
New-Item -ItemType Directory -Force -Path $captureDir | Out-Null
|
||||
# Chain via a distinct intermediate file (reading+writing the same path drops the second
|
||||
# hook on the mixed-mode native image). Final dll carries both hooks with distinct Phase
|
||||
# strings: WCF.WriteMessage.Body and WCF.ReadMessage.Body.
|
||||
$writeOnly = Join-Path $captureDir "aahClientManaged.write.dll"
|
||||
dotnet run --no-build -c $Configuration --project $reProj -- `
|
||||
instrument-wcf-writemessage (Join-Path $repoRoot "current\aahClientManaged.dll") $writeOnly | Out-Null
|
||||
dotnet run --no-build -c $Configuration --project $reProj -- `
|
||||
instrument-wcf-readmessage $writeOnly $instrDll | Out-Null
|
||||
|
||||
Write-Host "== Staging current-copy ==" -ForegroundColor Cyan
|
||||
robocopy (Join-Path $repoRoot "current") $currentCopy /MIR /NJH /NJS /NDL /NP /NC /NS | Out-Null
|
||||
Copy-Item -Force $instrDll (Join-Path $currentCopy "aahClientManaged.dll")
|
||||
Copy-Item -Force $instrSourceDll (Join-Path $currentCopy "AVEVA.Historian.ReverseInstrumentation.dll")
|
||||
|
||||
$harnessDll = Join-Path $currentCopy "aahClientManaged.dll"
|
||||
if (Test-Path $capturePath) { Remove-Item -Force $capturePath }
|
||||
$env:AVEVA_HISTORIAN_RE_CAPTURE = $capturePath
|
||||
|
||||
Write-Host "== Capturing event-send ==" -ForegroundColor Green
|
||||
$harnessArgs = @(
|
||||
"--scenario", "event-send",
|
||||
"--event-send-confirm",
|
||||
"--server-name", $ServerName,
|
||||
"--tcp-port", "$TcpPort",
|
||||
"--event-type", $EventType,
|
||||
"--event-send-flush-seconds", "$FlushSeconds",
|
||||
"--current-dir", $currentCopy,
|
||||
"--managed-dll-path", $harnessDll
|
||||
)
|
||||
|
||||
$harnessJson = $null
|
||||
try {
|
||||
$prevEap = $ErrorActionPreference
|
||||
$ErrorActionPreference = "Continue"
|
||||
$harnessJson = & dotnet run --no-build -c $Configuration --project $harnessProj -- @harnessArgs 2>&1
|
||||
} catch {
|
||||
Write-Host " (event-send raised: $($_.Exception.Message))" -ForegroundColor Yellow
|
||||
} finally {
|
||||
$ErrorActionPreference = $prevEap
|
||||
}
|
||||
|
||||
Remove-Item Env:\AVEVA_HISTORIAN_RE_CAPTURE -ErrorAction SilentlyContinue
|
||||
|
||||
$recCount = if (Test-Path $capturePath) { (Get-Content $capturePath | Where-Object { $_.Trim() }).Count } else { 0 }
|
||||
Write-Host "`n== Capture summary ==" -ForegroundColor Cyan
|
||||
Write-Host " -> $recCount records -> $capturePath"
|
||||
Write-Host "Harness output (look for AddStreamedEvent Success / ErrorCode):" -ForegroundColor Cyan
|
||||
$harnessJson | Select-Object -Last 60
|
||||
Write-Host "`nDecode with: python scripts\decode-event-send-capture.py" -ForegroundColor Cyan
|
||||
@@ -0,0 +1,94 @@
|
||||
"""Decide whether native event-send (HCAL R2.1) rides WCF or the storage-engine pipe.
|
||||
|
||||
Reads the both-hooks capture produced by scripts/Capture-EventSend.ps1 and, for every
|
||||
outgoing WCF.WriteMessage.Body, tries to recognise the SOAP action / operation name. It
|
||||
then renders a verdict:
|
||||
|
||||
* If a storage/event delivery op (AddStreamValues / EnqueueEventDataPacket /
|
||||
OpenEventConnection / StartStorage / AddS2 / AddStreamValues2) appears on the WRITE path,
|
||||
event-send is a WCF op → M2 is implementable over WCF and that body carries the
|
||||
PackToVtq event value blob to decode (R2.2).
|
||||
* If NO such op appears on the WRITE path, the queued event was delivered via the
|
||||
storage-engine shared-memory pipe (not WCF) — M2 is architecturally blocked as a
|
||||
pure-managed-WCF SDK, the same conclusion as the revision-write path.
|
||||
|
||||
Output is diagnostic. Sanitize before copying into docs/.
|
||||
"""
|
||||
import base64
|
||||
import json
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
REPO_ROOT = Path(__file__).resolve().parent.parent
|
||||
CAPTURE = (REPO_ROOT / "artifacts" / "reverse-engineering"
|
||||
/ "instrumented-wcf-event-send" / "event-send-capture-latest.ndjson")
|
||||
|
||||
# Operation-name markers we care about. The MDAS binary SOAP body carries the action /
|
||||
# operation name as readable text (ASCII and/or UTF-16LE). We scan for both encodings.
|
||||
EVENT_OR_STORAGE_OPS = [
|
||||
"AddStreamValues2", "AddStreamValues", "EnqueueEventDataPacket", "OpenEventConnection2",
|
||||
"OpenEventConnection", "StartStorage", "AddS2", "ForwardEventSnapshot", "AddStreamedValue",
|
||||
"AddNonStreamValues",
|
||||
]
|
||||
# Other ops we expect to see on a healthy event-send connection (auth/open/registration),
|
||||
# printed for context so a "no event op" result is clearly "delivery left WCF", not "nothing ran".
|
||||
KNOWN_OPS = [
|
||||
"GetV", "ValCl", "Open2", "OpenConnection", "GETHI", "GetSystemParameter",
|
||||
"UpdC3", "UpdateClientStatus3", "RTag2", "RegisterTags2", "EnsT2", "EnsureTags2",
|
||||
"IsOriginalAllowed", "StartQuery", "GetInterfaceVersion",
|
||||
]
|
||||
|
||||
|
||||
def find_ops(body, candidates):
|
||||
hits = []
|
||||
for op in candidates:
|
||||
a = op.encode("ascii")
|
||||
u = op.encode("utf-16-le")
|
||||
if a in body or u in body:
|
||||
hits.append(op)
|
||||
return hits
|
||||
|
||||
|
||||
def main() -> int:
|
||||
if not CAPTURE.exists():
|
||||
print(f"Capture not found: {CAPTURE}")
|
||||
print("Run: scripts/Capture-EventSend.ps1")
|
||||
return 1
|
||||
|
||||
with CAPTURE.open(encoding="utf-8-sig") as fh:
|
||||
records = [json.loads(line) for line in fh if line.strip()]
|
||||
|
||||
writes = [r for r in records if r.get("Phase") == "WCF.WriteMessage.Body"]
|
||||
reads = [r for r in records if r.get("Phase") == "WCF.ReadMessage.Body"]
|
||||
print(f"Records: {len(records)} (write={len(writes)} read={len(reads)})\n")
|
||||
|
||||
event_write_hits = []
|
||||
print("== Outgoing WCF.WriteMessage.Body ops ==")
|
||||
for i, r in enumerate(writes):
|
||||
body = base64.b64decode(r["Base64"])
|
||||
known = find_ops(body, KNOWN_OPS)
|
||||
event = find_ops(body, EVENT_OR_STORAGE_OPS)
|
||||
if event:
|
||||
event_write_hits.extend(event)
|
||||
label = ", ".join(event + known) or "<no recognized op>"
|
||||
flag = " <<< EVENT/STORAGE OP" if event else ""
|
||||
print(f" write[{i:02d}] {len(body):6d}B {label}{flag}")
|
||||
|
||||
print("\n== Verdict ==")
|
||||
if event_write_hits:
|
||||
uniq = sorted(set(event_write_hits))
|
||||
print(f" EVENT/STORAGE op(s) on the WCF WRITE path: {uniq}")
|
||||
print(" => event-send IS a WCF op. M2 viable over WCF; decode the PackToVtq value")
|
||||
print(" blob in that body for R2.2.")
|
||||
return 0
|
||||
|
||||
print(" NO event/storage delivery op on the WCF WRITE path.")
|
||||
print(" => the queued event did NOT leave via WCF. If the native AddStreamedValue")
|
||||
print(" returned success (see harness JSON), delivery used the storage-engine")
|
||||
print(" shared-memory pipe — M2 is blocked as a pure-managed-WCF SDK, same as the")
|
||||
print(" revision-write path (docs/plans/revision-write-path.md).")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,114 @@
|
||||
"""Decode the AddS2 (AddStreamValues2) pBuf event-VTQ blob captured for event-send (R2.2).
|
||||
|
||||
Extracts the `pBuf` parameter from the AddS2 WriteMessage body in the event-send capture
|
||||
and hex-dumps it, annotating windows that match the known test event so the
|
||||
HistorianEvent.PackToVtq framing can be read off and inverted into a managed serializer.
|
||||
|
||||
Known test event (from scripts/Capture-EventSend.ps1 defaults):
|
||||
Type="User.Write", Namespace="RetestSdkEventSend",
|
||||
properties: Source="RetestSdkEventSend", TestMarker="histsdk-R2.1-capture"
|
||||
|
||||
Output is diagnostic. Sanitize before copying into docs/.
|
||||
"""
|
||||
import base64
|
||||
import json
|
||||
import struct
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
REPO_ROOT = Path(__file__).resolve().parent.parent
|
||||
CAPTURE = (REPO_ROOT / "artifacts" / "reverse-engineering"
|
||||
/ "instrumented-wcf-event-send" / "event-send-capture-latest.ndjson")
|
||||
|
||||
PARAM = b"pBuf"
|
||||
ADDS2 = b"AddS2"
|
||||
|
||||
|
||||
def extract_param(body, param):
|
||||
i = body.find(param)
|
||||
if i < 0:
|
||||
return None
|
||||
i += len(param)
|
||||
# Skip the closing of the element name / attributes until a binary length marker.
|
||||
# MDAS length markers: 0x9E (1-byte len), 0x9F (2-byte len), 0xA0 (2-byte len+1).
|
||||
for scan in range(i, min(i + 16, len(body))):
|
||||
marker = body[scan]
|
||||
if marker == 0x9E:
|
||||
length = body[scan + 1]
|
||||
return body[scan + 2:scan + 2 + length]
|
||||
if marker == 0x9F:
|
||||
length = int.from_bytes(body[scan + 1:scan + 3], "little")
|
||||
return body[scan + 3:scan + 3 + length]
|
||||
if marker == 0xA0:
|
||||
length = int.from_bytes(body[scan + 1:scan + 3], "little")
|
||||
return body[scan + 3:scan + 3 + length + 1]
|
||||
return None
|
||||
|
||||
|
||||
def main() -> int:
|
||||
if not CAPTURE.exists():
|
||||
print(f"Capture not found: {CAPTURE}")
|
||||
return 1
|
||||
|
||||
with CAPTURE.open(encoding="utf-8-sig") as fh:
|
||||
records = [json.loads(line) for line in fh if line.strip()]
|
||||
|
||||
body = None
|
||||
for r in records:
|
||||
if r.get("Phase") != "WCF.WriteMessage.Body":
|
||||
continue
|
||||
b = base64.b64decode(r["Base64"])
|
||||
if ADDS2 in b:
|
||||
body = b
|
||||
break
|
||||
|
||||
if body is None:
|
||||
print("No AddS2 WriteMessage body found.")
|
||||
return 2
|
||||
|
||||
pbuf = extract_param(body, PARAM)
|
||||
if pbuf is None:
|
||||
print("Found AddS2 body but could not extract pBuf. Full body hex dump:")
|
||||
pbuf = body
|
||||
|
||||
print(f"pBuf: {len(pbuf)} bytes\n")
|
||||
for off in range(0, len(pbuf), 16):
|
||||
chunk = pbuf[off:off + 16]
|
||||
hp = " ".join(f"{c:02X}" for c in chunk)
|
||||
ap = "".join(chr(c) if 32 <= c < 127 else "." for c in chunk)
|
||||
print(f" {off:04X} {hp:<48} |{ap}|")
|
||||
|
||||
print("\n== ASCII strings (len>=3) ==")
|
||||
cur = []
|
||||
start = 0
|
||||
for i, c in enumerate(pbuf):
|
||||
if 32 <= c < 127:
|
||||
if not cur:
|
||||
start = i
|
||||
cur.append(chr(c))
|
||||
else:
|
||||
if len(cur) >= 3:
|
||||
print(f" 0x{start:04X} {''.join(cur)!r}")
|
||||
cur = []
|
||||
if len(cur) >= 3:
|
||||
print(f" 0x{start:04X} {''.join(cur)!r}")
|
||||
|
||||
print("\n== UTF-16LE strings (len>=3) ==")
|
||||
i = 0
|
||||
while i < len(pbuf) - 1:
|
||||
j = i
|
||||
chars = []
|
||||
while j < len(pbuf) - 1 and 32 <= pbuf[j] < 127 and pbuf[j + 1] == 0:
|
||||
chars.append(chr(pbuf[j]))
|
||||
j += 2
|
||||
if len(chars) >= 3:
|
||||
print(f" 0x{i:04X} {''.join(chars)!r}")
|
||||
i = j
|
||||
else:
|
||||
i += 1
|
||||
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
Reference in New Issue
Block a user