cfc8d44e3a
New SDK surface:
HistorianClient.EnsureTagAsync(HistorianTagDefinition)
HistorianClient.DeleteTagAsync(string tagName)
Plumbing:
src/AVEVA.Historian.Client/Models/HistorianTagDefinition.cs
Public input model — TagName/Description/EngineeringUnit/DataType/MinEU/MaxEU.
Currently only HistorianDataType.Float is live-verified.
src/AVEVA.Historian.Client/Wcf/HistorianTagWriteProtocol.cs
SerializeAnalogCTagMetadata produces 146-byte payload byte-for-byte
identical to the captured native EnsT2(Float) request.
SerializeDeleteTagNames produces ushort 0x6751 + ushort 1 + uint count
+ per-tag (uint charCount + UTF-16 chars).
src/AVEVA.Historian.Client/Wcf/HistorianWcfTagWriteOrchestrator.cs
Both EnsT2 and DelT run the full Stat-priming chain captured for the
analog flow (UpdC3 + Stat.GetV ×3 + Stat.GETHI ×2 + 7× GetSystemParameter
+ Trx.GetV + Retr.GetV).
src/AVEVA.Historian.Client/Wcf/HistorianWcfTagClient.cs
MapDataType extended to accept tag-origin marker 0xC7 (SDK-created tags).
Tests:
5 golden-byte tests (HistorianTagWriteProtocolTests):
SerializeAnalogCTagMetadata byte-for-byte match against captured 146-byte fixture
SerializeAnalogCTagMetadata produces different bytes for different inputs
SerializeDeleteTagNames single-tag matches captured shape
SerializeDeleteTagNames multi-tag appends each
SerializeDeleteTagNames empty list throws
1 live integration test (gated by HISTORIAN_WRITE_SANDBOX_TAG):
EnsureTagAsync_AndDeleteTagAsync_RoundTrip_AgainstLocalHistorian
EnsureTagAsync creates the sandbox tag, GetTagMetadataAsync reads it
back. 130/130 tests pass.
Harness improvements:
--write-delete-after now runs DelT independently of AddStreamedValue
outcome.
HistorianTagStatusList constructed correctly for DeleteTags reflection
call (previous StringCollection attempt failed with TypeMismatch).
Known DelT gap: SDK's DeleteTagAsync returns true but server-side
cascading deletion does not always complete (the row remains in
Runtime.dbo.Tag). The captured native flow's DelT removes the tag
cleanly (verified via harness --write-delete-after), so something
around the WCF DelT call is missing from our orchestrator. Documented
as known issue with SMC-based cleanup as workaround.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
710 lines
32 KiB
Markdown
710 lines
32 KiB
Markdown
# Plan: Reverse-Engineering Write Commands
|
||
|
||
Status: **PHASE 2 PARTIALLY EXECUTED on 2026-05-04** — write-scenario
|
||
harness extension built and captured the full EnsT2(Float) wire byte
|
||
sequence against a real sandbox tag. AddS2 is blocked client-side by
|
||
"Tag not added to server" (error 168) — the native `AddStreamedValue`
|
||
refuses to send because the tag isn't in the server's session cache,
|
||
even though `EnsT2` created it in the Runtime DB. AddS2 wire bytes
|
||
**not yet captured**; needs a separate session to resolve the
|
||
post-EnsT2 registration prereq (likely RTag2 with the analog tag GUID,
|
||
mirroring the event flow's RTag2(CmEventTagId)).
|
||
|
||
## Phase 2 results
|
||
|
||
**Sandbox tag created** in Runtime DB: `RetestSdkWriteSandbox`,
|
||
wwTagKey=240, DateCreated=2026-05-04 07:49:50. Single dedicated tag
|
||
per safety §1; no other tags touched.
|
||
|
||
**`tools/AVEVA.Historian.NativeTraceHarness/Program.cs` extended** with
|
||
`--scenario write`:
|
||
|
||
- New args: `--write-sandbox-tag <name>` (default
|
||
`RetestSdkWriteSandbox`; refuses any name that doesn't start with
|
||
`RetestSdkWrite`), `--write-value <numeric>` (default 42.5),
|
||
`--write-data-type <name>` (default Float), `--write-delete-after`
|
||
(best-effort cleanup).
|
||
- Toggles `ConnectionArgs.ReadOnly` to false when scenario is `write`
|
||
(otherwise the connection rejects writes with error 132 "Operation
|
||
is not enabled").
|
||
- Calls `ArchestrA.HistorianAccess.AddTag` (drives `EnsT2` on the wire),
|
||
then `ArchestrA.HistorianAccess.AddStreamedValue` (would drive
|
||
`AddS2` but currently aborts client-side at error 168).
|
||
- Resolves the actual `wwTagKey` via SQL when `AddTag` returns 0
|
||
because the tag already exists from a prior session.
|
||
- Public `AddStreamedValue` overload selector: instance method whose
|
||
signature is `(HistorianDataValue, …, HistorianAccessError&)` —
|
||
picks the simplest dispatcher that's actually reflectable (the
|
||
4-param impl is private and not visible to reflection).
|
||
|
||
**Captures landed** at
|
||
`artifacts/reverse-engineering/instrumented-wcf-writemessage-writes/bothmessage-write-capture-latest.ndjson`
|
||
(46 records: 23 outgoing + 23 incoming). Same priming chain as the
|
||
event flow:
|
||
|
||
```
|
||
Hist.GetV → Hist.GetI ×2 → Hist.ValCl ×2 → Hist.Open2 →
|
||
Stat.GetV ×2 → Stat.GETHI ×2 → Hist.UpdC3 →
|
||
Stat.GetSystemParameter ×7 → Trx.GetV → Stat.GetV → Retr.GetV →
|
||
Hist.EnsT2(Float) → Hist.Close2
|
||
```
|
||
|
||
No `RTag2`. The chain identical to the event flow except the EnsT2
|
||
payload is the analog CTagMetadata instead of the event one, and there
|
||
is NO RTag2 between Open2 and EnsT2 (events used RTag2 to register
|
||
`CmEventTagId`).
|
||
|
||
**Native EnsT2(Float) request body** (record 42, 322 bytes total; the
|
||
146-byte CTagMetadata `InBuff` payload is the new evidence target):
|
||
|
||
```text
|
||
67 03 00 01 00 00 00 04 C6 02 01 00 00 00 00 00
|
||
00 00 00 00 00 00 00 00 00 00 00 09 15 00 52 65
|
||
74 65 73 74 53 64 6B 57 72 69 74 65 53 61 6E 64
|
||
62 6F 78 FF FF FF FF FF FF FF FF FF FF FF FF FF
|
||
FF FF FF 09 18 00 53 44 4B 20 77 72 69 74 65 2D
|
||
52 45 20 73 61 6E 64 62 6F 78 20 74 61 67 09 04
|
||
00 4D 44 41 53 02 01 01 00 00 00 01 E8 03 00 00
|
||
D6 00 0E 4F BC DB DC 01 1A 03 09 04 00 74 65 73
|
||
74 10 27 00 00 00 00 00 00 00 00 F0 3F FE 00 01
|
||
01 01
|
||
```
|
||
|
||
Visible fields (still being decoded against the
|
||
`CTagUtil.ConvertTagMetadataToHistorianTag` IL at token `0x060055CE`):
|
||
|
||
- `09 15 00 RetestSdkWriteSandbox` (compact ASCII tag name, len 21)
|
||
- 16 bytes of `FF` — possibly a placeholder/sentinel for `CommonArchestraEventTypeId`-equivalent that's not used for analog
|
||
- `09 18 00 SDK write-RE sandbox tag` (compact ASCII description, len 24)
|
||
- `09 04 00 MDAS` (compact ASCII metadata provider)
|
||
- `09 04 00 test` (compact ASCII engineering unit)
|
||
- `0E 4F BC DB DC 01 1A 03` byte-pattern looks like an Int64 FILETIME (date-created ~2026)
|
||
- `10 27 00 00` = uint32 0x2710 = 10000 (storage-related)
|
||
- `00 00 00 00 00 00 F0 3F` = double 1.0 (likely IntegralDivisor or similar scaling)
|
||
- `FE 00 01 01 01` = trailer (matches event tag's `2F 27 01 01 01` shape)
|
||
|
||
**Decoder script** at `scripts/decode-write-capture.py` for the next
|
||
session.
|
||
|
||
## Phase 2 follow-on findings (2026-05-04, second pass)
|
||
|
||
**The AddS2 prereq is architectural, not protocol-level.** Three follow-up
|
||
attempts to trigger AddS2 from the sandbox harness all hit a client-side
|
||
gate before any AddS2 byte reaches the wire:
|
||
|
||
1. **TagKey synthetic→real override.** First attempt used the placeholder
|
||
`TagKey=10000000` returned by `HistorianAccess.AddTag`. Native
|
||
`AddStreamedValue` refused with error 168 "Tag not added to server".
|
||
The harness now ALWAYS resolves the real `wwTagKey` from
|
||
`Runtime.dbo.Tag` after AddTag (logged as `TagKeyOverride: Synthetic→RealFromSql`).
|
||
Result: error code shifts to **129 "Tag not found in cache"**.
|
||
|
||
2. **Server-cache settle wait.** Inserted up to 8s sleep between AddTag and
|
||
AddStreamedValue (configurable via `--write-resync-wait-seconds`). The
|
||
wait period contains 2× UpdC3 + 2× Trx/GetV keep-alives but no
|
||
server-side cache update — error 129 persists.
|
||
|
||
3. **Fresh process / fresh connection.** Skipped AddTag entirely
|
||
(`--write-skip-add-tag`) and ran AddStreamedValue alone against the
|
||
already-existing sandbox tag. New native client instance, new
|
||
client-side cache, new server session. **Same error 129 — no AddS2
|
||
bytes sent on wire.** Capture confirms 44 records ending in Close2.
|
||
|
||
**Interpretation.** The Historian engine's runtime tag cache only
|
||
ingests tags from configured IOServers / Application Server data pipelines,
|
||
not from `HistorianAccess.AddTag`-only client flows. `HistorianAccess.AddTag`
|
||
populates `Runtime.dbo.Tag` (we confirmed wwTagKey=240 was created) but
|
||
does not register the tag with the live cache that `AddStreamedValue`
|
||
checks. That registration happens server-side when an upstream data
|
||
producer (an OPC driver, the AnE event subsystem, the Application Server
|
||
attribute store, etc.) claims the tag.
|
||
|
||
For SDK purposes this means **`WriteValueAsync` cannot be implemented as
|
||
a generic client API against this server architecture.** The SDK's writeable
|
||
surface is realistically:
|
||
|
||
- ✅ `EnsureTagAsync` (drives EnsT2 — 146-byte payload captured)
|
||
- ✅ `DeleteTagAsync` (drives DelT — not yet captured but should be straightforward)
|
||
- ❌ `WriteValueAsync` — won't work as designed; the server gates the
|
||
data path on tags being live in its in-memory cache
|
||
- ❓ `WriteRevisionAsync` — `HistorianAccess.AddRevisionValuesBegin/Value/End`
|
||
may use a different code path (intended for editing existing historized
|
||
data); needs a separate capture against an existing tag with stored history
|
||
|
||
Phase 2 effective deliverables:
|
||
|
||
- ✅ NativeTraceHarness `--scenario write` extension
|
||
- ✅ EnsT2(Float) 146-byte CTagMetadata wire bytes
|
||
- ✅ Sandbox tag `RetestSdkWriteSandbox` in Runtime DB (wwTagKey=240)
|
||
- ⏸ AddS2 — blocked architecturally; **not just a protocol gap**
|
||
- ⏸ DelT — not yet captured (need `--write-delete-after` run)
|
||
- ⏸ Revision write path — separate capture needed against a historized
|
||
tag
|
||
|
||
## Phase 3 partial (2026-05-04) — EnsureTagAsync live, DeleteTagAsync partial
|
||
|
||
`HistorianTagWriteProtocol` + `HistorianWcfTagWriteOrchestrator` +
|
||
`HistorianClient.EnsureTagAsync`/`DeleteTagAsync` landed:
|
||
|
||
- `HistorianTagDefinition` public model (TagName/Description/EngineeringUnit/
|
||
DataType/MinEU/MaxEU; only `Float` data type currently supported live).
|
||
- `HistorianTagWriteProtocol.SerializeAnalogCTagMetadata` — produces 146-byte
|
||
payload byte-for-byte identical to the captured native EnsT2(Float) request.
|
||
- `HistorianTagWriteProtocol.SerializeDeleteTagNames` — `[ushort 0x6751,
|
||
ushort 1, uint count, per-tag (uint charCount + UTF-16 chars)]`.
|
||
- `HistorianWcfTagWriteOrchestrator` — both EnsT2 and DelT run the full
|
||
Stat-priming chain captured for the analog flow (UpdC3 + Stat.GetV ×3 +
|
||
Stat.GETHI ×2 + 7× GetSystemParameter + Trx.GetV + Retr.GetV).
|
||
- New tag-origin marker `0xC7` added to `MapDataType` (SDK-created tags have
|
||
byte 1 = 0xC7, distinct from 0xCF system / 0xC3 MDAS-routed).
|
||
|
||
Golden-byte tests (5): EnsT2(Float) byte-for-byte match against the captured
|
||
146-byte fixture; DelT(single tag) byte-for-byte; DelT(multi-tag); empty list
|
||
throws; different-inputs-produce-different-bytes.
|
||
|
||
Live integration test
|
||
(`EnsureTagAsync_AndDeleteTagAsync_RoundTrip_AgainstLocalHistorian`,
|
||
gated by `HISTORIAN_WRITE_SANDBOX_TAG=RetestSdkWriteSandbox`): EnsureTagAsync
|
||
followed by GetTagMetadataAsync confirms the sandbox tag is created in
|
||
the Runtime DB. Test passes 130/130 in the full suite.
|
||
|
||
**Known DelT gap.** SDK's DeleteTagAsync currently returns true but the
|
||
server-side cascading deletion does not always complete — the row remains
|
||
in `Runtime.dbo.Tag` even after the call returns. The captured native flow's
|
||
DelT removes the tag cleanly (verified via the harness with
|
||
`--write-delete-after`), so something the native code does between or
|
||
around the WCF DelT call is missing from our orchestrator. The harness
|
||
cleanup path remains the documented workaround for sandbox housekeeping.
|
||
|
||
## Phase 2 remaining work (revised — narrower scope)
|
||
|
||
1. Decode the 146-byte EnsT2(Float) CTagMetadata against the IL of
|
||
`CTagUtil.ConvertTagMetadataToHistorianTag` (token `0x060055CE`),
|
||
then implement `HistorianAddTagsProtocol.SerializeAnalogCTagMetadata`.
|
||
Same approach for discrete/string variants — capture each by passing
|
||
`--write-data-type Discrete` / `String` to the harness.
|
||
2. Capture DelT wire bytes by running the harness with
|
||
`--write-delete-after`.
|
||
3. Implement public `EnsureTagAsync` + `DeleteTagAsync` only. **Drop
|
||
`WriteValueAsync` from this plan.**
|
||
4. (Stretch) probe `AddRevisionValuesBegin/Value/End` against a tag that
|
||
IS in the server cache (e.g., SysTimeSec) to see whether the revision
|
||
path bypasses the cache check.
|
||
|
||
`WriteValueAsync` is now an OPEN QUESTION: is the only viable path for
|
||
client-driven writes the AVEVA REST API or the Application Server SDK?
|
||
File a separate plan for that investigation if SDK consumers actually
|
||
need data-write support.
|
||
|
||
## Phase 1 findings (recorded here, not implementing)
|
||
|
||
### §3.4 ModifyData/DeleteData — ELIMINATED FROM SCOPE
|
||
|
||
`methods aahClientManaged.dll` returns no managed wrapper for any of:
|
||
`EditValue`, `ModifyValue`, `EditData`, `DeleteData`, `ModifyData`,
|
||
`OverwriteData`. Per the plan's §3.4 disposition rule, this op is
|
||
REST-only / SMC-only and remains out of scope for the SDK.
|
||
|
||
### §4.a Native serializers identified (token IDs for future Phase 2)
|
||
|
||
The wrapper does have managed-public write API:
|
||
|
||
| Public method | Token | Used for |
|
||
|---|---|---|
|
||
| `ArchestrA.HistorianAccess.AddTag` | `0x0600619A` | Creates a new tag (drives `EnsT2`) |
|
||
| `ArchestrA.HistorianAccess.AddStreamedValue` | `0x0600618C/D/E` (3 overloads) | Pushes one timestamped value (drives `AddS2`) |
|
||
| `ArchestrA.HistorianAccess.AddNonStreamedValue` | `0x0600618F/90` (2 overloads) | Pushes one timestamped value, non-stream mode |
|
||
| `ArchestrA.HistorianAccess.DeleteTags` | `0x060061A4` | Removes tags (drives `DelT`) |
|
||
| `ArchestrA.HistorianAccess.AddVersionedStreamedValue` | `0x0600616F` | Pushes versioned value (rev edit) |
|
||
| `ArchestrA.HistorianAccess.AddRevisionValuesBegin/Value/End/AddRevisionValues` | `0x06006175-77, 0x0600617F` | Multi-row revision write (replaces `ModifyData` use case) |
|
||
|
||
So even though the engine doesn't expose `ModifyData` over WCF, the
|
||
**revision-write path** (`AddRevisionValuesBegin → AddRevisionValue * N →
|
||
AddRevisionValuesEnd`) covers the bulk-modify use case. This is a NEW
|
||
discovery worth folding into the Phase 2 scope.
|
||
|
||
Native serializer for `EnsT2(analog/discrete/string)`:
|
||
**`<Module>.CTagUtil.ConvertTagMetadataToHistorianTag`** at token
|
||
`0x060055CE` (412 IL instructions, 10 locals). Calls `CTagMetadata.GetUnit`,
|
||
`GetMessage0`, `GetMessage1`, `GetMaxLength`, `GetMinRaw`, `GetMaxRaw`,
|
||
`GetMinEU`, `GetMaxEU`, `GetIntegralDivisor`, `GetDefaultTagRate`,
|
||
`GetRolloverValue`, plus `CDataType.IsAnalog/IsWideString/GetRawType/
|
||
GetTagType` — i.e. every field the analog `CTagMetadata` shape would
|
||
need is wired through this method. Decoding it line-by-line **OR**
|
||
capturing live wire bytes against a sandbox tag are the two ways
|
||
forward.
|
||
|
||
WCF wrapper for `AddS2`: **`<Module>.CHistoryConnectionWCF.AddStreamValuesToHistorian`**
|
||
at token `0x0600404C`. Confirms the on-wire shape is
|
||
`IHistoryServiceContract2.AddStreamValues2(string handle, byte[] pBuf,
|
||
out byte[] errorBuffer)` — matches our existing contract. Handle is
|
||
the same Open2 v6 session GUID we already extract.
|
||
|
||
### Phase 2 chicken-and-egg resolved
|
||
|
||
Per §5 ordering: §3.1 (EnsT2) must come before §3.2 (AddS2) because
|
||
AddS2 needs an existing tag. The sandbox tag itself is created BY
|
||
the first §3.1 EnsT2 test. So the very first write-flow run creates
|
||
`RetestSdkWriteSandbox`. No SMC required — the chain is closed.
|
||
|
||
### Open question (was §8.6) answered
|
||
|
||
The wrapper exposes `AddStreamedValue` AND `AddNonStreamedValue`.
|
||
The latter is the documented path for backfilling values older than
|
||
`RealTimeWindow`. So the SDK should expose both modes, not just
|
||
`AddStreamedValue`. Update the success criteria for `AddS2`
|
||
accordingly.
|
||
|
||
### Phase 2 next steps (NOT EXECUTED in this session)
|
||
|
||
1. Extend `tools/AVEVA.Historian.NativeTraceHarness/Program.cs` with a
|
||
`--scenario write` that calls `HistorianAccess.AddTag` (creating
|
||
`RetestSdkWriteSandbox` if absent) followed by
|
||
`HistorianAccess.AddStreamedValue`. New args:
|
||
`--write-sandbox-tag <name>` (default: `RetestSdkWriteSandbox`),
|
||
`--write-value <numeric>`, `--write-data-type analog|discrete|string`.
|
||
2. Run the harness with `instrument-wcf-writemessage` +
|
||
`instrument-wcf-readmessage` instrumented copies of
|
||
`aahClientManaged.dll` to capture the full write flow.
|
||
3. Decode `EnsT2(analog)` `InBuff` bytes against the IL of
|
||
`CTagUtil.ConvertTagMetadataToHistorianTag` (token `0x060055CE`).
|
||
4. Decode `AddS2` `pBuf` bytes against the IL of
|
||
`CHistoryConnectionWCF.AddStreamValuesToHistorian` (token
|
||
`0x0600404C`).
|
||
5. Implement `WriteValueAsync`, `EnsureTagAsync`, `DeleteTagAsync` per
|
||
§4.e of the original plan; live tests gated by
|
||
`HISTORIAN_WRITE_SANDBOX_TAG`.
|
||
|
||
Phase 2 was deferred because (a) it requires extending the harness
|
||
(non-trivial scaffolding) and (b) per safety §1, even sandbox-tag
|
||
writes warrant explicit operator approval before the first run. The
|
||
operator decides whether to proceed; if yes, the instructions above
|
||
are executable as-is.
|
||
|
||
---
|
||
|
||
Original plan content below.
|
||
|
||
|
||
|
||
## 1. Goal
|
||
|
||
"Write commands work" means the production SDK at
|
||
`src/AVEVA.Historian.Client/` performs these operations end-to-end
|
||
against a live AVEVA Historian, with parsed responses, golden-byte
|
||
unit tests, and gated live integration tests.
|
||
|
||
In scope:
|
||
|
||
1. **`AddS2` (`IHistoryServiceContract2.AddStreamValues2`)** — push
|
||
one or more timestamped samples for an existing historized tag.
|
||
Primary use case: an OPC UA driver pushing values to the
|
||
Historian.
|
||
2. **`EnsT2` (`IHistoryServiceContract2.EnsureTags2`) for
|
||
analog/discrete/string data tags** — partially decoded for the
|
||
`CM_EVENT` AnE-event tag in
|
||
`src/AVEVA.Historian.Client/Wcf/HistorianAddTagsProtocol.cs`. The
|
||
`CTagMetadata` byte layout for `CDataType` ∈ {1, 2, 3, 4} is the
|
||
new evidence target.
|
||
3. **`DelT` (`IHistoryServiceContract2.DeleteTags`)** — needed for
|
||
safe sandbox cleanup during RE.
|
||
4. **`ModifyData` / `DeleteData`** — only if §3.4 method discovery
|
||
confirms a managed WCF op exists.
|
||
|
||
Out of scope: tag-extended-properties (`AddTEx` / `DelTep`),
|
||
`ExKey`, `SetSFP`, snapshot send (`SendSnapshotBegin/End/Snapshot`),
|
||
tag-id-pair maintenance, shard splits, flush ops, all
|
||
`IStorageServiceContract` writes (engine-internal — see §6.d), event
|
||
writes (events come from AVEVA AnE, we only read them), schema
|
||
changes (forbidden over the wire).
|
||
|
||
## 2. Safety Constraints
|
||
|
||
The Runtime DB is production data even on `localhost`. `AddS2`
|
||
writes are persistent — they go to compressed history blocks and
|
||
cannot be removed through any client-facing surface.
|
||
|
||
Hard rules:
|
||
|
||
1. **Single dedicated sandbox tag.** Add env var
|
||
`HISTORIAN_WRITE_SANDBOX_TAG = "RetestSdkWriteSandbox"`. Live
|
||
write tests refuse to run when unset, even when other
|
||
`HISTORIAN_*` vars are set.
|
||
2. **Never write to** any tag named in `HISTORIAN_TEST_TAG`,
|
||
`HISTORIAN_TAG_FILTER`, the docs, the test fixtures, or the
|
||
captured RE ndjson. The read fixture
|
||
`OtOpcUaParityTest_001.Counter` is OFF-LIMITS for writes.
|
||
3. **Documented rollback.** Every write session records its time
|
||
window to
|
||
`artifacts/reverse-engineering/write-sandbox-window-<stamp>.json`
|
||
so SQL `SELECT * FROM History WHERE wwTagKey = ? AND DateTime
|
||
BETWEEN @s AND @e` can identify exactly which rows the session
|
||
inserted. Tag rollback is via decoded `DelT` (§3.3) once
|
||
available, or manually via System Management Console until then.
|
||
4. **Time bounds on writes.** Every `AddS2` test uses
|
||
`DateTime.UtcNow` ± a small offset, so writes always land inside
|
||
the live `RealTimeWindow` / `FutureTimeThreshold` system
|
||
parameters and cannot accidentally overwrite older blocks.
|
||
5. **No customer / corporate hosts.** `localhost` only.
|
||
6. **Sanitization scan after every session:**
|
||
`rg -n "(?i)(password|credential|secret|token|<known-sensitive-host>|<known-sensitive-machine>|<known-sensitive-user>)" docs\reverse-engineering scripts tools docs\plans`.
|
||
|
||
Soft rules:
|
||
|
||
- Use a separate captures dir
|
||
(`artifacts/reverse-engineering/instrumented-wcf-writemessage-writes/`)
|
||
so write captures don't contaminate the existing read/event
|
||
ndjson.
|
||
- New integration tests follow the existing gating pattern in
|
||
`tests/AVEVA.Historian.Client.Tests/HistorianClientIntegrationTests.cs`
|
||
(`Skip = ...` when env var unset).
|
||
|
||
## 3. Discovery Workstreams
|
||
|
||
### 3.1 EnsT2 for analog/discrete/string tags (priority 1)
|
||
|
||
- WCF op: `aa/Hist/EnsT2`.
|
||
- Contract:
|
||
`src/AVEVA.Historian.Client/Wcf/Contracts/IHistoryServiceContract2.cs:82-89`,
|
||
already declared with `[MessageParameter(Name = "InBuff" / "OutBuff")]`.
|
||
- Existing code: `HistorianAddTagsProtocol.SerializeCmEventCTagMetadata`
|
||
builds the `CDataType=5` (event) shape.
|
||
- Missing: the `CTagMetadata` byte layout for `CDataType ∈ {1, 2,
|
||
3, 4}` (analog double, discrete, string, analog int per the
|
||
type-code table in `data-query-request-ctor-il-latest.txt`);
|
||
whether the optional-mask `0x0086` and the 5-byte trailer
|
||
`2F 27 01 01 01` change per type; analog engineering-units / range
|
||
/ deadband fields (likely populate the bytes that are zero in the
|
||
event-tag fixture).
|
||
|
||
### 3.2 AddS2 stream values (priority 1)
|
||
|
||
- WCF op: `aa/Hist/AddS2`.
|
||
- Contract:
|
||
`src/AVEVA.Historian.Client/Wcf/Contracts/IHistoryServiceContract2.cs:75-80`,
|
||
already has `[MessageParameter(Name = "pBuf")]`. **Audit
|
||
requirement:** verify against `ildasm aahClientAccessPoint.exe`
|
||
that `Handle` and `errorBuffer` parameter names also match — the
|
||
handoff's parameter-name-mismatch class has bitten ~30 ops.
|
||
- Missing: entire `pBuf` byte layout (likely `UInt16 version + UInt32
|
||
sampleCount + N × {tagId GUID, FILETIME, qualityByte, value typed
|
||
by CDataType}`); whether `Handle` is the same Open2 v6 session GUID
|
||
as `UpdC3`/`RTag2`/`EnsT2`; the auth-chain prereqs (event flow
|
||
needed Stat priming + Trx/Stat/Retr `GetV` between RTag2 and EnsT2;
|
||
writes may have a different chain); success vs error response
|
||
shape.
|
||
|
||
### 3.3 DelT tag deletion (priority 2 — needed for safe RE)
|
||
|
||
- WCF op: `aa/Hist/DelT`.
|
||
- Contract:
|
||
`src/AVEVA.Historian.Client/Wcf/Contracts/IHistoryServiceContract2.cs:21-30`.
|
||
- Missing: `tagNames` byte layout (likely length-prefixed
|
||
compact-ASCII per the handoff convention); whether server refuses
|
||
to delete tags with stored history or cascades; whether `DelT` is
|
||
sufficient to fully unregister or leaves orphan rows in
|
||
`Runtime.dbo.Tag`.
|
||
|
||
### 3.4 ModifyData / DeleteData (priority 3 — exists?)
|
||
|
||
No corresponding WCF op is currently declared. **First step:** static
|
||
inspection to confirm any managed wrapper exists.
|
||
|
||
```powershell
|
||
dotnet run --no-build --project tools\AVEVA.Historian.ReverseEngineering -- methods current\aahClientManaged.dll EditValue
|
||
dotnet run --no-build --project tools\AVEVA.Historian.ReverseEngineering -- methods current\aahClientManaged.dll ModifyValue
|
||
dotnet run --no-build --project tools\AVEVA.Historian.ReverseEngineering -- methods current\aahClientManaged.dll EditData
|
||
dotnet run --no-build --project tools\AVEVA.Historian.ReverseEngineering -- methods current\aahClientManaged.dll DeleteData
|
||
```
|
||
|
||
If no managed wrapper exists, this op is REST-only / SMC-only —
|
||
mark as **out of scope** in this doc. Otherwise decode like
|
||
§3.1/§3.2.
|
||
|
||
Parallelism: 3.1 and 3.3 can be developed in parallel because the
|
||
operator can create the sandbox tag manually via SMC while SDK code
|
||
is being written. 3.2 cannot meaningfully proceed until 3.1 (or the
|
||
manual tag) exists. 3.4 method discovery is cheap and may eliminate
|
||
its own scope.
|
||
|
||
## 4. RE Steps in Execution Order
|
||
|
||
For each workstream above, run these five steps. Mirrors the read
|
||
+ event flows that recovered the existing protocol.
|
||
|
||
### 4.a Static method discovery
|
||
|
||
Find the native serializer:
|
||
|
||
```powershell
|
||
dotnet run --no-build --project tools\AVEVA.Historian.ReverseEngineering -- methods current\aahClientManaged.dll AddS
|
||
dotnet run --no-build --project tools\AVEVA.Historian.ReverseEngineering -- methods current\aahClientManaged.dll EnsureTag
|
||
dotnet run --no-build --project tools\AVEVA.Historian.ReverseEngineering -- methods current\aahClientManaged.dll DeleteTag
|
||
```
|
||
|
||
Dump IL for each method of interest:
|
||
|
||
```powershell
|
||
dotnet run --no-build --project tools\AVEVA.Historian.ReverseEngineering -- dnlib-method --instructions current\aahClientManaged.dll <Type::Method>
|
||
```
|
||
|
||
Save sanitized excerpts to
|
||
`docs/reverse-engineering/dnlib-<op>-il-latest.txt`.
|
||
|
||
### 4.b Wire-byte capture for the request
|
||
|
||
Same IL-rewrite tooling that captured the 27 outgoing event calls:
|
||
|
||
```powershell
|
||
$captureDir = "artifacts\reverse-engineering\instrumented-wcf-writemessage-writes"
|
||
New-Item -ItemType Directory -Force -Path $captureDir | Out-Null
|
||
dotnet run --no-build --project tools\AVEVA.Historian.ReverseEngineering -- instrument-wcf-writemessage current\aahClientManaged.dll "$captureDir\aahClientManaged.dll"
|
||
Copy-Item -Force "$captureDir\aahClientManaged.dll" "$captureDir\current-copy\aahClientManaged.dll"
|
||
$env:AVEVA_HISTORIAN_RE_CAPTURE = (Resolve-Path $captureDir).Path + "\writemessage-capture-write-latest.ndjson"
|
||
```
|
||
|
||
A new harness scenario `--scenario write` needs to be added to
|
||
`tools/AVEVA.Historian.NativeTraceHarness` to drive the native
|
||
wrapper's `AddStreamValues2` against the sandbox tag. Suggested
|
||
new args: `--write-sandbox-tag`, `--write-value`.
|
||
|
||
### 4.c Wire-byte capture for the response
|
||
|
||
Symmetric `instrument-wcf-readmessage`:
|
||
|
||
```powershell
|
||
dotnet run --no-build --project tools\AVEVA.Historian.ReverseEngineering -- instrument-wcf-readmessage current\aahClientManaged.dll "$captureDir\aahClientManaged.dll"
|
||
```
|
||
|
||
The success response for `AddS2` is just `<AddS2Result>true</…>` +
|
||
empty `errorBuffer`. **Capture at least one negative case** (write
|
||
to non-existent tag, or write with malformed CDataType) so the
|
||
orchestrator can surface diagnostics like
|
||
`HistorianWcfEventOrchestrator.LastErrorBufferDescription`.
|
||
|
||
### 4.d Decode against IL
|
||
|
||
Strip SOAP/MDAS envelope; align byte offsets against the native
|
||
serializer IL from 4.a (the `ldc.i4 / call WriteByte` sequence
|
||
makes field order and constants explicit); cross-reference the
|
||
`CDataType` table from `data-query-request-ctor-il-latest.txt` to
|
||
interpret typed value bytes; write a parser-and-builder pair and
|
||
verify against the captured bytes before committing.
|
||
|
||
### 4.e Implement managed serializer + tests
|
||
|
||
New code under `src/AVEVA.Historian.Client/Wcf/`:
|
||
|
||
- `HistorianAddStreamValuesProtocol.cs` — `Serialize(...)` returns
|
||
`byte[] pBuf`, mirroring `HistorianAddTagsProtocol`.
|
||
- Extend (or split) `HistorianAddTagsProtocol` for the analog /
|
||
discrete / string `EnsT2` shapes.
|
||
- `HistorianWcfWriteOrchestrator.cs` — chains `Hist.GetV →
|
||
Hist.ValCl × 2 → Hist.Open2 → UpdC3 → priming chain (TBD per
|
||
§3.2) → AddS2 loop → Close2`.
|
||
|
||
Public surface on `HistorianClient`:
|
||
|
||
- `WriteValueAsync(tag, value, timestampUtc, quality)`
|
||
- `WriteValuesAsync(IReadOnlyList<HistorianSampleWrite>)`
|
||
- `EnsureTagAsync(HistorianTagDefinition)`
|
||
- `DeleteTagAsync(string tagName)`
|
||
|
||
Until evidence supports each path, throw
|
||
`ProtocolEvidenceMissingException` (mirrors the existing read
|
||
guardrail).
|
||
|
||
Unit tests under `tests/AVEVA.Historian.Client.Tests/Wcf/`:
|
||
|
||
- `WcfAddStreamValuesProtocolTests` — golden-byte tests for one
|
||
analog, one discrete, one string write.
|
||
- `WcfEnsureTagsProtocolTests` — golden-byte tests for the
|
||
analog/discrete/string `CTagMetadata` shapes.
|
||
- Extend `ProtocolGuardrailTests` so any not-yet-implemented write
|
||
path still throws `ProtocolEvidenceMissingException`.
|
||
|
||
Live integration tests in `HistorianClientIntegrationTests.cs`,
|
||
gated on `HISTORIAN_WRITE_SANDBOX_TAG`:
|
||
`WriteValueAsync_WithinDocumentedWindow_PersistsToHistorianDb`
|
||
writes a unique value, reads it back via `ReadRawAsync`, and
|
||
verifies via direct `sqlcmd` to the History extension table.
|
||
|
||
## 5. Order of Operations
|
||
|
||
```
|
||
3.4 method discovery (cheap; may eliminate scope)
|
||
│
|
||
▼
|
||
3.1 EnsT2 (analog/discrete/string) ──► sandbox tag exists
|
||
│
|
||
├─────────────────────────────┐
|
||
▼ ▼
|
||
3.2 AddS2 (priority 1) 3.3 DelT (sandbox cleanup)
|
||
│
|
||
▼
|
||
3.4 ModifyData/DeleteData (only if 3.4 confirmed scope)
|
||
│
|
||
▼
|
||
public surface, golden-byte tests, integration tests
|
||
```
|
||
|
||
3.2 is the headline win and depends only on 3.1 (or a manually
|
||
created sandbox tag). 3.3 must land before any commit that
|
||
programmatically creates new tags; until then, manual SMC deletion
|
||
is the documented rollback.
|
||
|
||
## 6. Risks and Mitigations
|
||
|
||
### 6.a Auth chain may differ for writes
|
||
|
||
Reads use `Hist.Open2(ConnectionMode = 0x402)`. Events use the same
|
||
`0x402` plus a Stat-priming chain. Writes may need a different
|
||
mode (the handoff notes `0x501` was an unverified guess for
|
||
events; writes may legitimately need `0x401` or another value).
|
||
|
||
Mitigation: capture the *full* WriteMessage sequence for a native
|
||
write session (not just `AddS2`) to see what `Open2` payload and
|
||
priming calls the native wrapper sends.
|
||
|
||
### 6.b Server-side session-table requirement
|
||
|
||
Writes may require `RTag2` after `EnsT2` and before `AddS2` (the
|
||
event flow needs `RTag2(CmEventTagId)`). The "tag identifier" the
|
||
server returns from `EnsT2` may differ from the GUID the client
|
||
seeded.
|
||
|
||
Mitigation: capture the analog `EnsT2` `OutBuff` (event flow's was
|
||
a 45-byte echo) and verify whether subsequent `AddS2` payloads
|
||
reference the client-seeded GUID, the server-returned GUID, or a
|
||
numeric `wwTagKey`. SQL ground truth: `SELECT TagName, wwTagKey
|
||
FROM Tag WHERE TagName = '...'`.
|
||
|
||
### 6.c Silent-success failure mode
|
||
|
||
`AddS2` may return `true` but no row appears in the History
|
||
extension table — the engine silently drops samples outside the
|
||
`FutureTimeThreshold` / `RealTimeWindow` system parameters (which
|
||
the event flow now reads).
|
||
|
||
Mitigation: always write at `DateTime.UtcNow`; cross-check with
|
||
SQL after every test:
|
||
|
||
```sql
|
||
SELECT TOP 5 DateTime, Value, QualityDetail
|
||
FROM History
|
||
WHERE wwTagKey = (SELECT wwTagKey FROM Tag WHERE TagName = @sandbox)
|
||
AND DateTime BETWEEN @windowStart AND @windowEnd
|
||
ORDER BY DateTime DESC;
|
||
```
|
||
|
||
Surface `FutureTimeThreshold` / `RealTimeWindow` via existing
|
||
`GetSystemParameterAsync` so failures are diagnosable.
|
||
|
||
### 6.d Storage service vs History service
|
||
|
||
`IStorageServiceContract` also exposes `AddT/AddS/AddS2/DelT`. The
|
||
working hypothesis is that `/Hist` is client-facing and `/Stor` is
|
||
engine-internal, but it's not yet verified.
|
||
|
||
Mitigation: the WriteMessage capture (§4.b) shows the actual
|
||
service path on the wire. If it goes to `/Stor`, update the
|
||
orchestrator. Do NOT preemptively implement against both.
|
||
|
||
### 6.e Parameter-name mismatches
|
||
|
||
Handoff already flagged `EnsT`, `EnsT2`, `RTag2`, `ExKey`, `StJb`,
|
||
`GtJb` for the same `inBuff`/`inputBuffer` mismatch class that
|
||
broke reads for weeks. Until each is audited against the server
|
||
contract, requests bind to null and the server NREs.
|
||
|
||
Mitigation: before the first write WriteMessage capture, run an
|
||
`ildasm` audit against `aahClientAccessPoint.exe` for the exact
|
||
parameter names of `EnsT2`, `AddS2`, and `DelT`, and reconcile
|
||
against the existing `[MessageParameter]` attributes.
|
||
|
||
### 6.f Customer-data exposure in capture files
|
||
|
||
Write captures contain the sandbox tag name and any value the test
|
||
wrote. Not secrets, but noise.
|
||
|
||
Mitigation: keep all
|
||
`instrumented-wcf-writemessage-writes/` artifacts under
|
||
`artifacts/` (already gitignored). Sanitize tag names to
|
||
`<sandbox-tag>` before committing decoded bytes into
|
||
`docs/reverse-engineering/`.
|
||
|
||
## 7. Success Criteria
|
||
|
||
Per op:
|
||
|
||
- **`EnsT2(analog)`**: `EnsureTagAsync(new HistorianTagDefinition {
|
||
Name = sandbox, DataType = Analog })` returns success;
|
||
`sqlcmd -E -S . -d Runtime -Q "SELECT TagName FROM Tag WHERE
|
||
TagName = '...'"` returns one row.
|
||
- **`EnsT2(discrete, string)`**: same shape with corresponding
|
||
`DataType`; SQL check uses `DiscreteTag` / `StringTag` view.
|
||
- **`AddS2`**: `WriteValueAsync(sandbox, 42.0, DateTime.UtcNow)`
|
||
returns success; `ReadRawAsync` returns the value;
|
||
`SELECT TOP 1 Value FROM History WHERE wwTagKey = ? AND DateTime
|
||
BETWEEN ? AND ?` returns the same value.
|
||
- **`DelT`**: `DeleteTagAsync(sandbox)` returns success and SQL
|
||
returns zero rows from `Tag`.
|
||
- **`ModifyData` / `DeleteData`**: deferred until §3.4 method
|
||
discovery confirms scope.
|
||
|
||
Cross-cutting:
|
||
|
||
- All new code in `src/AVEVA.Historian.Client/` is pure managed
|
||
.NET 10. No new P/Invoke beyond the existing `HistorianSspiClient`.
|
||
- Every new op has a golden-byte unit test.
|
||
- `dotnet test .\Histsdk.slnx --no-build --logger
|
||
"console;verbosity=minimal"` passes 100%.
|
||
- With `HISTORIAN_HOST=localhost`,
|
||
`HISTORIAN_WRITE_SANDBOX_TAG=RetestSdkWriteSandbox` set, write
|
||
integration tests pass and leave zero residue (test `Dispose`
|
||
calls `DelT` for cleanup).
|
||
- Sanitization scan returns no real secrets.
|
||
- `CLAUDE.md` "Required SDK Surface" updated to add the new write
|
||
ops — this is a SCOPE CHANGE that must land *alongside* the
|
||
evidence, not before. Do not update the SDK surface doc until
|
||
3.1 + 3.2 are at least live-test-green.
|
||
|
||
## 8. Open Questions
|
||
|
||
1. Does `AddS2` go through `/Hist` or `/Stor` on the wire?
|
||
2. Does the sandbox tag need pre-configuration via System
|
||
Management Console once before `EnsT2` will accept it from a
|
||
client (e.g. for `Storage` / `wwDomain` rows the wire protocol
|
||
may not be able to populate)?
|
||
3. What `ConnectionMode` does the native wrapper use for write
|
||
sessions — `0x402` (read mode reused), `0x401`, or something
|
||
else?
|
||
4. Does `EnsT2(analog)` require any optional Archestra
|
||
engineering-units fields, or are they purely cosmetic? Affects
|
||
how minimal `HistorianTagDefinition` can be.
|
||
5. Server-side throttles on writes (max samples per AddS2, max
|
||
calls per second) — need to surface as batching guidance?
|
||
6. What does the server return when `AddS2` is called with a
|
||
timestamp older than the tag's earliest stored block? Some
|
||
historians silently drop, some error, some accept-and-overwrite.
|
||
7. Does the SDK expose write quality as the same
|
||
`HistorianSample.Quality` enum used on reads, or a smaller
|
||
subset (good/bad)?
|
||
8. Is there a managed-side `DelT` path at all? If
|
||
`aahClientManaged` only exposes deletion via SMC, §3.3 is
|
||
"manual SMC only" and must be documented as such.
|
||
|
||
## 9. Docs To Update Once Each Workstream Lands
|
||
|
||
- `CLAUDE.md` "Required SDK Surface" — add `WriteValueAsync`,
|
||
`EnsureTagAsync`, `DeleteTagAsync` once 3.1+3.2+3.3 land.
|
||
- `AGENTS.md` "Required SDK Surface" — same; update the "alarm-event
|
||
write path is dormant" note.
|
||
- `docs/reverse-engineering/handoff.md` — add a "Write-flow prereqs"
|
||
section symmetric to the existing "Event-flow prereqs".
|
||
- `docs/reverse-engineering/wcf-contract-evidence.md` — add evidence
|
||
rows for `EnsT2(analog/discrete/string)`, `AddS2`, `DelT`.
|
||
- `docs/reverse-engineering/implementation-status.md` — flip
|
||
status from "out of scope" to "implemented".
|
||
- `README.md` — operation status table.
|