histsdk/docs/plans/write-commands-reverse-engineering.md

Plan: Reverse-Engineering Write Commands

Status: PLAN ONLY (no implementation yet). Extends the read/event work in docs/reverse-engineering/handoff.md (2026-05-04).

1. Goal

"Write commands work" means the production SDK at src/AVEVA.Historian.Client/ performs these operations end-to-end against a live AVEVA Historian, with parsed responses, golden-byte unit tests, and gated live integration tests.

In scope:

1. AddS2 (IHistoryServiceContract2.AddStreamValues2) — push one or more timestamped samples for an existing historized tag. Primary use case: an OPC UA driver pushing values to the Historian.
2. EnsT2 (IHistoryServiceContract2.EnsureTags2) for analog/discrete/string data tags — partially decoded for the CM_EVENT AnE-event tag in src/AVEVA.Historian.Client/Wcf/HistorianAddTagsProtocol.cs. The CTagMetadata byte layout for CDataType ∈ {1, 2, 3, 4} is the new evidence target.
3. DelT (IHistoryServiceContract2.DeleteTags) — needed for safe sandbox cleanup during RE.
4. ModifyData / DeleteData — only if §3.4 method discovery confirms a managed WCF op exists.

Out of scope: tag-extended-properties (AddTEx / DelTep), ExKey, SetSFP, snapshot send (SendSnapshotBegin/End/Snapshot), tag-id-pair maintenance, shard splits, flush ops, all IStorageServiceContract writes (engine-internal — see §6.d), event writes (events come from AVEVA AnE, we only read them), schema changes (forbidden over the wire).

2. Safety Constraints

The Runtime DB is production data even on localhost. AddS2 writes are persistent — they go to compressed history blocks and cannot be removed through any client-facing surface.

Hard rules:

1. Single dedicated sandbox tag. Add env var HISTORIAN_WRITE_SANDBOX_TAG = "RetestSdkWriteSandbox". Live write tests refuse to run when unset, even when other HISTORIAN_* vars are set.
2. Never write to any tag named in HISTORIAN_TEST_TAG, HISTORIAN_TAG_FILTER, the docs, the test fixtures, or the captured RE ndjson. The read fixture OtOpcUaParityTest_001.Counter is OFF-LIMITS for writes.
3. Documented rollback. Every write session records its time window to artifacts/reverse-engineering/write-sandbox-window-<stamp>.json so SQL SELECT * FROM History WHERE wwTagKey = ? AND DateTime BETWEEN @s AND @e can identify exactly which rows the session inserted. Tag rollback is via decoded DelT (§3.3) once available, or manually via System Management Console until then.
4. Time bounds on writes. Every AddS2 test uses DateTime.UtcNow ± a small offset, so writes always land inside the live RealTimeWindow / FutureTimeThreshold system parameters and cannot accidentally overwrite older blocks.
5. No customer / corporate hosts. localhost only.
6. Sanitization scan after every session: rg -n "(?i)(password|credential|secret|token|<known-sensitive-host>|<known-sensitive-machine>|<known-sensitive-user>)" docs\reverse-engineering scripts tools docs\plans.

Soft rules:

• Use a separate captures dir (artifacts/reverse-engineering/instrumented-wcf-writemessage-writes/) so write captures don't contaminate the existing read/event ndjson.
• New integration tests follow the existing gating pattern in tests/AVEVA.Historian.Client.Tests/HistorianClientIntegrationTests.cs (Skip = ... when env var unset).

3. Discovery Workstreams

3.1 EnsT2 for analog/discrete/string tags (priority 1)

• WCF op: aa/Hist/EnsT2.
• Contract: src/AVEVA.Historian.Client/Wcf/Contracts/IHistoryServiceContract2.cs:82-89, already declared with [MessageParameter(Name = "InBuff" / "OutBuff")].
• Existing code: HistorianAddTagsProtocol.SerializeCmEventCTagMetadata builds the CDataType=5 (event) shape.
• Missing: the CTagMetadata byte layout for CDataType ∈ {1, 2, 3, 4} (analog double, discrete, string, analog int per the type-code table in data-query-request-ctor-il-latest.txt); whether the optional-mask 0x0086 and the 5-byte trailer 2F 27 01 01 01 change per type; analog engineering-units / range / deadband fields (likely populate the bytes that are zero in the event-tag fixture).

3.2 AddS2 stream values (priority 1)

• WCF op: aa/Hist/AddS2.
• Contract: src/AVEVA.Historian.Client/Wcf/Contracts/IHistoryServiceContract2.cs:75-80, already has [MessageParameter(Name = "pBuf")]. Audit requirement: verify against ildasm aahClientAccessPoint.exe that Handle and errorBuffer parameter names also match — the handoff's parameter-name-mismatch class has bitten ~30 ops.
• Missing: entire pBuf byte layout (likely UInt16 version + UInt32 sampleCount + N × {tagId GUID, FILETIME, qualityByte, value typed by CDataType}); whether Handle is the same Open2 v6 session GUID as UpdC3/RTag2/EnsT2; the auth-chain prereqs (event flow needed Stat priming + Trx/Stat/Retr GetV between RTag2 and EnsT2; writes may have a different chain); success vs error response shape.

3.3 DelT tag deletion (priority 2 — needed for safe RE)

• WCF op: aa/Hist/DelT.
• Contract: src/AVEVA.Historian.Client/Wcf/Contracts/IHistoryServiceContract2.cs:21-30.
• Missing: tagNames byte layout (likely length-prefixed compact-ASCII per the handoff convention); whether server refuses to delete tags with stored history or cascades; whether DelT is sufficient to fully unregister or leaves orphan rows in Runtime.dbo.Tag.

3.4 ModifyData / DeleteData (priority 3 — exists?)

No corresponding WCF op is currently declared. First step: static inspection to confirm any managed wrapper exists.

dotnet run --no-build --project tools\AVEVA.Historian.ReverseEngineering -- methods current\aahClientManaged.dll EditValue
dotnet run --no-build --project tools\AVEVA.Historian.ReverseEngineering -- methods current\aahClientManaged.dll ModifyValue
dotnet run --no-build --project tools\AVEVA.Historian.ReverseEngineering -- methods current\aahClientManaged.dll EditData
dotnet run --no-build --project tools\AVEVA.Historian.ReverseEngineering -- methods current\aahClientManaged.dll DeleteData

If no managed wrapper exists, this op is REST-only / SMC-only — mark as out of scope in this doc. Otherwise decode like §3.1/§3.2.

Parallelism: 3.1 and 3.3 can be developed in parallel because the operator can create the sandbox tag manually via SMC while SDK code is being written. 3.2 cannot meaningfully proceed until 3.1 (or the manual tag) exists. 3.4 method discovery is cheap and may eliminate its own scope.

4. RE Steps in Execution Order

For each workstream above, run these five steps. Mirrors the read

• event flows that recovered the existing protocol.

4.a Static method discovery

Find the native serializer:

dotnet run --no-build --project tools\AVEVA.Historian.ReverseEngineering -- methods current\aahClientManaged.dll AddS
dotnet run --no-build --project tools\AVEVA.Historian.ReverseEngineering -- methods current\aahClientManaged.dll EnsureTag
dotnet run --no-build --project tools\AVEVA.Historian.ReverseEngineering -- methods current\aahClientManaged.dll DeleteTag

Dump IL for each method of interest:

dotnet run --no-build --project tools\AVEVA.Historian.ReverseEngineering -- dnlib-method --instructions current\aahClientManaged.dll <Type::Method>

Save sanitized excerpts to docs/reverse-engineering/dnlib-<op>-il-latest.txt.

4.b Wire-byte capture for the request

Same IL-rewrite tooling that captured the 27 outgoing event calls:

$captureDir = "artifacts\reverse-engineering\instrumented-wcf-writemessage-writes"
New-Item -ItemType Directory -Force -Path $captureDir | Out-Null
dotnet run --no-build --project tools\AVEVA.Historian.ReverseEngineering -- instrument-wcf-writemessage current\aahClientManaged.dll "$captureDir\aahClientManaged.dll"
Copy-Item -Force "$captureDir\aahClientManaged.dll" "$captureDir\current-copy\aahClientManaged.dll"
$env:AVEVA_HISTORIAN_RE_CAPTURE = (Resolve-Path $captureDir).Path + "\writemessage-capture-write-latest.ndjson"

A new harness scenario --scenario write needs to be added to tools/AVEVA.Historian.NativeTraceHarness to drive the native wrapper's AddStreamValues2 against the sandbox tag. Suggested new args: --write-sandbox-tag, --write-value.

4.c Wire-byte capture for the response

Symmetric instrument-wcf-readmessage:

dotnet run --no-build --project tools\AVEVA.Historian.ReverseEngineering -- instrument-wcf-readmessage current\aahClientManaged.dll "$captureDir\aahClientManaged.dll"

The success response for AddS2 is just <AddS2Result>true</…> + empty errorBuffer. Capture at least one negative case (write to non-existent tag, or write with malformed CDataType) so the orchestrator can surface diagnostics like HistorianWcfEventOrchestrator.LastErrorBufferDescription.

4.d Decode against IL

Strip SOAP/MDAS envelope; align byte offsets against the native serializer IL from 4.a (the ldc.i4 / call WriteByte sequence makes field order and constants explicit); cross-reference the CDataType table from data-query-request-ctor-il-latest.txt to interpret typed value bytes; write a parser-and-builder pair and verify against the captured bytes before committing.

4.e Implement managed serializer + tests

New code under src/AVEVA.Historian.Client/Wcf/:

• HistorianAddStreamValuesProtocol.cs — Serialize(...) returns byte[] pBuf, mirroring HistorianAddTagsProtocol.
• Extend (or split) HistorianAddTagsProtocol for the analog / discrete / string EnsT2 shapes.
• HistorianWcfWriteOrchestrator.cs — chains Hist.GetV → Hist.ValCl × 2 → Hist.Open2 → UpdC3 → priming chain (TBD per §3.2) → AddS2 loop → Close2.

Public surface on HistorianClient:

• WriteValueAsync(tag, value, timestampUtc, quality)
• WriteValuesAsync(IReadOnlyList<HistorianSampleWrite>)
• EnsureTagAsync(HistorianTagDefinition)
• DeleteTagAsync(string tagName)

Until evidence supports each path, throw ProtocolEvidenceMissingException (mirrors the existing read guardrail).

Unit tests under tests/AVEVA.Historian.Client.Tests/Wcf/:

• WcfAddStreamValuesProtocolTests — golden-byte tests for one analog, one discrete, one string write.
• WcfEnsureTagsProtocolTests — golden-byte tests for the analog/discrete/string CTagMetadata shapes.
• Extend ProtocolGuardrailTests so any not-yet-implemented write path still throws ProtocolEvidenceMissingException.

Live integration tests in HistorianClientIntegrationTests.cs, gated on HISTORIAN_WRITE_SANDBOX_TAG: WriteValueAsync_WithinDocumentedWindow_PersistsToHistorianDb writes a unique value, reads it back via ReadRawAsync, and verifies via direct sqlcmd to the History extension table.

5. Order of Operations

3.4 method discovery (cheap; may eliminate scope)
        │
        ▼
3.1 EnsT2 (analog/discrete/string)  ──► sandbox tag exists
        │
        ├─────────────────────────────┐
        ▼                             ▼
3.2 AddS2 (priority 1)         3.3 DelT (sandbox cleanup)
        │
        ▼
3.4 ModifyData/DeleteData (only if 3.4 confirmed scope)
        │
        ▼
public surface, golden-byte tests, integration tests

3.2 is the headline win and depends only on 3.1 (or a manually created sandbox tag). 3.3 must land before any commit that programmatically creates new tags; until then, manual SMC deletion is the documented rollback.

6. Risks and Mitigations

6.a Auth chain may differ for writes

Reads use Hist.Open2(ConnectionMode = 0x402). Events use the same 0x402 plus a Stat-priming chain. Writes may need a different mode (the handoff notes 0x501 was an unverified guess for events; writes may legitimately need 0x401 or another value).

Mitigation: capture the full WriteMessage sequence for a native write session (not just AddS2) to see what Open2 payload and priming calls the native wrapper sends.

6.b Server-side session-table requirement

Writes may require RTag2 after EnsT2 and before AddS2 (the event flow needs RTag2(CmEventTagId)). The "tag identifier" the server returns from EnsT2 may differ from the GUID the client seeded.

Mitigation: capture the analog EnsT2 OutBuff (event flow's was a 45-byte echo) and verify whether subsequent AddS2 payloads reference the client-seeded GUID, the server-returned GUID, or a numeric wwTagKey. SQL ground truth: SELECT TagName, wwTagKey FROM Tag WHERE TagName = '...'.

6.c Silent-success failure mode

AddS2 may return true but no row appears in the History extension table — the engine silently drops samples outside the FutureTimeThreshold / RealTimeWindow system parameters (which the event flow now reads).

Mitigation: always write at DateTime.UtcNow; cross-check with SQL after every test:

SELECT TOP 5 DateTime, Value, QualityDetail
FROM History
WHERE wwTagKey = (SELECT wwTagKey FROM Tag WHERE TagName = @sandbox)
  AND DateTime BETWEEN @windowStart AND @windowEnd
ORDER BY DateTime DESC;

Surface FutureTimeThreshold / RealTimeWindow via existing GetSystemParameterAsync so failures are diagnosable.

6.d Storage service vs History service

IStorageServiceContract also exposes AddT/AddS/AddS2/DelT. The working hypothesis is that /Hist is client-facing and /Stor is engine-internal, but it's not yet verified.

Mitigation: the WriteMessage capture (§4.b) shows the actual service path on the wire. If it goes to /Stor, update the orchestrator. Do NOT preemptively implement against both.

6.e Parameter-name mismatches

Handoff already flagged EnsT, EnsT2, RTag2, ExKey, StJb, GtJb for the same inBuff/inputBuffer mismatch class that broke reads for weeks. Until each is audited against the server contract, requests bind to null and the server NREs.

Mitigation: before the first write WriteMessage capture, run an ildasm audit against aahClientAccessPoint.exe for the exact parameter names of EnsT2, AddS2, and DelT, and reconcile against the existing [MessageParameter] attributes.

6.f Customer-data exposure in capture files

Write captures contain the sandbox tag name and any value the test wrote. Not secrets, but noise.

Mitigation: keep all instrumented-wcf-writemessage-writes/ artifacts under artifacts/ (already gitignored). Sanitize tag names to <sandbox-tag> before committing decoded bytes into docs/reverse-engineering/.

7. Success Criteria

Per op:

• EnsT2(analog): EnsureTagAsync(new HistorianTagDefinition { Name = sandbox, DataType = Analog }) returns success; sqlcmd -E -S . -d Runtime -Q "SELECT TagName FROM Tag WHERE TagName = '...'" returns one row.
• EnsT2(discrete, string): same shape with corresponding DataType; SQL check uses DiscreteTag / StringTag view.
• AddS2: WriteValueAsync(sandbox, 42.0, DateTime.UtcNow) returns success; ReadRawAsync returns the value; SELECT TOP 1 Value FROM History WHERE wwTagKey = ? AND DateTime BETWEEN ? AND ? returns the same value.
• DelT: DeleteTagAsync(sandbox) returns success and SQL returns zero rows from Tag.
• ModifyData / DeleteData: deferred until §3.4 method discovery confirms scope.

Cross-cutting:

• All new code in src/AVEVA.Historian.Client/ is pure managed .NET 10. No new P/Invoke beyond the existing HistorianSspiClient.
• Every new op has a golden-byte unit test.
• dotnet test .\Histsdk.slnx --no-build --logger "console;verbosity=minimal" passes 100%.
• With HISTORIAN_HOST=localhost, HISTORIAN_WRITE_SANDBOX_TAG=RetestSdkWriteSandbox set, write integration tests pass and leave zero residue (test Dispose calls DelT for cleanup).
• Sanitization scan returns no real secrets.
• CLAUDE.md "Required SDK Surface" updated to add the new write ops — this is a SCOPE CHANGE that must land alongside the evidence, not before. Do not update the SDK surface doc until 3.1 + 3.2 are at least live-test-green.

8. Open Questions

1. Does AddS2 go through /Hist or /Stor on the wire?
2. Does the sandbox tag need pre-configuration via System Management Console once before EnsT2 will accept it from a client (e.g. for Storage / wwDomain rows the wire protocol may not be able to populate)?
3. What ConnectionMode does the native wrapper use for write sessions — 0x402 (read mode reused), 0x401, or something else?
4. Does EnsT2(analog) require any optional Archestra engineering-units fields, or are they purely cosmetic? Affects how minimal HistorianTagDefinition can be.
5. Server-side throttles on writes (max samples per AddS2, max calls per second) — need to surface as batching guidance?
6. What does the server return when AddS2 is called with a timestamp older than the tag's earliest stored block? Some historians silently drop, some error, some accept-and-overwrite.
7. Does the SDK expose write quality as the same HistorianSample.Quality enum used on reads, or a smaller subset (good/bad)?
8. Is there a managed-side DelT path at all? If aahClientManaged only exposes deletion via SMC, §3.3 is "manual SMC only" and must be documented as such.

9. Docs To Update Once Each Workstream Lands

• CLAUDE.md "Required SDK Surface" — add WriteValueAsync, EnsureTagAsync, DeleteTagAsync once 3.1+3.2+3.3 land.
• AGENTS.md "Required SDK Surface" — same; update the "alarm-event write path is dormant" note.
• docs/reverse-engineering/handoff.md — add a "Write-flow prereqs" section symmetric to the existing "Event-flow prereqs".
• docs/reverse-engineering/wcf-contract-evidence.md — add evidence rows for EnsT2(analog/discrete/string), AddS2, DelT.
• docs/reverse-engineering/implementation-status.md — flip status from "out of scope" to "implemented".
• README.md — operation status table.