histsdk/docs/plans/write-commands-reverse-engineering.md

# Plan: Reverse-Engineering Write Commands

## Status (2026-05-04, post-ApplyScaling landing)

Phase 2 is **complete**. The write surface that the server actually
supports for managed clients is implemented and live-verified:

- `EnsureTagAsync` for analog tags (Float, Double, Int2, Int4, UInt4),
  with optional `ApplyScaling=true` for distinct MinRaw/MaxRaw
  persistence (`AnalogTag.Scaling=1`).
- `DeleteTagAsync` for any tag created via the SDK.

Architecturally blocked / out of scope:

- **`AddS2` (write samples)** — server's runtime cache only ingests
  from configured IOServers / Application Server pipelines, not from
  client-only `AddTag` flows. The native wrapper hits the same wall;
  this is a server architecture decision, not a protocol gap.
- **Discrete / String / Int1 / Int8 / UInt8 tag types** — fail at
  native `HistorianAccess.AddTag` before any wire bytes leave the
  client. Likely require a different code path (`AddTagExtendedProperties`
  or pre-population via SMC); not investigated.

This plan covers the residual workstreams.

## Workstreams

### A. Documentation closeout

Status docs across the repo still describe write commands as
"in progress" or speculate about a non-existent `UpdateTags`
operation. Closing those out so future agents don't re-walk the
same dead ends.

| Step | File | Action |
|---|---|---|
| A1 | `docs/reverse-engineering/handoff.md` | Add a "Write-flow status" note marking EnsT2/DelT live; remove stale write-blocker callouts |
| A2 | `docs/reverse-engineering/implementation-status.md` | Flip EnsT2 / DelT rows from "out of scope" to "implemented"; add ApplyScaling row |
| A3 | `docs/reverse-engineering/wcf-contract-evidence.md` | Add evidence rows for `EnsT2(analog)` and `DelT` pointing at the captured fixtures |
| A4 | `README.md` | Operation-status table reflects the two write ops |

### B. EnsT2 idempotency / update behavior

We don't currently know what happens when `EnsureTagAsync` is called
against a tag name that already exists with different fields. Three
plausible outcomes: server errors, server silently updates, server
no-ops. This affects how callers should think about the API
(create-only vs upsert).

| Step | Action |
|---|---|
| B1 | Add a live integration test that calls `EnsureTagAsync` twice on the same tag name with different `MinEU/MaxEU/Description`; query SQL after each call to capture observed behavior |
| B2 | Document the observed contract in `HistorianTagDefinition` doc-comment and (if surprising) in CLAUDE.md |

### C. Expose currently-hardcoded CTagMetadata fields

The serializer hardcodes `StorageRate=1000ms`, `StorageType=Cyclic`,
`IntegralDivisor=1.0`, and a few flag-block bytes. The server accepts
those defaults so existing tests pass, but customers building tags
with non-default rates can't currently express that.

| Step | Field | Effort | Notes |
|---|---|---|---|
| C1 | `StorageRate` (uint32 ms) | small — wire field is already at a known offset, just plumb a parameter through | Default stays 1000ms |
| C2 | `StorageType` (Cyclic / Delta) | medium — need a comparison capture to find which byte in the flag block encodes it | Deferred unless customer asks |
| C3 | `IntegralDivisor` (double) | small — wire field already known | Deferred unless customer asks |

C1 is the only one I'm executing in this round. C2/C3 are listed
for completeness; pick them up when there's a concrete request.

### D. Deferred — no current evidence or customer ask

| ID | Item | Why deferred |
|---|---|---|
| D1 | `AddTagExtendedProperties` / `DeleteTagExtendedProperties` | No wire captures yet; no customer ask |
| D2 | `AddRevisionValuesBegin/Value/End` (revision-write path) | Multi-step capture needed against an existing historized tag; complex; no customer ask |
| D3 | Discrete/String/Int1/Int8/UInt8 EnsT2 root cause | Native `AddTag` fails for these — likely requires an entirely different code path; would need a fresh capture and IL walkthrough |

## Parallelism

| Track | Files touched | Conflicts with |
|---|---|---|
| A1 | `docs/reverse-engineering/handoff.md` | none |
| A2 | `docs/reverse-engineering/implementation-status.md` | none |
| A3 | `docs/reverse-engineering/wcf-contract-evidence.md` | none |
| A4 | `README.md` | none |
| B  | `tests/AVEVA.Historian.Client.Tests/HistorianClientIntegrationTests.cs`, `src/AVEVA.Historian.Client/Models/HistorianTagDefinition.cs` | C1 (same `HistorianTagDefinition` file) |
| C1 | `src/AVEVA.Historian.Client/Models/HistorianTagDefinition.cs`, `src/AVEVA.Historian.Client/Wcf/HistorianTagWriteProtocol.cs`, `src/AVEVA.Historian.Client/Wcf/HistorianWcfTagWriteOrchestrator.cs`, `tests/AVEVA.Historian.Client.Tests/HistorianTagWriteProtocolTests.cs`, `tests/AVEVA.Historian.Client.Tests/HistorianClientIntegrationTests.cs` | B (same `HistorianTagDefinition` and integration-test file) |

**Concurrency-safe groupings:**

- A1–A4 are pairwise independent and pairwise independent of B and C1 → can be assigned to four agents in parallel.
- B and C1 both touch `HistorianTagDefinition.cs`. Sequence them: B first (it just adds a doc-comment) then C1 (which adds a field).

In a single-agent execution (this session), the order is: A* batched
edits → B → C1 → build/test → commit.

## Success Criteria

- A1–A4: documentation reflects the actual current write surface and
  removes references to non-existent operations (`UpdateTags`).
- B: a passing live test that asserts the observed double-EnsT2
  behavior, plus a doc-comment update on `HistorianTagDefinition`.
- C1: `HistorianTagDefinition.StorageRateMs` field exposed, default
  preserves existing wire output, golden test for a non-default rate,
  live test that creates a tag with a non-default rate and asserts
  via SQL that `Tag.CurrentEditorUserKey` (or the storage-rate column,
  TBD) reflects the value.
- All 165+ tests still pass; no regression in existing live tests.

---

## Appendix: Prior Phase Notes

The historical phase logs that drove the implementation are preserved
below for context. They describe the path from "write surface
unknown" to "write surface implemented and live-verified" as it
unfolded through 2026-05-04. Anything in this appendix that contradicts
the current Status section above is superseded.

### Phase 1 findings (recorded, not implementing)

#### §3.4 ModifyData/DeleteData — ELIMINATED FROM SCOPE

`methods aahClientManaged.dll` returns no managed wrapper for any of:
`EditValue`, `ModifyValue`, `EditData`, `DeleteData`, `ModifyData`,
`OverwriteData`. Per the plan's §3.4 disposition rule, this op is
REST-only / SMC-only and remains out of scope for the SDK.

#### §4.a Native serializers identified

The wrapper does have managed-public write API:

| Public method | Token | Used for |
|---|---|---|
| `ArchestrA.HistorianAccess.AddTag` | `0x0600619A` | Creates a new tag (drives `EnsT2`) |
| `ArchestrA.HistorianAccess.AddStreamedValue` | `0x0600618C/D/E` (3 overloads) | Pushes one timestamped value (drives `AddS2`) |
| `ArchestrA.HistorianAccess.AddNonStreamedValue` | `0x0600618F/90` (2 overloads) | Pushes one timestamped value, non-stream mode |
| `ArchestrA.HistorianAccess.DeleteTags` | `0x060061A4` | Removes tags (drives `DelT`) |
| `ArchestrA.HistorianAccess.AddVersionedStreamedValue` | `0x0600616F` | Pushes versioned value (rev edit) |
| `ArchestrA.HistorianAccess.AddRevisionValuesBegin/Value/End/AddRevisionValues` | `0x06006175-77, 0x0600617F` | Multi-row revision write |

Native serializer for `EnsT2`:
`<Module>.CTagUtil.ConvertTagMetadataToHistorianTag` at token
`0x060055CE`. WCF wrapper for `AddS2`:
`<Module>.CHistoryConnectionWCF.AddStreamValuesToHistorian` at token
`0x0600404C`.

### Phase 2 follow-on findings (2026-05-04, second pass)

**The AddS2 prereq is architectural, not protocol-level.** Three
follow-up attempts to trigger AddS2 from the sandbox harness all hit
a client-side gate before any AddS2 byte reaches the wire:

1. TagKey synthetic→real override: even with the real `wwTagKey` the
   server returns error 129 "Tag not found in cache".
2. Server-cache settle wait of 8s: error 129 persists.
3. Fresh process / fresh connection (skip AddTag): error 129; no AddS2
   bytes sent on wire.

The Historian engine's runtime tag cache only ingests tags from
configured IOServers / Application Server pipelines, not from
`HistorianAccess.AddTag`-only flows. `WriteValueAsync` cannot be
implemented as a generic client API against this server architecture.

### Phase 2 results (write captures + EnsureTagAsync/DeleteTagAsync)

EnsT2 + DelT priming chain captured (no `RTag2` between Open2 and
EnsT2):

```
Hist.GetV → Hist.GetI ×2 → Hist.ValCl ×2 → Hist.Open2 →
Stat.GetV ×2 → Stat.GETHI ×2 → Hist.UpdC3 →
Stat.GetSystemParameter ×7 → Trx.GetV → Stat.GetV → Retr.GetV →
Hist.EnsT2 → Hist.Close2
```

Open2 with `NativeIntegratedWriteEnabledConnectionMode = 0x401`
(Process | Write | IntegratedSecurity) is required — read-mode
`0x402` makes the server return err 132 `OperationNotEnabled`
silently. The analog Float `CTagMetadata` payload is 144 bytes with a
leading `0x4E` marker byte and a 2-byte trailer `FE xx` where the
second byte is the ApplyScaling flag (`00` = false, `01` = true).

### ApplyScaling resolution (2026-05-04)

Earlier docs claimed "MinRaw is mirrored to MinEU — server quirk,
not SDK bug". That conclusion was based on tests that always set
ApplyScaling=false on the native side. Re-running with
`set_ApplyScaling(true)` on the harness and capturing wire bytes for
both values revealed:

- ApplyScaling=false → trailer = `FE 00` → server mirrors MinRaw→MinEU,
  sets `AnalogTag.Scaling=0`
- ApplyScaling=true  → trailer = `FE 01` → server persists distinct
  MinRaw/MaxRaw, sets `AnalogTag.Scaling=1`

The `IHistoryServiceContract2` surface has **no `UpdateTags`
operation**. Distinct MinRaw/MaxRaw persistence is achieved entirely
by toggling that one byte in the EnsT2 payload. The SDK now exposes
this via `HistorianTagDefinition.ApplyScaling`.

Capture artifacts:
`artifacts/reverse-engineering/apply-scaling-experiment/enst2-applyscaling-{false,true}.ndjson`.

### Original goal section (preserved for historical reference)

"Write commands work" was originally defined as the four ops:
`EnsT2`, `AddS2`, `DelT`, and `ModifyData/DeleteData`. The realized
scope is `EnsT2 + DelT` only. AddS2 is permanently blocked by
server architecture; ModifyData/DeleteData were eliminated by
static analysis (no managed wrapper exists). The
`AddRevisionValuesBegin/Value/End` chain remains a stretch goal
(item D2 in the current plan) — it was never investigated because
no SDK consumer has asked for revision writes.

### Original safety rules (still applicable)

- Single dedicated sandbox tag per RE session, name must start with
  `RetestSdkWrite`.
- Never write to any tag named in `HISTORIAN_TEST_TAG`,
  `HISTORIAN_TAG_FILTER`, the docs, or the captured RE ndjson.
- Time bounds on writes: every test uses `DateTime.UtcNow` so writes
  land inside the live `RealTimeWindow` / `FutureTimeThreshold`.
- `localhost` only; no customer / corporate hosts.
- Sanitization scan after every session.
- Write captures live in `artifacts/reverse-engineering/instrumented-wcf-writemessage-writes/`
  (gitignored).