dohertj2/histsdk
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
33 Commits 1 Branch 0 Tags
main
Commit Graph

7 Commits

Author SHA1 Message Date
Joseph Doherty 5ce62a5900 Wire ApplyScaling, StorageRate; close out write-commands plan 
ApplyScaling (HistorianTagDefinition.ApplyScaling):
The EnsT2 trailer's second byte controls server-side scaling — `FE 00`
mirrors MinRaw to MinEU and sets AnalogTag.Scaling=0; `FE 01` persists
distinct MinRaw/MaxRaw and sets Scaling=1. Decoded by toggling
set_ApplyScaling on the native harness and capturing the wire bytes for
both values with identical inputs. The earlier docs claimed
EnsureTagAsync needed a follow-up "UpdateTags" call; the WCF surface has
no such operation — toggling that one byte is the whole fix.

StorageRate (HistorianTagDefinition.StorageRateMs):
Serializer accepts a non-default rate, validated empirically against
the live server which only accepts quantized values
(1000/5000/10000/60000/300000 ms).

EnsureTagAsync upsert semantics:
Second call on the same tag name with different fields succeeds and
updates Description, MinEU, MaxEU, MinRaw, MaxRaw, Scaling in place
(verified by direct SQL inspection in a live test).

Plan + doc closeout:
write-commands-reverse-engineering.md rewritten as a current-state
plan with three workstreams (A doc closeout / B idempotency / C1
StorageRate) and a parallelism table; prior phase notes preserved as
appendix. handoff.md, implementation-status.md, wcf-contract-evidence.md,
README.md updated to remove "writes are out of scope" / non-existent
UpdateTags references and document the actual EnsT2 wire format
including the `FE xx` trailer.

Reverse-engineering harness gains --write-apply-scaling and a SQL
post-check that prints the persisted AnalogTag bounds so future RE
sessions can verify wire→DB causality without leaving the harness.

169/169 tests pass (was 165; +4 new tests covering ApplyScaling,
StorageRate golden bytes, StorageRate live persistence, and
EnsureTagAsync upsert semantics).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-04 22:04:27 -04:00
dohertj2 200493c990 DelT investigation: wire-byte parity is necessary but not sufficient 
Investigation step 1 — wire-byte parity check. Captured native DelT
sends ref input values statusSize=1 + status=null (encoded as .nil
on the wire). SDK was passing statusSize=0 + status=[] (empty array).
Updated SDK to match native input values.

Investigation step 2 — verified DelT still doesn't work standalone.
With the ref-input fix, SDK DelT now returns false (instead of the
previous true-with-no-effect). Tag continues to persist in
Runtime.dbo.Tag. So the wire-byte parity fix moved the symptom but
didn't resolve the root cause.

Investigation step 3 — discovered EnsureTagAsync is ALSO silently
broken. Byte-for-byte wire matches captured native EnsT2 (golden test
still passes), but the call returns false and does NOT create the tag
in the DB. The earlier "EnsureTagAsync round-trip test passing" was
relying on the persistent tag from the broken DelT — a false
positive.

Two distinct issues remain:
1. EnsT2 silently fails server-side (returns false; no tag created)
2. DelT returns false even with native-matching wire bytes

Test adjusted to no longer assert that EnsureTagAsync actually
creates the tag (because it currently doesn't). Test still exercises
the SDK call path to confirm it doesn't throw.

Next-session diagnostic: write a custom IClientMessageInspector for
the SDK's WCF channel that captures outgoing DelT/EnsT2 bytes to a
file. Compare byte-for-byte (offset by offset, not just per-field)
against captured native to isolate the difference.

130/130 tests pass.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-04 08:51:24 -04:00
dohertj2 cfc8d44e3a Implement EnsureTagAsync (live-verified) + DeleteTagAsync (DelT semantics partial) 
New SDK surface:
  HistorianClient.EnsureTagAsync(HistorianTagDefinition)
  HistorianClient.DeleteTagAsync(string tagName)

Plumbing:
  src/AVEVA.Historian.Client/Models/HistorianTagDefinition.cs
    Public input model — TagName/Description/EngineeringUnit/DataType/MinEU/MaxEU.
    Currently only HistorianDataType.Float is live-verified.

  src/AVEVA.Historian.Client/Wcf/HistorianTagWriteProtocol.cs
    SerializeAnalogCTagMetadata produces 146-byte payload byte-for-byte
    identical to the captured native EnsT2(Float) request.
    SerializeDeleteTagNames produces ushort 0x6751 + ushort 1 + uint count
    + per-tag (uint charCount + UTF-16 chars).

  src/AVEVA.Historian.Client/Wcf/HistorianWcfTagWriteOrchestrator.cs
    Both EnsT2 and DelT run the full Stat-priming chain captured for the
    analog flow (UpdC3 + Stat.GetV ×3 + Stat.GETHI ×2 + 7× GetSystemParameter
    + Trx.GetV + Retr.GetV).

  src/AVEVA.Historian.Client/Wcf/HistorianWcfTagClient.cs
    MapDataType extended to accept tag-origin marker 0xC7 (SDK-created tags).

Tests:
  5 golden-byte tests (HistorianTagWriteProtocolTests):
    SerializeAnalogCTagMetadata byte-for-byte match against captured 146-byte fixture
    SerializeAnalogCTagMetadata produces different bytes for different inputs
    SerializeDeleteTagNames single-tag matches captured shape
    SerializeDeleteTagNames multi-tag appends each
    SerializeDeleteTagNames empty list throws

  1 live integration test (gated by HISTORIAN_WRITE_SANDBOX_TAG):
    EnsureTagAsync_AndDeleteTagAsync_RoundTrip_AgainstLocalHistorian
    EnsureTagAsync creates the sandbox tag, GetTagMetadataAsync reads it
    back. 130/130 tests pass.

Harness improvements:
  --write-delete-after now runs DelT independently of AddStreamedValue
  outcome.
  HistorianTagStatusList constructed correctly for DeleteTags reflection
  call (previous StringCollection attempt failed with TypeMismatch).

Known DelT gap: SDK's DeleteTagAsync returns true but server-side
cascading deletion does not always complete (the row remains in
Runtime.dbo.Tag). The captured native flow's DelT removes the tag
cleanly (verified via harness --write-delete-after), so something
around the WCF DelT call is missing from our orchestrator. Documented
as known issue with SMC-based cleanup as workaround.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-04 08:33:21 -04:00
dohertj2 b3d22befd0 write-commands plan: AddS2 prereq is architectural - not implementable as generic client write 
Three follow-up attempts to satisfy the AddS2 server-cache prereq all
failed at the same client-side gate before any AddS2 byte reached the
wire:

1. TagKey synthetic→real override. First attempt used the placeholder
   TagKey=10000000 returned by HistorianAccess.AddTag. Native
   AddStreamedValue refused with error 168 "Tag not added to server".
   Harness now ALWAYS resolves the real wwTagKey from Runtime.dbo.Tag
   after AddTag (logged as TagKeyOverride: Synthetic→RealFromSql).
   Error code shifted to 129 "Tag not found in cache" — request now
   reaches the server but the server's in-memory tag cache doesn't
   know about the new tag.

2. Server-cache settle wait. Up to 8s sleep between AddTag and
   AddStreamedValue (--write-resync-wait-seconds N). Wait period
   contains 2× UpdC3 + 2× Trx/GetV keep-alives but no server-side
   cache update — error 129 persists.

3. Fresh process / fresh connection. Skipped AddTag entirely
   (--write-skip-add-tag) and ran AddStreamedValue alone against the
   already-existing sandbox tag. New native client instance, new
   client-side cache, new server session. SAME error 129 — no AddS2
   bytes sent on wire. Capture confirms 44 records ending in Close2.

Interpretation: the Historian engine's runtime tag cache only ingests
tags from configured IOServers / Application Server data pipelines,
not from HistorianAccess.AddTag-only client flows. AddTag populates
Runtime.dbo.Tag (wwTagKey=240 was created) but doesn't register the
tag with the live cache that AddStreamedValue checks. That
registration happens server-side when an upstream data producer (an
OPC driver, AnE event subsystem, Application Server attribute store)
claims the tag.

WriteValueAsync therefore CANNOT be implemented as a generic client
API against this server architecture. The SDK's realistic writeable
surface is now narrowed to EnsureTagAsync + DeleteTagAsync only.

Harness changes:
- --write-skip-add-tag    skip the AddTag call (for fresh-cache test)
- --write-skip-add-value  skip the AddStreamedValue call (capture EnsT2 only)
- --write-resync-wait-seconds N  sleep N seconds between AddTag and
                                  AddStreamedValue (default 0)
- TagKey lookup now ALWAYS hits SQL after AddTag, not just when
  the synthetic key is 0.

Plan doc updated with full Phase 2 follow-on findings + revised
remaining work (4-item checklist focused on EnsureTagAsync/
DeleteTagAsync, plus a stretch goal of probing AddRevisionValues*
against an existing-tag).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-04 08:06:28 -04:00
dohertj2 b5f9a71fe7 write-commands plan: Phase 2 partial - capture EnsT2(Float) wire bytes 
Per plan §1 in scope: EnsT2 for analog tags, AddS2, DelT.
Per plan §2 safety: localhost only, single sandbox tag
RetestSdkWriteSandbox, harness refuses any name not starting with
RetestSdkWrite, time-bounded writes, ReadOnly=false only when scenario
is "write".

Phase 2 actually executed:

1. tools/AVEVA.Historian.NativeTraceHarness/Program.cs extended with
   --scenario write. New args:
     --write-sandbox-tag <name>  (default RetestSdkWriteSandbox)
     --write-value <numeric>     (default 42.5)
     --write-data-type <name>    (default Float)
     --write-delete-after        (best-effort cleanup)
   Toggles ConnectionArgs.ReadOnly=false when scenario is "write" so
   the connection accepts the write attempt instead of rejecting at
   the harness boundary with error 132 "Operation is not enabled".

2. Sandbox tag RetestSdkWriteSandbox created in Runtime DB
   (wwTagKey=240, AcquisitionType=2 Manual, StorageType=1 Cyclic)
   via the harness's AddTag call. Single dedicated tag per safety §1.

3. Captured the full write-flow wire sequence at
   artifacts/reverse-engineering/instrumented-wcf-writemessage-writes/
   bothmessage-write-capture-latest.ndjson (46 records, 23 outgoing +
   23 incoming).

   The chain is identical to the event flow except:
     - EnsT2 payload is the 146-byte analog CTagMetadata instead of
       the 83-byte event one
     - NO RTag2 between Open2 and EnsT2 (events used RTag2 with
       CmEventTagId)

4. The 146-byte analog CTagMetadata layout is dumped in the plan doc
   for layout decoding. Visible fields (still being aligned against
   CTagUtil.ConvertTagMetadataToHistorianTag IL at token 0x060055CE):
     - tag name "RetestSdkWriteSandbox" (compact ASCII, len 21)
     - 16 bytes of FF (CommonArchestraEventTypeId placeholder unused
       for analog?)
     - description "SDK write-RE sandbox tag" (compact ASCII, len 24)
     - metadata provider "MDAS" (compact ASCII)
     - engineering unit "test" (compact ASCII)
     - Int64 FILETIME (date-created, year 2026)
     - uint32 0x2710 = 10000 (storage-related, possibly StorageRate)
     - double 1.0 (likely IntegralDivisor or scaling factor)
     - 5-byte trailer FE 00 01 01 01 (matches event tag's
       2F 27 01 01 01 shape)

5. AddS2 BLOCKED CLIENT-SIDE at error 168 "Tag not added to server".
   Native AddStreamedValue refuses to send because the tag isn't in
   the server's session cache, even though EnsT2 created it in the
   Runtime DB. Likely needs RTag2(analog tag GUID) prereq similar
   to the event flow's RTag2(CmEventTagId), or one of
   aahClientCommon.CHistStorage.AddTagidPairs (token 0x0600202F) or
   AddTagsWithServerTagId (token 0x06002026). AddS2 wire bytes NOT
   captured this session.

6. scripts/decode-write-capture.py — sanitized decoder for the
   capture, walks the 46 records and dumps the EnsT2 InBuff bytes
   for layout work. No identity strings; only sandbox-chosen values
   appear in output.

Phase 2 remaining work documented in the plan doc as a 5-item
checklist for the next session:
  1. Decode the AddS2 prereq (likely RTag2 with analog tag GUID).
  2. Capture AddS2 wire bytes once prereq is satisfied.
  3. Implement HistorianAddTagsProtocol.SerializeAnalog/Discrete/
     String CTagMetadata variants.
  4. Implement HistorianAddStreamValuesProtocol.Serialize.
  5. Implement public surface: EnsureTagAsync, WriteValueAsync,
     DeleteTagAsync (golden-byte + gated live integration tests).

No SDK source changed — implementation deferred until AddS2 wire
bytes are in hand.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-04 07:55:27 -04:00
dohertj2 e3c003d978 write-commands plan: execute Phase 1 discovery (no DB writes) 
Phase 1 of docs/plans/write-commands-reverse-engineering.md is purely
discovery — static IL inspection, public-API enumeration, scope
elimination — and produces no DB writes. Phase 2 requires extending
NativeTraceHarness with a write scenario AND explicit operator approval
per the plan's safety rule §1, so it is deferred to a separate session.

Phase 1 findings recorded inline in the plan's status header:

1. §3.4 ModifyData/DeleteData eliminated — no managed wrapper exists.
   `methods` returns zero hits for EditValue, ModifyValue, EditData,
   DeleteData, ModifyData, OverwriteData. Per the plan's own §3.4
   disposition rule, this op is REST/SMC-only.

2. §4.a native serializer tokens identified for Phase 2:

   Public managed-wrapper write API in ArchestrA.HistorianAccess:
   - AddTag (0x0600619A)
   - AddStreamedValue × 3 overloads (0x0600618C/D/E)
   - AddNonStreamedValue × 2 overloads (0x0600618F/90)
   - DeleteTags (0x060061A4)
   - AddRevisionValuesBegin / AddRevisionValue / AddRevisionValuesEnd
     / AddRevisionValues (0x06006175-77, 0x0600617F)
     — covers the bulk-modify use case the eliminated ModifyData would
     have served. Worth folding into Phase 2 scope.

   Native serializers:
   - CTagUtil.ConvertTagMetadataToHistorianTag (0x060055CE) — 412 IL
     instructions; builds CTagMetadata for analog/discrete/string.
     Calls every CTagMetadata accessor we'd need to surface
     (Min/MaxRaw, Min/MaxEU, Unit, Message0/1, MaxLength,
     IntegralDivisor, DefaultTagRate, RolloverValue) plus all the
     CDataType predicates already decoded.
   - CHistoryConnectionWCF.AddStreamValuesToHistorian (0x0600404C)
     — confirms the on-wire shape matches our existing
     IHistoryServiceContract2.AddStreamValues2 declaration.

3. Chicken-and-egg resolved: the first §3.1 EnsT2 test creates the
   sandbox tag itself, so no SMC step is needed.

4. Open question §8.6 answered — the wrapper exposes both
   AddStreamedValue and AddNonStreamedValue, so the SDK should
   eventually surface both real-time and backfill write modes.

Phase 2 next steps recorded inline in the plan as a 5-item executable
checklist (extend harness with --scenario write, capture, decode
EnsT2(analog) + AddS2 bytes, implement public surface, gated live
tests).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-04 07:41:34 -04:00
dohertj2 6f01b83313 Plan two reverse-engineering campaigns: write commands + store/forward cache 
docs/plans/write-commands-reverse-engineering.md (425 lines):
  Plan for adding WriteValueAsync (AddS2 stream values), EnsureTags2 for
  analog/discrete/string tags, and DelT for sandbox cleanup. Hard safety
  rules center on a dedicated sandbox tag gated by env var, time-bounded
  writes, SQL ground-truth verification per session, explicit rollback.
  Five-step RE workflow mirrors the read/event decode (static IL discovery
  -> instrument-wcf-writemessage capture -> instrument-wcf-readmessage
  capture -> byte/IL alignment -> managed serializer + golden-byte tests).
  Risks call out auth-chain unknowns, parameter-name-mismatch class,
  silent-success failure modes, History-vs-Storage service question.

docs/plans/store-forward-cache-reverse-engineering.md (501 lines):
  Plan for replacing the synthesized GetStoreForwardStatusAsync with a
  real implementation. Architecture investigation already partially
  answered via IL inspection during planning: ArchestrA.HistorianAccess.
  GetStoreForwardStatus (token 0x06006187) reads an in-process C struct
  via calli to mdas_GetStorageStatus, kept current by server-pushed WCF
  callbacks (IStatusServiceContract2.SetStoreForwardEvent). CSFConnection.
  GetSFPipeName indicates a separate Named Pipe sidecar exists when SF
  is configured. Five parallelizable discovery workstreams, six concrete
  RE steps with cited tokens, eight risks, eight success criteria.

Both plans deliberately produce no code changes and no captures. They
exist so the next implementer can start with full context.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-04 07:16:32 -04:00