histsdk/docs/plans/hcal-roadmap.md

HCAL modern-.NET client — implementation roadmap

Ordered, actionable plan to grow histsdk from "reads + basic config" into a broad HCAL replacement, built on the 2023 R2 gRPC transport. Derived from hcal-capability-matrix.md; event details in histevents.md.

Move to the repo's docs/plans/ when execution starts. Each work item lands as: a protocol serializer/parser + golden-byte unit test + an env-gated live integration test against the local Historian.

Guiding principles

1. gRPC-first. New ops go on the RemoteGrpc transport (clean protobuf envelope); the inner bytes blob is the only thing to RE. Keep WCF as the legacy/Windows path.
2. Two tests per op, always. A golden-byte test (deterministic, no server) and a gated live test (HISTORIAN_GRPC_HOST / HISTORIAN_HOST). No op is "done" without both.
3. Version-pin, fail closed. Read server version at connect; gate every byte serializer on it; throw ProtocolEvidenceMissingException on mismatch — never best-effort parse.
4. Capture once, encode forever. For CAPTURE-tier items, instrument one native call, save a sanitized fixture under fixtures/protocol/, then implement against the fixture.
5. Ship per milestone. Each milestone is independently releasable.

Effort: S ≈ days · M ≈ ~1 week · L ≈ weeks. Estimates are incremental on histsdk's existing infra (auth chain, transport, frame primitives, test harness).

---

Milestone 0 — Foundation: full gRPC parity for the DONE surface (M)

Goal: everything already working over WCF also works over RemoteGrpc, so the whole read/browse/status surface is Windows-free and the gRPC stack is the default path.

ID	Work	gRPC op	Files	Verify	Effort
R0.1	Route browse over gRPC	Retrieval.StartTagQuery/QueryTag or GetTagInfosFromName	Grpc/HistorianGrpcReadOrchestrator (+ new …GrpcBrowseClient), Historian2020ProtocolDialect	browse tags live over gRPC	S
R0.2	Route tag metadata over gRPC	Retrieval.GetTagInfosFromName	dialect + grpc client	metadata matches WCF result	S
R0.3	Route status/system-param over gRPC	Status.GetSystemParameter, Status.GetHistorianConsoleStatus	new Grpc/HistorianGrpcStatusClient	system param + conn status live	S
R0.4	Probe over gRPC	*.GetInterfaceVersion	grpc clients	ProbeAsync Windows-free	XS
R0.5	Capture harness for gRPC payloads	n/a	reuse instrument-wcf-* tooling (same byte blobs) + add a grpc-call-dump helper	dump any request/response bytes to a fixture	S
R0.6	Version gate	server version at connect	HistorianClientOptions, orchestrators	mismatched version → throws	S

Acceptance: the entire Phase-0 capability set runs end-to-end over RemoteGrpc (incl. Linux), no WCF on the path. 188+ unit tests green; live gRPC integration suite green.

---

Milestone 1 — Cheap surface completion (TRIVIAL/BOUNDED) (M–L total)

Goal: knock out the remaining read/config surface. Order = ascending payload difficulty.

1a. Trivial (XS–S each, no new payload format)

ID	Capability	gRPC op	Notes
R1.1	ExecuteSqlCommandAsync	Retrieval.ExecuteSqlCommand	string in → iRetValue + status; thin
R1.2	GetRuntimeParameterAsync	Status.GetRuntimeParameter	mirror GetSystemParameter
R1.3	GetServerTimeZoneAsync	Status.GetSystemTimeZoneName	string out

1b. Bounded (decode one bytes payload; S–M each)

ID	Capability	gRPC op	Payload to decode	Depends
R1.4	GetHistorianInfoAsync	Status.GetHistorianInfo	GETHI buffer (partly decoded; incl. EventStorageMode@514)	R0.5
R1.5	Extended-property read	Retrieval.GetTagExtendedPropertiesFromName	TEP result buffer	R0.5
R1.6	Localized-property read	Retrieval.GetTagLocalizedPropertiesFromName	localized buffer	R0.5
R1.7	Event filters	filter bytes in Retrieval.StartEventQuery	filter predicate encoding (name/op/value)	R0.5
R1.8	Analog-summary query	Retrieval.StartQuery (summary mode)	summary row layout	—
R1.9	State-summary query	Retrieval.StartQuery (state mode)	state-summary row layout	—

1c. Bounded config writes (S–M each)

ID	Capability	gRPC op	Payload	Notes
R1.10	RenameTagsAsync	History rename op	rename request buffer	AllowRenameTags already probed
R1.11	Extended-property write	History.AddTagExtendedProperties (+ groups) / DeleteTagExtendedProperties	TEP serialize	mirror analog CTagMetadata discipline
R1.12	Localized-property write	History.AddTagLocalizedProperties / DeleteTagLocalizedProperties	localized serialize
R1.13	Non-analog tag create (string/discrete)	History.EnsureTags	distinct CTagMetadata variant	⚠ native AddTag rejected some types — confirm server path first; may be GATED

Acceptance: read + browse + metadata + system/status + property R/W + summaries + event-filtered reads + rename all live-verified over gRPC.

---

Milestone 2 — Event sending (CAPTURE) (S–M) ← headline gap

Goal: SendEventAsync(HistorianEvent). Path fully mapped in histevents.md; one capture away.

ID	Work	Detail
R2.1	Capture the event value blob	Instrument CCommonArchestraEventValue::PackToVtq (or dump the VTQ value bytes) on a live AddStreamedValue(HistorianEvent); save sanitized fixture
R2.2	HistorianEventWriteProtocol	Serialize header (ReceivedTime, EventType, EventTime, Id, RevisionVersion, IsUpdate/IsDelete, Namespace) + typed property bag — inverse of HistorianEventRowProtocol (reuse typemarkers 0x02/0x10/0x18/0x31/0x43/…)
R2.3	Event write orchestrator	Open Event connection (write mode) → register CM_EVENT (already have) → Storage.AddStreamValues with the event VTQ
R2.4	Public API	HistorianClient.SendEventAsync(HistorianEvent) (+ HistorianEvent model: Type, EventTime, property bag)
R2.5	Round-trip test	Send an event → read it back via StartEventQuery / v_AlarmEventHistory2; golden-byte on R2.2

Acceptance: an event sent from histsdk appears in the historian and is read back with matching Type + properties. Now practical — Historian is installed locally.

---

Milestone 3 — Historical / non-streamed value writes (BOUNDED) (M)

Goal: insert original historical VTQs (backfill), the path that is NOT the gated cache push.

ID	Work	gRPC op
R3.1	Decode non-streamed VTQ packet	Transaction.AddNonStreamValuesBegin/AddNonStreamValues/End
R3.2	AddHistoricalValuesAsync	batched begin→values→end
R3.3	Ingest-permission validation	confirm the target accepts original-data insert (distinct from AddS2 cache wall)

Acceptance: historical points inserted and read back. Document clearly where this differs from (gated) streaming sample writes.

---

Milestone 4 — HARD subsystems (deferred / optional) (L each)

Only if the use case demands them. Each is a real subsystem, not an op.

ID	Capability	Approach	Risk
R4.1	Store-and-forward	Pragmatic local queue (durable outbox + replay on reconnect) rather than bit-faithful SF cache + Forward*Snapshot. Faithful SF = decode SF cache format + snapshot framing + recovery log	high; consider "good enough"
R4.2	Revision / edit writes	AddRevisionValue(s) go via the non-WCF storage-engine pipe (STransactPipeClient2) — separate transport RE	high
R4.3	Real store-forward status	duplex push (SetStoreForwardEvent) or a decoded pull endpoint — see store-forward plan	medium
R4.4	Multi-historian / redundancy	client-side orchestration over N single-historian sessions (failover, ReSyncTags, partner watchdog) — build last	medium

---

Won't-do from the client (GATED)

• Streaming process-sample writes (AddStreamedValue(HistorianDataValue) / AddS2): runtime cache only ingests from configured IOServer/AppServer pipelines. Confirm your ingestion architecture instead of pursuing this.

---

Cross-cutting workstreams (run alongside all milestones)

• CW-1 Capture tooling (enables R0.5, R1.x, R2.1): one reusable "call op → dump request/response bytes → sanitized fixture" path. Highest leverage — do first.
• CW-2 Version compatibility: matrix of tested Historian versions; serializers keyed by version; CI gate.
• CW-3 Cross-platform CI: run the gRPC suite on Linux/macOS (transport is portable; explicit-cred auth path).
• CW-4 Fixtures discipline: every new op ships a fixtures/protocol/<op>/ golden file; sanitize hostnames/tags/GUIDs before commit.
• CW-5 Public API shape: keep the modern surface (async, IAsyncEnumerable, cancellation, options record, DI-friendly) consistent as the surface grows.

---

Sequencing (critical path)

CW-1 capture tooling ─┐
M0 gRPC parity ───────┼─→ M1 cheap surface ─→ M2 event send ─→ M3 historical writes ─→ (M4 optional)
R0.6 version gate ────┘

Recommended first sprint: CW-1 + M0 (R0.1–R0.6) → a fully Windows-free, version-safe gRPC client at today's capability. Second sprint: M1a + M2 (cheap wins + the headline event-send). M3/M4 as demand dictates.

One-glance status

Milestone	Tier	Effort	Value	When
M0 gRPC parity + capture tooling	foundation	M	unblocks everything, Windows-free	now
M1 cheap surface	TRIVIAL/BOUNDED	M–L	most remaining read/config	next
M2 event send	CAPTURE	S–M	headline write capability	next
M3 historical writes	BOUNDED	M	backfill	on demand
M4 SF / revisions / redundancy	HARD	L×N	parity completeness	defer