lmxopcua/docs/plans/2026-06-26-otopcua-historian-backend-design.md

OtOpcUa ↔ HistorianGateway Historian Backend — Design

Date: 2026-06-26 Status: Design approved; implementation in two plans (see end). Repos: ~/Desktop/HistorianGateway (gateway + client lib), ~/Desktop/OtOpcUa (OPC UA server consumer)

---

1. Goal

Make HistorianGateway the historian read/write backend for the OtOpcUa OPC UA server, serving two distinct use cases:

1. Read of historic values for mxaccessgw-served (Galaxy) tags & alarms. Galaxy tags are already historized by AVEVA's own IOServer/AppServer pipeline; OtOpcUa serves their history to OPC UA HistoryRead clients by reading them back through the gateway.
2. Full read/write historian backend for non-mxaccessgw tags & alarms (Modbus / S7 / AB / TwinCAT / FOCAS / scripted-alarm sources). These are not historized by AVEVA, so OtOpcUa records their live value changes and alarm events into the historian through the gateway, then reads them back through the same path.

The vehicle, per decision, is a dedicated .NET gRPC client library for the gateway — ZB.MOM.WW.HistorianGateway.Client — built "similar to the mxaccessgw client" (ZB.MOM.WW.MxGateway.Client), which OtOpcUa consumes as a Gitea-feed package.

2. Locked decisions (from brainstorming)

Decision	Choice
Write model for non-Galaxy tags	Continuous historization — OtOpcUa records live value changes automatically
Relation to existing Wonderware TCP-sidecar backend	Replace it — gateway becomes the sole historian backend; retire the Wonderware driver projects
Alarm/event history	In scope for v1 — HistoryReadEvents from the gateway + route OtOpcUa alarm events to SendEvent
Client library location & consumption	In the gateway repo (clients/dotnet/), published to the Gitea feed; OtOpcUa references Contracts + Client as packages (mirrors how it already consumes ZB.MOM.WW.GalaxyRepository @ 0.2.0)
Continuous-historization durability	Mirror the gateway's StoreForward design — an OtOpcUa-side crash-safe FasterLog append-only outbox (so values buffer durably when the gateway itself is unreachable)
Deliverable	One design doc, two implementation plans (gateway-client plan; OtOpcUa-integration plan)

3. Why this is tractable — the seams already exist

OtOpcUa's historian integration was designed for pluggable backends. The gateway slots into seams that are already in place; only two genuinely-new pieces are required (the recorder and tag provisioning).

OtOpcUa seam	File	Role for us
IHistorianDataSource	src/Core/…Core.Abstractions/Historian/IHistorianDataSource.cs	Read surface (ReadRaw/ReadProcessed/ReadAtTime/ReadEvents + GetHealthSnapshot); wired into the NodeManager's HistoryReadRawModified/HistoryReadEvents overrides
IAlarmHistorianWriter	src/Core/…Core.AlarmHistorian/IAlarmHistorianSink.cs	Alarm-event write surface (WriteBatchAsync(batch)); already fronted by SqliteStoreAndForwardSink
AddServerHistorian(cfg, factory)	src/Server/…Runtime/ServiceCollectionExtensions.cs	Generic over Func<ServerHistorianOptions, IServiceProvider, IHistorianDataSource> — swap the factory, zero change to Runtime/OpcUaServer
AddAlarmHistorian(cfg, writerFactory)	same	Generic over the IAlarmHistorianWriter factory — swap to the gateway writer
DependencyMuxActor	src/Server/…Runtime/VirtualTags/DependencyMuxActor.cs	Value-change fan-out (RegisterInterest + AttributeValuePublished) — the tap point for continuous historization
AddressSpaceApplier.Apply()	src/Server/…OpcUaServer/AddressSpaceApplier.cs	Per-tag iteration over plan.AddedEquipmentTags.Where(IsHistorized) — the hook for EnsureTags provisioning

Currently these seams are filled by WonderwareHistorianClient (a single class implementing both IHistorianDataSource and IAlarmHistorianWriter over a bespoke TCP FrameChannel to an ArchestrA-SDK sidecar) — exactly the COM-bound approach HistorianGateway was built to replace.

4. Gateway gRPC surface vs. OtOpcUa needs

The gateway's historian_gateway.v1 contract already covers the surface. Mapping:

OtOpcUa need	Gateway RPC	Notes
ReadRawAsync	HistorianRead.ReadRaw (stream)	direct
ReadProcessedAsync	HistorianRead.ReadAggregate (stream)	HistoryAggregateType → RetrievalMode mapping (§6)
ReadAtTimeAsync	HistorianRead.ReadAtTime (unary)	direct
ReadEventsAsync	HistorianRead.ReadEvents (stream)	needs gateway RuntimeDb:EventReadsEnabled=true (C2 SQL path) + source-name filter (gateway gap §5)
continuous value write	HistorianWrite.WriteLiveValues	SQL live path; needs gateway RuntimeDb:Enabled=true; numeric/analog only (§7)
alarm event write	HistorianWrite.SendEvent	maps AlarmHistorianEvent → HistorianEvent
tag provisioning	HistorianTags.EnsureTags	DriverDataType → HistorianDataType mapping (§6)
health/diagnostics	HistorianStatus.Probe / GetConnectionStatus	feeds GetHealthSnapshot()

Galaxy hierarchy browse (GalaxyRepository service) is not needed here — OtOpcUa already gets Galaxy hierarchy via mxaccessgw's GalaxyRepositoryClient.

5. What gets added to HistorianGateway

1. ZB.MOM.WW.HistorianGateway.Client (NEW, clients/dotnet/). Clones the MxGatewayClient pattern: HistorianGatewayClient.Create(options) owning a GrpcChannel over a SocketsHttpHandler (TLS, connect timeout), Polly resilience pipeline (retry transient codes only), histgw_<id>_<secret> bearer key attached in the authorization metadata header, typed exception hierarchy, and wrappers for all five services (unary → Task<T>, streaming → IAsyncEnumerable<T>). Packable NuGet, references the Contracts project.
2. Make ZB.MOM.WW.HistorianGateway.Contracts packable + publish to the Gitea feed (it has no packaging props today). Mirrors ZB.MOM.WW.MxGateway.Contracts @ 0.1.x. This is what lets the Client and OtOpcUa consume generated historian_gateway.v1 types as a package.
3. SQL ReadEvents source-name filter (small enhancement, coordinated with the in-flight feat/sql-readevents branch). The SQL event-read path is currently time-range-only (per-property filter → Unimplemented); add Source_Object filtering so OtOpcUa's ReadEventsAsync(sourceName, …) is server-filtered rather than full-window + client-side filter.
4. Optional smoke CLI (…Client.Cli) mirroring mxgw cli — manual live checks.
5. Deployment/config prerequisites (no code): the gateway OtOpcUa points at must run with RuntimeDb:Enabled=true (WriteLiveValues) and RuntimeDb:EventReadsEnabled=true (alarm reads). Provision an API key carrying historian:read, historian:write, historian:tags:write.

6. Mapping tables (single source of truth for the mappers)

HistoryAggregateType (OPC UA) → RetrievalMode (gateway). Mirror the existing WonderwareHistorianClient.ReadProcessedAsync mapping as the authoritative reference; expected:

HistoryAggregateType	RetrievalMode
Average	TimeWeightedAverage
Minimum	MinimumWithTime
Maximum	MaximumWithTime
Total	Integral
Count	Counter (verify against Wonderware client; may have no exact native mode)

DriverDataType (OtOpcUa) → HistorianDataType (gateway), for EnsureTags/WriteLiveValues. Constrained by which writes are server-proven (CLAUDE.md: write-captured = Int1/2/4/8, UInt4/8, Float, Double):

DriverDataType	HistorianDataType	Write status
Boolean	Int1	proven
Int16	Int2	proven
Int32	Int4	proven
Int64	Int8	proven
UInt16	UInt4 (fallback — UInt2 write is deferred upstream)	proven via fallback
UInt32	UInt4	proven
UInt64	UInt8	proven
Float32	Float	proven
Float64	Double	proven
String	SingleByteString	deferred — gated upstream; not historized in v1
DateTime	FileTime	deferred — not on the analog write path
Reference	(string)	deferred

HistorianSample → DataValueSnapshot: Value ← numeric/string value; StatusCode ← quality translated to OPC UA status (reuse Wonderware client's quality translation); SourceTimestampUtc ← sample timestamp; ServerTimestampUtc ← received/processing time.

HistorianEvent → HistoricalEvent: EventId ← id; SourceName ← source_name; EventTimeUtc ← event_time; ReceivedTimeUtc ← received_time; Message ← properties (rendered); Severity ← properties (Priority/Severity) mapped to OPC UA 1–1000.

AlarmHistorianEvent → HistorianEvent (SendEvent): source_name ← EquipmentPath; event_time ← TimestampUtc; type ← AlarmTypeName; rich fields (AlarmName, EventKind, Severity, User, Comment, Message) carried in the properties map.

7. New OtOpcUa components

NEW  src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway/
       GatewayHistorianDataSource   : IHistorianDataSource    — read adapter over the client
       GatewayAlarmHistorianWriter  : IAlarmHistorianWriter    — SendEvent; behind existing SqliteStoreAndForwardSink
       GatewayTagProvisioner        : IHistorianProvisioning   — EnsureTags (NEW interface)
       Mappers                      — the §6 tables, with matrix-guard unit tests
NEW  ContinuousHistorizationRecorder (Runtime actor + FasterLog outbox)
       - registers RegisterInterest with DependencyMuxActor for historized non-Galaxy tag refs
       - appends each AttributeValuePublished to a crash-safe FasterLog outbox (PerEntry/Periodic
         commit, mirroring the gateway's FasterLogOutboxStore)
       - background drainer batches → client.WriteLiveValues; commits/truncates on ack; backoff on
         failure; outbox-full → drop-oldest + metric
HOOK AddressSpaceApplier.Apply() — for plan.AddedEquipmentTags.Where(IsHistorized) →
       provisioner.EnsureTags (non-blocking; failures logged + counted, never block publish)
SWAP Program.cs — AddServerHistorian + AddAlarmHistorian factories construct the Gateway-backed impls
CONF ServerHistorian options reshaped to gateway form (Endpoint / ApiKey / Tls); drop SharedSecret
RETIRE src/Drivers/*Wonderware* (3 src + 2 test projects) after live validation

8. Data flow

Use case 1 — Galaxy tag history read: UA HistoryRead → OtOpcUaNodeManager.HistoryReadRawModified → GatewayHistorianDataSource.ReadRaw → client.ReadRaw → gateway → AVEVA historian (already historized by AVEVA IOServer).

Use case 2 — non-Galaxy tag record + read:

• Provision (deploy): AddressSpaceApplier.Apply → GatewayTagProvisioner.EnsureTags → client.EnsureTags → gateway.
• Record (runtime): driver value change → DriverInstanceActor.AttributeValuePublished → DependencyMuxActor → ContinuousHistorizationRecorder → FasterLog outbox → drainer → client.WriteLiveValues → gateway (SQL live path).
• Read back: same path as use case 1.
• Alarms: ScriptedAlarmEngine → HistorianAdapterActor → SqliteStoreAndForwardSink → GatewayAlarmHistorianWriter.WriteBatchAsync → client.SendEvent; alarm-history read via GatewayHistorianDataSource.ReadEvents → client.ReadEvents.

9. Error handling

• Client: RpcException → typed hierarchy (HistorianGatewayException, …AuthenticationException/Unauthenticated, …AuthorizationException/PermissionDenied, …UnavailableException/Unavailable). Polly retries transient codes only.
• Read adapter: quality → OPC UA StatusCode inside the data source; empty windows are not faults; backend errors surface as Bad snapshots, never crash a HistoryRead.
• Provisioning: non-blocking — log + count failures; address-space publish always proceeds.
• Recorder: append-to-outbox is the durable boundary; drain failures back off; outbox-full → drop-oldest + metric; health via GetHealthSnapshot + meter.

10. Testing

• Client lib: fake-transport unit tests (clone mxaccessgw FakeGatewayTransport) — auth-header attach, retry, streaming, exception mapping; golden proto round-trips; smoke CLI.
• OtOpcUa adapter: unit tests with a fake IHistorianGatewayClient for every §6 mapper (matrix-guard so a new enum member fails the build); recorder tested against fake outbox + fake client (batch/drain/outage/drop); provisioning hook over a synthetic plan.
• Live (env-gated, skips without VPN): reuse the wonder-sql-vd03 fixture — Galaxy-tag read round-trip; write→read round-trip on a HistGW.LiveTest.* tag; alarm SendEvent→ReadEvents.

11. Verify-live risks (settle during implementation, not now)

1. Galaxy-tag → historian-tag identity — does OtOpcUa's historianTagname (tag_name.Attribute) match the AVEVA historian tag name? Confirm against wonder-sql-vd03 early.
2. UInt16 / String / DateTime write gaps — continuous historization is numeric-analog only in v1; documented mappings/fallbacks in §6, not silent drops.
3. Alarm-history reads depend on feat/sql-readevents landing + gateway RuntimeDb:EventReadsEnabled=true; the source-name filter (§5.3) is the one coordinated gateway enhancement.
4. WriteLiveValues requires gateway RuntimeDb:Enabled=true and an EnsureTags-provisioned tag.
5. received_time UTC semantics on the SQL event/value paths (local vs UTC; EventTimeUTCOffsetMins) — inherit whatever the feat/sql-readevents work establishes.

12. Implementation plans

• Plan 1 — Gateway client (docs/plans/2026-06-26-historian-gateway-client.md): Contracts packable + publish → client options/channel/auth → Polly + exception mapping → per-service wrappers → fake-transport tests → CLI → SQL-ReadEvents source filter (coordinated) → live smoke.
• Plan 2 — OtOpcUa integration (docs/plans/2026-06-26-otopcua-historian-gateway-integration.md): new Gateway driver project → mappers (matrix-guard) → read adapter + AddServerHistorian swap → alarm writer adapter + AddAlarmHistorian swap → ReadEvents alarm-history → continuous- historization recorder (FasterLog outbox) → EnsureTags provisioning hook → retire Wonderware → live-validate. (Authored here; relocates into ~/Desktop/OtOpcUa/docs/plans/ on its own branch when that phase starts, to avoid entangling OtOpcUa's current in-flight working tree.)

Plan 1 is a prerequisite for Plan 2 (OtOpcUa consumes the published Client package). Within Plan 2, the read path (phases through AddServerHistorian swap) is independently shippable and validates use case 1 before any write code lands.