lmxopcua/docs/plans/2026-06-26-otopcua-historian-backend-design.md

# OtOpcUa ↔ HistorianGateway Historian Backend — Design

**Date:** 2026-06-26
**Status:** Design approved; implementation in two plans (see end).
**Repos:** `~/Desktop/HistorianGateway` (gateway + client lib), `~/Desktop/OtOpcUa` (OPC UA server consumer)

---

## 1. Goal

Make **HistorianGateway** the historian read/write backend for the **OtOpcUa** OPC UA server,
serving two distinct use cases:

1. **Read of historic values for mxaccessgw-served (Galaxy) tags & alarms.** Galaxy tags are
   already historized by AVEVA's own IOServer/AppServer pipeline; OtOpcUa serves their history
   to OPC UA `HistoryRead` clients by reading them back through the gateway.
2. **Full read/write historian backend for non-mxaccessgw tags & alarms** (Modbus / S7 / AB /
   TwinCAT / FOCAS / scripted-alarm sources). These are *not* historized by AVEVA, so OtOpcUa
   **records** their live value changes and alarm events into the historian through the gateway,
   then reads them back through the same path.

The vehicle, per decision, is a **dedicated .NET gRPC client library** for the gateway —
`ZB.MOM.WW.HistorianGateway.Client` — built "similar to the mxaccessgw client"
(`ZB.MOM.WW.MxGateway.Client`), which OtOpcUa consumes as a Gitea-feed package.

## 2. Locked decisions (from brainstorming)

| Decision | Choice |
|---|---|
| Write model for non-Galaxy tags | **Continuous historization** — OtOpcUa records live value changes automatically |
| Relation to existing Wonderware TCP-sidecar backend | **Replace it** — gateway becomes the sole historian backend; retire the Wonderware driver projects |
| Alarm/event history | **In scope for v1** — `HistoryReadEvents` from the gateway + route OtOpcUa alarm events to `SendEvent` |
| Client library location & consumption | **In the gateway repo (`clients/dotnet/`), published to the Gitea feed**; OtOpcUa references Contracts + Client as packages (mirrors how it already consumes `ZB.MOM.WW.GalaxyRepository @ 0.2.0`) |
| Continuous-historization durability | **Mirror the gateway's StoreForward design** — an OtOpcUa-side crash-safe FasterLog append-only outbox (so values buffer durably when the *gateway itself* is unreachable) |
| Deliverable | **One design doc, two implementation plans** (gateway-client plan; OtOpcUa-integration plan) |

## 3. Why this is tractable — the seams already exist

OtOpcUa's historian integration was designed for pluggable backends. The gateway slots into
seams that are already in place; only two genuinely-new pieces are required (the recorder and tag
provisioning).

| OtOpcUa seam | File | Role for us |
|---|---|---|
| `IHistorianDataSource` | `src/Core/…Core.Abstractions/Historian/IHistorianDataSource.cs` | Read surface (`ReadRaw/ReadProcessed/ReadAtTime/ReadEvents` + `GetHealthSnapshot`); wired into the NodeManager's `HistoryReadRawModified`/`HistoryReadEvents` overrides |
| `IAlarmHistorianWriter` | `src/Core/…Core.AlarmHistorian/IAlarmHistorianSink.cs` | Alarm-event write surface (`WriteBatchAsync(batch)`); already fronted by `SqliteStoreAndForwardSink` |
| `AddServerHistorian(cfg, factory)` | `src/Server/…Runtime/ServiceCollectionExtensions.cs` | Generic over `Func<ServerHistorianOptions, IServiceProvider, IHistorianDataSource>` — swap the factory, zero change to Runtime/OpcUaServer |
| `AddAlarmHistorian(cfg, writerFactory)` | same | Generic over the `IAlarmHistorianWriter` factory — swap to the gateway writer |
| `DependencyMuxActor` | `src/Server/…Runtime/VirtualTags/DependencyMuxActor.cs` | Value-change fan-out (`RegisterInterest` + `AttributeValuePublished`) — the tap point for continuous historization |
| `AddressSpaceApplier.Apply()` | `src/Server/…OpcUaServer/AddressSpaceApplier.cs` | Per-tag iteration over `plan.AddedEquipmentTags.Where(IsHistorized)` — the hook for `EnsureTags` provisioning |

**Currently** these seams are filled by `WonderwareHistorianClient` (a single class implementing
both `IHistorianDataSource` and `IAlarmHistorianWriter` over a bespoke **TCP FrameChannel** to an
ArchestrA-SDK sidecar) — exactly the COM-bound approach HistorianGateway was built to replace.

## 4. Gateway gRPC surface vs. OtOpcUa needs

The gateway's `historian_gateway.v1` contract already covers the surface. Mapping:

| OtOpcUa need | Gateway RPC | Notes |
|---|---|---|
| `ReadRawAsync` | `HistorianRead.ReadRaw` (stream) | direct |
| `ReadProcessedAsync` | `HistorianRead.ReadAggregate` (stream) | `HistoryAggregateType` → `RetrievalMode` mapping (§6) |
| `ReadAtTimeAsync` | `HistorianRead.ReadAtTime` (unary) | direct |
| `ReadEventsAsync` | `HistorianRead.ReadEvents` (stream) | needs gateway `RuntimeDb:EventReadsEnabled=true` (C2 SQL path) + source-name filter (gateway gap §5) |
| continuous value write | `HistorianWrite.WriteLiveValues` | SQL live path; needs gateway `RuntimeDb:Enabled=true`; numeric/analog only (§7) |
| alarm event write | `HistorianWrite.SendEvent` | maps `AlarmHistorianEvent` → `HistorianEvent` |
| tag provisioning | `HistorianTags.EnsureTags` | `DriverDataType` → `HistorianDataType` mapping (§6) |
| health/diagnostics | `HistorianStatus.Probe` / `GetConnectionStatus` | feeds `GetHealthSnapshot()` |

Galaxy hierarchy browse (`GalaxyRepository` service) is **not** needed here — OtOpcUa already gets
Galaxy hierarchy via mxaccessgw's `GalaxyRepositoryClient`.

## 5. What gets added to HistorianGateway

1. **`ZB.MOM.WW.HistorianGateway.Client`** (NEW, `clients/dotnet/`). Clones the `MxGatewayClient`
   pattern: `HistorianGatewayClient.Create(options)` owning a `GrpcChannel` over a
   `SocketsHttpHandler` (TLS, connect timeout), Polly resilience pipeline (retry transient codes
   only), `histgw_<id>_<secret>` bearer key attached in the `authorization` metadata header,
   typed exception hierarchy, and wrappers for all five services (unary → `Task<T>`, streaming →
   `IAsyncEnumerable<T>`). Packable NuGet, references the Contracts project.
2. **Make `ZB.MOM.WW.HistorianGateway.Contracts` packable + publish to the Gitea feed** (it has no
   packaging props today). Mirrors `ZB.MOM.WW.MxGateway.Contracts @ 0.1.x`. This is what lets the
   Client and OtOpcUa consume generated `historian_gateway.v1` types as a package.
3. **SQL `ReadEvents` source-name filter** (small enhancement, coordinated with the in-flight
   `feat/sql-readevents` branch). The SQL event-read path is currently time-range-only
   (per-property filter → `Unimplemented`); add `Source_Object` filtering so OtOpcUa's
   `ReadEventsAsync(sourceName, …)` is server-filtered rather than full-window + client-side filter.
4. **Optional smoke CLI** (`…Client.Cli`) mirroring `mxgw` cli — manual live checks.
5. **Deployment/config prerequisites** (no code): the gateway OtOpcUa points at must run with
   `RuntimeDb:Enabled=true` (WriteLiveValues) **and** `RuntimeDb:EventReadsEnabled=true` (alarm
   reads). Provision an API key carrying `historian:read`, `historian:write`,
   `historian:tags:write`.

## 6. Mapping tables (single source of truth for the mappers)

**`HistoryAggregateType` (OPC UA) → `RetrievalMode` (gateway).** Mirror the existing
`WonderwareHistorianClient.ReadProcessedAsync` mapping as the authoritative reference; expected:

| `HistoryAggregateType` | `RetrievalMode` |
|---|---|
| `Average` | `TimeWeightedAverage` |
| `Minimum` | `MinimumWithTime` |
| `Maximum` | `MaximumWithTime` |
| `Total` | `Integral` |
| `Count` | `Counter` *(verify against Wonderware client; may have no exact native mode)* |

**`DriverDataType` (OtOpcUa) → `HistorianDataType` (gateway), for `EnsureTags`/`WriteLiveValues`.**
Constrained by which writes are server-proven (CLAUDE.md: write-captured = Int1/2/4/8, UInt4/8,
Float, Double):

| `DriverDataType` | `HistorianDataType` | Write status |
|---|---|---|
| `Boolean` | `Int1` | proven |
| `Int16` | `Int2` | proven |
| `Int32` | `Int4` | proven |
| `Int64` | `Int8` | proven |
| `UInt16` | `UInt4` *(fallback — UInt2 write is deferred upstream)* | proven via fallback |
| `UInt32` | `UInt4` | proven |
| `UInt64` | `UInt8` | proven |
| `Float32` | `Float` | proven |
| `Float64` | `Double` | proven |
| `String` | `SingleByteString` | **deferred — gated upstream; not historized in v1** |
| `DateTime` | `FileTime` | **deferred — not on the analog write path** |
| `Reference` | (string) | **deferred** |

**`HistorianSample` → `DataValueSnapshot`:** `Value` ← numeric/string value; `StatusCode` ←
quality translated to OPC UA status (reuse Wonderware client's quality translation);
`SourceTimestampUtc` ← sample timestamp; `ServerTimestampUtc` ← received/processing time.

**`HistorianEvent` → `HistoricalEvent`:** `EventId` ← id; `SourceName` ← source_name;
`EventTimeUtc` ← event_time; `ReceivedTimeUtc` ← received_time; `Message` ← properties (rendered);
`Severity` ← properties (Priority/Severity) mapped to OPC UA 1–1000.

**`AlarmHistorianEvent` → `HistorianEvent` (SendEvent):** `source_name` ← `EquipmentPath`;
`event_time` ← `TimestampUtc`; `type` ← `AlarmTypeName`; rich fields (`AlarmName`, `EventKind`,
`Severity`, `User`, `Comment`, `Message`) carried in the `properties` map.

## 7. New OtOpcUa components

```
NEW  src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway/
       GatewayHistorianDataSource   : IHistorianDataSource    — read adapter over the client
       GatewayAlarmHistorianWriter  : IAlarmHistorianWriter    — SendEvent; behind existing SqliteStoreAndForwardSink
       GatewayTagProvisioner        : IHistorianProvisioning   — EnsureTags (NEW interface)
       Mappers                      — the §6 tables, with matrix-guard unit tests
NEW  ContinuousHistorizationRecorder (Runtime actor + FasterLog outbox)
       - registers RegisterInterest with DependencyMuxActor for historized non-Galaxy tag refs
       - appends each AttributeValuePublished to a crash-safe FasterLog outbox (PerEntry/Periodic
         commit, mirroring the gateway's FasterLogOutboxStore)
       - background drainer batches → client.WriteLiveValues; commits/truncates on ack; backoff on
         failure; outbox-full → drop-oldest + metric
HOOK AddressSpaceApplier.Apply() — for plan.AddedEquipmentTags.Where(IsHistorized) →
       provisioner.EnsureTags (non-blocking; failures logged + counted, never block publish)
SWAP Program.cs — AddServerHistorian + AddAlarmHistorian factories construct the Gateway-backed impls
CONF ServerHistorian options reshaped to gateway form (Endpoint / ApiKey / Tls); drop SharedSecret
RETIRE src/Drivers/*Wonderware* (3 src + 2 test projects) after live validation
```

## 8. Data flow

**Use case 1 — Galaxy tag history read:**
`UA HistoryRead → OtOpcUaNodeManager.HistoryReadRawModified → GatewayHistorianDataSource.ReadRaw →
client.ReadRaw → gateway → AVEVA historian (already historized by AVEVA IOServer)`.

**Use case 2 — non-Galaxy tag record + read:**
- *Provision (deploy):* `AddressSpaceApplier.Apply → GatewayTagProvisioner.EnsureTags → client.EnsureTags → gateway`.
- *Record (runtime):* `driver value change → DriverInstanceActor.AttributeValuePublished → DependencyMuxActor → ContinuousHistorizationRecorder → FasterLog outbox → drainer → client.WriteLiveValues → gateway (SQL live path)`.
- *Read back:* same path as use case 1.
- *Alarms:* `ScriptedAlarmEngine → HistorianAdapterActor → SqliteStoreAndForwardSink → GatewayAlarmHistorianWriter.WriteBatchAsync → client.SendEvent`; alarm-history read via `GatewayHistorianDataSource.ReadEvents → client.ReadEvents`.

## 9. Error handling

- **Client:** `RpcException` → typed hierarchy (`HistorianGatewayException`,
  `…AuthenticationException`/`Unauthenticated`, `…AuthorizationException`/`PermissionDenied`,
  `…UnavailableException`/`Unavailable`). Polly retries transient codes only.
- **Read adapter:** quality → OPC UA `StatusCode` inside the data source; empty windows are not
  faults; backend errors surface as `Bad` snapshots, never crash a `HistoryRead`.
- **Provisioning:** non-blocking — log + count failures; address-space publish always proceeds.
- **Recorder:** append-to-outbox is the durable boundary; drain failures back off; outbox-full →
  drop-oldest + metric; health via `GetHealthSnapshot` + meter.

## 10. Testing

- **Client lib:** fake-transport unit tests (clone mxaccessgw `FakeGatewayTransport`) — auth-header
  attach, retry, streaming, exception mapping; golden proto round-trips; smoke CLI.
- **OtOpcUa adapter:** unit tests with a fake `IHistorianGatewayClient` for every §6 mapper
  (matrix-guard so a new enum member fails the build); recorder tested against fake outbox + fake
  client (batch/drain/outage/drop); provisioning hook over a synthetic plan.
- **Live (env-gated, skips without VPN):** reuse the `wonder-sql-vd03` fixture — Galaxy-tag read
  round-trip; write→read round-trip on a `HistGW.LiveTest.*` tag; alarm `SendEvent`→`ReadEvents`.

## 11. Verify-live risks (settle during implementation, not now)

1. **Galaxy-tag → historian-tag identity** — does OtOpcUa's `historianTagname`
   (`tag_name.Attribute`) match the AVEVA historian tag name? Confirm against `wonder-sql-vd03` early.
2. **UInt16 / String / DateTime write gaps** — continuous historization is numeric-analog only in
   v1; documented mappings/fallbacks in §6, not silent drops.
3. **Alarm-history reads depend on `feat/sql-readevents`** landing + gateway
   `RuntimeDb:EventReadsEnabled=true`; the source-name filter (§5.3) is the one coordinated gateway
   enhancement.
4. **`WriteLiveValues` requires gateway `RuntimeDb:Enabled=true`** and an `EnsureTags`-provisioned tag.
5. **`received_time` UTC semantics** on the SQL event/value paths (local vs UTC; `EventTimeUTCOffsetMins`)
   — inherit whatever the `feat/sql-readevents` work establishes.

## 12. Implementation plans

- **Plan 1 — Gateway client (`docs/plans/2026-06-26-historian-gateway-client.md`):** Contracts
  packable + publish → client options/channel/auth → Polly + exception mapping → per-service
  wrappers → fake-transport tests → CLI → SQL-ReadEvents source filter (coordinated) → live smoke.
- **Plan 2 — OtOpcUa integration (`docs/plans/2026-06-26-otopcua-historian-gateway-integration.md`):**
  new Gateway driver project → mappers (matrix-guard) → read adapter + `AddServerHistorian` swap →
  alarm writer adapter + `AddAlarmHistorian` swap → `ReadEvents` alarm-history → continuous-
  historization recorder (FasterLog outbox) → `EnsureTags` provisioning hook → retire Wonderware →
  live-validate. (Authored here; relocates into `~/Desktop/OtOpcUa/docs/plans/` on its own branch
  when that phase starts, to avoid entangling OtOpcUa's current in-flight working tree.)

Plan 1 is a prerequisite for Plan 2 (OtOpcUa consumes the published Client package). Within Plan 2,
the read path (phases through `AddServerHistorian` swap) is independently shippable and validates
use case 1 before any write code lands.