docs(historian-gateway): relocate OtOpcUa↔HistorianGateway integration plan + design
Claude-Session: https://claude.ai/code/session_012SDSQ3AcaXqPcBtDESBRii
This commit is contained in:
Joseph Doherty
@@ -0,0 +1,220 @@
|
||||
# OtOpcUa ↔ HistorianGateway Historian Backend — Design
|
||||
|
||||
**Date:** 2026-06-26
|
||||
**Status:** Design approved; implementation in two plans (see end).
|
||||
**Repos:** `~/Desktop/HistorianGateway` (gateway + client lib), `~/Desktop/OtOpcUa` (OPC UA server consumer)
|
||||
|
||||
---
|
||||
|
||||
## 1. Goal
|
||||
|
||||
Make **HistorianGateway** the historian read/write backend for the **OtOpcUa** OPC UA server,
|
||||
serving two distinct use cases:
|
||||
|
||||
1. **Read of historic values for mxaccessgw-served (Galaxy) tags & alarms.** Galaxy tags are
|
||||
already historized by AVEVA's own IOServer/AppServer pipeline; OtOpcUa serves their history
|
||||
to OPC UA `HistoryRead` clients by reading them back through the gateway.
|
||||
2. **Full read/write historian backend for non-mxaccessgw tags & alarms** (Modbus / S7 / AB /
|
||||
TwinCAT / FOCAS / scripted-alarm sources). These are *not* historized by AVEVA, so OtOpcUa
|
||||
**records** their live value changes and alarm events into the historian through the gateway,
|
||||
then reads them back through the same path.
|
||||
|
||||
The vehicle, per decision, is a **dedicated .NET gRPC client library** for the gateway —
|
||||
`ZB.MOM.WW.HistorianGateway.Client` — built "similar to the mxaccessgw client"
|
||||
(`ZB.MOM.WW.MxGateway.Client`), which OtOpcUa consumes as a Gitea-feed package.
|
||||
|
||||
## 2. Locked decisions (from brainstorming)
|
||||
|
||||
| Decision | Choice |
|
||||
|---|---|
|
||||
| Write model for non-Galaxy tags | **Continuous historization** — OtOpcUa records live value changes automatically |
|
||||
| Relation to existing Wonderware TCP-sidecar backend | **Replace it** — gateway becomes the sole historian backend; retire the Wonderware driver projects |
|
||||
| Alarm/event history | **In scope for v1** — `HistoryReadEvents` from the gateway + route OtOpcUa alarm events to `SendEvent` |
|
||||
| Client library location & consumption | **In the gateway repo (`clients/dotnet/`), published to the Gitea feed**; OtOpcUa references Contracts + Client as packages (mirrors how it already consumes `ZB.MOM.WW.GalaxyRepository @ 0.2.0`) |
|
||||
| Continuous-historization durability | **Mirror the gateway's StoreForward design** — an OtOpcUa-side crash-safe FasterLog append-only outbox (so values buffer durably when the *gateway itself* is unreachable) |
|
||||
| Deliverable | **One design doc, two implementation plans** (gateway-client plan; OtOpcUa-integration plan) |
|
||||
|
||||
## 3. Why this is tractable — the seams already exist
|
||||
|
||||
OtOpcUa's historian integration was designed for pluggable backends. The gateway slots into
|
||||
seams that are already in place; only two genuinely-new pieces are required (the recorder and tag
|
||||
provisioning).
|
||||
|
||||
| OtOpcUa seam | File | Role for us |
|
||||
|---|---|---|
|
||||
| `IHistorianDataSource` | `src/Core/…Core.Abstractions/Historian/IHistorianDataSource.cs` | Read surface (`ReadRaw/ReadProcessed/ReadAtTime/ReadEvents` + `GetHealthSnapshot`); wired into the NodeManager's `HistoryReadRawModified`/`HistoryReadEvents` overrides |
|
||||
| `IAlarmHistorianWriter` | `src/Core/…Core.AlarmHistorian/IAlarmHistorianSink.cs` | Alarm-event write surface (`WriteBatchAsync(batch)`); already fronted by `SqliteStoreAndForwardSink` |
|
||||
| `AddServerHistorian(cfg, factory)` | `src/Server/…Runtime/ServiceCollectionExtensions.cs` | Generic over `Func<ServerHistorianOptions, IServiceProvider, IHistorianDataSource>` — swap the factory, zero change to Runtime/OpcUaServer |
|
||||
| `AddAlarmHistorian(cfg, writerFactory)` | same | Generic over the `IAlarmHistorianWriter` factory — swap to the gateway writer |
|
||||
| `DependencyMuxActor` | `src/Server/…Runtime/VirtualTags/DependencyMuxActor.cs` | Value-change fan-out (`RegisterInterest` + `AttributeValuePublished`) — the tap point for continuous historization |
|
||||
| `AddressSpaceApplier.Apply()` | `src/Server/…OpcUaServer/AddressSpaceApplier.cs` | Per-tag iteration over `plan.AddedEquipmentTags.Where(IsHistorized)` — the hook for `EnsureTags` provisioning |
|
||||
|
||||
**Currently** these seams are filled by `WonderwareHistorianClient` (a single class implementing
|
||||
both `IHistorianDataSource` and `IAlarmHistorianWriter` over a bespoke **TCP FrameChannel** to an
|
||||
ArchestrA-SDK sidecar) — exactly the COM-bound approach HistorianGateway was built to replace.
|
||||
|
||||
## 4. Gateway gRPC surface vs. OtOpcUa needs
|
||||
|
||||
The gateway's `historian_gateway.v1` contract already covers the surface. Mapping:
|
||||
|
||||
| OtOpcUa need | Gateway RPC | Notes |
|
||||
|---|---|---|
|
||||
| `ReadRawAsync` | `HistorianRead.ReadRaw` (stream) | direct |
|
||||
| `ReadProcessedAsync` | `HistorianRead.ReadAggregate` (stream) | `HistoryAggregateType` → `RetrievalMode` mapping (§6) |
|
||||
| `ReadAtTimeAsync` | `HistorianRead.ReadAtTime` (unary) | direct |
|
||||
| `ReadEventsAsync` | `HistorianRead.ReadEvents` (stream) | needs gateway `RuntimeDb:EventReadsEnabled=true` (C2 SQL path) + source-name filter (gateway gap §5) |
|
||||
| continuous value write | `HistorianWrite.WriteLiveValues` | SQL live path; needs gateway `RuntimeDb:Enabled=true`; numeric/analog only (§7) |
|
||||
| alarm event write | `HistorianWrite.SendEvent` | maps `AlarmHistorianEvent` → `HistorianEvent` |
|
||||
| tag provisioning | `HistorianTags.EnsureTags` | `DriverDataType` → `HistorianDataType` mapping (§6) |
|
||||
| health/diagnostics | `HistorianStatus.Probe` / `GetConnectionStatus` | feeds `GetHealthSnapshot()` |
|
||||
|
||||
Galaxy hierarchy browse (`GalaxyRepository` service) is **not** needed here — OtOpcUa already gets
|
||||
Galaxy hierarchy via mxaccessgw's `GalaxyRepositoryClient`.
|
||||
|
||||
## 5. What gets added to HistorianGateway
|
||||
|
||||
1. **`ZB.MOM.WW.HistorianGateway.Client`** (NEW, `clients/dotnet/`). Clones the `MxGatewayClient`
|
||||
pattern: `HistorianGatewayClient.Create(options)` owning a `GrpcChannel` over a
|
||||
`SocketsHttpHandler` (TLS, connect timeout), Polly resilience pipeline (retry transient codes
|
||||
only), `histgw_<id>_<secret>` bearer key attached in the `authorization` metadata header,
|
||||
typed exception hierarchy, and wrappers for all five services (unary → `Task<T>`, streaming →
|
||||
`IAsyncEnumerable<T>`). Packable NuGet, references the Contracts project.
|
||||
2. **Make `ZB.MOM.WW.HistorianGateway.Contracts` packable + publish to the Gitea feed** (it has no
|
||||
packaging props today). Mirrors `ZB.MOM.WW.MxGateway.Contracts @ 0.1.x`. This is what lets the
|
||||
Client and OtOpcUa consume generated `historian_gateway.v1` types as a package.
|
||||
3. **SQL `ReadEvents` source-name filter** (small enhancement, coordinated with the in-flight
|
||||
`feat/sql-readevents` branch). The SQL event-read path is currently time-range-only
|
||||
(per-property filter → `Unimplemented`); add `Source_Object` filtering so OtOpcUa's
|
||||
`ReadEventsAsync(sourceName, …)` is server-filtered rather than full-window + client-side filter.
|
||||
4. **Optional smoke CLI** (`…Client.Cli`) mirroring `mxgw` cli — manual live checks.
|
||||
5. **Deployment/config prerequisites** (no code): the gateway OtOpcUa points at must run with
|
||||
`RuntimeDb:Enabled=true` (WriteLiveValues) **and** `RuntimeDb:EventReadsEnabled=true` (alarm
|
||||
reads). Provision an API key carrying `historian:read`, `historian:write`,
|
||||
`historian:tags:write`.
|
||||
|
||||
## 6. Mapping tables (single source of truth for the mappers)
|
||||
|
||||
**`HistoryAggregateType` (OPC UA) → `RetrievalMode` (gateway).** Mirror the existing
|
||||
`WonderwareHistorianClient.ReadProcessedAsync` mapping as the authoritative reference; expected:
|
||||
|
||||
| `HistoryAggregateType` | `RetrievalMode` |
|
||||
|---|---|
|
||||
| `Average` | `TimeWeightedAverage` |
|
||||
| `Minimum` | `MinimumWithTime` |
|
||||
| `Maximum` | `MaximumWithTime` |
|
||||
| `Total` | `Integral` |
|
||||
| `Count` | `Counter` *(verify against Wonderware client; may have no exact native mode)* |
|
||||
|
||||
**`DriverDataType` (OtOpcUa) → `HistorianDataType` (gateway), for `EnsureTags`/`WriteLiveValues`.**
|
||||
Constrained by which writes are server-proven (CLAUDE.md: write-captured = Int1/2/4/8, UInt4/8,
|
||||
Float, Double):
|
||||
|
||||
| `DriverDataType` | `HistorianDataType` | Write status |
|
||||
|---|---|---|
|
||||
| `Boolean` | `Int1` | proven |
|
||||
| `Int16` | `Int2` | proven |
|
||||
| `Int32` | `Int4` | proven |
|
||||
| `Int64` | `Int8` | proven |
|
||||
| `UInt16` | `UInt4` *(fallback — UInt2 write is deferred upstream)* | proven via fallback |
|
||||
| `UInt32` | `UInt4` | proven |
|
||||
| `UInt64` | `UInt8` | proven |
|
||||
| `Float32` | `Float` | proven |
|
||||
| `Float64` | `Double` | proven |
|
||||
| `String` | `SingleByteString` | **deferred — gated upstream; not historized in v1** |
|
||||
| `DateTime` | `FileTime` | **deferred — not on the analog write path** |
|
||||
| `Reference` | (string) | **deferred** |
|
||||
|
||||
**`HistorianSample` → `DataValueSnapshot`:** `Value` ← numeric/string value; `StatusCode` ←
|
||||
quality translated to OPC UA status (reuse Wonderware client's quality translation);
|
||||
`SourceTimestampUtc` ← sample timestamp; `ServerTimestampUtc` ← received/processing time.
|
||||
|
||||
**`HistorianEvent` → `HistoricalEvent`:** `EventId` ← id; `SourceName` ← source_name;
|
||||
`EventTimeUtc` ← event_time; `ReceivedTimeUtc` ← received_time; `Message` ← properties (rendered);
|
||||
`Severity` ← properties (Priority/Severity) mapped to OPC UA 1–1000.
|
||||
|
||||
**`AlarmHistorianEvent` → `HistorianEvent` (SendEvent):** `source_name` ← `EquipmentPath`;
|
||||
`event_time` ← `TimestampUtc`; `type` ← `AlarmTypeName`; rich fields (`AlarmName`, `EventKind`,
|
||||
`Severity`, `User`, `Comment`, `Message`) carried in the `properties` map.
|
||||
|
||||
## 7. New OtOpcUa components
|
||||
|
||||
```
|
||||
NEW src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway/
|
||||
GatewayHistorianDataSource : IHistorianDataSource — read adapter over the client
|
||||
GatewayAlarmHistorianWriter : IAlarmHistorianWriter — SendEvent; behind existing SqliteStoreAndForwardSink
|
||||
GatewayTagProvisioner : IHistorianProvisioning — EnsureTags (NEW interface)
|
||||
Mappers — the §6 tables, with matrix-guard unit tests
|
||||
NEW ContinuousHistorizationRecorder (Runtime actor + FasterLog outbox)
|
||||
- registers RegisterInterest with DependencyMuxActor for historized non-Galaxy tag refs
|
||||
- appends each AttributeValuePublished to a crash-safe FasterLog outbox (PerEntry/Periodic
|
||||
commit, mirroring the gateway's FasterLogOutboxStore)
|
||||
- background drainer batches → client.WriteLiveValues; commits/truncates on ack; backoff on
|
||||
failure; outbox-full → drop-oldest + metric
|
||||
HOOK AddressSpaceApplier.Apply() — for plan.AddedEquipmentTags.Where(IsHistorized) →
|
||||
provisioner.EnsureTags (non-blocking; failures logged + counted, never block publish)
|
||||
SWAP Program.cs — AddServerHistorian + AddAlarmHistorian factories construct the Gateway-backed impls
|
||||
CONF ServerHistorian options reshaped to gateway form (Endpoint / ApiKey / Tls); drop SharedSecret
|
||||
RETIRE src/Drivers/*Wonderware* (3 src + 2 test projects) after live validation
|
||||
```
|
||||
|
||||
## 8. Data flow
|
||||
|
||||
**Use case 1 — Galaxy tag history read:**
|
||||
`UA HistoryRead → OtOpcUaNodeManager.HistoryReadRawModified → GatewayHistorianDataSource.ReadRaw →
|
||||
client.ReadRaw → gateway → AVEVA historian (already historized by AVEVA IOServer)`.
|
||||
|
||||
**Use case 2 — non-Galaxy tag record + read:**
|
||||
- *Provision (deploy):* `AddressSpaceApplier.Apply → GatewayTagProvisioner.EnsureTags → client.EnsureTags → gateway`.
|
||||
- *Record (runtime):* `driver value change → DriverInstanceActor.AttributeValuePublished → DependencyMuxActor → ContinuousHistorizationRecorder → FasterLog outbox → drainer → client.WriteLiveValues → gateway (SQL live path)`.
|
||||
- *Read back:* same path as use case 1.
|
||||
- *Alarms:* `ScriptedAlarmEngine → HistorianAdapterActor → SqliteStoreAndForwardSink → GatewayAlarmHistorianWriter.WriteBatchAsync → client.SendEvent`; alarm-history read via `GatewayHistorianDataSource.ReadEvents → client.ReadEvents`.
|
||||
|
||||
## 9. Error handling
|
||||
|
||||
- **Client:** `RpcException` → typed hierarchy (`HistorianGatewayException`,
|
||||
`…AuthenticationException`/`Unauthenticated`, `…AuthorizationException`/`PermissionDenied`,
|
||||
`…UnavailableException`/`Unavailable`). Polly retries transient codes only.
|
||||
- **Read adapter:** quality → OPC UA `StatusCode` inside the data source; empty windows are not
|
||||
faults; backend errors surface as `Bad` snapshots, never crash a `HistoryRead`.
|
||||
- **Provisioning:** non-blocking — log + count failures; address-space publish always proceeds.
|
||||
- **Recorder:** append-to-outbox is the durable boundary; drain failures back off; outbox-full →
|
||||
drop-oldest + metric; health via `GetHealthSnapshot` + meter.
|
||||
|
||||
## 10. Testing
|
||||
|
||||
- **Client lib:** fake-transport unit tests (clone mxaccessgw `FakeGatewayTransport`) — auth-header
|
||||
attach, retry, streaming, exception mapping; golden proto round-trips; smoke CLI.
|
||||
- **OtOpcUa adapter:** unit tests with a fake `IHistorianGatewayClient` for every §6 mapper
|
||||
(matrix-guard so a new enum member fails the build); recorder tested against fake outbox + fake
|
||||
client (batch/drain/outage/drop); provisioning hook over a synthetic plan.
|
||||
- **Live (env-gated, skips without VPN):** reuse the `wonder-sql-vd03` fixture — Galaxy-tag read
|
||||
round-trip; write→read round-trip on a `HistGW.LiveTest.*` tag; alarm `SendEvent`→`ReadEvents`.
|
||||
|
||||
## 11. Verify-live risks (settle during implementation, not now)
|
||||
|
||||
1. **Galaxy-tag → historian-tag identity** — does OtOpcUa's `historianTagname`
|
||||
(`tag_name.Attribute`) match the AVEVA historian tag name? Confirm against `wonder-sql-vd03` early.
|
||||
2. **UInt16 / String / DateTime write gaps** — continuous historization is numeric-analog only in
|
||||
v1; documented mappings/fallbacks in §6, not silent drops.
|
||||
3. **Alarm-history reads depend on `feat/sql-readevents`** landing + gateway
|
||||
`RuntimeDb:EventReadsEnabled=true`; the source-name filter (§5.3) is the one coordinated gateway
|
||||
enhancement.
|
||||
4. **`WriteLiveValues` requires gateway `RuntimeDb:Enabled=true`** and an `EnsureTags`-provisioned tag.
|
||||
5. **`received_time` UTC semantics** on the SQL event/value paths (local vs UTC; `EventTimeUTCOffsetMins`)
|
||||
— inherit whatever the `feat/sql-readevents` work establishes.
|
||||
|
||||
## 12. Implementation plans
|
||||
|
||||
- **Plan 1 — Gateway client (`docs/plans/2026-06-26-historian-gateway-client.md`):** Contracts
|
||||
packable + publish → client options/channel/auth → Polly + exception mapping → per-service
|
||||
wrappers → fake-transport tests → CLI → SQL-ReadEvents source filter (coordinated) → live smoke.
|
||||
- **Plan 2 — OtOpcUa integration (`docs/plans/2026-06-26-otopcua-historian-gateway-integration.md`):**
|
||||
new Gateway driver project → mappers (matrix-guard) → read adapter + `AddServerHistorian` swap →
|
||||
alarm writer adapter + `AddAlarmHistorian` swap → `ReadEvents` alarm-history → continuous-
|
||||
historization recorder (FasterLog outbox) → `EnsureTags` provisioning hook → retire Wonderware →
|
||||
live-validate. (Authored here; relocates into `~/Desktop/OtOpcUa/docs/plans/` on its own branch
|
||||
when that phase starts, to avoid entangling OtOpcUa's current in-flight working tree.)
|
||||
|
||||
Plan 1 is a prerequisite for Plan 2 (OtOpcUa consumes the published Client package). Within Plan 2,
|
||||
the read path (phases through `AddServerHistorian` swap) is independently shippable and validates
|
||||
use case 1 before any write code lands.
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,27 @@
|
||||
{
|
||||
"planPath": "docs/plans/2026-06-26-otopcua-historian-gateway-integration.md",
|
||||
"tasks": [
|
||||
{ "id": 0, "subject": "Task 1: Consume gateway packages + scaffold Gateway driver project", "status": "pending", "blockedBy": [] },
|
||||
{ "id": 1, "subject": "Task 2: HistoryAggregateType->RetrievalMode mapper (matrix-guarded)", "status": "pending", "blockedBy": [0] },
|
||||
{ "id": 2, "subject": "Task 3: DriverDataType->HistorianDataType mapper + write-gap fallbacks (matrix-guarded)", "status": "pending", "blockedBy": [0] },
|
||||
{ "id": 3, "subject": "Task 4: HistorianSample/Aggregate->DataValueSnapshot + quality mapper", "status": "pending", "blockedBy": [0] },
|
||||
{ "id": 4, "subject": "Task 5: HistorianEvent->HistoricalEvent mapper (+ severity)", "status": "pending", "blockedBy": [0] },
|
||||
{ "id": 5, "subject": "Task 6: AlarmHistorianEvent->HistorianEvent mapper (SendEvent)", "status": "pending", "blockedBy": [0] },
|
||||
{ "id": 6, "subject": "Task 7: GatewayHistorianDataSource read paths (raw/processed/at-time)", "status": "pending", "blockedBy": [1, 3] },
|
||||
{ "id": 7, "subject": "Task 8: GetHealthSnapshot via Probe/GetConnectionStatus", "status": "pending", "blockedBy": [6] },
|
||||
{ "id": 8, "subject": "Task 9: Reshape ServerHistorianOptions to gateway form", "status": "pending", "blockedBy": [0] },
|
||||
{ "id": 9, "subject": "Task 10: Swap AddServerHistorian factory in Program.cs (READ CUTOVER)", "status": "pending", "blockedBy": [6, 8] },
|
||||
{ "id": 10, "subject": "Task 11: ReadEventsAsync alarm-history on the data source", "status": "pending", "blockedBy": [6, 4] },
|
||||
{ "id": 11, "subject": "Task 12: GatewayAlarmHistorianWriter (SendEvent + outcome mapping)", "status": "pending", "blockedBy": [9, 5] },
|
||||
{ "id": 12, "subject": "Task 13: Swap AddAlarmHistorian factory in Program.cs", "status": "pending", "blockedBy": [11] },
|
||||
{ "id": 13, "subject": "Task 14: IHistorianProvisioning + GatewayTagProvisioner (EnsureTags)", "status": "pending", "blockedBy": [9, 2] },
|
||||
{ "id": 14, "subject": "Task 15: Hook provisioning into AddressSpaceApplier.Apply()", "status": "pending", "blockedBy": [13] },
|
||||
{ "id": 15, "subject": "Task 16: FasterLog historization outbox store", "status": "pending", "blockedBy": [9] },
|
||||
{ "id": 16, "subject": "Task 17: ContinuousHistorizationRecorder actor", "status": "pending", "blockedBy": [15, 9] },
|
||||
{ "id": 17, "subject": "Task 18: Wire recorder into DI + hosted lifecycle", "status": "pending", "blockedBy": [16] },
|
||||
{ "id": 18, "subject": "Task 19: Retire Wonderware historian projects", "status": "pending", "blockedBy": [9, 12, 17, 19] },
|
||||
{ "id": 19, "subject": "Task 20: Env-gated live validation vs wonder-sql-vd03", "status": "pending", "blockedBy": [9, 10, 12, 17] },
|
||||
{ "id": 20, "subject": "Task 21: Documentation (CLAUDE.md, appsettings, README)", "status": "pending", "blockedBy": [18] }
|
||||
],
|
||||
"lastUpdated": "2026-06-26"
|
||||
}
|
||||
Reference in New Issue
Block a user