docs(historian-gateway): relocate OtOpcUa↔HistorianGateway integration plan + design

Claude-Session: https://claude.ai/code/session_012SDSQ3AcaXqPcBtDESBRii

docs/plans/2026-06-26-otopcua-historian-backend-design.md

@@ -0,0 +1,220 @@
+# OtOpcUa ↔ HistorianGateway Historian Backend — Design
+
+**Date:** 2026-06-26
+**Status:** Design approved; implementation in two plans (see end).
+**Repos:** `~/Desktop/HistorianGateway` (gateway + client lib), `~/Desktop/OtOpcUa` (OPC UA server consumer)
+
+---
+
+## 1. Goal
+
+Make **HistorianGateway** the historian read/write backend for the **OtOpcUa** OPC UA server,
+serving two distinct use cases:
+
+1. **Read of historic values for mxaccessgw-served (Galaxy) tags & alarms.** Galaxy tags are
+   already historized by AVEVA's own IOServer/AppServer pipeline; OtOpcUa serves their history
+   to OPC UA `HistoryRead` clients by reading them back through the gateway.
+2. **Full read/write historian backend for non-mxaccessgw tags & alarms** (Modbus / S7 / AB /
+   TwinCAT / FOCAS / scripted-alarm sources). These are *not* historized by AVEVA, so OtOpcUa
+   **records** their live value changes and alarm events into the historian through the gateway,
+   then reads them back through the same path.
+
+The vehicle, per decision, is a **dedicated .NET gRPC client library** for the gateway —
+`ZB.MOM.WW.HistorianGateway.Client` — built "similar to the mxaccessgw client"
+(`ZB.MOM.WW.MxGateway.Client`), which OtOpcUa consumes as a Gitea-feed package.
+
+## 2. Locked decisions (from brainstorming)
+
+| Decision | Choice |
+|---|---|
+| Write model for non-Galaxy tags | **Continuous historization** — OtOpcUa records live value changes automatically |
+| Relation to existing Wonderware TCP-sidecar backend | **Replace it** — gateway becomes the sole historian backend; retire the Wonderware driver projects |
+| Alarm/event history | **In scope for v1** — `HistoryReadEvents` from the gateway + route OtOpcUa alarm events to `SendEvent` |
+| Client library location & consumption | **In the gateway repo (`clients/dotnet/`), published to the Gitea feed**; OtOpcUa references Contracts + Client as packages (mirrors how it already consumes `ZB.MOM.WW.GalaxyRepository @ 0.2.0`) |
+| Continuous-historization durability | **Mirror the gateway's StoreForward design** — an OtOpcUa-side crash-safe FasterLog append-only outbox (so values buffer durably when the *gateway itself* is unreachable) |
+| Deliverable | **One design doc, two implementation plans** (gateway-client plan; OtOpcUa-integration plan) |
+
+## 3. Why this is tractable — the seams already exist
+
+OtOpcUa's historian integration was designed for pluggable backends. The gateway slots into
+seams that are already in place; only two genuinely-new pieces are required (the recorder and tag
+provisioning).
+
+| OtOpcUa seam | File | Role for us |
+|---|---|---|
+| `IHistorianDataSource` | `src/Core/…Core.Abstractions/Historian/IHistorianDataSource.cs` | Read surface (`ReadRaw/ReadProcessed/ReadAtTime/ReadEvents` + `GetHealthSnapshot`); wired into the NodeManager's `HistoryReadRawModified`/`HistoryReadEvents` overrides |
+| `IAlarmHistorianWriter` | `src/Core/…Core.AlarmHistorian/IAlarmHistorianSink.cs` | Alarm-event write surface (`WriteBatchAsync(batch)`); already fronted by `SqliteStoreAndForwardSink` |
+| `AddServerHistorian(cfg, factory)` | `src/Server/…Runtime/ServiceCollectionExtensions.cs` | Generic over `Func<ServerHistorianOptions, IServiceProvider, IHistorianDataSource>` — swap the factory, zero change to Runtime/OpcUaServer |
+| `AddAlarmHistorian(cfg, writerFactory)` | same | Generic over the `IAlarmHistorianWriter` factory — swap to the gateway writer |
+| `DependencyMuxActor` | `src/Server/…Runtime/VirtualTags/DependencyMuxActor.cs` | Value-change fan-out (`RegisterInterest` + `AttributeValuePublished`) — the tap point for continuous historization |
+| `AddressSpaceApplier.Apply()` | `src/Server/…OpcUaServer/AddressSpaceApplier.cs` | Per-tag iteration over `plan.AddedEquipmentTags.Where(IsHistorized)` — the hook for `EnsureTags` provisioning |
+
+**Currently** these seams are filled by `WonderwareHistorianClient` (a single class implementing
+both `IHistorianDataSource` and `IAlarmHistorianWriter` over a bespoke **TCP FrameChannel** to an
+ArchestrA-SDK sidecar) — exactly the COM-bound approach HistorianGateway was built to replace.
+
+## 4. Gateway gRPC surface vs. OtOpcUa needs
+
+The gateway's `historian_gateway.v1` contract already covers the surface. Mapping:
+
+| OtOpcUa need | Gateway RPC | Notes |
+|---|---|---|
+| `ReadRawAsync` | `HistorianRead.ReadRaw` (stream) | direct |
+| `ReadProcessedAsync` | `HistorianRead.ReadAggregate` (stream) | `HistoryAggregateType` → `RetrievalMode` mapping (§6) |
+| `ReadAtTimeAsync` | `HistorianRead.ReadAtTime` (unary) | direct |
+| `ReadEventsAsync` | `HistorianRead.ReadEvents` (stream) | needs gateway `RuntimeDb:EventReadsEnabled=true` (C2 SQL path) + source-name filter (gateway gap §5) |
+| continuous value write | `HistorianWrite.WriteLiveValues` | SQL live path; needs gateway `RuntimeDb:Enabled=true`; numeric/analog only (§7) |
+| alarm event write | `HistorianWrite.SendEvent` | maps `AlarmHistorianEvent` → `HistorianEvent` |
+| tag provisioning | `HistorianTags.EnsureTags` | `DriverDataType` → `HistorianDataType` mapping (§6) |
+| health/diagnostics | `HistorianStatus.Probe` / `GetConnectionStatus` | feeds `GetHealthSnapshot()` |
+
+Galaxy hierarchy browse (`GalaxyRepository` service) is **not** needed here — OtOpcUa already gets
+Galaxy hierarchy via mxaccessgw's `GalaxyRepositoryClient`.
+
+## 5. What gets added to HistorianGateway
+
+1. **`ZB.MOM.WW.HistorianGateway.Client`** (NEW, `clients/dotnet/`). Clones the `MxGatewayClient`
+   pattern: `HistorianGatewayClient.Create(options)` owning a `GrpcChannel` over a
+   `SocketsHttpHandler` (TLS, connect timeout), Polly resilience pipeline (retry transient codes
+   only), `histgw_<id>_<secret>` bearer key attached in the `authorization` metadata header,
+   typed exception hierarchy, and wrappers for all five services (unary → `Task<T>`, streaming →
+   `IAsyncEnumerable<T>`). Packable NuGet, references the Contracts project.
+2. **Make `ZB.MOM.WW.HistorianGateway.Contracts` packable + publish to the Gitea feed** (it has no
+   packaging props today). Mirrors `ZB.MOM.WW.MxGateway.Contracts @ 0.1.x`. This is what lets the
+   Client and OtOpcUa consume generated `historian_gateway.v1` types as a package.
+3. **SQL `ReadEvents` source-name filter** (small enhancement, coordinated with the in-flight
+   `feat/sql-readevents` branch). The SQL event-read path is currently time-range-only
+   (per-property filter → `Unimplemented`); add `Source_Object` filtering so OtOpcUa's
+   `ReadEventsAsync(sourceName, …)` is server-filtered rather than full-window + client-side filter.
+4. **Optional smoke CLI** (`…Client.Cli`) mirroring `mxgw` cli — manual live checks.
+5. **Deployment/config prerequisites** (no code): the gateway OtOpcUa points at must run with
+   `RuntimeDb:Enabled=true` (WriteLiveValues) **and** `RuntimeDb:EventReadsEnabled=true` (alarm
+   reads). Provision an API key carrying `historian:read`, `historian:write`,
+   `historian:tags:write`.
+
+## 6. Mapping tables (single source of truth for the mappers)
+
+**`HistoryAggregateType` (OPC UA) → `RetrievalMode` (gateway).** Mirror the existing
+`WonderwareHistorianClient.ReadProcessedAsync` mapping as the authoritative reference; expected:
+
+| `HistoryAggregateType` | `RetrievalMode` |
+|---|---|
+| `Average` | `TimeWeightedAverage` |
+| `Minimum` | `MinimumWithTime` |
+| `Maximum` | `MaximumWithTime` |
+| `Total` | `Integral` |
+| `Count` | `Counter` *(verify against Wonderware client; may have no exact native mode)* |
+
+**`DriverDataType` (OtOpcUa) → `HistorianDataType` (gateway), for `EnsureTags`/`WriteLiveValues`.**
+Constrained by which writes are server-proven (CLAUDE.md: write-captured = Int1/2/4/8, UInt4/8,
+Float, Double):
+
+| `DriverDataType` | `HistorianDataType` | Write status |
+|---|---|---|
+| `Boolean` | `Int1` | proven |
+| `Int16` | `Int2` | proven |
+| `Int32` | `Int4` | proven |
+| `Int64` | `Int8` | proven |
+| `UInt16` | `UInt4` *(fallback — UInt2 write is deferred upstream)* | proven via fallback |
+| `UInt32` | `UInt4` | proven |
+| `UInt64` | `UInt8` | proven |
+| `Float32` | `Float` | proven |
+| `Float64` | `Double` | proven |
+| `String` | `SingleByteString` | **deferred — gated upstream; not historized in v1** |
+| `DateTime` | `FileTime` | **deferred — not on the analog write path** |
+| `Reference` | (string) | **deferred** |
+
+**`HistorianSample` → `DataValueSnapshot`:** `Value` ← numeric/string value; `StatusCode` ←
+quality translated to OPC UA status (reuse Wonderware client's quality translation);
+`SourceTimestampUtc` ← sample timestamp; `ServerTimestampUtc` ← received/processing time.
+
+**`HistorianEvent` → `HistoricalEvent`:** `EventId` ← id; `SourceName` ← source_name;
+`EventTimeUtc` ← event_time; `ReceivedTimeUtc` ← received_time; `Message` ← properties (rendered);
+`Severity` ← properties (Priority/Severity) mapped to OPC UA 1–1000.
+
+**`AlarmHistorianEvent` → `HistorianEvent` (SendEvent):** `source_name` ← `EquipmentPath`;
+`event_time` ← `TimestampUtc`; `type` ← `AlarmTypeName`; rich fields (`AlarmName`, `EventKind`,
+`Severity`, `User`, `Comment`, `Message`) carried in the `properties` map.
+
+## 7. New OtOpcUa components
+
+```
+NEW  src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway/
+       GatewayHistorianDataSource   : IHistorianDataSource    — read adapter over the client
+       GatewayAlarmHistorianWriter  : IAlarmHistorianWriter    — SendEvent; behind existing SqliteStoreAndForwardSink
+       GatewayTagProvisioner        : IHistorianProvisioning   — EnsureTags (NEW interface)
+       Mappers                      — the §6 tables, with matrix-guard unit tests
+NEW  ContinuousHistorizationRecorder (Runtime actor + FasterLog outbox)
+       - registers RegisterInterest with DependencyMuxActor for historized non-Galaxy tag refs
+       - appends each AttributeValuePublished to a crash-safe FasterLog outbox (PerEntry/Periodic
+         commit, mirroring the gateway's FasterLogOutboxStore)
+       - background drainer batches → client.WriteLiveValues; commits/truncates on ack; backoff on
+         failure; outbox-full → drop-oldest + metric
+HOOK AddressSpaceApplier.Apply() — for plan.AddedEquipmentTags.Where(IsHistorized) →
+       provisioner.EnsureTags (non-blocking; failures logged + counted, never block publish)
+SWAP Program.cs — AddServerHistorian + AddAlarmHistorian factories construct the Gateway-backed impls
+CONF ServerHistorian options reshaped to gateway form (Endpoint / ApiKey / Tls); drop SharedSecret
+RETIRE src/Drivers/*Wonderware* (3 src + 2 test projects) after live validation
+```
+
+## 8. Data flow
+
+**Use case 1 — Galaxy tag history read:**
+`UA HistoryRead → OtOpcUaNodeManager.HistoryReadRawModified → GatewayHistorianDataSource.ReadRaw →
+client.ReadRaw → gateway → AVEVA historian (already historized by AVEVA IOServer)`.
+
+**Use case 2 — non-Galaxy tag record + read:**
+- *Provision (deploy):* `AddressSpaceApplier.Apply → GatewayTagProvisioner.EnsureTags → client.EnsureTags → gateway`.
+- *Record (runtime):* `driver value change → DriverInstanceActor.AttributeValuePublished → DependencyMuxActor → ContinuousHistorizationRecorder → FasterLog outbox → drainer → client.WriteLiveValues → gateway (SQL live path)`.
+- *Read back:* same path as use case 1.
+- *Alarms:* `ScriptedAlarmEngine → HistorianAdapterActor → SqliteStoreAndForwardSink → GatewayAlarmHistorianWriter.WriteBatchAsync → client.SendEvent`; alarm-history read via `GatewayHistorianDataSource.ReadEvents → client.ReadEvents`.
+
+## 9. Error handling
+
+- **Client:** `RpcException` → typed hierarchy (`HistorianGatewayException`,
+  `…AuthenticationException`/`Unauthenticated`, `…AuthorizationException`/`PermissionDenied`,
+  `…UnavailableException`/`Unavailable`). Polly retries transient codes only.
+- **Read adapter:** quality → OPC UA `StatusCode` inside the data source; empty windows are not
+  faults; backend errors surface as `Bad` snapshots, never crash a `HistoryRead`.
+- **Provisioning:** non-blocking — log + count failures; address-space publish always proceeds.
+- **Recorder:** append-to-outbox is the durable boundary; drain failures back off; outbox-full →
+  drop-oldest + metric; health via `GetHealthSnapshot` + meter.
+
+## 10. Testing
+
+- **Client lib:** fake-transport unit tests (clone mxaccessgw `FakeGatewayTransport`) — auth-header
+  attach, retry, streaming, exception mapping; golden proto round-trips; smoke CLI.
+- **OtOpcUa adapter:** unit tests with a fake `IHistorianGatewayClient` for every §6 mapper
+  (matrix-guard so a new enum member fails the build); recorder tested against fake outbox + fake
+  client (batch/drain/outage/drop); provisioning hook over a synthetic plan.
+- **Live (env-gated, skips without VPN):** reuse the `wonder-sql-vd03` fixture — Galaxy-tag read
+  round-trip; write→read round-trip on a `HistGW.LiveTest.*` tag; alarm `SendEvent`→`ReadEvents`.
+
+## 11. Verify-live risks (settle during implementation, not now)
+
+1. **Galaxy-tag → historian-tag identity** — does OtOpcUa's `historianTagname`
+   (`tag_name.Attribute`) match the AVEVA historian tag name? Confirm against `wonder-sql-vd03` early.
+2. **UInt16 / String / DateTime write gaps** — continuous historization is numeric-analog only in
+   v1; documented mappings/fallbacks in §6, not silent drops.
+3. **Alarm-history reads depend on `feat/sql-readevents`** landing + gateway
+   `RuntimeDb:EventReadsEnabled=true`; the source-name filter (§5.3) is the one coordinated gateway
+   enhancement.
+4. **`WriteLiveValues` requires gateway `RuntimeDb:Enabled=true`** and an `EnsureTags`-provisioned tag.
+5. **`received_time` UTC semantics** on the SQL event/value paths (local vs UTC; `EventTimeUTCOffsetMins`)
+   — inherit whatever the `feat/sql-readevents` work establishes.
+
+## 12. Implementation plans
+
+- **Plan 1 — Gateway client (`docs/plans/2026-06-26-historian-gateway-client.md`):** Contracts
+  packable + publish → client options/channel/auth → Polly + exception mapping → per-service
+  wrappers → fake-transport tests → CLI → SQL-ReadEvents source filter (coordinated) → live smoke.
+- **Plan 2 — OtOpcUa integration (`docs/plans/2026-06-26-otopcua-historian-gateway-integration.md`):**
+  new Gateway driver project → mappers (matrix-guard) → read adapter + `AddServerHistorian` swap →
+  alarm writer adapter + `AddAlarmHistorian` swap → `ReadEvents` alarm-history → continuous-
+  historization recorder (FasterLog outbox) → `EnsureTags` provisioning hook → retire Wonderware →
+  live-validate. (Authored here; relocates into `~/Desktop/OtOpcUa/docs/plans/` on its own branch
+  when that phase starts, to avoid entangling OtOpcUa's current in-flight working tree.)
+
+Plan 1 is a prerequisite for Plan 2 (OtOpcUa consumes the published Client package). Within Plan 2,
+the read path (phases through `AddServerHistorian` swap) is independently shippable and validates
+use case 1 before any write code lands.

docs/plans/2026-06-26-otopcua-historian-gateway-integration.md.tasks.json

@@ -0,0 +1,27 @@
+{
+  "planPath": "docs/plans/2026-06-26-otopcua-historian-gateway-integration.md",
+  "tasks": [
+    { "id": 0,  "subject": "Task 1: Consume gateway packages + scaffold Gateway driver project", "status": "pending", "blockedBy": [] },
+    { "id": 1,  "subject": "Task 2: HistoryAggregateType->RetrievalMode mapper (matrix-guarded)", "status": "pending", "blockedBy": [0] },
+    { "id": 2,  "subject": "Task 3: DriverDataType->HistorianDataType mapper + write-gap fallbacks (matrix-guarded)", "status": "pending", "blockedBy": [0] },
+    { "id": 3,  "subject": "Task 4: HistorianSample/Aggregate->DataValueSnapshot + quality mapper", "status": "pending", "blockedBy": [0] },
+    { "id": 4,  "subject": "Task 5: HistorianEvent->HistoricalEvent mapper (+ severity)", "status": "pending", "blockedBy": [0] },
+    { "id": 5,  "subject": "Task 6: AlarmHistorianEvent->HistorianEvent mapper (SendEvent)", "status": "pending", "blockedBy": [0] },
+    { "id": 6,  "subject": "Task 7: GatewayHistorianDataSource read paths (raw/processed/at-time)", "status": "pending", "blockedBy": [1, 3] },
+    { "id": 7,  "subject": "Task 8: GetHealthSnapshot via Probe/GetConnectionStatus", "status": "pending", "blockedBy": [6] },
+    { "id": 8,  "subject": "Task 9: Reshape ServerHistorianOptions to gateway form", "status": "pending", "blockedBy": [0] },
+    { "id": 9,  "subject": "Task 10: Swap AddServerHistorian factory in Program.cs (READ CUTOVER)", "status": "pending", "blockedBy": [6, 8] },
+    { "id": 10, "subject": "Task 11: ReadEventsAsync alarm-history on the data source", "status": "pending", "blockedBy": [6, 4] },
+    { "id": 11, "subject": "Task 12: GatewayAlarmHistorianWriter (SendEvent + outcome mapping)", "status": "pending", "blockedBy": [9, 5] },
+    { "id": 12, "subject": "Task 13: Swap AddAlarmHistorian factory in Program.cs", "status": "pending", "blockedBy": [11] },
+    { "id": 13, "subject": "Task 14: IHistorianProvisioning + GatewayTagProvisioner (EnsureTags)", "status": "pending", "blockedBy": [9, 2] },
+    { "id": 14, "subject": "Task 15: Hook provisioning into AddressSpaceApplier.Apply()", "status": "pending", "blockedBy": [13] },
+    { "id": 15, "subject": "Task 16: FasterLog historization outbox store", "status": "pending", "blockedBy": [9] },
+    { "id": 16, "subject": "Task 17: ContinuousHistorizationRecorder actor", "status": "pending", "blockedBy": [15, 9] },
+    { "id": 17, "subject": "Task 18: Wire recorder into DI + hosted lifecycle", "status": "pending", "blockedBy": [16] },
+    { "id": 18, "subject": "Task 19: Retire Wonderware historian projects", "status": "pending", "blockedBy": [9, 12, 17, 19] },
+    { "id": 19, "subject": "Task 20: Env-gated live validation vs wonder-sql-vd03", "status": "pending", "blockedBy": [9, 10, 12, 17] },
+    { "id": 20, "subject": "Task 21: Documentation (CLAUDE.md, appsettings, README)", "status": "pending", "blockedBy": [18] }
+  ],
+  "lastUpdated": "2026-06-26"
+}