lmxopcua/HISTORIAN-GATEWAY-INTEGRATION-ISSUES.md

# HistorianGateway backend — issues identified

**Date:** 2026-06-27
**Context:** Installed the `ZB.MOM.WW.HistorianGateway` sidecar on **wonder-app-vd03** and updated OtOpcUa to
`origin/master` (`245316d8`, PR #423 "HistorianGateway as the OtOpcUa historian backend"), then smoke-tested
the OtOpcUa→gateway link live. These are the issues found during that work.

> **File/line references are against `origin/master` @ `245316d8`** (the build that contains the gateway
> backend — `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway/`). They will not resolve on older
> local checkouts that predate PR #423.

> **Live evidence** came from the gateway's Prometheus `/metrics` on the box: an OPC UA `historyread` on a
> historized tag produced `historian_gateway_grpc_requests_total{method="ReadRaw",service="HistorianRead"}=1`
> and `historian_session_pool_warm_sessions=1`, while a historized-tag **deploy** produced **zero**
> `EnsureTags` calls.

---

## Issue 1 — `GatewayTagProvisioner` is never wired; EnsureTags auto-provisioning is dormant (HIGH, confirmed)

**Summary.** PR #423 ships `GatewayTagProvisioner` (the `IHistorianProvisioning` implementation that calls the
gateway's `EnsureTags`) and unit tests for it, but it is **never registered in DI and never passed into the
`AddressSpaceApplier`**. At runtime the applier uses the no-op `NullHistorianProvisioning`, so **deploying
historized tags does NOT provision them in the historian**. The documented "EnsureTags auto-provisions
historian tags on deploy" behaviour does not occur.

**Root cause (code).**
- `AddressSpaceApplier` constructor coalesces a missing provisioner to the no-op:
  - `src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/AddressSpaceApplier.cs:63`
    `_provisioning = provisioning ?? NullHistorianProvisioning.Instance;`
- The **only** construction site omits the `provisioning:` argument:
  - `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/ServiceCollectionExtensions.cs:258`
    ```csharp
    var applier = new AddressSpaceApplier(
        addressSpaceSink,
        loggerFactory.CreateLogger<AddressSpaceApplier>(),
        historizedSubscriptions: historizedSubscriptions);   // <-- no provisioning:
    ```
- **No `IHistorianProvisioning` / `GatewayTagProvisioner` registration exists anywhere** in the repo
  (`grep` over `origin/master` finds only the interface definition, `NullHistorianProvisioning`, and the
  `GatewayTagProvisioner` constructor definition itself — no `AddSingleton<IHistorianProvisioning, …>` and
  no `new GatewayTagProvisioner(…)`).
- Consequently `ProvisionHistorizedTags` (called unconditionally at
  `AddressSpaceApplier.cs:208`, body at `:226`, dispatch at `:261`) builds the request list and invokes
  `_provisioning.EnsureTagsAsync(...)` on the **no-op**, which returns without contacting the gateway.

**Live evidence.** On wonder-app-vd03: flagged the `parts-count` equipment tag `isHistorized:true`, deployed,
and restarted the host. The gateway `/metrics` showed **no** gRPC request series for `HistorianTags`/`EnsureTags`
(zero gateway calls). By contrast, an explicit OPC UA `historyread` on the same tag *did* reach the gateway
(`ReadRaw`), confirming the read path is wired but the provisioning path is not.

**Impact.**
- Historized tags are never created in the historian on deploy.
- When **continuous historization** (`ContinuousHistorization:Enabled=true`) is later turned on, the recorder
  drains live values to historian tags that were never provisioned — those writes will fail / land nowhere
  unless the tags are created by some other means.
- Docs (`docs/Historian.md`, repo `CLAUDE.md`, and the umbrella scadaproj `CLAUDE.md`) describe provisioning
  as active; that claim is currently false. See Issue 4.

**Recommended fix (two parts).**
1. **Register the provisioner** in the Host alongside the existing gateway wiring — in
   `src/Server/ZB.MOM.WW.OtOpcUa.Host/Program.cs`, inside the `if (serverHistorianOptions.Enabled)` block that
   already calls `AddServerHistorian` / `AddAlarmHistorian` — e.g.
   ```csharp
   builder.Services.AddSingleton<IHistorianProvisioning>(sp =>
       new GatewayTagProvisioner(
           HistorianGatewayClientAdapter.Create(serverHistorianOptions, sp.GetRequiredService<ILoggerFactory>()),
           sp.GetRequiredService<ILogger<GatewayTagProvisioner>>()));
   ```
   (mirror the `GatewayHistorianValueWriter` registration pattern already in that block).
2. **Pass it into the applier** at `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/ServiceCollectionExtensions.cs:258`
   by resolving it the same way sibling deps are resolved there (`resolver.GetService<…>()`):
   ```csharp
   var provisioning = resolver.GetService<IHistorianProvisioning>();
   var applier = new AddressSpaceApplier(
       addressSpaceSink,
       loggerFactory.CreateLogger<AddressSpaceApplier>(),
       provisioning: provisioning,
       historizedSubscriptions: historizedSubscriptions);
   ```
3. **Add a wiring/integration test** asserting `EnsureTags` is invoked on a historized-tag deploy. The existing
   `GatewayTagProvisioner` unit tests pass but do not cover the DI wiring, which is exactly the gap that let
   this ship dormant.

---

## Issue 2 — Provisioning success is logged nothing; the dormant path is invisible (LOW, observability)

**Summary.** The provisioning continuation in `AddressSpaceApplier` only emits a log line when
`result.Failed > 0 || result.Skipped > 0`. A fully-successful provisioning (`Ensured>0, Failed=0, Skipped=0`)
**and** the no-op `NullHistorianProvisioning` path both log **nothing**. This is what made Issue 1 invisible:
there was no log line either way, so "no provisioning log" could not be distinguished from "provisioning ran
and fully succeeded."

**Location.** `src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/AddressSpaceApplier.cs` — the
`EnsureTagsAsync(...).ContinueWith(...)` block (~lines 261–283); the Information log is gated behind
`if (result.Failed > 0 || result.Skipped > 0)`.

**Recommended fix.** Emit an Information tally on every dispatch (e.g. `requested/ensured/skipped/failed`),
or at minimum whenever `Ensured > 0`, so a successful provisioning is observable and a no-op is detectable.

---

## Issue 3 — HistoryRead on a historized-but-unprovisioned tag blocks for the full CallTimeout (~30s) (MEDIUM, needs investigation)

**Summary.** An OPC UA `historyread` on a tag that is historized but has no corresponding historian
tag/data returned only after ~30 s — the gateway's `ReadRaw` recorded
`historian_gateway_grpc_request_duration_seconds_sum ≈ 30.5` for the single call, matching the
`ServerHistorian:CallTimeout` default of `00:00:30`. The read did not fail fast on an unknown tag; it ran to
the timeout and then errored (`outcome="error"`).

**Notes / attribution.**
- The slow leg is the gateway→historian read of a non-existent tag; the OtOpcUa-facing symptom is a 30 s OPC UA
  HistoryRead latency governed by `ServerHistorianOptions.CallTimeout` (default 30 s, `…Runtime/Historian/ServerHistorianOptions.cs`).
- This is **coupled to Issue 1**: because provisioning never runs, freshly-historized tags don't exist in the
  historian, so every HistoryRead on them hits this slow path until tags/data exist.
- Single data point (one unprovisioned tag); confirm whether the slow path is the historian, the gateway, or the
  client-side timeout before changing behaviour.

**Recommended action.** Primarily fix Issue 1 so historian tags exist. Separately, consider a shorter default
HistoryRead timeout and/or fast-fail when the historian reports an unknown tag, so a missing tag doesn't hang an
OPC UA HistoryRead for 30 s.

---

## Issue 4 — Docs describe provisioning as active (LOW, documentation)

`docs/Historian.md`, the repo `CLAUDE.md`, and the umbrella `scadaproj/CLAUDE.md` state that the gateway backend
"auto-provisions historian tags on deploy (EnsureTags)". Given Issue 1, that is inaccurate in the shipped build.
Update the docs once Issue 1 is fixed, or flag the provisioning behaviour as not-yet-wired in the interim.

---

## Issue 5 — Provisioning covers only *newly-added* tags; flagging an existing tag for historization never provisions it (MEDIUM, follow-up to the Issue 1 fix)

**Status:** found while live-verifying the Issue 1 fix (commit `257214f7`) on wonder-app-vd03.

**Summary.** The Issue 1 fix correctly wires `GatewayTagProvisioner` and is confirmed deployed (the new
`"...provisioning completed (dispatched…"` log literal is present in the deployed `OpcUaServer.dll`). However,
`ProvisionHistorizedTags` still iterates **only** `plan.AddedEquipmentTags`:

- `src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/AddressSpaceApplier.cs` (`ProvisionHistorizedTags`):
  `foreach (var tag in plan.AddedEquipmentTags)` — `ChangedEquipmentTags` is **not** considered.

So when an operator flags an **existing** tag `isHistorized:true` and deploys, that tag is classified
**Changed** (its config changed; the tag already existed) and is **never provisioned**. This is compounded by
the bootstrap "restore served state" behaviour: a host restart re-applies the served address space
**incrementally**, so an existing tag comes back as *unchanged/changed*, never *Added* — i.e. a restart does
not provision it either.

**Live evidence.** Flagging `parts-count` (an existing tag) historized + deploy + restart produced **zero**
`EnsureTags` calls at the gateway (`/metrics` had no `HistorianTags/EnsureTags` series). The read path on the
same tag *did* reach the gateway (`HistorianRead/ReadRaw`), confirming the tag was historized and the gateway
link works — only provisioning was skipped. (A raw `dbo.Tag` INSERT of a brand-new historized tag did not
materialise either, because a raw insert bypasses the normal compose/validate path — so the clean way to
exercise the *added* path is to author a new historized tag via the AdminUI.)

**Impact.** The most common operator action — take an existing tag and turn on historization — does **not**
auto-provision the historian tag. Continuous historization for such a tag would then write to a tag that does
not exist. Only genuinely **new** historized tags get provisioned.

**Recommended fix.** Provision the union of added **and** newly-historized changed tags: also iterate
`plan.ChangedEquipmentTags` (or compute "tags whose `IsHistorized` transitioned false→true") and include them
in the `EnsureTags` request. `EnsureTags` is idempotent, so re-ensuring an already-present tag is safe. Add a
test that flips an existing tag to historized and asserts `EnsureTags` is invoked.

---

## Pre-existing observation (not introduced by this work)

- **Serilog log path is service-CWD-relative.** On wonder-app-vd03 the host writes to
  `C:\Windows\System32\logs\otopcua-<date>.log` (because the service CWD is `System32`), not under the install
  directory. This was noted in prior sessions; re-observed here. Not a gateway-integration issue, but it makes
  on-box diagnosis awkward and is worth fixing (anchor the Serilog file sink to an absolute/content-root path).

---

## What is working (for context)

The OtOpcUa↔gateway **read and credential path is confirmed working end-to-end** on wonder-app-vd03: an OPC UA
HistoryRead routed through `GatewayHistorianDataSource` to the gateway over h2c, authenticated with a
peppered-HMAC API key (`historian:read` scope), and leased a real historian session (the gateway reached
wonder-sql-vd03). Only the **provisioning** half (Issue 1) is unwired.