lmxopcua/docs/plans/2026-06-27-historian-gateway-integration-issues.md

HistorianGateway backend — issues identified

Date: 2026-06-27 Context: Installed the ZB.MOM.WW.HistorianGateway sidecar on wonder-app-vd03 and updated OtOpcUa to origin/master (245316d8, PR #423 "HistorianGateway as the OtOpcUa historian backend"), then smoke-tested the OtOpcUa→gateway link live. These are the issues found during that work.

File/line references are against origin/master @ 245316d8 (the build that contains the gateway backend — src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway/). They will not resolve on older local checkouts that predate PR #423.

Live evidence came from the gateway's Prometheus /metrics on the box: an OPC UA historyread on a historized tag produced historian_gateway_grpc_requests_total{method="ReadRaw",service="HistorianRead"}=1 and historian_session_pool_warm_sessions=1, while a historized-tag deploy produced zero EnsureTags calls.

---

Issue 1 — GatewayTagProvisioner is never wired; EnsureTags auto-provisioning is dormant (HIGH, confirmed)

Summary. PR #423 ships GatewayTagProvisioner (the IHistorianProvisioning implementation that calls the gateway's EnsureTags) and unit tests for it, but it is never registered in DI and never passed into the AddressSpaceApplier. At runtime the applier uses the no-op NullHistorianProvisioning, so deploying historized tags does NOT provision them in the historian. The documented "EnsureTags auto-provisions historian tags on deploy" behaviour does not occur.

Root cause (code).

• AddressSpaceApplier constructor coalesces a missing provisioner to the no-op:
  • src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/AddressSpaceApplier.cs:63 _provisioning = provisioning ?? NullHistorianProvisioning.Instance;
• The only construction site omits the provisioning: argument:
  • src/Server/ZB.MOM.WW.OtOpcUa.Runtime/ServiceCollectionExtensions.cs:258

    var applier = new AddressSpaceApplier(
        addressSpaceSink,
        loggerFactory.CreateLogger<AddressSpaceApplier>(),
        historizedSubscriptions: historizedSubscriptions);   // <-- no provisioning:
    

• No IHistorianProvisioning / GatewayTagProvisioner registration exists anywhere in the repo (grep over origin/master finds only the interface definition, NullHistorianProvisioning, and the GatewayTagProvisioner constructor definition itself — no AddSingleton<IHistorianProvisioning, …> and no new GatewayTagProvisioner(…)).
• Consequently ProvisionHistorizedTags (called unconditionally at AddressSpaceApplier.cs:208, body at :226, dispatch at :261) builds the request list and invokes _provisioning.EnsureTagsAsync(...) on the no-op, which returns without contacting the gateway.

Live evidence. On wonder-app-vd03: flagged the parts-count equipment tag isHistorized:true, deployed, and restarted the host. The gateway /metrics showed no gRPC request series for HistorianTags/EnsureTags (zero gateway calls). By contrast, an explicit OPC UA historyread on the same tag did reach the gateway (ReadRaw), confirming the read path is wired but the provisioning path is not.

Impact.

• Historized tags are never created in the historian on deploy.
• When continuous historization (ContinuousHistorization:Enabled=true) is later turned on, the recorder drains live values to historian tags that were never provisioned — those writes will fail / land nowhere unless the tags are created by some other means.
• Docs (docs/Historian.md, repo CLAUDE.md, and the umbrella scadaproj CLAUDE.md) describe provisioning as active; that claim is currently false. See Issue 4.

Recommended fix (two parts).

1. Register the provisioner in the Host alongside the existing gateway wiring — in src/Server/ZB.MOM.WW.OtOpcUa.Host/Program.cs, inside the if (serverHistorianOptions.Enabled) block that already calls AddServerHistorian / AddAlarmHistorian — e.g.

   builder.Services.AddSingleton<IHistorianProvisioning>(sp =>
       new GatewayTagProvisioner(
           HistorianGatewayClientAdapter.Create(serverHistorianOptions, sp.GetRequiredService<ILoggerFactory>()),
           sp.GetRequiredService<ILogger<GatewayTagProvisioner>>()));
   

   (mirror the GatewayHistorianValueWriter registration pattern already in that block).
2. Pass it into the applier at src/Server/ZB.MOM.WW.OtOpcUa.Runtime/ServiceCollectionExtensions.cs:258 by resolving it the same way sibling deps are resolved there (resolver.GetService<…>()):

   var provisioning = resolver.GetService<IHistorianProvisioning>();
   var applier = new AddressSpaceApplier(
       addressSpaceSink,
       loggerFactory.CreateLogger<AddressSpaceApplier>(),
       provisioning: provisioning,
       historizedSubscriptions: historizedSubscriptions);
   

3. Add a wiring/integration test asserting EnsureTags is invoked on a historized-tag deploy. The existing GatewayTagProvisioner unit tests pass but do not cover the DI wiring, which is exactly the gap that let this ship dormant.

---

Issue 2 — Provisioning success is logged nothing; the dormant path is invisible (LOW, observability)

Summary. The provisioning continuation in AddressSpaceApplier only emits a log line when result.Failed > 0 || result.Skipped > 0. A fully-successful provisioning (Ensured>0, Failed=0, Skipped=0) and the no-op NullHistorianProvisioning path both log nothing. This is what made Issue 1 invisible: there was no log line either way, so "no provisioning log" could not be distinguished from "provisioning ran and fully succeeded."

Location. src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/AddressSpaceApplier.cs — the EnsureTagsAsync(...).ContinueWith(...) block (~lines 261–283); the Information log is gated behind if (result.Failed > 0 || result.Skipped > 0).

Recommended fix. Emit an Information tally on every dispatch (e.g. requested/ensured/skipped/failed), or at minimum whenever Ensured > 0, so a successful provisioning is observable and a no-op is detectable.

---

Issue 3 — HistoryRead on a historized-but-unprovisioned tag blocks for the full CallTimeout (~30s) (MEDIUM, needs investigation)

Summary. An OPC UA historyread on a tag that is historized but has no corresponding historian tag/data returned only after ~30 s — the gateway's ReadRaw recorded historian_gateway_grpc_request_duration_seconds_sum ≈ 30.5 for the single call, matching the ServerHistorian:CallTimeout default of 00:00:30. The read did not fail fast on an unknown tag; it ran to the timeout and then errored (outcome="error").

Notes / attribution.

• The slow leg is the gateway→historian read of a non-existent tag; the OtOpcUa-facing symptom is a 30 s OPC UA HistoryRead latency governed by ServerHistorianOptions.CallTimeout (default 30 s, …Runtime/Historian/ServerHistorianOptions.cs).
• This is coupled to Issue 1: because provisioning never runs, freshly-historized tags don't exist in the historian, so every HistoryRead on them hits this slow path until tags/data exist.
• Single data point (one unprovisioned tag); confirm whether the slow path is the historian, the gateway, or the client-side timeout before changing behaviour.

Recommended action. Primarily fix Issue 1 so historian tags exist. Separately, consider a shorter default HistoryRead timeout and/or fast-fail when the historian reports an unknown tag, so a missing tag doesn't hang an OPC UA HistoryRead for 30 s.

---

Issue 4 — Docs describe provisioning as active (LOW, documentation)

docs/Historian.md, the repo CLAUDE.md, and the umbrella scadaproj/CLAUDE.md state that the gateway backend "auto-provisions historian tags on deploy (EnsureTags)". Given Issue 1, that is inaccurate in the shipped build. Update the docs once Issue 1 is fixed, or flag the provisioning behaviour as not-yet-wired in the interim.

---

Issue 5 — Provisioning covers only newly-added tags; flagging an existing tag for historization never provisions it (MEDIUM, follow-up to the Issue 1 fix)

Status: found while live-verifying the Issue 1 fix (commit 257214f7) on wonder-app-vd03.

Summary. The Issue 1 fix correctly wires GatewayTagProvisioner and is confirmed deployed (the new "...provisioning completed (dispatched…" log literal is present in the deployed OpcUaServer.dll). However, ProvisionHistorizedTags still iterates only plan.AddedEquipmentTags:

• src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/AddressSpaceApplier.cs (ProvisionHistorizedTags): foreach (var tag in plan.AddedEquipmentTags) — ChangedEquipmentTags is not considered.

So when an operator flags an existing tag isHistorized:true and deploys, that tag is classified Changed (its config changed; the tag already existed) and is never provisioned. This is compounded by the bootstrap "restore served state" behaviour: a host restart re-applies the served address space incrementally, so an existing tag comes back as unchanged/changed, never Added — i.e. a restart does not provision it either.

Live evidence. Flagging parts-count (an existing tag) historized + deploy + restart produced zero EnsureTags calls at the gateway (/metrics had no HistorianTags/EnsureTags series). The read path on the same tag did reach the gateway (HistorianRead/ReadRaw), confirming the tag was historized and the gateway link works — only provisioning was skipped. (A raw dbo.Tag INSERT of a brand-new historized tag did not materialise either, because a raw insert bypasses the normal compose/validate path — so the clean way to exercise the added path is to author a new historized tag via the AdminUI.)

Impact. The most common operator action — take an existing tag and turn on historization — does not auto-provision the historian tag. Continuous historization for such a tag would then write to a tag that does not exist. Only genuinely new historized tags get provisioned.

Recommended fix. Provision the union of added and newly-historized changed tags: also iterate plan.ChangedEquipmentTags (or compute "tags whose IsHistorized transitioned false→true") and include them in the EnsureTags request. EnsureTags is idempotent, so re-ensuring an already-present tag is safe. Add a test that flips an existing tag to historized and asserts EnsureTags is invoked.

---

Pre-existing observation (not introduced by this work)

• Serilog log path is service-CWD-relative. On wonder-app-vd03 the host writes to C:\Windows\System32\logs\otopcua-<date>.log (because the service CWD is System32), not under the install directory. This was noted in prior sessions; re-observed here. Not a gateway-integration issue, but it makes on-box diagnosis awkward and is worth fixing (anchor the Serilog file sink to an absolute/content-root path).

---

What is working (for context)

The OtOpcUa↔gateway read and credential path is confirmed working end-to-end on wonder-app-vd03: an OPC UA HistoryRead routed through GatewayHistorianDataSource to the gateway over h2c, authenticated with a peppered-HMAC API key (historian:read scope), and leased a real historian session (the gateway reached wonder-sql-vd03). Only the provisioning half (Issue 1) is unwired.