chore(r2-12): untrack/archive root planning files; honor pending.md's never-stage rule (07/U-4)
- pending.md, current.md: git rm --cached (untrack, keep on disk) + root-anchored .gitignore entries — making the files' own declared 'never stage' rule true. - looseends.md: deleted (2026-05-18 snapshot; sole open item + all follow-ups target the retired Wonderware historian sidecar rig, moot since the gateway cutover; recoverable from history). - stillpending.md -> docs/plans/2026-06-18-stillpending-backlog-audit.md (git mv; referenced by docs/Historian.md + memory index, preserved under dated-plan convention). - HISTORIAN-GATEWAY-INTEGRATION-ISSUES.md -> docs/plans/2026-06-27-historian-gateway-integration-issues.md (git mv). - docs/Historian.md §'Closed backlog §2 items' repointed at the archived path. Historical docs/plans/* references to the bare stillpending.md filename are frozen point-in-time records, left untouched per the documentation-audit convention.
This commit is contained in:
Joseph Doherty
@@ -0,0 +1,183 @@
|
||||
# HistorianGateway backend — issues identified
|
||||
|
||||
**Date:** 2026-06-27
|
||||
**Context:** Installed the `ZB.MOM.WW.HistorianGateway` sidecar on **wonder-app-vd03** and updated OtOpcUa to
|
||||
`origin/master` (`245316d8`, PR #423 "HistorianGateway as the OtOpcUa historian backend"), then smoke-tested
|
||||
the OtOpcUa→gateway link live. These are the issues found during that work.
|
||||
|
||||
> **File/line references are against `origin/master` @ `245316d8`** (the build that contains the gateway
|
||||
> backend — `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway/`). They will not resolve on older
|
||||
> local checkouts that predate PR #423.
|
||||
|
||||
> **Live evidence** came from the gateway's Prometheus `/metrics` on the box: an OPC UA `historyread` on a
|
||||
> historized tag produced `historian_gateway_grpc_requests_total{method="ReadRaw",service="HistorianRead"}=1`
|
||||
> and `historian_session_pool_warm_sessions=1`, while a historized-tag **deploy** produced **zero**
|
||||
> `EnsureTags` calls.
|
||||
|
||||
---
|
||||
|
||||
## Issue 1 — `GatewayTagProvisioner` is never wired; EnsureTags auto-provisioning is dormant (HIGH, confirmed)
|
||||
|
||||
**Summary.** PR #423 ships `GatewayTagProvisioner` (the `IHistorianProvisioning` implementation that calls the
|
||||
gateway's `EnsureTags`) and unit tests for it, but it is **never registered in DI and never passed into the
|
||||
`AddressSpaceApplier`**. At runtime the applier uses the no-op `NullHistorianProvisioning`, so **deploying
|
||||
historized tags does NOT provision them in the historian**. The documented "EnsureTags auto-provisions
|
||||
historian tags on deploy" behaviour does not occur.
|
||||
|
||||
**Root cause (code).**
|
||||
- `AddressSpaceApplier` constructor coalesces a missing provisioner to the no-op:
|
||||
- `src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/AddressSpaceApplier.cs:63`
|
||||
`_provisioning = provisioning ?? NullHistorianProvisioning.Instance;`
|
||||
- The **only** construction site omits the `provisioning:` argument:
|
||||
- `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/ServiceCollectionExtensions.cs:258`
|
||||
```csharp
|
||||
var applier = new AddressSpaceApplier(
|
||||
addressSpaceSink,
|
||||
loggerFactory.CreateLogger<AddressSpaceApplier>(),
|
||||
historizedSubscriptions: historizedSubscriptions); // <-- no provisioning:
|
||||
```
|
||||
- **No `IHistorianProvisioning` / `GatewayTagProvisioner` registration exists anywhere** in the repo
|
||||
(`grep` over `origin/master` finds only the interface definition, `NullHistorianProvisioning`, and the
|
||||
`GatewayTagProvisioner` constructor definition itself — no `AddSingleton<IHistorianProvisioning, …>` and
|
||||
no `new GatewayTagProvisioner(…)`).
|
||||
- Consequently `ProvisionHistorizedTags` (called unconditionally at
|
||||
`AddressSpaceApplier.cs:208`, body at `:226`, dispatch at `:261`) builds the request list and invokes
|
||||
`_provisioning.EnsureTagsAsync(...)` on the **no-op**, which returns without contacting the gateway.
|
||||
|
||||
**Live evidence.** On wonder-app-vd03: flagged the `parts-count` equipment tag `isHistorized:true`, deployed,
|
||||
and restarted the host. The gateway `/metrics` showed **no** gRPC request series for `HistorianTags`/`EnsureTags`
|
||||
(zero gateway calls). By contrast, an explicit OPC UA `historyread` on the same tag *did* reach the gateway
|
||||
(`ReadRaw`), confirming the read path is wired but the provisioning path is not.
|
||||
|
||||
**Impact.**
|
||||
- Historized tags are never created in the historian on deploy.
|
||||
- When **continuous historization** (`ContinuousHistorization:Enabled=true`) is later turned on, the recorder
|
||||
drains live values to historian tags that were never provisioned — those writes will fail / land nowhere
|
||||
unless the tags are created by some other means.
|
||||
- Docs (`docs/Historian.md`, repo `CLAUDE.md`, and the umbrella scadaproj `CLAUDE.md`) describe provisioning
|
||||
as active; that claim is currently false. See Issue 4.
|
||||
|
||||
**Recommended fix (two parts).**
|
||||
1. **Register the provisioner** in the Host alongside the existing gateway wiring — in
|
||||
`src/Server/ZB.MOM.WW.OtOpcUa.Host/Program.cs`, inside the `if (serverHistorianOptions.Enabled)` block that
|
||||
already calls `AddServerHistorian` / `AddAlarmHistorian` — e.g.
|
||||
```csharp
|
||||
builder.Services.AddSingleton<IHistorianProvisioning>(sp =>
|
||||
new GatewayTagProvisioner(
|
||||
HistorianGatewayClientAdapter.Create(serverHistorianOptions, sp.GetRequiredService<ILoggerFactory>()),
|
||||
sp.GetRequiredService<ILogger<GatewayTagProvisioner>>()));
|
||||
```
|
||||
(mirror the `GatewayHistorianValueWriter` registration pattern already in that block).
|
||||
2. **Pass it into the applier** at `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/ServiceCollectionExtensions.cs:258`
|
||||
by resolving it the same way sibling deps are resolved there (`resolver.GetService<…>()`):
|
||||
```csharp
|
||||
var provisioning = resolver.GetService<IHistorianProvisioning>();
|
||||
var applier = new AddressSpaceApplier(
|
||||
addressSpaceSink,
|
||||
loggerFactory.CreateLogger<AddressSpaceApplier>(),
|
||||
provisioning: provisioning,
|
||||
historizedSubscriptions: historizedSubscriptions);
|
||||
```
|
||||
3. **Add a wiring/integration test** asserting `EnsureTags` is invoked on a historized-tag deploy. The existing
|
||||
`GatewayTagProvisioner` unit tests pass but do not cover the DI wiring, which is exactly the gap that let
|
||||
this ship dormant.
|
||||
|
||||
---
|
||||
|
||||
## Issue 2 — Provisioning success is logged nothing; the dormant path is invisible (LOW, observability)
|
||||
|
||||
**Summary.** The provisioning continuation in `AddressSpaceApplier` only emits a log line when
|
||||
`result.Failed > 0 || result.Skipped > 0`. A fully-successful provisioning (`Ensured>0, Failed=0, Skipped=0`)
|
||||
**and** the no-op `NullHistorianProvisioning` path both log **nothing**. This is what made Issue 1 invisible:
|
||||
there was no log line either way, so "no provisioning log" could not be distinguished from "provisioning ran
|
||||
and fully succeeded."
|
||||
|
||||
**Location.** `src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/AddressSpaceApplier.cs` — the
|
||||
`EnsureTagsAsync(...).ContinueWith(...)` block (~lines 261–283); the Information log is gated behind
|
||||
`if (result.Failed > 0 || result.Skipped > 0)`.
|
||||
|
||||
**Recommended fix.** Emit an Information tally on every dispatch (e.g. `requested/ensured/skipped/failed`),
|
||||
or at minimum whenever `Ensured > 0`, so a successful provisioning is observable and a no-op is detectable.
|
||||
|
||||
---
|
||||
|
||||
## Issue 3 — HistoryRead on a historized-but-unprovisioned tag blocks for the full CallTimeout (~30s) (MEDIUM, needs investigation)
|
||||
|
||||
**Summary.** An OPC UA `historyread` on a tag that is historized but has no corresponding historian
|
||||
tag/data returned only after ~30 s — the gateway's `ReadRaw` recorded
|
||||
`historian_gateway_grpc_request_duration_seconds_sum ≈ 30.5` for the single call, matching the
|
||||
`ServerHistorian:CallTimeout` default of `00:00:30`. The read did not fail fast on an unknown tag; it ran to
|
||||
the timeout and then errored (`outcome="error"`).
|
||||
|
||||
**Notes / attribution.**
|
||||
- The slow leg is the gateway→historian read of a non-existent tag; the OtOpcUa-facing symptom is a 30 s OPC UA
|
||||
HistoryRead latency governed by `ServerHistorianOptions.CallTimeout` (default 30 s, `…Runtime/Historian/ServerHistorianOptions.cs`).
|
||||
- This is **coupled to Issue 1**: because provisioning never runs, freshly-historized tags don't exist in the
|
||||
historian, so every HistoryRead on them hits this slow path until tags/data exist.
|
||||
- Single data point (one unprovisioned tag); confirm whether the slow path is the historian, the gateway, or the
|
||||
client-side timeout before changing behaviour.
|
||||
|
||||
**Recommended action.** Primarily fix Issue 1 so historian tags exist. Separately, consider a shorter default
|
||||
HistoryRead timeout and/or fast-fail when the historian reports an unknown tag, so a missing tag doesn't hang an
|
||||
OPC UA HistoryRead for 30 s.
|
||||
|
||||
---
|
||||
|
||||
## Issue 4 — Docs describe provisioning as active (LOW, documentation)
|
||||
|
||||
`docs/Historian.md`, the repo `CLAUDE.md`, and the umbrella `scadaproj/CLAUDE.md` state that the gateway backend
|
||||
"auto-provisions historian tags on deploy (EnsureTags)". Given Issue 1, that is inaccurate in the shipped build.
|
||||
Update the docs once Issue 1 is fixed, or flag the provisioning behaviour as not-yet-wired in the interim.
|
||||
|
||||
---
|
||||
|
||||
## Issue 5 — Provisioning covers only *newly-added* tags; flagging an existing tag for historization never provisions it (MEDIUM, follow-up to the Issue 1 fix)
|
||||
|
||||
**Status:** found while live-verifying the Issue 1 fix (commit `257214f7`) on wonder-app-vd03.
|
||||
|
||||
**Summary.** The Issue 1 fix correctly wires `GatewayTagProvisioner` and is confirmed deployed (the new
|
||||
`"...provisioning completed (dispatched…"` log literal is present in the deployed `OpcUaServer.dll`). However,
|
||||
`ProvisionHistorizedTags` still iterates **only** `plan.AddedEquipmentTags`:
|
||||
|
||||
- `src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/AddressSpaceApplier.cs` (`ProvisionHistorizedTags`):
|
||||
`foreach (var tag in plan.AddedEquipmentTags)` — `ChangedEquipmentTags` is **not** considered.
|
||||
|
||||
So when an operator flags an **existing** tag `isHistorized:true` and deploys, that tag is classified
|
||||
**Changed** (its config changed; the tag already existed) and is **never provisioned**. This is compounded by
|
||||
the bootstrap "restore served state" behaviour: a host restart re-applies the served address space
|
||||
**incrementally**, so an existing tag comes back as *unchanged/changed*, never *Added* — i.e. a restart does
|
||||
not provision it either.
|
||||
|
||||
**Live evidence.** Flagging `parts-count` (an existing tag) historized + deploy + restart produced **zero**
|
||||
`EnsureTags` calls at the gateway (`/metrics` had no `HistorianTags/EnsureTags` series). The read path on the
|
||||
same tag *did* reach the gateway (`HistorianRead/ReadRaw`), confirming the tag was historized and the gateway
|
||||
link works — only provisioning was skipped. (A raw `dbo.Tag` INSERT of a brand-new historized tag did not
|
||||
materialise either, because a raw insert bypasses the normal compose/validate path — so the clean way to
|
||||
exercise the *added* path is to author a new historized tag via the AdminUI.)
|
||||
|
||||
**Impact.** The most common operator action — take an existing tag and turn on historization — does **not**
|
||||
auto-provision the historian tag. Continuous historization for such a tag would then write to a tag that does
|
||||
not exist. Only genuinely **new** historized tags get provisioned.
|
||||
|
||||
**Recommended fix.** Provision the union of added **and** newly-historized changed tags: also iterate
|
||||
`plan.ChangedEquipmentTags` (or compute "tags whose `IsHistorized` transitioned false→true") and include them
|
||||
in the `EnsureTags` request. `EnsureTags` is idempotent, so re-ensuring an already-present tag is safe. Add a
|
||||
test that flips an existing tag to historized and asserts `EnsureTags` is invoked.
|
||||
|
||||
---
|
||||
|
||||
## Pre-existing observation (not introduced by this work)
|
||||
|
||||
- **Serilog log path is service-CWD-relative.** On wonder-app-vd03 the host writes to
|
||||
`C:\Windows\System32\logs\otopcua-<date>.log` (because the service CWD is `System32`), not under the install
|
||||
directory. This was noted in prior sessions; re-observed here. Not a gateway-integration issue, but it makes
|
||||
on-box diagnosis awkward and is worth fixing (anchor the Serilog file sink to an absolute/content-root path).
|
||||
|
||||
---
|
||||
|
||||
## What is working (for context)
|
||||
|
||||
The OtOpcUa↔gateway **read and credential path is confirmed working end-to-end** on wonder-app-vd03: an OPC UA
|
||||
HistoryRead routed through `GatewayHistorianDataSource` to the gateway over h2c, authenticated with a
|
||||
peppered-HMAC API key (`historian:read` scope), and leased a real historian session (the gateway reached
|
||||
wonder-sql-vd03). Only the **provisioning** half (Issue 1) is unwired.
|
||||
Reference in New Issue
Block a user