dohertj2/lmxopcua
1
0
Fork 0
Code Issues 5 Pull Requests Actions Packages Projects Releases Wiki Activity

Historian: no read-only binding to an existing AVEVA tag; EnsureTags upserts (retypes, re-describes) it on every add #899

New Issue
Closed
opened 2026-11-13 00:43:30 -05:00 by dohertj2 · 1 comment
dohertj2 commented 2026-11-13 00:43:30 -05:00
Owner
Copy Link
Copy Source

There is no way to expose an existing AVEVA historian tag's history on an OtOpcUa node without OtOpcUa rewriting that tag.

  • A node is history-readable only as an authored raw tag with isHistorized: true (+ historianTagname).
  • Every added (or renamed/retyped) historized raw tag fires IHistorianProvisioning.EnsureTagsAsync → GatewayTagProvisioner → gateway EnsureTags → Tags.AddAsync. That is an upsert: it sets the DataType (Boolean→Int2, Int32→Int4, …), StorageRate 1000 ms, IntegralDivisor 1 and Description = the OtOpcUa tag name.
  • The only off switch is ServerHistorian:Enabled=false, which also disables HistoryRead. Typical gateway keys carry historian:tags:write (the deployed OtOpcUa-wonder key on vd03 does).

Example. Binding an Int32 node to SysTimeSec (AVEVA UInt2 system tag) would re-ensure it as Int4 with OtOpcUa's description. Binding a Boolean node to SysPulse (discrete) would push it as an Int2 analog. This happens again on every host start, because the first plan adds every tag.

This blocked the Int32/Boolean part of the #700 deployed-endpoint gate on DEV vd03. That gate used only types the applier happens not to provision (#898) and therefore stayed zero-write. Evidence: docs/plans/evidence/2026-11-13-historian-700-vd03/vd03-endpoint-gate.md.

Proposal. A per-tag opt-out in TagConfig, e.g. "historianProvision": false (or a mode read-only), that keeps HistoryRead and skips EnsureTags and continuous-historization capture for that tag. Optionally also a server switch ServerHistorian:ProvisionTags (default true). Either should be honoured by the provisioner, the recorder and the validator/AdminUI (CSV column + editor toggle). Consider also making the default safer: refuse to upsert a tag that already exists with a different type unless explicitly allowed.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi

There is no way to expose an **existing** AVEVA historian tag's history on an OtOpcUa node without OtOpcUa **rewriting that tag**. - A node is history-readable only as an authored raw tag with `isHistorized: true` (+ `historianTagname`). - Every added (or renamed/retyped) historized raw tag fires `IHistorianProvisioning.EnsureTagsAsync` → `GatewayTagProvisioner` → gateway `EnsureTags` → `Tags.AddAsync`. That is an **upsert**: it sets the DataType (`Boolean`→Int2, `Int32`→Int4, …), StorageRate 1000 ms, IntegralDivisor 1 and Description = the OtOpcUa tag name. - The only off switch is `ServerHistorian:Enabled=false`, which also disables HistoryRead. Typical gateway keys carry `historian:tags:write` (the deployed `OtOpcUa-wonder` key on vd03 does). **Example.** Binding an `Int32` node to `SysTimeSec` (AVEVA UInt2 system tag) would re-ensure it as Int4 with OtOpcUa's description. Binding a `Boolean` node to `SysPulse` (discrete) would push it as an Int2 analog. This happens again on every host start, because the first plan adds every tag. This blocked the Int32/Boolean part of the #700 deployed-endpoint gate on DEV vd03. That gate used only types the applier happens not to provision (#898) and therefore stayed zero-write. Evidence: `docs/plans/evidence/2026-11-13-historian-700-vd03/vd03-endpoint-gate.md`. **Proposal.** A per-tag opt-out in TagConfig, e.g. `"historianProvision": false` (or a mode `read-only`), that keeps HistoryRead and skips EnsureTags and continuous-historization capture for that tag. Optionally also a server switch `ServerHistorian:ProvisionTags` (default `true`). Either should be honoured by the provisioner, the recorder and the validator/AdminUI (CSV column + editor toggle). Consider also making the default safer: refuse to upsert a tag that already exists with a different type unless explicitly allowed. Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
dohertj2 added the type:bugpriority:p2 labels 2026-11-13 00:43:30 -05:00
dohertj2 commented 2026-11-13 01:17:30 -05:00
Author
Owner
Copy Link
Copy Source

Design note and fix (lands together with #898)

Problem. EnsureTags is an upsert. Every added historized raw tag, which on a host start means every one, was provisioned. Binding a node to an existing AVEVA tag through historianTagname therefore re-typed and re-described that tag. The only off switch was ServerHistorian:Enabled=false, which also turns off reads.

Design. There are two orthogonal opt-outs. The default for ordinary owned tags is unchanged.

  1. Per tag: TagConfig.historianReadOnly: true. This is a read-only binding.
    • The node keeps Historizing=true and the HistoryRead bit, and reads the named historian tag.
    • The applier never puts it in an EnsureTags request, whether it was added or changed (AddressSpaceApplier.IsOwnedHistorizedValue).
    • The applier never feeds it to the continuous-historization recorder, so no WriteLiveValues reach it. OtOpcUa issues no write of any kind for that tag.
    • Clearing the flag makes the tag owned. It is then provisioned and recorded like a historization flip.
    • Flow:
      • TagConfigIntent.Parse sets HistorianReadOnly (only a JSON true counts).
      • The composer copies it into RawTagPlan.HistorianReadOnly, which takes part in equality and the hash.
      • The artifact carries the TagConfig verbatim. The deploy-artifact parity test asserts the flag survives artifact decode, and two entries were added to the golden corpus.
      • The applier reads it for provisioning and for the recorder feed.
    • It is node-inert. The classifier lists it in the surgical whitelist, the applier sends nothing to the sink for a flag-only change, and UNS reference nodes do not mirror it. UnsReferenceBackingParityTests records the reason.
    • Validation and authoring.
      • A non-boolean value (for example the string "true") is the deploy error HistorianReadOnlyInvalid. The runtime would read such a value as owned and upsert someone else's tag, so it must not deploy.
      • In the AdminUI, the CSV has a new common column HistorianReadOnly next to HistorianTagname. The export writes it, and the TagConfigJson residual strips the key. The import parses it strictly: a non-boolean cell is a row error.
      • TagHistorizeConfig.Read uses the runtime's own TagConfigIntent.Parse, so what the AdminUI shows is what the runtime does.
      • There is no modal toggle. isHistorized and historianTagname have none in the v3 raw modal either; they are authored in the CSV or the raw JSON. This keeps the new flag consistent with them.
  2. Server-wide: ServerHistorian:ProvisionTags. The values are case-insensitive, and an unknown value fails host start.
    • CreateOrUpdate (default) is the upsert, as before.
    • CreateOnly is Gateway only; any other backend fails host start. Each tag is checked with TagExistsAsync (a read) first. An existing tag is left alone and counted skipped. A lookup that fails is counted failed and the tag is not sent, so the policy fails closed.
    • Off keeps the no-op provisioner. Reads and recording are unaffected.
    • CreateOnly still records into existing tags. Only historianReadOnly stops recording.

Default for an explicit historianTagname: unchanged (owned). I considered making an override read-only by default and decided against it. An override does not mean "someone else's tag". The AdminUI's raw-rename warning tells authors to set historianTagname to pin an owned tag's history across RawPath renames, so overrides on owned tags are common. A read-only default would silently stop both provisioning and recording for all of them. Binding to a foreign tag is therefore an explicit opt-in, per tag or server-wide.

Not covered. On the SqlServer and QuestDB backends, HistoryUpdate on a read-only node is unchanged. Those stores are OtOpcUa's own tables, and HistoryUpdate is an explicit client operation governed by ACLs. The Gateway backend has no HistoryUpdate, so on AVEVA the binding is fully read-only.

Tests (all with a capturing or fake provisioner and gateway; nothing calls EnsureTags against a real historian):

  • AddressSpaceApplierProvisioningTests:
    • A read-only tag never reaches EnsureTags, either added or changed (flip to read-only, rename while read-only). An owned positive control and an override-without-flag tag (the default) both provision.
    • Clearing the flag provisions the tag.
    • A read-only tag is never fed to the recorder. Flipping the flag removes or adds the ref.
  • GatewayTagProvisionerTests for CreateOnly:
    • An existing tag is not sent and an absent one is.
    • If every tag exists, there is no EnsureTags call.
    • A failed lookup is not sent; a gRPC NotFound counts as absent.
    • Non-historizable types are never looked up.
    • The default policy never looks a tag up.
  • Policy wiring and validation: GatewayHistorianProviderTests, AddHistorianProvisioningTests (Off keeps the Null provisioner), ServerHistorianOptionsValidatorTests (unknown value; CreateOnly on SqlServer or QuestDB).
  • Parsing and authoring:
    • TagConfigIntentTests
    • DraftValidatorHistorianReadOnlyTests
    • TagHistorizeConfigTests
    • RawTagCsvRoundTripTests
    • AddressSpaceChangeClassifierDualNamespaceTests: a flag-only change is surgical.
    • DeploymentArtifactRawUnsParityTests
  • Mutation-checked: each guard was reverted on its own and the tests above failed. Results are in the closing comment.

Docs: docs/Historian.md gains §"Read-only bindings to existing historian tags (#899)", the ProvisionTags row and the upgrade warning. .claude/rules/historian.md, docs/Raw.md and docs/Uns.md are updated too.

Mutation results. Each guard was reverted on its own; every one was caught. The test projects were built and then run filtered.

Reverted guard Result
read-only check in IsOwnedHistorizedValue killed (3)
HistorianReadOnly in RawTagPlan.Equals killed by the new planner test (2)
composer copies the flag killed (artifact parity)
CreateOnly filter killed (3)
CreateOnly fail-closed on a lookup error killed (1)
policy reaches the provisioner killed (2)
Off keeps the Null provisioner killed (2)
CreateOnly is Gateway-only (options validator) killed (2)
HistorianReadOnlyInvalid rule killed (3)
CSV export column killed (1)
AdminUI read uses the runtime parse killed (1)

Verification.

  • Release build: 0 warnings, 0 errors.
  • Green: OpcUaServer.Tests 1290, Commons 377, Configuration 285, Core.Abstractions 801, AdminUI 2170, Historian.Gateway 593, Historian.SqlServer 45, Historian.QuestDb 88, and the Host historian-options validator 37.
  • Runtime.Tests: 1040 passed. One unrelated timing test (DriverHostActorSourceRemovedTests) failed under load and passes when run alone (5/5).

Landed on master as 6c4d2990. No live run: EnsureTags is a write, so the provisioning paths were verified only against fakes and recording doubles. Nothing was deployed.

## Design note and fix (lands together with #898) **Problem.** `EnsureTags` is an upsert. Every added historized raw tag, which on a host start means every one, was provisioned. Binding a node to an existing AVEVA tag through `historianTagname` therefore re-typed and re-described that tag. The only off switch was `ServerHistorian:Enabled=false`, which also turns off reads. **Design.** There are two orthogonal opt-outs. The default for ordinary owned tags is unchanged. 1. **Per tag: `TagConfig.historianReadOnly: true`.** This is a read-only binding. - The node keeps `Historizing=true` and the HistoryRead bit, and reads the named historian tag. - The applier never puts it in an `EnsureTags` request, whether it was added or changed (`AddressSpaceApplier.IsOwnedHistorizedValue`). - The applier never feeds it to the continuous-historization recorder, so no `WriteLiveValues` reach it. OtOpcUa issues no write of any kind for that tag. - Clearing the flag makes the tag owned. It is then provisioned and recorded like a historization flip. - Flow: - `TagConfigIntent.Parse` sets `HistorianReadOnly` (only a JSON `true` counts). - The composer copies it into `RawTagPlan.HistorianReadOnly`, which takes part in equality and the hash. - The artifact carries the `TagConfig` verbatim. The deploy-artifact parity test asserts the flag survives artifact decode, and two entries were added to the golden corpus. - The applier reads it for provisioning and for the recorder feed. - It is node-inert. The classifier lists it in the surgical whitelist, the applier sends nothing to the sink for a flag-only change, and UNS reference nodes do not mirror it. `UnsReferenceBackingParityTests` records the reason. - **Validation and authoring.** - A non-boolean value (for example the string `"true"`) is the deploy error `HistorianReadOnlyInvalid`. The runtime would read such a value as *owned* and upsert someone else's tag, so it must not deploy. - In the AdminUI, the CSV has a new common column `HistorianReadOnly` next to `HistorianTagname`. The export writes it, and the `TagConfigJson` residual strips the key. The import parses it strictly: a non-boolean cell is a row error. - `TagHistorizeConfig.Read` uses the runtime's own `TagConfigIntent.Parse`, so what the AdminUI shows is what the runtime does. - There is no modal toggle. `isHistorized` and `historianTagname` have none in the v3 raw modal either; they are authored in the CSV or the raw JSON. This keeps the new flag consistent with them. 2. **Server-wide: `ServerHistorian:ProvisionTags`.** The values are case-insensitive, and an unknown value fails host start. - `CreateOrUpdate` (default) is the upsert, as before. - `CreateOnly` is **Gateway only**; any other backend fails host start. Each tag is checked with `TagExistsAsync` (a read) first. An existing tag is left alone and counted `skipped`. A lookup that fails is counted `failed` and the tag is **not** sent, so the policy fails closed. - `Off` keeps the no-op provisioner. Reads and recording are unaffected. - `CreateOnly` still records into existing tags. Only `historianReadOnly` stops recording. **Default for an explicit `historianTagname`: unchanged (owned).** I considered making an override read-only by default and decided against it. An override does not mean "someone else's tag". The AdminUI's raw-rename warning tells authors to set `historianTagname` to pin an owned tag's history across RawPath renames, so overrides on owned tags are common. A read-only default would silently stop both provisioning and recording for all of them. Binding to a foreign tag is therefore an explicit opt-in, per tag or server-wide. **Not covered.** On the SqlServer and QuestDB backends, HistoryUpdate on a read-only node is unchanged. Those stores are OtOpcUa's own tables, and HistoryUpdate is an explicit client operation governed by ACLs. The Gateway backend has no HistoryUpdate, so on AVEVA the binding is fully read-only. **Tests** (all with a capturing or fake provisioner and gateway; nothing calls `EnsureTags` against a real historian): - `AddressSpaceApplierProvisioningTests`: - A read-only tag never reaches EnsureTags, either added or changed (flip to read-only, rename while read-only). An owned positive control and an override-without-flag tag (the default) both provision. - Clearing the flag provisions the tag. - A read-only tag is never fed to the recorder. Flipping the flag removes or adds the ref. - `GatewayTagProvisionerTests` for CreateOnly: - An existing tag is not sent and an absent one is. - If every tag exists, there is no EnsureTags call. - A failed lookup is not sent; a gRPC NotFound counts as absent. - Non-historizable types are never looked up. - The default policy never looks a tag up. - Policy wiring and validation: `GatewayHistorianProviderTests`, `AddHistorianProvisioningTests` (`Off` keeps the Null provisioner), `ServerHistorianOptionsValidatorTests` (unknown value; CreateOnly on SqlServer or QuestDB). - Parsing and authoring: - `TagConfigIntentTests` - `DraftValidatorHistorianReadOnlyTests` - `TagHistorizeConfigTests` - `RawTagCsvRoundTripTests` - `AddressSpaceChangeClassifierDualNamespaceTests`: a flag-only change is surgical. - `DeploymentArtifactRawUnsParityTests` - Mutation-checked: each guard was reverted on its own and the tests above failed. Results are in the closing comment. Docs: `docs/Historian.md` gains §"Read-only bindings to existing historian tags (#899)", the `ProvisionTags` row and the upgrade warning. `.claude/rules/historian.md`, `docs/Raw.md` and `docs/Uns.md` are updated too. **Mutation results.** Each guard was reverted on its own; every one was caught. The test projects were built and then run filtered. | Reverted guard | Result | |---|---| | read-only check in `IsOwnedHistorizedValue` | killed (3) | | `HistorianReadOnly` in `RawTagPlan.Equals` | killed by the new planner test (2) | | composer copies the flag | killed (artifact parity) | | CreateOnly filter | killed (3) | | CreateOnly fail-closed on a lookup error | killed (1) | | policy reaches the provisioner | killed (2) | | `Off` keeps the Null provisioner | killed (2) | | CreateOnly is Gateway-only (options validator) | killed (2) | | `HistorianReadOnlyInvalid` rule | killed (3) | | CSV export column | killed (1) | | AdminUI read uses the runtime parse | killed (1) | **Verification.** - Release build: 0 warnings, 0 errors. - Green: OpcUaServer.Tests 1290, Commons 377, Configuration 285, Core.Abstractions 801, AdminUI 2170, Historian.Gateway 593, Historian.SqlServer 45, Historian.QuestDb 88, and the Host historian-options validator 37. - Runtime.Tests: 1040 passed. One unrelated timing test (`DriverHostActorSourceRemovedTests`) failed under load and passes when run alone (5/5). Landed on master as `6c4d2990`. No live run: `EnsureTags` is a write, so the provisioning paths were verified only against fakes and recording doubles. Nothing was deployed.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
auto-managed
queue management
 blocked
Blocked by dependency or external hold
 cross-driver
queue management
 cross-repo
Needs coordinated change in another family repo (wire contract etc.)
 driver/abcip
queue management
 driver/ablegacy
queue management
 driver/focas
queue management
 driver/mqtt
MQTT / Sparkplug B broker-subscribe driver
 driver/opcuaclient
queue management
 driver/s7
queue management
 driver/twincat
queue management
 migrated
Migrated from a markdown backlog (2026-11 standardization)
 phase/1
queue management
 phase/2
queue management
 phase/3
queue management
 phase/4
queue management
 phase/5
queue management
 phase/6
queue management
 priority:p0
Must-have / blocking
 priority:p1
High priority
 priority:p2
Normal priority
 priority:p3
Low priority / nice to have
 queue/blocked
queue management
 queue/done
queue management
 queue/failed
queue management
 queue/in-progress
queue management
 queue/queued
queue management
 type:bug
Defect in shipped behaviour
 type:debt
Technical debt / hygiene
 type:deploy-gate
Rollout/deploy/migration step not yet done on an environment
 type:docs
Documentation work
 type:feature
Feature implementation
 type:followup
Follow-up deferred from a plan, PR or gate
 type:infra
Infrastructure and scaffolding
 type:review-finding
Unfixed finding from an architecture/code review
 type:test
Test work
No Label priority:p2 type:bug
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
dohertj2 (dohertj2)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: dohertj2/lmxopcua#899
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?