dohertj2/lmxopcua
1
0
Fork 0
Code Issues 5 Pull Requests Actions Packages Projects Releases Wiki Activity

Map HistorianGateway AGGREGATE_NOT_SUPPORTED_FOR_TAG (FailedPrecondition) to BadAggregateNotSupported without counting health #900

New Issue
Closed
opened 2026-11-13 00:59:16 -05:00 by dohertj2 · 5 comments
dohertj2 commented 2026-11-13 00:59:16 -05:00
Owner
Copy Link
Copy Source

Follow-up to #897 (fixed up front for Boolean nodes in ee39c169) and the gateway side historiangw#135 (merged to historiangw main @ a220ec0).

What the gateway now sends. When AVEVA refuses an aggregate retrieval mode for the tag (AVEVA 2023 R2: TimeWeightedAverage / MinimumWithTime / MaximumWithTime / Integral / Slope on a discrete or string tag; native CustomError 85 at StartQuery), ReadAggregate fails with:

  • gRPC FailedPrecondition, Status.Detail = HistorianAggregateNotSupportedException;
  • a google.rpc.ErrorInfo in grpc-status-details-bin: reason AGGREGATE_NOT_SUPPORTED_FOR_TAG, domain historian_gateway.v1, metadata retrieval_mode (e.g. RETRIEVAL_MODE_TIME_WEIGHTED_AVERAGE) and native_error_code (85).

A genuine historian fault stays Internal with no ErrorInfo. Live, read-only on wonder-sql-vd03 (2026-11-13): SysPulse TWA/Integral gave the new status; SysTimeSec TWA/Integral and SysPulse Cyclic were still served.

Ask (OtOpcUa). In GatewayHistorianDataSource (native processed read, ReadProcessed… catch around RecordReadFailure(ex)), recognise that refusal and:

  1. answer the node BadAggregateNotSupported (Part 11 §6.5.4), not BadHistoryOperationUnsupported;
  2. do not call RecordReadFailure / count it in the historian health snapshot (it is the tag's configuration, not a backend failure) and do not treat it as transport.

This covers the residual #897 case: a node declared numeric whose AVEVA tag is actually discrete still reaches AVEVA.

How to detect it.

  • With Client 1.1.0 (planned; unpublished — the owner publishes Contracts + Client 1.1.0): HistorianGatewayException.ErrorReason == HistorianGatewayErrorReasons.AggregateNotSupportedForTag (also ErrorDomain, ErrorMetadata; contract enum HistorianErrorReason).
  • Without the bump (Client 1.0.0 pin): ex.StatusCode == FailedPrecondition and the inner RpcException's GetRpcStatus()?.GetDetail<Google.Rpc.ErrorInfo>()?.Reason == "AGGREGATE_NOT_SUPPORTED_FOR_TAG" (needs Grpc.StatusProto). Match on the reason, not on FailedPrecondition alone: the gateway also uses FailedPrecondition for other things (live writes disabled, a historian-side FailedPrecondition passed through).

Compatibility. An older gateway still sends Internal for this case, so the existing handling must stay. Don't bump the ZB.MOM.WW.HistorianGateway.Client pin until 1.1.0 is on the feed.

Tests to add. Extend DiscreteTagTimeWeightedWireTests (AVEVA-shaped fake gateway): a Double/Int node whose fake gateway answers FailedPrecondition + ErrorInfo → BadAggregateNotSupported, 0 health failures. Positive control: an Internal answer still counts as a failure. Re-run the AVEVA live gate against a gateway at or after a220ec0.

Follow-up to #897 (fixed up front for Boolean nodes in `ee39c169`) and the gateway side **historiangw#135** (merged to historiangw `main` @ `a220ec0`). **What the gateway now sends.** When AVEVA refuses an aggregate retrieval mode for the tag (AVEVA 2023 R2: TimeWeightedAverage / MinimumWithTime / MaximumWithTime / Integral / Slope on a discrete or string tag; native CustomError 85 at `StartQuery`), `ReadAggregate` fails with: - gRPC **`FailedPrecondition`**, `Status.Detail` = `HistorianAggregateNotSupportedException`; - a `google.rpc.ErrorInfo` in `grpc-status-details-bin`: reason **`AGGREGATE_NOT_SUPPORTED_FOR_TAG`**, domain `historian_gateway.v1`, metadata `retrieval_mode` (e.g. `RETRIEVAL_MODE_TIME_WEIGHTED_AVERAGE`) and `native_error_code` (`85`). A genuine historian fault stays `Internal` with no ErrorInfo. Live, read-only on wonder-sql-vd03 (2026-11-13): `SysPulse` TWA/Integral gave the new status; `SysTimeSec` TWA/Integral and `SysPulse` Cyclic were still served. **Ask (OtOpcUa).** In `GatewayHistorianDataSource` (native processed read, `ReadProcessed…` catch around `RecordReadFailure(ex)`), recognise that refusal and: 1. answer the node **`BadAggregateNotSupported`** (Part 11 §6.5.4), not `BadHistoryOperationUnsupported`; 2. **do not** call `RecordReadFailure` / count it in the historian health snapshot (it is the tag's configuration, not a backend failure) and do not treat it as transport. This covers the residual #897 case: a node declared numeric whose AVEVA tag is actually discrete still reaches AVEVA. **How to detect it.** - With Client **1.1.0** (planned; **unpublished** — the owner publishes Contracts + Client 1.1.0): `HistorianGatewayException.ErrorReason == HistorianGatewayErrorReasons.AggregateNotSupportedForTag` (also `ErrorDomain`, `ErrorMetadata`; contract enum `HistorianErrorReason`). - Without the bump (Client 1.0.0 pin): `ex.StatusCode == FailedPrecondition` and the inner `RpcException`'s `GetRpcStatus()?.GetDetail<Google.Rpc.ErrorInfo>()?.Reason == "AGGREGATE_NOT_SUPPORTED_FOR_TAG"` (needs `Grpc.StatusProto`). Match on the reason, not on FailedPrecondition alone: the gateway also uses FailedPrecondition for other things (live writes disabled, a historian-side FailedPrecondition passed through). **Compatibility.** An older gateway still sends `Internal` for this case, so the existing handling must stay. Don't bump the `ZB.MOM.WW.HistorianGateway.Client` pin until 1.1.0 is on the feed. **Tests to add.** Extend `DiscreteTagTimeWeightedWireTests` (AVEVA-shaped fake gateway): a Double/Int node whose fake gateway answers FailedPrecondition + ErrorInfo → `BadAggregateNotSupported`, 0 health failures. Positive control: an `Internal` answer still counts as a failure. Re-run the AVEVA live gate against a gateway at or after `a220ec0`.
dohertj2 added the type:followuppriority:p2cross-repo labels 2026-11-13 00:59:16 -05:00
dohertj2 commented 2026-11-13 01:32:15 -05:00
Author
Owner
Copy Link
Copy Source

Fixed in efaf3435 on master.

What changed. If the gateway refuses the aggregate with FailedPrecondition and the google.rpc.ErrorInfo reason AGGREGATE_NOT_SUPPORTED_FOR_TAG (domain historian_gateway.v1), the native processed read in GatewayHistorianDataSource (ReadNativeAsync) now throws HistoryAggregateNotSupportedException. The node gets BadAggregateNotSupported. RecordReadFailure is not called and the read is not treated as transport. Nothing else changes: any other FailedPrecondition, and every Internal (which is what an older gateway sends for this case), still gets BadHistoryOperationUnsupported and counts as a failure.

Detection (one place: GatewayErrorReasons). The Client pin stays at 1.0.0. Client 1.0.0's MapRpcException keeps the gateway's RpcException, trailers included, as the InnerException of the HistorianGatewayException it throws. So ReasonOf walks the exception chain and reads GetRpcStatus()?.GetDetail<Google.Rpc.ErrorInfo>(). This needs Grpc.StatusProto 2.76.0, added through central package management. It comes from nuget.org, matches the Grpc.Core.Api version, and pulls in Google.Api.CommonProtos 2.16.0, which accepts Google.Protobuf 3.34.1. A malformed trailer or a different domain counts as no reason. When Client 1.1.0 is on the feed, ReasonOf becomes (ex as HistorianGatewayException)?.ErrorReason (one line) and the StatusProto reference can be removed. The code comment says the same.

Tests.

  • GatewayAggregateNotSupportedWireTests (new): an in-process gRPC gateway sends the real status plus the ErrorInfo shape (copied from historiangw HistorianRpcFaults.ToRpcException). The test reads through the published Client 1.0.0 via the production adapter. TimeAverage and Total give BadAggregateNotSupported and 0 health failures. These controls are rethrown, are not IOException, and count as failures: Internal; FailedPrecondition with no ErrorInfo; another reason; another domain; the right reason on Internal. A positive control is served.
  • DiscreteTagTimeWeightedWireTests: over a real SDK session, a Double node and an Int32 node on the discrete tag give BadAggregateNotSupported and 0 failures. Control: the older gateway's Internal answer keeps its status and counts.
  • AvevaGatewayAggregateRefusalLiveGateTests: an opt-in, read-only live gate (HISTGW_GATEWAY_ENDPOINT/_APIKEY + OTOPCUA_AVEVA_GATE_900=true).
  • Mutation checks: removing the catch, ignoring the status, ignoring the domain, ignoring the reason, or removing the inner-exception walk each fail at least one test.
  • Release build has 0 warnings. Historian.Gateway.Tests: 601 passed, 6 skipped. OpcUaServer.IntegrationTests: 323 passed, 38 skipped.

Live gate, read-only (2026-11-13). I ran a private HistorianGateway container built from a220ec0 (API-key auth disabled, live writes off, removed afterwards) against wonder-sql-vd03:

  • Double and Int32 nodes on SysPulse, TimeAverage and Total: BadAggregateNotSupported, health failures unchanged at 0.
  • SysTimeSec TimeAverage: Good, 2 rows.
  • SysPulse Average: served.

The gateway saw only ReadAggregate, ReadRaw and GetTagMetadata. Nothing was written and no tags were created. The rig builds no provisioner or recorder. One unplanned negative control: a stale build with the catch disabled answered the same live read BadHistoryOperationUnsupported.

docs/Historian.md (Aggregates by node DataType) is updated. Deployment: vd03 and QA need both this OtOpcUa build and a HistorianGateway at or after a220ec0 before the behaviour shows. Against today's deployed gateways the answer is unchanged.

Fixed in `efaf3435` on `master`. **What changed.** If the gateway refuses the aggregate with FailedPrecondition and the `google.rpc.ErrorInfo` reason `AGGREGATE_NOT_SUPPORTED_FOR_TAG` (domain `historian_gateway.v1`), the native processed read in `GatewayHistorianDataSource` (`ReadNativeAsync`) now throws `HistoryAggregateNotSupportedException`. The node gets **`BadAggregateNotSupported`**. `RecordReadFailure` is not called and the read is not treated as transport. Nothing else changes: any other FailedPrecondition, and every `Internal` (which is what an older gateway sends for this case), still gets `BadHistoryOperationUnsupported` and counts as a failure. **Detection (one place: `GatewayErrorReasons`).** The Client pin stays at **1.0.0**. Client 1.0.0's `MapRpcException` keeps the gateway's `RpcException`, trailers included, as the `InnerException` of the `HistorianGatewayException` it throws. So `ReasonOf` walks the exception chain and reads `GetRpcStatus()?.GetDetail<Google.Rpc.ErrorInfo>()`. This needs **`Grpc.StatusProto` 2.76.0**, added through central package management. It comes from nuget.org, matches the Grpc.Core.Api version, and pulls in Google.Api.CommonProtos 2.16.0, which accepts Google.Protobuf 3.34.1. A malformed trailer or a different domain counts as no reason. **When Client 1.1.0 is on the feed**, `ReasonOf` becomes `(ex as HistorianGatewayException)?.ErrorReason` (one line) and the StatusProto reference can be removed. The code comment says the same. **Tests.** - `GatewayAggregateNotSupportedWireTests` (new): an in-process gRPC gateway sends the real status plus the ErrorInfo shape (copied from historiangw `HistorianRpcFaults.ToRpcException`). The test reads through the **published Client 1.0.0** via the production adapter. TimeAverage and Total give `BadAggregateNotSupported` and 0 health failures. These controls are rethrown, are not IOException, and count as failures: Internal; FailedPrecondition with no ErrorInfo; another reason; another domain; the right reason on Internal. A positive control is served. - `DiscreteTagTimeWeightedWireTests`: over a real SDK session, a Double node and an Int32 node on the discrete tag give `BadAggregateNotSupported` and 0 failures. Control: the older gateway's `Internal` answer keeps its status and counts. - `AvevaGatewayAggregateRefusalLiveGateTests`: an opt-in, read-only live gate (`HISTGW_GATEWAY_ENDPOINT`/`_APIKEY` + `OTOPCUA_AVEVA_GATE_900=true`). - Mutation checks: removing the catch, ignoring the status, ignoring the domain, ignoring the reason, or removing the inner-exception walk each fail at least one test. - Release build has 0 warnings. Historian.Gateway.Tests: 601 passed, 6 skipped. OpcUaServer.IntegrationTests: 323 passed, 38 skipped. **Live gate, read-only (2026-11-13).** I ran a private HistorianGateway container built from `a220ec0` (API-key auth disabled, live writes off, removed afterwards) against wonder-sql-vd03: - Double and Int32 nodes on `SysPulse`, TimeAverage and Total: **BadAggregateNotSupported**, health failures unchanged at 0. - `SysTimeSec` TimeAverage: Good, 2 rows. - `SysPulse` Average: served. The gateway saw only `ReadAggregate`, `ReadRaw` and `GetTagMetadata`. Nothing was written and no tags were created. The rig builds no provisioner or recorder. One unplanned negative control: a stale build with the catch disabled answered the same live read `BadHistoryOperationUnsupported`. `docs/Historian.md` (Aggregates by node DataType) is updated. **Deployment:** vd03 and QA need both this OtOpcUa build and a HistorianGateway at or after `a220ec0` before the behaviour shows. Against today's deployed gateways the answer is unchanged.
dohertj2 commented 2026-11-13 01:36:25 -05:00
Author
Owner
Copy Link
Copy Source

Contracts + Client 1.1.0 published to the Gitea feed on 2026-11-13: release commit 6365f34 (tag v1.1.0) on historiangw main. Verified by restoring from the feed into a throwaway project: the SHA256 matches the packed artifacts and HistorianGatewayErrorReasons.AggregateNotSupportedForTag compiles. Client 1.1.0 depends on Grpc.StatusProto 2.76.0. OtOpcUa still pins 1.0.0; switching to 1.1.0 is the one-line ReasonOf change noted in lmxopcua#900.

**Contracts + Client 1.1.0 published** to the Gitea feed on 2026-11-13: release commit `6365f34` (tag `v1.1.0`) on historiangw main. Verified by restoring from the feed into a throwaway project: the SHA256 matches the packed artifacts and `HistorianGatewayErrorReasons.AggregateNotSupportedForTag` compiles. Client 1.1.0 depends on `Grpc.StatusProto` 2.76.0. OtOpcUa still pins 1.0.0; switching to 1.1.0 is the one-line `ReasonOf` change noted in lmxopcua#900.
dohertj2 commented 2026-11-13 03:37:16 -05:00
Author
Owner
Copy Link
Copy Source

The HistorianGateway side of this is now deployed on DEV and QA (2026-11-13, historiangw 6365f34, historiangw#135). It runs on vd03, vq04 and vq03, so c9640f5c's #900 mapping is now reachable on both tiers.

On all three boxes, ReadAggregate TWA on the discrete SysPulse, called with OtOpcUa's key, now returns FailedPrecondition with ErrorInfo reason AGGREGATE_NOT_SUPPORTED_FOR_TAG (domain historian_gateway.v1) in grpc-status-details-bin. The previous builds returned Internal. OtOpcUa's historian probe kept running through each ~6.5 s swap, and :9000/health/ready stayed 200.

Record: historiangw docs/deploy/2026-11-13-vd03-qa-6365f34-record.md.

The HistorianGateway side of this is now deployed on DEV and QA (2026-11-13, historiangw `6365f34`, historiangw#135). It runs on vd03, vq04 and vq03, so `c9640f5c`'s #900 mapping is now reachable on both tiers. On all three boxes, `ReadAggregate` TWA on the discrete `SysPulse`, called with OtOpcUa's key, now returns `FailedPrecondition` with ErrorInfo reason `AGGREGATE_NOT_SUPPORTED_FOR_TAG` (domain `historian_gateway.v1`) in `grpc-status-details-bin`. The previous builds returned `Internal`. OtOpcUa's historian probe kept running through each ~6.5 s swap, and `:9000/health/ready` stayed 200. Record: historiangw `docs/deploy/2026-11-13-vd03-qa-6365f34-record.md`.
dohertj2 commented 2026-11-13 07:17:58 -05:00
Author
Owner
Copy Link
Copy Source

Moved to HistorianGateway Client/Contracts 1.1.0 on master 76e73e7d (historiangw#135, release 6365f34).

  • GatewayErrorReasons now reads Client 1.1.0's typed HistorianGatewayException.StatusCode / ErrorReason / ErrorDomain (constants from HistorianGatewayErrorReasons). It no longer parses grpc-status-details-bin itself, and the direct Grpc.StatusProto pin and reference are gone (the client brings it in). There is no trailer fallback: the published client maps every RpcException to a HistorianGatewayException.
  • Behaviour is unchanged: FailedPrecondition + AGGREGATE_NOT_SUPPORTED_FOR_TAG (domain historian_gateway.v1) gives BadAggregateNotSupported, and every other shape keeps its handling.
  • New GatewayErrorReasonsTests (11), whose exceptions carry the reason only on the typed properties. The wire tests and DiscreteTagTimeWeightedWireTests pass through the published 1.1.0 client. Four mutations were each caught: status check dropped, domain check dropped, never-refusal, and a revert to the old trailer implementation.
  • Release build 0 warnings. Gateway.Tests 611/0/6, Core.AlarmHistorian.Tests 53/0, Sql.Common.Tests 300/0, Runtime.Tests 1063/0/3, OpcUaServer.Tests 490/0, OpcUaServer.IntegrationTests (aggregate wire subset) 10/0/1.

Not deployed. It ships with the next deploy after 47f0b98c.

**Moved to HistorianGateway Client/Contracts 1.1.0** on master `76e73e7d` (historiangw#135, release `6365f34`). - `GatewayErrorReasons` now reads Client 1.1.0's typed `HistorianGatewayException.StatusCode` / `ErrorReason` / `ErrorDomain` (constants from `HistorianGatewayErrorReasons`). It no longer parses `grpc-status-details-bin` itself, and the direct `Grpc.StatusProto` pin and reference are gone (the client brings it in). There is no trailer fallback: the published client maps every `RpcException` to a `HistorianGatewayException`. - Behaviour is unchanged: `FailedPrecondition` + `AGGREGATE_NOT_SUPPORTED_FOR_TAG` (domain `historian_gateway.v1`) gives `BadAggregateNotSupported`, and every other shape keeps its handling. - New `GatewayErrorReasonsTests` (11), whose exceptions carry the reason only on the typed properties. The wire tests and `DiscreteTagTimeWeightedWireTests` pass through the published 1.1.0 client. Four mutations were each caught: status check dropped, domain check dropped, never-refusal, and a revert to the old trailer implementation. - Release build 0 warnings. Gateway.Tests 611/0/6, Core.AlarmHistorian.Tests 53/0, Sql.Common.Tests 300/0, Runtime.Tests 1063/0/3, OpcUaServer.Tests 490/0, OpcUaServer.IntegrationTests (aggregate wire subset) 10/0/1. Not deployed. It ships with the next deploy after `47f0b98c`.
dohertj2 commented 2026-11-13 07:53:52 -05:00
Author
Owner
Copy Link
Copy Source

Deployed to DEV vd03 and the QA pair vq03/vq04 on 2026-11-13. The build is master 76e73e7d: HistorianGateway Client/Contracts 1.1.0 and the typed HistorianGatewayException reason. Record: docs/deploy/2026-11-13-vd03-qa-to-76e73e7d.md (5baf05f6).

  • Packages: live deps.json on all three nodes shows ZB.MOM.WW.HistorianGateway.Client/1.1.0. HG on both tiers is 6365f34 (the 1.1.0 server).
  • Added gate (vd03): the historian path still answers. On HG :5220/metrics, HistorianStatus/Probe ok went 494 (12:27:17Z) → 504 (12:31:52Z) → 536 (12:47:45Z). The boot log has Historian gateway probe: the gateway is reachable. The other counters did not move: BrowseTagNames 1, ReadRaw 1, ReadAggregate ok 1 / error 2. There is no EnsureTags, AddTagExtendedProperties, WriteLiveValues, AddHistoricalValues or SendEvent(s) series.
  • QA: vq03 HG Probe went 988 → 1028 with nothing else changing, and both nodes log the probe as reachable on every start.
  • Not exercised live: the AGGREGATE_NOT_SUPPORTED_FOR_TAG → BadAggregateNotSupported mapping. Neither tier has historized tags. The mapping is covered by GatewayErrorReasonsTests and the wire tests.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi

**Deployed** to DEV vd03 and the QA pair vq03/vq04 on 2026-11-13. The build is master `76e73e7d`: HistorianGateway Client/Contracts **1.1.0** and the typed `HistorianGatewayException` reason. Record: `docs/deploy/2026-11-13-vd03-qa-to-76e73e7d.md` (`5baf05f6`). - **Packages:** live `deps.json` on all three nodes shows `ZB.MOM.WW.HistorianGateway.Client/1.1.0`. HG on both tiers is `6365f34` (the 1.1.0 server). - **Added gate (vd03): the historian path still answers.** On HG `:5220/metrics`, `HistorianStatus/Probe` ok went 494 (12:27:17Z) → 504 (12:31:52Z) → 536 (12:47:45Z). The boot log has `Historian gateway probe: the gateway is reachable.` The other counters did not move: BrowseTagNames 1, ReadRaw 1, ReadAggregate ok 1 / error 2. **There is no `EnsureTags`, `AddTagExtendedProperties`, `WriteLiveValues`, `AddHistoricalValues` or `SendEvent(s)` series.** - **QA:** vq03 HG Probe went 988 → 1028 with nothing else changing, and both nodes log the probe as reachable on every start. - **Not exercised live:** the `AGGREGATE_NOT_SUPPORTED_FOR_TAG` → `BadAggregateNotSupported` mapping. Neither tier has historized tags. The mapping is covered by `GatewayErrorReasonsTests` and the wire tests. Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
auto-managed
queue management
 blocked
Blocked by dependency or external hold
 cross-driver
queue management
 cross-repo
Needs coordinated change in another family repo (wire contract etc.)
 driver/abcip
queue management
 driver/ablegacy
queue management
 driver/focas
queue management
 driver/mqtt
MQTT / Sparkplug B broker-subscribe driver
 driver/opcuaclient
queue management
 driver/s7
queue management
 driver/twincat
queue management
 migrated
Migrated from a markdown backlog (2026-11 standardization)
 phase/1
queue management
 phase/2
queue management
 phase/3
queue management
 phase/4
queue management
 phase/5
queue management
 phase/6
queue management
 priority:p0
Must-have / blocking
 priority:p1
High priority
 priority:p2
Normal priority
 priority:p3
Low priority / nice to have
 queue/blocked
queue management
 queue/done
queue management
 queue/failed
queue management
 queue/in-progress
queue management
 queue/queued
queue management
 type:bug
Defect in shipped behaviour
 type:debt
Technical debt / hygiene
 type:deploy-gate
Rollout/deploy/migration step not yet done on an environment
 type:docs
Documentation work
 type:feature
Feature implementation
 type:followup
Follow-up deferred from a plan, PR or gate
 type:infra
Infrastructure and scaffolding
 type:review-finding
Unfixed finding from an architecture/code review
 type:test
Test work
No Label cross-repo priority:p2 type:followup
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
dohertj2 (dohertj2)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: dohertj2/lmxopcua#900
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?