diff --git a/docs/Historian.md b/docs/Historian.md index ced29a8ca..41315775b 100644 --- a/docs/Historian.md +++ b/docs/Historian.md @@ -200,7 +200,7 @@ Processed read is `BadAggregateNotSupported` and no history aggregate functions | Key | Type | Default | Description | |---|---|---|---| -| `Gateway:EnableNativeTimeWeighted` | bool | `true` | Serve HistoryRead-Processed `TimeAverage` / `Total` through AVEVA's native retrieval modes. ⚠️ **On by default since #694 (client-visible: both are advertised and served again).** The gateway interval fix historiangw#27 (gateway server `9f711c1`+) is live-verified: forward and reverse per-interval values, clipped, sub-millisecond clipped and fractional-millisecond intervals against a step-held reference, and all-NULL intervals ⇒ `BadNoData` ([evidence](plans/evidence/2026-11-12-historian-694-700-746/aveva-gate.md)). `false` is the kill switch for a gateway older than the fix (it answers one Good-status whole-span value on the earliest interval): both are then `BadAggregateNotSupported` and **not advertised**. On a discrete (Boolean) AVEVA tag AVEVA refuses both modes and the node answers `BadHistoryOperationUnsupported` (counted as a backend failure). | +| `Gateway:EnableNativeTimeWeighted` | bool | `true` | Serve HistoryRead-Processed `TimeAverage` / `Total` through AVEVA's native retrieval modes. ⚠️ **On by default since #694 (client-visible: both are advertised and served again).** The gateway interval fix historiangw#27 (gateway server `9f711c1`+) is live-verified: forward and reverse per-interval values, clipped, sub-millisecond clipped and fractional-millisecond intervals against a step-held reference, and all-NULL intervals ⇒ `BadNoData` ([evidence](plans/evidence/2026-11-12-historian-694-700-746/aveva-gate.md)). `false` is the kill switch for a gateway older than the fix (it answers one Good-status whole-span value on the earliest interval): both are then `BadAggregateNotSupported` and **not advertised**. On a discrete (Boolean) AVEVA tag AVEVA refuses both modes and the node answers `BadHistoryOperationUnsupported` (counted as a backend failure; #897). | | `Gateway:ProbeInterval` | TimeSpan | `00:00:30` | How often the read client calls the gateway's `ProbeAsync` for the historian health snapshot (`ProcessConnectionOpen` / `EventConnectionOpen`, shown on `/hosts`). The first probe runs at start-up, so a node that has served no HistoryRead yet — or only writes (continuous historization) — reports a healthy gateway as open; it used to read closed until some client happened to read. The latest observation (probe or read outcome) wins; probes never count as queries or failures. Zero/negative = no probe (read outcomes only, the previous behaviour). | | `Gateway:EventDrainBudget` | int | `10000` | Since #746 a capped HistoryReadEvents read sends cap + 1 as the server-side `max_events` and the gateway completes the stream there, so this budget applies only to the uncapped fallback read (an ambiguous server-capped answer — foreign rows filtered out of a stream the server cut — or a gateway refusing the cap) and to a stream that runs past the `max_events` it was sent. It is the number of stream rows such a read consumes (and discards) past its page cap before it abandons the gateway stream; an **abandoned** stream cancels the RPC and the gateway evicts its pooled historian session (the next read reconnects; concurrent reads on that session fail). The usual small overshoot is drained to the stream's end; past the budget the stream is abandoned, so a page over a window far larger than the page does not stream the whole window. Rows of another source never count before the "more events" signal is seen, so an abandoned stream never answers a page as complete. `0` = abandon at the cap (the previous behaviour); negative counts as `0` (a `Validate()` warning). | diff --git a/docs/plans/evidence/2026-11-12-historian-694-700-746/aveva-gate-801-4c8e977e.json.gz b/docs/plans/evidence/2026-11-12-historian-694-700-746/aveva-gate-801-4c8e977e.json.gz new file mode 100644 index 000000000..07663406d Binary files /dev/null and b/docs/plans/evidence/2026-11-12-historian-694-700-746/aveva-gate-801-4c8e977e.json.gz differ diff --git a/docs/plans/evidence/2026-11-12-historian-694-700-746/aveva-gate-followups-4c8e977e.json.gz b/docs/plans/evidence/2026-11-12-historian-694-700-746/aveva-gate-followups-4c8e977e.json.gz new file mode 100644 index 000000000..b9ad8f89c Binary files /dev/null and b/docs/plans/evidence/2026-11-12-historian-694-700-746/aveva-gate-followups-4c8e977e.json.gz differ diff --git a/docs/plans/evidence/2026-11-12-historian-694-700-746/aveva-gate-followups-mutation-adapter-no-max-events.json.gz b/docs/plans/evidence/2026-11-12-historian-694-700-746/aveva-gate-followups-mutation-adapter-no-max-events.json.gz new file mode 100644 index 000000000..891f69e2c Binary files /dev/null and b/docs/plans/evidence/2026-11-12-historian-694-700-746/aveva-gate-followups-mutation-adapter-no-max-events.json.gz differ diff --git a/docs/plans/evidence/2026-11-12-historian-694-700-746/aveva-gate-strict-4c8e977e.json.gz b/docs/plans/evidence/2026-11-12-historian-694-700-746/aveva-gate-strict-4c8e977e.json.gz new file mode 100644 index 000000000..d831c47d6 Binary files /dev/null and b/docs/plans/evidence/2026-11-12-historian-694-700-746/aveva-gate-strict-4c8e977e.json.gz differ diff --git a/docs/plans/evidence/2026-11-12-historian-694-700-746/aveva-gate.md b/docs/plans/evidence/2026-11-12-historian-694-700-746/aveva-gate.md new file mode 100644 index 000000000..aac838599 --- /dev/null +++ b/docs/plans/evidence/2026-11-12-historian-694-700-746/aveva-gate.md @@ -0,0 +1,114 @@ +# #694 / #700 / #746: real AVEVA history through a HistorianGateway (2026-11-12) + +**Scope.** The live gates of [#694](https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/694) item 3 (native TimeAverage / +Total after historiangw#27), [#700](https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/700) (the deployed-endpoint +acceptance list) and [#746](https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/746) item 1 (events paging, TimeAverage), +plus the code from `4c8e977e` that they gate (native pair on by default, the whole-millisecond grid, `SendEvents` batches, +server-side `max_events`). + +**Result.** Every case passed. Native TimeAverage / Total are correct in both directions, including clipped, sub-millisecond +clipped and fractional-millisecond intervals, and all-NULL intervals are `BadNoData`. The gate also found two defects in the +native path that unit fakes had hidden; both are fixed in `4c8e977e` (see [Findings](#findings)). + +## Setup + +| | | +|---|---| +| OtOpcUa | `4c8e977e` (this change on `origin/master` `00a5be26`), OPC UA SDK server **2.0.1.0**, HistorianGateway Client/Contracts **1.0.0**. It was pushed as **`53978ca6`** after a rebase onto `09b373e4`; the commits in between (#615, #614, #755, #829, #893, #895, #896) change no historian code. | +| How | `AvevaGatewayHistorianFollowupsLiveGateTests` and `AvevaGatewayHistoryLiveGateTests` (#801) in `OpcUaServer.IntegrationTests`. Both boot the OtOpcUa server in-process on the Mac, wire the **production** Gateway backend (`HistorianBackendRegistry` → `GatewayHistorianDataSource`, default `ServerHistorianOptions`), and read over a real SDK client session (`None`, anonymous, ACL off). This is the server code path a deployed node runs, but in the test process, not a Host service. | +| Gateway | A local HistorianGateway published from `main` **`dffb5e2`** (server code = the deployed DEV build `ea9f21c` plus the Telemetry 0.3.x bump only). It runs in Docker on the Mac (`zb-historiangw:h694-dffb5e2`, container `zb-historiangw-h694`, `:5233` h2c) with `ApiKeys__Mode=Disabled`, `Historian__LiveWritesEnabled=false`, store-forward and redundancy off. Over the VPN it talks gRPC/TLS to the AVEVA **2023 R2** historian on wonder-sql-vd03, with the historian account from the existing gitignored `histgw.secrets.env` (never printed). No deployed service was touched, and nothing ran on a wonder box. | +| Strict fixture | A second local container (`zb-historiangw-h694-strict`, `:5237`) of the same image with `Historian__ReadLimits__AllowStreamingBeyondCeiling=false`. The test restarts it with `MaxValuesCeiling` 1,000 → 500 → 5,000 (`OTOPCUA_AVEVA_GATE_STRICT_SET_CEILING`, a `docker rm -f` + `docker run` script). | +| Writes | **None.** Only HistoryRead (raw, processed, at-time, events) was sent. No tag was created, no value or event was written, `Z28061` was not used. The gateway had live writes disabled. | +| Tags | `SysTimeSec` (UInt2, 1 Hz 0..59, integrated by AVEVA as a step), `SysPulse` (discrete, toggles each minute), `ZIPSEN.WTC1` (Float, ~2 s). It is stored NULL at 2026-11-10 11:32:36.073 (platform stop), with nothing until 11:38:34.853. Events: `DevWinPlatform`, 2026-11-05 15:00 → 2026-11-12 00:00 (72 events). | +| Runs | Follow-ups gate 2026-11-13 04:58:13Z, strict fixture 05:01:32Z, #801 gate straight after. All three passed with every in-test check green. | + +## #694 item 3: native TimeAverage / Total + +`SysTimeSec` window 2026-11-11 14:00:00Z. Every bucket of every case is checked three ways: wire = backend (value, status, +stamp); the bucket count against the Part 13 grid; the value against a **step-held integral of the tag's own raw rows** +computed in the test (independent of AVEVA's aggregate). Each of the 18 cases below is run for both TimeAverage and Total. + +| Case | Window / interval | Result | +|---|---|---| +| forward, whole | 14:00:00 → 14:02:00, 20 s | 6 buckets: TimeAverage 9.5 / 29.5 / 49.5 ×2, Total 190 / 590 / 990 ×2. They equal the step reference, are stamped with the earlier edge and carry `0x00000401`. | +| reverse, whole | 14:02:00 → 14:00:00 | Newest first, stamped with the later edge, values = the forward ones reversed | +| forward, clipped 7 s | 14:00:00 → 14:02:07 | Last interval [14:02:00, 14:02:07) = 3 / 21, status `0x00000405` (Partial) | +| reverse, clipped 7 s | 14:02:07 → 14:00:00 | Grid steps back from 14:02:07. The earliest interval (14:00:00, 14:00:07] = 3 / 21 Partial; the others are 35.5 / 36.5 / 16.5 (710 / 730 / 330). | +| forward, **sub-ms clipped** | 14:00:00 → 14:00:40.0004 | Last interval [40, 40.0004) = **40 / 0.016** (= 40 × 0.0004 s), Partial | +| reverse, **sub-ms clipped** | 14:00:43.0004 → 14:00:03 | 32.5004 / 12.5004, then the 0.4 ms interval (3, 3.0004] = **3 / 0.0012**, Partial | +| forward, non-whole-ms clipped | 14:00:00 → 14:00:43.0004 | Last interval [40, 43.0004) = 41.000266… / **123.0172** (see Finding 2) | +| forward / reverse, **fractional ms** | 14:00:03.997 ↔ 14:00:04.003, 1.5 ms | 4 buckets 3, 3, 4, 4 (Total 0.0045, 0.0045, 0.006, 0.006), on the 1.5 ms grid across the 14:00:04 step (see Finding 1) | +| **all-NULL** (ZIPSEN.WTC1) | 11:31 → 11:40, 60 s, both directions | 11:33–11:37: `BadNoData \| Calculated` (`0x809B0401`), null. 11:32 and 11:38 (the NULL plus some good time): Uncertain `0x40A40401` with AVEVA's value. The rest are Good. The reverse read gives the same intervals, newest first. | + +`HistoryServerCapabilities/AggregateFunctions` advertises the twelve, TimeAverage and Total included. The #801 gate +(244 cases) now serves TimeAverage / Total on its own windows and tags (wire = backend) and still refuses the other five +time-weighted aggregates. + +## #700: deployed-endpoint acceptance + +| Check | Result | +|---|---| +| Missing tag (`OtOpcUa.Gate694.DoesNotExist`) | Raw `GoodNoData` 0 rows. Average and TimeAverage: 3 buckets, each `BadNoData` (`0x809B0401`), null. AtTime: 2 × `BadNoData`. | +| Existing tag Raw / AtTime / Processed | Int32 node (`SysTimeSec`): 10 raw rows 0..9, AtTime stored 0 and midpoint 1 (`0x00000402` Interpolated), Average and TimeAverage 2 / 7. Boolean node (`SysPulse`): false/true/… each minute, Average 0.333 / 0.5. Float node (`ZIPSEN.WTC1`): raw, AtTime, Average and TimeAverage. Every case wire = backend. | +| **DataType matching** | Raw and AtTime values arrive as the node's DataType: **Int32**, **Boolean** and **Single** (Float). Processed values are Double on every node. | +| Future AtTime | now + 30 min / + 1 h / + 2 h: the requested timestamps come back in request order, as Uncertain (`0x40A40402`) holds of the last value. 505 ms. | +| Transport outage (backend pointed at a closed port) | Raw, Processed Average, Processed TimeAverage, AtTime and Events: `BadNoCommunication`. Modified: `BadHistoryOperationUnsupported`. TimeAverage2: `BadAggregateNotSupported`. The unsupported statuses are kept during the outage. Positive control: the same session reads 30 rows once the backend is back. | +| Strict ceiling (1,000), stopped-tag reverse | End-only read of the latest 3 before 2026-11-10 11:38:00 on `ZIPSEN.WTC1`, ~5.4 min after it stopped: the NULL at 11:32:36.073, then 11:32:33.620 and 11:32:31.619. Exact against a streaming-gateway reference, newest first. **10 gateway `ReadRaw` calls, 1.47 s**, against #696's unit regression bound of ≤ 12 calls. That bound is for a 1-day gap over 7,200 rows; the live gap here is minutes, so the count is evidence for this case only. A paged reverse read across the whole outage (page 37, 3 pages) equals the reference reversed. | +| Lowered / raised limits, aggregate cooldown | One long-lived data source. Ceiling 1,000: raw over 1,800 rows takes 2 pages, Average over them is `BadTooManyOperations`. Lowered to 500: 4 pages, learned from the next refusal. Raised to 5,000: Average is still `BadTooManyOperations` 2.9 s after the last refusal (the 30 s cooldown), then served Good (29.5) after it. Raw re-probes at once: 1 page. | + +## #746 item 1: events paging and TimeAverage + +The events data source was built by the production factory with **`EventDrainBudget = 0`**. Without the server-side cap, a +truncated page would then abandon (cancel) its gateway stream at the cap. + +| Check | Result | +|---|---| +| Paging | `DevWinPlatform`: 72 events. 1-event pages (72 pages) and 7-event pages (11 pages) equal the whole read; 5-event reverse pages (15 pages) are newest first. | +| Streams | **0 abandoned**, 0 ambiguous re-reads. The gateway log for the run holds 106 `ReadEvents` calls and **0** `OperationCanceledException`. | +| Mutation (live) | The same run with the adapter not sending `max_events`: **81 abandoned streams** (the check failed as intended) and 66 `OperationCanceledException` in the gateway log ([report](aveva-gate-followups-mutation-adapter-no-max-events.json.gz)). | +| TimeAverage | Served (no longer refused): see #694 above. | + +Not covered live: `SendEvents` batches. They need historian writes, which this gate does not make; they are covered by the +deterministic `GatewayAlarmWriterBatchTests`. + +## Findings + +1. **Fractional-millisecond intervals were misplaced (fixed).** The gateway sends `max(1, round(ms))`, so a 1.5 ms + interval reached AVEVA as 2 ms. The live probe answered 2 ms rows, and `NativeAggregatePlan.Place` filed them on the + 1.5 ms grid. Since `4c8e977e` a non-whole-ms grid is read one interval per request (at most 1,000). +2. **A non-whole-ms one-interval request lost its tail (fixed).** The clipped [40, 43.0004) was sent at its own width, + 3.0004 s, which reached AVEVA as 3,000 ms. AVEVA answered [40, 43) plus [43, 43.0004), and the first row won: Total 123 + instead of 123.0172 (first run of this gate). One-interval requests are now sent rounded **up** to whole ms. AVEVA then + answers one row clipped at the request end, which gives 123.0172 above. +3. **TimeAverage / Total on a discrete AVEVA tag** (`SysPulse`): AVEVA refuses `StartQuery` (UnknownException 85), the + gateway answers `Internal`, and the node answers **`BadHistoryOperationUnsupported`**. That is a Bad status, never a value, + but the read counts as a backend failure in the health snapshot. Before `4c8e977e` the pair was refused up front with + `BadAggregateNotSupported`. Follow-up: [#897](https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/897). +4. AVEVA integrates an integer tag (`SysTimeSec`) as a step (TimeAverage of 0..19 over 20 s = 9.5). Float tags are sloped + (see docs/Historian.md). The step reference above is therefore exact for `SysTimeSec`. + +## Files + +- [`aveva-gate-followups-4c8e977e.json.gz`](aveva-gate-followups-4c8e977e.json.gz): 65 cases, sha256 of the JSON `fc4891a788e5…`. +- [`aveva-gate-strict-4c8e977e.json.gz`](aveva-gate-strict-4c8e977e.json.gz): sha256 `58f86f79798a…`. +- [`aveva-gate-801-4c8e977e.json.gz`](aveva-gate-801-4c8e977e.json.gz): the #801 gate on this build, 244 cases, sha256 `b75b8e0c227d…`. +- [`aveva-gate-followups-mutation-adapter-no-max-events.json.gz`](aveva-gate-followups-mutation-adapter-no-max-events.json.gz): the live mutation run, sha256 `051b84f9e0b4…`. + +## Reproduce + +```bash +# HistorianGateway main, published framework-dependent, in a runtime-only aspnet:10.0 image (see HistorianGateway/docker/Dockerfile) +dotnet publish src/ZB.MOM.WW.HistorianGateway.Server/ZB.MOM.WW.HistorianGateway.Server.csproj -c Release -o /hgpub -p:UseAppHost=false +docker run -d --name zb-historiangw-h694 --env-file HistorianGateway/docker/histgw.secrets.env --env-file \ + -p 5232:5220 -p 5233:5221 +# gw.env: the docker-compose environment plus ApiKeys__Mode=Disabled, Historian__LiveWritesEnabled=false, Dashboard__DisableLogin=true +# (the second --env-file wins). Strict fixture: the same with +# -e Historian__ReadLimits__MaxValuesCeiling= -e Historian__ReadLimits__AllowStreamingBeyondCeiling=false -p 5236:5220 -p 5237:5221 + +export HISTGW_GATEWAY_ENDPOINT=http://localhost:5233 HISTGW_GATEWAY_APIKEY=auth-disabled +export OTOPCUA_AVEVA_GATE_OUT=/tmp/aveva-gate OTOPCUA_AVEVA_GATE_LABEL=$(git rev-parse --short HEAD) +export OTOPCUA_AVEVA_GATE_STRICT_ENDPOINT=http://localhost:5237 OTOPCUA_AVEVA_GATE_STRICT_CONTAINER=zb-historiangw-h694-strict +export OTOPCUA_AVEVA_GATE_STRICT_SET_CEILING=