feat(historian-gateway): native TimeAverage/Total on by default, SendEvents batches, server-side max_events (#694, #746)
#694 item 3 — historiangw#27 is fixed and now live-verified through OtOpcUa: - ServerHistorian:Gateway:EnableNativeTimeWeighted defaults to true (false is the kill switch for a gateway older than 9f711c1). TimeAverage / Total are served and advertised again (12 aggregates). - The gateway sends the interval as max(1, round(ms)) and AVEVA clips its last interval at the request end (live). NativeAggregatePlan therefore sends a multi-interval grid only for whole-ms intervals; a fractional interval is read one interval per request (at most 1,000, else BadTooManyOperations), and every one-interval request goes out rounded UP to whole ms so AVEVA answers one clipped row. Live, a 3.0004 s clipped interval sent at its own width lost its last 0.4 ms (Total 123 vs 123.0172). #746 — HistorianGateway Contracts/Client 1.0.0 surface: - Alarm history: GatewayAlarmHistorianWriter sends SendEvents batches of <=200. A failed per-event result is re-sent alone via SendEvent (classified as before); a refused batch (InvalidArgument etc.) goes event by event so a poison event dead-letters alone; a systemic fault retries the rest without sending more; Unimplemented (older gateway) falls back to SendEvent for good. IHistorianGatewayClient.SendEventsAsync defaults to Unimplemented. - Events reads: a capped page sends cap + 1 as ReadEventsRequest.max_events, so the gateway completes the stream instead of the client cancelling it. An ambiguous capped answer (foreign rows filtered out of the server's cut) or a refused cap is re-read uncapped with the existing EventDrainBudget drain. Tests: GatewayServerCappedEventsTests, GatewayAlarmWriterBatchTests, GatewayNativeWholeMillisecondGridTests (gateway-faithful rounding fake), gate tests flipped to the new default; every new test mutation-checked (each mutation of the production change fails at least one). Live harness AvevaGatewayHistorianFollowupsLiveGateTests (read-only) for #694/#700/#746. Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
This commit is contained in:
@@ -50,7 +50,7 @@ historian tagname, which is the tag's driver `FullName`.
|
||||
The server dispatches all OPC UA HistoryRead to the registered `IHistorianDataSource` — the
|
||||
`GatewayHistorianDataSource` read client when enabled, else the `NullHistorianDataSource` default
|
||||
(historized nodes return `GoodNoData` for Raw/AtTime — Processed is `BadAggregateNotSupported` since #682). Supported variants: Raw (⚠️ #687, client-visible: Gateway raw reads no longer return AVEVA's synthetic start-boundary row — asked from 1 ms early (epoch-floored) with `NumValuesPerNode + 16` rows (unclamped; 0 ⇒ `int.MaxValue`) and filtered by timestamp — so an empty window is `GoodNoData`, not a fabricated value at StartTime; the forward end is now exclusive (a point read inclusive); pre-window rows that crowd the budget trigger ONE re-read with room for them, else `BadTooManyOperations`; `ReturnBounds` is ignored on every backend; ⚠️ #690: a Gateway Bad row — a stored NULL, which histsdk sends as value 0 — now reads value **null** + Bad on Raw and AtTime, as on the SQL backends; ⚠️ #691.3: a strict gateway (`AllowStreamingBeyondCeiling=false`) that refuses a raw budget is re-asked ONCE at the ceiling its refusal names — raw and per-sample reads alike; the ceiling is then remembered per data source (`GatewayRowCeiling`: refusals count as gateway errors), re-learned from a refusal when lowered, and re-probed once at the full budget when a ceiling-bounded read comes back full (raised / streaming) — so an uncapped (`NumValuesPerNode = 0`) read of a small window is served; a ceiling-bounded stream that is not provably complete is `BadTooManyOperations`, never a truncated answer), Processed
|
||||
(backend-dependent: SqlServer/QuestDb serve nineteen; Gateway twelve = `GatewayHistorianDataSource.ServedAggregates` — ⚠️ #681, client-visible: the ten per-sample ones (Average, Count, Min/Max, Range, Start/End/Delta, Min/MaxActualTime) are evaluated from RAW gateway reads by the shared `Sql.Common` evaluator — identical to the SQL backends, Calculated bits, interval-start stamps, #677 reverse; Min/Max are no longer AVEVA modes, the read moves every raw sample of the window (a month of 1 Hz ⇒ `BadTooManyOperations` at the default `MaxTimeWeightedRawRows` cap), and every gateway stream is consumed to completion (an abandoned one makes the gateway evict its shared historian session) — plus TimeAverage/Total via AVEVA modes (interval-start stamped + Calculated; ⚠️ #685: AVEVA's extra pre-window row is dropped ⇒ one row per interval in `[Start, End)`, and an Uncertain (opc 64) interval keeps its value as `UncertainDataSubNormal` — AVEVA's TWA excludes NULL-gap time where the SQL backends bridge it; per-interval values still need gateway fix historiangw#27; ⚠️ #691.2: native reads use the SQL backends' grid in both directions via forward requests only — a REVERSE read (refused by the gateway before) is anchored at its later time (#677), and the clipped interval (earliest reverse, LATEST forward — which used to be averaged past EndTime) is its own exact-span read marked Partial unless Bad; an interval wider than the window is sent as the window, interval 0 as one interval; more than 100,000 intervals is refused up front (`BadHistoryOperationUnsupported`, as on the SQL backends — forward used to be served up to the gateway's 1,000,000); until historiangw#27 lands both directions return Good-status wrong values); the other time-weighted five stay `BadAggregateNotSupported` because the gateway has no prior-sample read for Part 13 bounds; `docs/Historian.md` §Aggregate set), and AtTime over historized variable nodes (⚠️ #693, client-visible: Gateway AtTime is evaluated from RAW reads by the shared `Sql.Common` `AtTimeEvaluator` — exact stored value at a stored time, null + Bad at/after a stored NULL, Uncertain hold before a Bad neighbour, linear + Interpolated bit, `BadNoData` before the first sample — never AVEVA's own interpolation (which answered 107 Good at a NULL, null+Bad at a Good sample after it, 99.99999999944 for 100, no Interpolated bit); `GatewayAtTimeReader`: one ≤1,024-row window read for the batch (jumps to far-apart times use 64-row reads) + a walk-back for the first time's prev (gallop ×4, whole-remainder read after two empties, then real BISECTION seeded by any full read's last row — a dense tag stopped a year ago ≈ 25–31 reads; the first cut regrew segments after a full read and gave up at 65) + an 18-row probe for the last time's next — 10 dense times = 1 read (exact) / 3 (interpolated), 10 hourly = 12; read cap 64 + 16 per distinct time (a runaway guard: a time in a gap of a bursty tag needs ≈ 6 reads; 4/time refused a 1 h/day tag asked daily ×30) and a strict ceiling < 4 rows ⇒ `BadTooManyOperations` with a message naming the limit; DateTime hints stepped, non-finite/out-of-range values under an integer hint unusable; all request bounds whole ms, rows counted only strictly inside a request; the node manager passes the node's scalar type (`ReadAtTimeAsync(…, TypeCode? valueType, …)` default-interface overload, ignored by SqlServer/QuestDB/Null) and the Gateway converts raw rows to it before evaluating — Boolean stepped, integers interpolated + rounded half-away and returned as the node's integer type, floats Double; no hint ⇒ Double; ⚠️ the shared evaluator used to box interpolated Int64 as Double (`cond ? (long)x : double` widens) — fixed for SQL backends too, and Shouldly `ShouldBe` hides numeric-type bugs (assert `ShouldBeOfType`); kill switch `ServerHistorian:GatewayAtTimeFromRaw` (default `true`); `docs/Historian.md` §Gateway AtTime); Events over
|
||||
(backend-dependent: SqlServer/QuestDb serve nineteen; Gateway twelve = `GatewayHistorianDataSource.ServedAggregates` — ⚠️ #681, client-visible: the ten per-sample ones (Average, Count, Min/Max, Range, Start/End/Delta, Min/MaxActualTime) are evaluated from RAW gateway reads by the shared `Sql.Common` evaluator — identical to the SQL backends, Calculated bits, interval-start stamps, #677 reverse; Min/Max are no longer AVEVA modes, the read moves every raw sample of the window (a month of 1 Hz ⇒ `BadTooManyOperations` at the default `MaxTimeWeightedRawRows` cap), and every gateway stream is consumed to completion (an abandoned one makes the gateway evict its shared historian session) — plus TimeAverage/Total via AVEVA modes (interval-start stamped + Calculated; ⚠️ #685: AVEVA's extra pre-window row is dropped ⇒ one row per interval in `[Start, End)`, and an Uncertain (opc 64) interval keeps its value as `UncertainDataSubNormal` — AVEVA's TWA excludes NULL-gap time where the SQL backends bridge it; per-interval values live-verified after gateway fix historiangw#27 (#694 — on by default again, `Gateway:EnableNativeTimeWeighted=false` is the kill switch; a non-whole-ms interval is read one interval per request, a one-interval request is sent rounded UP to whole ms so AVEVA answers one clipped row); ⚠️ #691.2: native reads use the SQL backends' grid in both directions via forward requests only — a REVERSE read (refused by the gateway before) is anchored at its later time (#677), and the clipped interval (earliest reverse, LATEST forward — which used to be averaged past EndTime) is its own exact-span read marked Partial unless Bad; an interval wider than the window is sent as the window, interval 0 as one interval; more than 100,000 intervals is refused up front (`BadHistoryOperationUnsupported`, as on the SQL backends — forward used to be served up to the gateway's 1,000,000); until historiangw#27 lands both directions return Good-status wrong values); the other time-weighted five stay `BadAggregateNotSupported` because the gateway has no prior-sample read for Part 13 bounds; `docs/Historian.md` §Aggregate set), and AtTime over historized variable nodes (⚠️ #693, client-visible: Gateway AtTime is evaluated from RAW reads by the shared `Sql.Common` `AtTimeEvaluator` — exact stored value at a stored time, null + Bad at/after a stored NULL, Uncertain hold before a Bad neighbour, linear + Interpolated bit, `BadNoData` before the first sample — never AVEVA's own interpolation (which answered 107 Good at a NULL, null+Bad at a Good sample after it, 99.99999999944 for 100, no Interpolated bit); `GatewayAtTimeReader`: one ≤1,024-row window read for the batch (jumps to far-apart times use 64-row reads) + a walk-back for the first time's prev (gallop ×4, whole-remainder read after two empties, then real BISECTION seeded by any full read's last row — a dense tag stopped a year ago ≈ 25–31 reads; the first cut regrew segments after a full read and gave up at 65) + an 18-row probe for the last time's next — 10 dense times = 1 read (exact) / 3 (interpolated), 10 hourly = 12; read cap 64 + 16 per distinct time (a runaway guard: a time in a gap of a bursty tag needs ≈ 6 reads; 4/time refused a 1 h/day tag asked daily ×30) and a strict ceiling < 4 rows ⇒ `BadTooManyOperations` with a message naming the limit; DateTime hints stepped, non-finite/out-of-range values under an integer hint unusable; all request bounds whole ms, rows counted only strictly inside a request; the node manager passes the node's scalar type (`ReadAtTimeAsync(…, TypeCode? valueType, …)` default-interface overload, ignored by SqlServer/QuestDB/Null) and the Gateway converts raw rows to it before evaluating — Boolean stepped, integers interpolated + rounded half-away and returned as the node's integer type, floats Double; no hint ⇒ Double; ⚠️ the shared evaluator used to box interpolated Int64 as Double (`cond ? (long)x : double` widens) — fixed for SQL backends too, and Shouldly `ShouldBe` hides numeric-type bugs (assert `ShouldBeOfType`); kill switch `ServerHistorian:GatewayAtTimeFromRaw` (default `true`); `docs/Historian.md` §Gateway AtTime); Events over
|
||||
alarm-owning equipment-folder event-notifier nodes. Reads are ungated (served from any redundancy node);
|
||||
authorization uses the standard `AccessLevels.HistoryRead` bit set at materialization.
|
||||
|
||||
@@ -76,7 +76,7 @@ authorization uses the standard `AccessLevels.HistoryRead` bit set at materializ
|
||||
|
||||
### Alarm-history path (`AlarmHistorian` section)
|
||||
|
||||
Alarm events are written through `GatewayAlarmHistorianWriter` (the gateway **`SendEvent`** path) behind
|
||||
Alarm events are written through `GatewayAlarmHistorianWriter` (gateway **`SendEvents`** batches of ≤200 since #746; `SendEvent` per event for a failed event, a refused batch or an older gateway) behind
|
||||
the durable **`LocalDbStoreAndForwardSink`** — `AlarmHistorian:Enabled=true` swaps the `NullAlarmHistorianSink`
|
||||
default for the store-and-forward queue, which buffers into the node's consolidated LocalDb as the
|
||||
replicated `alarm_sf_events` table (see the LocalDb section) and whose drain worker forwards batches to
|
||||
@@ -186,6 +186,13 @@ dotnet test --filter "Category=LiveIntegration"
|
||||
The live suite **skips cleanly** when these env vars are absent (safe to run offline on macOS). It is the
|
||||
gate the operator runs on the VPN before trusting the cutover.
|
||||
|
||||
⚠️ That `LiveIntegration` suite **writes** (sandbox tag, `SendEvent`). The **read-only** HistoryRead gates — run these
|
||||
when historian writes are not allowed — are `AvevaGatewayHistoryLiveGateTests` (#801) and
|
||||
`AvevaGatewayHistorianFollowupsLiveGateTests` (#694/#700/#746: native TimeAverage/Total vs a step reference, typed values,
|
||||
outage statuses, server-capped event paging, and a strict-ceiling fixture) in `OpcUaServer.IntegrationTests`. A local
|
||||
read-only gateway is enough: publish HistorianGateway `main` and run it with `ApiKeys__Mode=Disabled`,
|
||||
`Historian__LiveWritesEnabled=false` (recipe + evidence: `docs/plans/evidence/2026-11-12-historian-694-700-746/aveva-gate.md`).
|
||||
|
||||
### Continuous-historization value capture — wired and live-gated (was "KNOWN LIMITATION 2"; #491 CLOSED 2026-08-04)
|
||||
|
||||
The `ContinuousHistorizationRecorder` is fully wired (actor + FasterLog outbox + gateway value-writer +
|
||||
|
||||
Reference in New Issue
Block a user