feat(historian-gateway): native TimeAverage/Total on by default, SendEvents batches, server-side max_events (#694, #746)

#694 item 3 — historiangw#27 is fixed and now live-verified through OtOpcUa:
- ServerHistorian:Gateway:EnableNativeTimeWeighted defaults to true (false is
  the kill switch for a gateway older than 9f711c1). TimeAverage / Total are
  served and advertised again (12 aggregates).
- The gateway sends the interval as max(1, round(ms)) and AVEVA clips its last
  interval at the request end (live). NativeAggregatePlan therefore sends a
  multi-interval grid only for whole-ms intervals; a fractional interval is read
  one interval per request (at most 1,000, else BadTooManyOperations), and every
  one-interval request goes out rounded UP to whole ms so AVEVA answers one
  clipped row. Live, a 3.0004 s clipped interval sent at its own width lost its
  last 0.4 ms (Total 123 vs 123.0172).

#746 — HistorianGateway Contracts/Client 1.0.0 surface:
- Alarm history: GatewayAlarmHistorianWriter sends SendEvents batches of <=200.
  A failed per-event result is re-sent alone via SendEvent (classified as
  before); a refused batch (InvalidArgument etc.) goes event by event so a
  poison event dead-letters alone; a systemic fault retries the rest without
  sending more; Unimplemented (older gateway) falls back to SendEvent for good.
  IHistorianGatewayClient.SendEventsAsync defaults to Unimplemented.
- Events reads: a capped page sends cap + 1 as ReadEventsRequest.max_events, so
  the gateway completes the stream instead of the client cancelling it. An
  ambiguous capped answer (foreign rows filtered out of the server's cut) or a
  refused cap is re-read uncapped with the existing EventDrainBudget drain.

Tests: GatewayServerCappedEventsTests, GatewayAlarmWriterBatchTests,
GatewayNativeWholeMillisecondGridTests (gateway-faithful rounding fake), gate
tests flipped to the new default; every new test mutation-checked (each
mutation of the production change fails at least one). Live harness
AvevaGatewayHistorianFollowupsLiveGateTests (read-only) for #694/#700/#746.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi

docs/AlarmHistorian.md

@@ -53,7 +53,9 @@ unless noted.
 - **`IAlarmHistorianWriter`** — what the drain worker delegates writes to.
   `WriteBatchAsync(batch, ct)` returns one `HistorianWriteOutcome` per event,
   in order. Production binds this to `GatewayAlarmHistorianWriter` (the
-  HistorianGateway `SendEvent` path).
+  HistorianGateway `SendEvents` batch path — up to 200 events per RPC, #746 —
+  with `SendEvent` as the per-event fallback for a failed event, a refused
+  batch, or a gateway without the batch RPC).
 - **`HistorianWriteOutcome`** — per-event drain result: `Ack` (persisted,
   remove from queue), `RetryPlease` (transient failure — leave queued, retry
   after backoff), `PermanentFail` (malformed/unrecoverable — move to
@@ -241,7 +243,7 @@ writes across a redundant pair. `AlarmTransitionEvent` carries `AlarmTypeName`
 (the Part 9 subtype string) and `Comment` (the operator comment from the
 originating ack/shelve command) that populate the corresponding fields of
 `AlarmHistorianEvent`. `GatewayAlarmHistorianWriter` is the `IAlarmHistorianWriter`
-the drain worker delegates to (the gateway `SendEvent` path). See
+the drain worker delegates to (gateway `SendEvents` batches, `SendEvent` fallback). See
 [ServiceHosting.md](ServiceHosting.md) for the (external) HistorianGateway setup.
 
 **Scope:** scripted alarms only. Galaxy-native alarms historize via System
@@ -285,7 +287,7 @@ node's consolidated `LocalDb:Path` database (see
 
 | Key | Type | Default | Description |
 |---|---|---|---|
-| `Enabled` | bool | `false` | Enable the durable store-and-forward sink (drains to the HistorianGateway `SendEvent` path). `false` → `NullAlarmHistorianSink`. Requires LocalDb to be registered — an enabled sink with no `ILocalDb` fails at resolution rather than silently discarding events. |
+| `Enabled` | bool | `false` | Enable the durable store-and-forward sink (drains to the HistorianGateway `SendEvents` / `SendEvent` path). `false` → `NullAlarmHistorianSink`. Requires LocalDb to be registered — an enabled sink with no `ILocalDb` fails at resolution rather than silently discarding events. |
 | `BatchSize` | int | `100` | Max rows per drain cycle handed to `IAlarmHistorianWriter.WriteBatchAsync`. |
 | `DrainIntervalSeconds` | int | `5` | Seconds between drain-worker ticks. |
 | `Capacity` | long | `1000000` | Max queued rows before the sink evicts the oldest (data-loss signal via `EvictedCount`). |

docs/Historian.md

@@ -108,14 +108,14 @@ consumes the historian. The existing enable gates (`ServerHistorian:Enabled`, `A
 | Value kinds recorded | `Double`, `Int64`, `Boolean` (numeric analog only) | all: `Double`, `Int64`, `Boolean`, `String`, `DateTime`, plus `Empty` (a Bad sample with no value) | same as `SqlServer` |
 | Status code stored | Sent as DA quality but **discarded** by the gateway (C-002); AVEVA stores every sample Good | Stored bit-for-bit (`StatusCode` int column) | Stored bit-for-bit (unsigned, in a `LONG`) |
 | Comms-loss / driver-Bad publishes | Dropped, counted in `DroppedUnsupportedKind` | Stored as an `Empty` outage-marker sample carrying the Bad code (see [below](#typed-values-real-status-and-outage-markers)) | Stored as an `Empty` outage-marker sample, as on `SqlServer` |
-| Processed aggregates | **Ten** by default: the per-sample ones (Average, Count, Minimum, Maximum, Range, Start, End, Delta, Min/MaxActualTime) evaluated from raw reads with the SQL backends' evaluator — identical results (#681). TimeAverage and Total from AVEVA's native modes **only with `ServerHistorian:Gateway:EnableNativeTimeWeighted=true`** (default off: Good-status wrong values until historiangw#27). TimeAverage2 / Total2 / Interpolative / PercentGood / PercentBad / DurationGood / DurationBad: never | The nineteen in the [aggregate table](#aggregates-processed-reads) (incl. Part 13 time-weighted, #671, and DurationGood/DurationBad, #820) | The same nineteen, identical results |
+| Processed aggregates | **Twelve** by default: the per-sample ones (Average, Count, Minimum, Maximum, Range, Start, End, Delta, Min/MaxActualTime) evaluated from raw reads with the SQL backends' evaluator — identical results (#681) — plus TimeAverage and Total from AVEVA's native modes (on by default again since #694, after historiangw#27 was live-verified; `ServerHistorian:Gateway:EnableNativeTimeWeighted=false` is the kill switch). TimeAverage2 / Total2 / Interpolative / PercentGood / PercentBad / DurationGood / DurationBad: never | The nineteen in the [aggregate table](#aggregates-processed-reads) (incl. Part 13 time-weighted, #671, and DurationGood/DurationBad, #820) | The same nineteen, identical results |
 | Reverse / end-only Raw and Events reads | Emulated: walked back from bounded forward reads by the server (`ReverseRawReader` / `ReverseEventsReader`) — correct, but many round trips on a sparse tag | Native newest-first | Native newest-first |
-| Capped Events read | The cap is applied client-side (the gateway RPC has no max-events field); the stream past it is drained up to `Gateway:EventDrainBudget` rows | `TOP` | `LIMIT` |
+| Capped Events read | Server-side: cap + 1 sent as `ReadEventsRequest.max_events` (#746); the gateway completes the stream at it. Uncapped fallback (drained up to `Gateway:EventDrainBudget` rows) only for an ambiguous or refused cap | `TOP` | `LIMIT` |
 | Retention | AVEVA Historian's own | `SqlServer:RetentionDays` purge (opt-in) | `QuestDb:RetentionDays` TTL (opt-in) |
 | Health connection flags | `Gateway:ProbeInterval` probe + read outcomes | Read outcomes | Read outcomes |
 | HistoryUpdate | No (`BadHistoryOperationUnsupported`) | Yes (Insert / Replace / Update / DeleteRaw / DeleteAtTime) | Yes — soft delete (tombstones), see [below](#historyupdate-soft-delete) |
 | Modified-value history (`IsReadModified`, #668) | No (`BadHistoryOperationUnsupported`) | Yes — `SampleModified`, atomic with the change | Yes — `{prefix}sample_modified`, written before the change, marked applied after it (an orphan is not served) |
-| Alarm history | gateway `SendEvent`, one RPC per event, up to 8 in flight (no batch RPC yet) | `Event` table (insert-if-absent), multi-row statements of 150 events | `{prefix}event` table (dedup upsert on `(ts, event_id)`), multi-row INSERTs of 500 events |
+| Alarm history | gateway `SendEvents` batches of up to 200 events (#746); a failed event, a refused batch or a gateway without the RPC falls back to `SendEvent`, up to 8 in flight | `Event` table (insert-if-absent), multi-row statements of 150 events | `{prefix}event` table (dedup upsert on `(ts, event_id)`), multi-row INSERTs of 500 events |
 | Provisioning (`EnsureTags`) | numeric types only; others skipped | `MERGE` upsert of every request, all types | Upsert into `{prefix}tag`, all types |
 | Connection | `Endpoint` + `ApiKey` + TLS keys (top-level `ServerHistorian:*`) | `ServerHistorian:SqlServer:*` | `ServerHistorian:QuestDb:*` |
 
@@ -200,9 +200,9 @@ Processed read is `BadAggregateNotSupported` and no history aggregate functions
 
 | Key | Type | Default | Description |
 |---|---|---|---|
-| `Gateway:EnableNativeTimeWeighted` | bool | `false` | Serve HistoryRead-Processed `TimeAverage` / `Total` through AVEVA's native retrieval modes. ⚠️ **Off by default (client-visible):** the gateway sends AVEVA the interval in the wrong unit (historiangw#27), so AVEVA answers one Good-status whole-span value on the earliest interval and nothing for the rest — wrong values a client cannot tell from right ones. Off, both are `BadAggregateNotSupported` and are **not advertised** (`AggregateFunctions`), like the other five time-weighted aggregates on this backend. Turn it on only after a fixed gateway is live-verified. |
+| `Gateway:EnableNativeTimeWeighted` | bool | `true` | Serve HistoryRead-Processed `TimeAverage` / `Total` through AVEVA's native retrieval modes. ⚠️ **On by default since #694 (client-visible: both are advertised and served again).** The gateway interval fix historiangw#27 (gateway server `9f711c1`+) is live-verified: forward and reverse per-interval values, clipped, sub-millisecond clipped and fractional-millisecond intervals against a step-held reference, and all-NULL intervals ⇒ `BadNoData` ([evidence](plans/evidence/2026-11-12-historian-694-700-746/aveva-gate.md)). `false` is the kill switch for a gateway older than the fix (it answers one Good-status whole-span value on the earliest interval): both are then `BadAggregateNotSupported` and **not advertised**. On a discrete (Boolean) AVEVA tag AVEVA refuses both modes and the node answers `BadHistoryOperationUnsupported` (counted as a backend failure). |
 | `Gateway:ProbeInterval` | TimeSpan | `00:00:30` | How often the read client calls the gateway's `ProbeAsync` for the historian health snapshot (`ProcessConnectionOpen` / `EventConnectionOpen`, shown on `/hosts`). The first probe runs at start-up, so a node that has served no HistoryRead yet — or only writes (continuous historization) — reports a healthy gateway as open; it used to read closed until some client happened to read. The latest observation (probe or read outcome) wins; probes never count as queries or failures. Zero/negative = no probe (read outcomes only, the previous behaviour). |
-| `Gateway:EventDrainBudget` | int | `10000` | Stream rows a capped HistoryReadEvents read consumes (and discards) past its page cap before it abandons the gateway stream. The gateway's `ReadEvents` has no max-events field, so the cap is applied client-side; an **abandoned** stream cancels the RPC and the gateway evicts its pooled historian session (the next read reconnects; concurrent reads on that session fail). The usual small overshoot is drained to the stream's end; past the budget the stream is abandoned, so a page over a window far larger than the page does not stream the whole window. Rows of another source never count before the "more events" signal is seen, so an abandoned stream never answers a page as complete. `0` = abandon at the cap (the previous behaviour); negative counts as `0` (a `Validate()` warning). |
+| `Gateway:EventDrainBudget` | int | `10000` | Since #746 a capped HistoryReadEvents read sends cap + 1 as the server-side `max_events` and the gateway completes the stream there, so this budget applies only to the uncapped fallback read (an ambiguous server-capped answer — foreign rows filtered out of a stream the server cut — or a gateway refusing the cap) and to a stream that runs past the `max_events` it was sent. It is the number of stream rows such a read consumes (and discards) past its page cap before it abandons the gateway stream; an **abandoned** stream cancels the RPC and the gateway evicts its pooled historian session (the next read reconnects; concurrent reads on that session fail). The usual small overshoot is drained to the stream's end; past the budget the stream is abandoned, so a page over a window far larger than the page does not stream the whole window. Rows of another source never count before the "more events" signal is seen, so an abandoned stream never answers a page as complete. `0` = abandon at the cap (the previous behaviour); negative counts as `0` (a `Validate()` warning). |
 
 **`ServerHistorian:SqlServer`** (read only when `Backend=SqlServer`; bound by `HistorianBackendRegistry.BindSqlServer`):
 
@@ -1189,10 +1189,16 @@ says is complete carries no continuation point. `NumValuesPerNode == 0` stays un
 - **Tie exactness depends on the backend.** SqlServer and QuestDB order by `(EventTimeUtc, EventId)`, so the skip count is exact.
   The Gateway backend has no stable intra-timestamp order (and only a truncation flag), so ties at a page boundary of a
   **forward** read are best-effort there; a walked-back **reverse** read (below) sorts them by `EventId` itself and is exact.
-- **Gateway streams are drained, not abandoned.** The gateway's `ReadEvents` carries no max-events field, so a capped page is cut
-  client-side; the rest of the stream is consumed (uncollected) up to `ServerHistorian:Gateway:EventDrainBudget` rows (default
-  10 000) so the gateway does not evict its historian session for a cancelled stream. A page over a window much larger than the
-  budget still abandons its stream. A caller cancellation (client abort, `HistoryReadDeadline`) is not counted as a backend failure.
+- **Gateway streams complete, they are not abandoned (#746).** A capped page sends cap + 1 as `ReadEventsRequest.max_events`
+  (HistorianGateway Contracts/Client 0.4.0+): the gateway stops there and completes the stream normally, so it never evicts its
+  historian session for a cancelled stream, and the (cap + 1)-th event is the "more events" signal. Live: 1-, 7- and reverse 5-event
+  pages over 72 `DevWinPlatform` events with `EventDrainBudget = 0` abandoned **0** streams; the same run with the cap not sent
+  abandoned 81 ([evidence](plans/evidence/2026-11-12-historian-694-700-746/aveva-gate.md)). If the server cut the stream at cap + 1 but the client-side source filter dropped some of those rows (so
+  whether more matching events exist is unknown), or the gateway refuses the cap (`InvalidArgument`, a `MaxEventsCeiling` below
+  it), the page is re-read without a server cap and drained as before: the rest of the stream is consumed (uncollected) up to
+  `ServerHistorian:Gateway:EventDrainBudget` rows (default 10 000), past which the stream is abandoned. A gateway older than
+  `max_events` ignores it and the package client stops at cap + 1 itself (which cancels the stream). A caller cancellation
+  (client abort, `HistoryReadDeadline`) is not counted as a backend failure.
 - **Reverse event reads page newest-first (#675).** `StartTime > EndTime` reads the Part 11 reverse window `(EndTime, StartTime]`
   with events in DESCENDING `(EventTimeUtc, EventId)` order, so a cap keeps the **latest** N and a continuation point is issued
   when the source flags more. The direction is stored **in the cursor**: a reverse point resumes reverse (and a forward one
@@ -1354,12 +1360,12 @@ fixed:
   bounding — 6.08 Uncertain for the same samples, see [Time-weighted aggregates](#time-weighted-aggregates-part-13-671)).
   Extrapolation past the last sample is `Good` on AVEVA and `Uncertain` on the SQL backends. On all-Good spans the magnitudes
   match exactly (both sloped/trapezoidal).
-- ⚠️ **End-to-end correctness also needs the gateway fix historiangw#27:** the gateway sends the aggregate interval to AVEVA in
-  ticks where AVEVA reads milliseconds (10,000× too large), so today a native read returns one row for the whole window (plus the
-  pre-window row, now dropped). OtOpcUa does not work around it; per-interval values return once the gateway is fixed.
-- **Not verified live:** what AVEVA answers for an interval holding **only** NULL samples (no good time at all). If it marks such a
-  row Uncertain (`opc_quality` 64–191) with a placeholder value, that value is now passed through as Uncertain; the gate only
-  observed Uncertain rows that also held good time. Re-check on the VPN once historiangw#27 is fixed.
+- **End-to-end correctness needed the gateway fix historiangw#27** (the interval reached AVEVA in ticks where it reads
+  milliseconds, so a native read returned one row for the whole window). Fixed on the gateway (`9f711c1`) and live-verified through
+  OtOpcUa in #694 ([evidence](plans/evidence/2026-11-12-historian-694-700-746/aveva-gate.md)): per-interval values in both directions match a step-held integral of the tag's own raw rows exactly.
+- **All-NULL intervals (verified live, #694).** An interval holding no good time at all (a float tag stored NULL at 11:32:36 with
+  nothing until 11:38:34) is answered by AVEVA `opc_quality = 0` with NaN, mapped `BadNoData | Calculated` (`0x809B0401`), null
+  value; the intervals that hold the NULL *and* some good time are Uncertain (`0x40A40401`) with AVEVA's value.
 
 **Integral unit (#677).** AVEVA's integral is `∫ value dt` (seconds) ÷ the tag's `IntegralDivisor`. The proto3 `double` default for
 `HistorianTagDefinition.IntegralDivisor` is `0`, so `GatewayTagProvisioner` now sends **`IntegralDivisor = 1`** explicitly on every
@@ -1384,13 +1390,20 @@ gateway refuses `start > end` and `interval <= 0`):
   (#685). An interval AVEVA answers no row for is absent (the SQL backends answer it `BadNoData`).
 - ⚠️ **Interval cap (client-visible):** forward native reads now go through `BucketPlan` too, so a window of more than 100,000
   intervals (e.g. 1 s over 2 days) is refused up front with `BadHistoryOperationUnsupported`, as on the SQL backends — it used to be
-  served up to the gateway's own `MaxAggregateIntervals` (1,000,000). Not counted as a backend failure. A clipped interval shorter than
-  1 ms (a sub-ms window end) is sent as a sub-ms interval request; re-check it in the post-historiangw#27 live leg.
-- ⚠️ **Native TimeAverage / Total are off by default** (`ServerHistorian:Gateway:EnableNativeTimeWeighted=false` ⇒
-  `BadAggregateNotSupported`, unsent, unadvertised) because of the next point; everything in this section describes the opted-in path.
-- ⚠️ **Until historiangw#27 is fixed, native values are wrong in BOTH directions with a Good status:** AVEVA answers one row per request
-  (the whole span's value), which lands on that request's first interval as a Good, Calculated value while the other intervals are
-  absent. Pinned against a native-mode fake only — re-run the forward and reverse native legs live once the gateway is fixed. Since #681 every native bucket is stamped with its interval **start**
+  served up to the gateway's own `MaxAggregateIntervals` (1,000,000). Not counted as a backend failure.
+- ⚠️ **Whole milliseconds on the wire (#694, client-visible).** The gateway sends AVEVA the interval as `max(1, round(ms))` and AVEVA
+  clips its last interval at the request's end (both observed live). So: (1) a grid of several intervals is one request only when the
+  interval is a whole number of milliseconds — a fractional one (e.g. `ProcessingInterval = 1.5`) is read **one interval per
+  request**, at most 1,000 intervals (more ⇒ `BadTooManyOperations`, not a backend failure); before, AVEVA answered it on the rounded
+  grid and the buckets were misplaced. (2) A request for one interval — the clipped one, a window narrower than one interval, each
+  per-interval request — is sent with its width rounded **up** to whole milliseconds, so AVEVA answers one row clipped at the
+  request's end: sent at its own width, a 3.0004 s clipped interval reached AVEVA as 3,000 ms and its Total lost the last 0.4 ms
+  (123 instead of 123.0172 live). A sub-millisecond clipped interval is exact (live: 0.4 ms at value 40 ⇒ Total 0.016).
+- **Native TimeAverage / Total are on by default** since #694 (`ServerHistorian:Gateway:EnableNativeTimeWeighted=false` ⇒
+  `BadAggregateNotSupported`, unsent, unadvertised — the kill switch for a gateway older than historiangw#27).
+- **Live-verified (#694, [evidence](plans/evidence/2026-11-12-historian-694-700-746/aveva-gate.md)):** forward and reverse, whole, clipped, sub-ms clipped and fractional-ms intervals on `SysTimeSec`
+  (an integer tag AVEVA integrates as a step) equal a step-held integral of its raw rows, stamps and Partial bits included. Before
+  the gateway fix AVEVA answered one row per request (the whole span's value) on the first interval. Since #681 every native bucket is stamped with its interval **start**
 (Part 13; it was the interval end) and carries the Calculated info bits (`Good` ⇒ `0x00000401`, no data ⇒ `BadNoData | 0x401`).
 The live AVEVA gate (2026-10-12) confirmed `start_time` is the interval start (and the `value·s` unit). Previously noted as
 pending: it reads the aggregate row's `start_time`, which histsdk decodes from a row
@@ -1409,12 +1422,12 @@ field (`tail + 43`) its own reverse-engineering notes still mark unverified (His
 > statuses may differ from AVEVA's own modes; every per-sample result now carries the Calculated info bits; a per-sample read is
 > bounded by **`ServerHistorian:MaxTimeWeightedRawRows`** (in-window raw rows; past it ⇒ `BadTooManyOperations` for that node — read
 > a long window over a dense tag in shorter windows). `TimeAverage` and `Total` stay AVEVA-native (`TimeWeightedAverage` /
-> `Integral`) — ⚠️ **and are refused `BadAggregateNotSupported` unless `ServerHistorian:Gateway:EnableNativeTimeWeighted` is
-> `true`** (default `false`, until historiangw#27 is live-verified; the default set is the ten raw-evaluated aggregates,
+> `Integral`) — served by default since #694 (refused `BadAggregateNotSupported` with the
+> `ServerHistorian:Gateway:EnableNativeTimeWeighted=false` kill switch, which leaves the ten raw-evaluated aggregates,
 > `GatewayHistorianDataSource.RawEvaluatedAggregates`) — now stamped with the interval start (was its end) and carrying the Calculated bit; `TimeAverage2`, `Total2`,
 > `Interpolative`, `PercentGood`, `PercentBad`, `DurationGood` and `DurationBad` remain `BadAggregateNotSupported` on Gateway. The most the backend can serve is
-> `GatewayHistorianDataSource.ServedAggregates` (12); an instance serves and advertises `SupportedAggregates` — those 12 with the
-> flag on, the 10 raw-evaluated ones without it.
+> `GatewayHistorianDataSource.ServedAggregates` (12); an instance serves and advertises `SupportedAggregates` — those 12 by
+> default, the 10 raw-evaluated ones with the kill switch.
 
 **How the Gateway raw evaluation works (#681).** Forward `ReadRaw` reads only (the gateway streams oldest-first and refuses
 `end < start`; `BucketPlan` assembles a reverse result newest-first), O(buckets) memory, and **every gateway stream is consumed to
@@ -2681,7 +2694,7 @@ phases and are recorded here so future audits don't re-flag them.
 ## See also
 
 - [docs/plans/2026-06-14-galaxy-phase-c-historian-design.md](plans/2026-06-14-galaxy-phase-c-historian-design.md) — full design and implementation notes
-- [AlarmHistorian.md](AlarmHistorian.md) — alarm write path; drains to the same HistorianGateway (`SendEvent`)
+- [AlarmHistorian.md](AlarmHistorian.md) — alarm write path; drains to the same HistorianGateway (`SendEvents` batches, `SendEvent` fallback)
 - [AlarmTracking.md](AlarmTracking.md) — OPC UA Part 9 alarm surface (event history source)
 - [Client.CLI.md](Client.CLI.md) — full `historyread` flag reference
 - [ScriptedAlarms.md](ScriptedAlarms.md) §"Native driver alarms" — the Phase B `alarm` object in `TagConfig` (parallel carrier)