lmxopcua/.claude/rules/historian.md

paths

paths
src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.*/** src/Core/ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian/** src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/Historian/** src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Historian/** src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/** tests/**/*Historian*/** docs/Historian.md

Historian / HistoryRead (Gateway, SqlServer, QuestDb backends; alarm history; continuous historization; provisioning)

Moved verbatim from the root CLAUDE.md (2026-10-13) so it loads only when working in this area. Root CLAUDE.md § "Subsystem guides" indexes it. Edit it here, not in the root file.

Historian / HistoryRead

Backend: pluggable — ServerHistorian:Backend = Gateway (default), SqlServer or QuestDb (see the subsection below). As of the gateway-integration cutover, the Gateway backend's historian read, alarm-write, and continuous-historization paths are all served by the ZB.MOM.WW.HistorianGateway sidecar, consumed as the Gitea-feed ZB.MOM.WW.HistorianGateway.Client gRPC package (historian_gateway.v1) behind a thin IHistorianGatewayClient seam in ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway. The bespoke Wonderware TCP/ArchestrA sidecar projects and the vestigial Historian.Wonderware driver type were retired — there is no Wonderware backend in the tree anymore (see docs/drivers/Historian.Wonderware.md, now a retired stub).

A tag is historized by adding "isHistorized": true to its TagConfig JSON blob (authored in the raw-JSON textarea on the /uns TagModal); an optional "historianTagname" field overrides the default historian tagname, which is the tag's driver FullName.

Pluggable backends, SQL Server and HistoryUpdate (2026-10)

• Selection: ServerHistorian:Backend (Gateway default | SqlServer | QuestDb; unknown fails host start). HistorianBackendRegistry → IHistorianBackendProvider builds all five seams; the gateway keys are only required for Gateway. SqlServer uses ServerHistorian:SqlServer:{ConnectionString (required), Schema=hist, CommandTimeout=30s, AutoCreateSchema=true}.
• SQL Server (Driver.Historian.SqlServer): hist.Sample PK (TagId, SourceTimeUtc) — writes are insert-if-absent, so outbox redelivery is idempotent. Stores every value kind with its real status; per-sample aggregates (10, simple definitions) are pushed down to T-SQL with bucket math in 100 ns ticks; the Part 13 time-weighted seven (TimeAverage/TimeAverage2/Total/Total2/Interpolative, #671 — ⚠️ Total is now the time-weighted integral, no longer SUM — plus the duration-based PercentGood/PercentBad, #680) stream window + 4 bound rows into the shared TimeWeightedAggregator (both SQL backends, identical results; reverse Interpolative anchored at the later time; window capped by ServerHistorian:MaxTimeWeightedRawRows ⇒ BadTooManyOperations, percentages included). ⚠️ #680 status rules (client-visible): TimeAverage/Total are Uncertain if any part of the interval uses an Uncertain value, is bridged across a Bad sample, is extrapolated past the last sample (⇒ a live tag's latest interval is Uncertain) or is not covered; TimeAverage2/Total2 keep the SDK time-based rule (Uncertain time counts as Good) but never extrapolate past the last sample — that interval is Partial, later ones BadNoData (a report-by-exception tag with nothing after its last sample reads BadNoData until its next sample arrives). docs/Historian.md §"Time-weighted aggregates". ⚠️ #680 per-sample (client-visible): Count counts Good samples only (value-based status; an all-Bad interval is generic Bad, value null) and an empty interval inside the tag's data range is 0/Good (outside it still BadNoData); Start/End return the raw first/last sample of any quality — its own timestamp and status, value null when Bad; Delta is Uncertain only when the interval's first or last sample is unusable; ⚠️ #684: Average/Minimum/Maximum/Range/*ActualTime use Good samples only (was Good + Uncertain; an Uncertain-only interval is BadNoData), Min/Max-family status Uncertain on a Bad sample, an Uncertain value beyond the Good extreme (Part 13 — the SDK ignores that) or an Uncertain non-numeric sample, and only a Good String/DateTime sample refuses them (BadAggregateInvalidInputs; Delta still refuses any usable one); and every per-sample aggregate but Average sets end-of-data Partial on the interval holding the last sample (forward incl. its earlier edge, reverse strictly inside), never on a Bad result. The data-range lookup is a key seek on SQL Server and a tag symbol-index seek on QuestDB (#683 — {p}sample.tag is SYMBOL INDEX; the bootstrapper ADDs it to a pre-#683 table, idempotent), read BEFORE the bucket statement so a racing append can never yield a Good Count of 0 (a concurrent HistoryUpdate delete is the mirror case and still can, transiently). docs/Historian.md §"Aggregates". Docker-dev recipe, schema and aggregate semantics: docs/Historian.md §"SQL Server backend". Integration tests need HISTORIAN_SQL_TEST_CONNECTION. Retention (#667): opt-in SqlServer:RetentionDays (+ PurgeInterval 1h, PurgeBatchSize 4000) -> HistorianRetentionService purges old hist.Sample (per tag) / hist.Event in DELETE TOP(n) batches, Primary-gated on IHistorizationRoleView, one purger per DB via sp_getapplock; counter otopcua.historian.retention.purged{table}. Optional IHistorianBackendProvider.CreateRetention (default null). docs/Historian.md §Retention.
• QuestDB (Driver.Historian.QuestDb, ServerHistorian:QuestDb:*, Npgsql over PG wire; docker-dev overlay docker-compose.questdb.yml): same kinds/status/aggregates/HistoryUpdate as SqlServer, soft delete by tombstone (INSERT only — never UPDATE/DELETE; every read filters NOT deleted). ⚠️ Timestamps are µs (QuestDbConnection.Round/BindTimestamp; IHistorianDataSource.TimestampResolutionTicks = 10 steps HistoryRead paging), pre-1970 cannot be stored (writer drops, HistoryUpdate BadOutOfRange), every write waits for WAL visibility, TTL is enforced on write commits only, set Npgsql Maximum Pool Size < QuestDB's 64. docs/Historian.md §"QuestDB backend".
• ⚠️ Reverse processed reads are Part 13/SDK-anchored (#677, client-visible): every aggregate on SqlServer/QuestDB now anchors a reverse read at its later time — window (end, start], partial interval earliest, each stamped with its later edge (was: earlier-anchored, stamped earlier edge); on forward reads Partial also marks the interval the data starts inside (not Average — the SDK's rule, forward-only); a non-default client AggregateConfiguration ⇒ BadAggregateConfigurationRejected; Gateway provisioning sends IntegralDivisor = 1. SDK-parity tests: SdkAggregateParityTests. docs/Historian.md §Aggregates.
• ⚠️ Reverse / end-only RAW reads work on Gateway (#686, client-visible): IHistorianDataSource.SupportsReverseRaw (SqlServer/QuestDB/Null true, Gateway false, interface default false, guard-tested) — the node manager sends start > end only to a backend that declares it; otherwise ReverseRawReader serves it from FORWARD reads: uncapped ⇒ ONE forward read reversed in memory; capped ⇒ a bounded walk back in segments (ms-aligned request bounds, width aimed from observed density at ≈ N + 8 rows and carried page to page in the continuation point — #689: it used to aim at half MaxTieClusterOverfetch, ~32k rows per page on a dense tag; a read asks for 2N + 18 rows (a discovery segment ≤ 1,026), MaxTieClusterOverfetch is only the tie-cluster ceiling — a full read is re-read, never answered truncated; only more than max(1, MaxTieClusterOverfetch) samples within ONE MILLISECOND ⇒ BadTooManyOperations (per ms, not per timestamp — requests are ms-rounded, and letting a segment narrow below 1 ms stepped a page through one ms a tick per read, ~10k reads); walks no later than now + 1 day and never before the Unix epoch; #691: after two empty segments it asks once for the whole remainder, and the stretch a page proved empty rides the continuation point — the last pages of a deep page-through no longer each walk ~14 reads to 1970). The raw/events tie-path oversized-cluster refusal is now BadTooManyOperations too (was BadHistoryOperationUnsupported). Uncapped Gateway raw reads (forward too) used to send max_values = 0, which the client/gateway reject — now unbounded (int.MaxValue; a ceiling refusal ⇒ BadTooManyOperations), and a cancelled read is no longer a backend failure. Since #669 such reads failed the node on Gateway and counted as a backend fault (the gateway rejects start > end — GatewayHistorianDataSource.ReadRawAsync now refuses one unsent); before #669 an end-only Gateway read returned the OLDEST N ascending — now the latest N at/before EndTime, newest first. Events follow since #691.4: a reverse/end-only capped events read on Gateway (SupportsReverseEvents false; Null now reports true and answers empty) is walked back the same way (ties sorted by EventId — the gateway orders by time only; one extra event found ⇒ continuation point) — it used to be refused BadHistoryOperationUnsupported (explicit reverse) or answered with the OLDEST N ascending (end-only); needs the gateway's RuntimeDb:EventReadMaxRows 0/unlimited (it truncates silently). Uncapped Gateway events reads (forward too) used to throw "Array dimensions exceeded" (result list pre-sized to int.MaxValue) — fixed. Above a truncated read the walk gallops up from the newest row seen, then bisects (never below 1 ms): a burst that ended 100 days ago ≈ 18 reads, a dense tag that stopped days ago ≈ 26. docs/Historian.md §"Open-ended and reverse raw reads".
• Advertised aggregates (#682): IHistorianDataSource.SupportedAggregates (Gateway 4 via GatewayHistorianDataSource.ServedAggregates, SqlServer/QuestDB 17 via AggregateFamilies.Served, Null none; a null set ⇒ all) is the one authority for HistoryServerCapabilities/AggregateFunctions (i=11201 — only its folder references are trimmed) and the up-front BadAggregateNotSupported in HistoryRead-Processed. ⚠️ ServerCapabilities/AggregateFunctions (i=2997) is deliberately NOT trimmed: it keeps the SDK's 37 because live AggregateFilter subscriptions still compute them all. ⚠️ Historian disabled ⇒ Processed is now BadAggregateNotSupported, not GoodNoData. docs/Historian.md §Advertised aggregates.
• Raw ReturnBounds is honoured on every backend (#691, client-visible): Part 11 bounding values (§4.6 Table 1 is the test oracle) are composed in the server by HistoryBounds from each backend's raw reads — "at or before" = capped reverse read (native on SQL backends, the #686 ReverseRawReader walk-back on Gateway, never AVEVA's synthetic boundary row), "at or after" = capped forward read. Bounds are stored samples of any quality, count toward NumValuesPerNode, start bound on the first page / end bound on the last (pending end bound rides the continuation point); missing ⇒ Bad_BoundNotFound (null, at the bound time; a one-sided read's open side ±1 s from the previous value). ⚠️ Breaking for SDK-default clients: the .NET SDK's new ReadRawModifiedDetails() defaults BOTH ReturnBounds = true AND IsReadModified = true — a client that cleared only IsReadModified now gets bounds, and one that set neither is a modified read with bounds ⇒ Bad_InvalidArgument (§6.5.3.3); set both explicitly (Client.Shared does; tests building details must too). No data AND no bound found ⇒ GoodNoData, no values (the pre-bounds answer). On Gateway the at-or-after lookup ends at now + 1 day (never a year-9999 end to AVEVA). Kill switch ServerHistorian:HonorReturnBounds (default true; false = pre-#691). docs/Historian.md §Bounding values.
• ⚠️ Raw pages are full (#692, client-visible, every backend): each raw page asks the backend for N + skip + 1 rows (the boundary ties it will trim + one lookahead row) — a resumed page used to ask only N, trim the ties and come back short; at N=1 that was an EMPTY page + continuation point that then fell into the MaxTieClusterOverfetch + 1 over-fetch (live: 8 values = 17 pages / 25 reads / 524k rows ⇒ now 8/8/≤24). No lookahead row ⇒ last page, no CP (an exact multiple no longer ends on an empty page). Exception: an oversized tie cluster — its last slice is min(N, remaining) with a CP, and stepping past one at the end of data can give one empty final page. The over-fetch fires only when every row past the trimmed ties sits on the boundary timestamp; a cursor inside such a cluster is marked WithinTieCluster (next page goes straight to the over-fetch). SliceTieCluster resumes only strictly before EndTime (truncated to the backend resolution) — it used to resume AT the exclusive EndTime and return samples there. Bound placeholders carry an explicit marker (HistoryBounds.IsPlaceholder no longer matches a stored Bad_BoundNotFound/null row); the N=1 bounds probe carries its walk hint + resolved end bound into the cursor. Reviewer fuzz harnesses (paged vs unpaged over random tie series) are the right test for any paging change — repo tests passed while master mismatched 1-in-160.
• ⚠️ Strict-ceiling paging (#694): IHistorianDataSource.MaxRawValuesPerRead = L = learned gateway ceiling − 16 (0 on SQL backends/Null, guard-tested; 0 until a refusal names the ceiling). RawPageLimit.Plan(want, k, L) sizes every raw page so page + boundary ties + lookahead ≤ L (room = L − k − 1, signed; first page L−1, resumed L−k−1; uncapped N = 0 reads page too) — a capped read above the ceiling used to be BadTooManyOperations, now a ceiling-sized page + CP. One re-plan when the read taught a new limit or was refused for crowding (HistoryReadLimitExceededException.PreWindowRows, set only above the 16-row allowance; the page shrinks by it). No room on a resumed page ⇒ the tie-cluster over-fetch, bounded min(M, L−1); the slice size is never 0. The page path passes retryOnLearnedLimit: false to ReverseRawReader (the walk re-running at the old N cost 200–3,000 reads cold); bounds lookups keep it. ⚠️ A Gateway window starting at/before the Unix epoch counts AVEVA's synthetic row as pre-window — it used to set reachedEnd, so a ceiling-truncated read came back Good with no CP (forward-from-1970 = first page only; end-only past the data = the OLDEST rows as "latest"; pre-existing). The #687 crowding re-read now doubles its room up to 8 re-reads, never past a known ceiling, and its room is capped at MaxTieClusterOverfetch (passed to the Gateway source by the factory; past it ⇒ BadTooManyOperations with PreWindowRows). A raised ceiling is not re-probed by raw paging (pages stay small until another read re-probes or restart).
• Real status fix: the recorder used to stamp every sample Good (192); it now carries the driver's status. ⚠️ On the Gateway backend this changes nothing stored: the gateway discards quality (C-002), so AVEVA still stores Good.
• Comms loss is historized as an outage marker (Empty + BadNoCommunication, one per tag per outage) on SqlServer; dropped (DroppedUnsupportedKind) on Gateway. Outbox records are format v2 — drain or delete ContinuousHistorization:OutboxPath before downgrading a node.
• HistoryUpdate (SqlServer + QuestDb; Gateway ⇒ BadHistoryOperationUnsupported): Insert/Replace/Update/DeleteRaw/DeleteAtTime, Primary-only since #674 (off-Primary ⇒ BadInvalidState, after authorization AND after planning — an unservable request (Remove / IsDeleteModified / not historized) stays BadHistoryOperationUnsupported off-Primary; the verdict is IHistoryUpdateRoleView = the data-plane verdict but closed until this node's first redundancy snapshot, so a pair cannot both accept in the boot window; admin-only nodes register no view ⇒ no gate; reads ungated), items with a SourceTimestamp past now + ServerHistorian:HistoryUpdateMaxFutureSkew (default 1 day, ≤0 disables) ⇒ BadOutOfRange in the node manager (every backend), gated in both ACL modes — flag off needs the HistoryUpdate role from a Cluster-scoped grant (+ key re-check), flag on uses DataPlaneAuthorizer's HistoryUpdate operation; WriteOperate never implies it. The HistoryWrite AccessLevel bit is set only when an updater is wired at materialise time. docs/Historian.md §HistoryUpdate, docs/security.md. ⚠️ SqlServer schema v1 is accepted (#674) with AutoCreateSchema=false: writes/reads/events/retention work; modified reads + HistoryUpdate ⇒ node-level BadHistoryOperationUnsupported on every arm (the updater throws NotSupportedException, nothing sent; the HistoryWrite bit is still advertised — not knowable at materialise time), retention skips SampleModified, one Warning per boot (SqlServerSchemaBootstrapper.SupportsModifiedTrail; RequiredVersion = 1, ModifiedTrailVersion = 2). QuestDB: keep cairo.ttl.use.wall.clock=true; a QuestDB trail INSERT retried after a client timeout duplicates (no DEDUP key).
• Modified-value history (#668, Part 11): every applied HistoryUpdate item leaves a trail row (prior value, principal from the session's API key / anonymous, HistoryModificationType, time) — SqlServer hist.SampleModified (schema v2, written by OUTPUT … INTO in the same statement ⇒ atomic; ⚠️ no FK/CHECK/trigger on it, and AutoCreateSchema=false installs must run 002_sample_modified.sql before upgrading), QuestDB {p}sample_modified (append-only, written before the change ⇒ a crash leaves an orphan trail row, never an untracked change). IsReadModified=true HistoryRead serves it as HistoryModifiedData via IHistorianDataSource.ReadModifiedAsync (default ⇒ NotSupported ⇒ BadHistoryOperationUnsupported on Gateway/Null), paged on (SourceTime, ModifiedId), gated by plain HistoryRead. Live writes and retention never write it; retention/TTL purge it on source time. IHistorianDataUpdater methods take a HistoryUpdateContext. CLI historyread --modified. docs/Historian.md §Modified-value history.

Read path (ServerHistorian section)

The server dispatches all OPC UA HistoryRead to the registered IHistorianDataSource — the GatewayHistorianDataSource read client when enabled, else the NullHistorianDataSource default (historized nodes return GoodNoData for Raw/AtTime — Processed is BadAggregateNotSupported since #682). Supported variants: Raw (⚠️ #687, client-visible: Gateway raw reads no longer return AVEVA's synthetic start-boundary row — asked from 1 ms early (epoch-floored) with NumValuesPerNode + 16 rows (unclamped; 0 ⇒ int.MaxValue) and filtered by timestamp — so an empty window is GoodNoData, not a fabricated value at StartTime; the forward end is now exclusive (a point read inclusive); pre-window rows that crowd the budget trigger ONE re-read with room for them, else BadTooManyOperations; ReturnBounds is ignored on every backend; ⚠️ #690: a Gateway Bad row — a stored NULL, which histsdk sends as value 0 — now reads value null + Bad on Raw and AtTime, as on the SQL backends; ⚠️ #691.3: a strict gateway (AllowStreamingBeyondCeiling=false) that refuses a raw budget is re-asked ONCE at the ceiling its refusal names — raw and per-sample reads alike; the ceiling is then remembered per data source (GatewayRowCeiling: refusals count as gateway errors), re-learned from a refusal when lowered, and re-probed once at the full budget when a ceiling-bounded read comes back full (raised / streaming) — so an uncapped (NumValuesPerNode = 0) read of a small window is served; a ceiling-bounded stream that is not provably complete is BadTooManyOperations, never a truncated answer), Processed (backend-dependent: SqlServer/QuestDb serve seventeen; Gateway twelve = GatewayHistorianDataSource.ServedAggregates — ⚠️ #681, client-visible: the ten per-sample ones (Average, Count, Min/Max, Range, Start/End/Delta, Min/MaxActualTime) are evaluated from RAW gateway reads by the shared Sql.Common evaluator — identical to the SQL backends, Calculated bits, interval-start stamps, #677 reverse; Min/Max are no longer AVEVA modes, the read moves every raw sample of the window (a month of 1 Hz ⇒ BadTooManyOperations at the default MaxTimeWeightedRawRows cap), and every gateway stream is consumed to completion (an abandoned one makes the gateway evict its shared historian session) — plus TimeAverage/Total via AVEVA modes (interval-start stamped + Calculated; ⚠️ #685: AVEVA's extra pre-window row is dropped ⇒ one row per interval in [Start, End), and an Uncertain (opc 64) interval keeps its value as UncertainDataSubNormal — AVEVA's TWA excludes NULL-gap time where the SQL backends bridge it; per-interval values still need gateway fix historiangw#27; ⚠️ #691.2: native reads use the SQL backends' grid in both directions via forward requests only — a REVERSE read (refused by the gateway before) is anchored at its later time (#677), and the clipped interval (earliest reverse, LATEST forward — which used to be averaged past EndTime) is its own exact-span read marked Partial unless Bad; an interval wider than the window is sent as the window, interval 0 as one interval; more than 100,000 intervals is refused up front (BadHistoryOperationUnsupported, as on the SQL backends — forward used to be served up to the gateway's 1,000,000); until historiangw#27 lands both directions return Good-status wrong values); the other time-weighted five stay BadAggregateNotSupported because the gateway has no prior-sample read for Part 13 bounds; docs/Historian.md §Aggregate set), and AtTime over historized variable nodes (⚠️ #693, client-visible: Gateway AtTime is evaluated from RAW reads by the shared Sql.Common AtTimeEvaluator — exact stored value at a stored time, null + Bad at/after a stored NULL, Uncertain hold before a Bad neighbour, linear + Interpolated bit, BadNoData before the first sample — never AVEVA's own interpolation (which answered 107 Good at a NULL, null+Bad at a Good sample after it, 99.99999999944 for 100, no Interpolated bit); GatewayAtTimeReader: one ≤1,024-row window read for the batch (jumps to far-apart times use 64-row reads) + a walk-back for the first time's prev (gallop ×4, whole-remainder read after two empties, then real BISECTION seeded by any full read's last row — a dense tag stopped a year ago ≈ 25–31 reads; the first cut regrew segments after a full read and gave up at 65) + an 18-row probe for the last time's next — 10 dense times = 1 read (exact) / 3 (interpolated), 10 hourly = 12; read cap 64 + 16 per distinct time (a runaway guard: a time in a gap of a bursty tag needs ≈ 6 reads; 4/time refused a 1 h/day tag asked daily ×30) and a strict ceiling < 4 rows ⇒ BadTooManyOperations with a message naming the limit; DateTime hints stepped, non-finite/out-of-range values under an integer hint unusable; all request bounds whole ms, rows counted only strictly inside a request; the node manager passes the node's scalar type (ReadAtTimeAsync(…, TypeCode? valueType, …) default-interface overload, ignored by SqlServer/QuestDB/Null) and the Gateway converts raw rows to it before evaluating — Boolean stepped, integers interpolated + rounded half-away and returned as the node's integer type, floats Double; no hint ⇒ Double; ⚠️ the shared evaluator used to box interpolated Int64 as Double (cond ? (long)x : double widens) — fixed for SQL backends too, and Shouldly ShouldBe hides numeric-type bugs (assert ShouldBeOfType); kill switch ServerHistorian:GatewayAtTimeFromRaw (default true); docs/Historian.md §Gateway AtTime); Events over alarm-owning equipment-folder event-notifier nodes. Reads are ungated (served from any redundancy node); authorization uses the standard AccessLevels.HistoryRead bit set at materialization.

ServerHistorian appsettings keys (ServerHistorianOptions; Enabled defaults to false):

Key	Default	Notes
Enabled	false	true registers the gateway read client; false keeps NullHistorianDataSource
Endpoint	""	Absolute gateway URI, e.g. https://host:5222. Scheme selects transport (https:// = TLS, http:// = h2c)
ApiKey	""	Peppered-HMAC key histgw_<id>_<secret> sent as Authorization: Bearer. Supply via env ServerHistorian__ApiKey — never commit
UseTls	true	Connect over TLS; must match the Endpoint scheme
AllowUntrustedServerCertificate	false	Accept a self-signed / untrusted server cert (dev / on-prem only)
CaCertificatePath	null	PEM CA file pinning the gateway TLS chain; null/empty uses the OS trust store
CallTimeout	00:00:30	Per-call deadline for each unary gateway read
MaxTieClusterOverfetch	65536	Bounded over-fetch the HistoryRead-Raw paging uses to page within an oversized same-timestamp tie cluster (retained from the prior backend)
HonorReturnBounds	true	#691 kill switch: false ignores HistoryRead-Raw ReturnBounds as before (no bounding values, no modified-read Bad_InvalidArgument)
GatewayAtTimeFromRaw	true	#693 kill switch (Gateway only): false = AVEVA's own AtTime interpolation passed through, as before
EventReadHorizon	00:00:00	#694, opt-in and LOSSY: when > 0, HistoryReadEvents never reads events older than now − horizon (fresh windows clamped, both directions, all backends; a window wholly older ⇒ GoodNoData unsent). Bounds the deep "rest is empty" event probe (~3.8 s live, 30 s gateway SQL timeout unfiltered). No per-source earliest-event cache by design — late S&F drains would make it stale
HistoryReadBatchParallelism	4	Max nodes served concurrently within one HistoryRead batch (03/S3); ⌈N/P⌉ × CallTimeout worst case instead of sequential
MaxConcurrentHistoryReadBatches	16	Process-wide concurrent HistoryRead batch cap (03/S3); flood degrades to BadTooManyOperations. Total in-flight reads capped at 16 × 4 = 64
MaxTimeWeightedRawRows	1000000	SqlServer/QuestDb (#671): raw in-window rows one time-weighted aggregate read (incl. Total and, since #680, PercentGood/PercentBad) may stream into the C# evaluator — and, since #681, every Gateway per-sample read (raw-evaluated); window query capped at this + 1, past it ⇒ BadTooManyOperations for that node. ≤ 0 = unbounded. Gateway (#688): the cap + 18 budget is clamped at the gateway's default MaxValuesCeiling (1,000,000) while the cap is ≤ it, so the default no longer trips a strict (AllowStreamingBeyondCeiling=false) gateway
HistoryReadDeadline	00:01:00	Per-request HistoryRead wall-clock deadline (03/S3); a node still in flight surfaces BadTimeout. Non-positive = unbounded

Alarm-history path (AlarmHistorian section)

Alarm events are written through GatewayAlarmHistorianWriter (the gateway SendEvent path) behind the durable LocalDbStoreAndForwardSink — AlarmHistorian:Enabled=true swaps the NullAlarmHistorianSink default for the store-and-forward queue, which buffers into the node's consolidated LocalDb as the replicated alarm_sf_events table (see the LocalDb section) and whose drain worker forwards batches to the gateway and uses per-event outcomes to decide retry vs. dead-letter (never throws). Only the node holding the Primary role drains (a gated node reports HistorianDrainState.NotPrimary). The AlarmHistorian section carries only the Enabled gate + the queue knobs (DrainIntervalSeconds, Capacity, DeadLetterRetentionDays, BatchSize, MaxAttempts) — the downstream gateway connection (endpoint/key/TLS) is sourced from the ServerHistorian section. Alarm-history ReadEvents requires the target gateway deployed with RuntimeDb:EventReadsEnabled=true (the C2 SQL event-read workaround).

Continuous historization (ContinuousHistorization section)

When ContinuousHistorization:Enabled=true and ServerHistorian is configured, the Host builds a durable, crash-safe FasterLog outbox (FasterLogHistorizationOutbox) + a gateway-backed IHistorianValueWriter, and WithOtOpcUaRuntimeActors spawns the ContinuousHistorizationRecorder. The recorder taps the per-node dependency-mux value fan-out, appends each numeric value to the outbox (the crash boundary), and drains the outbox to the gateway's SQL live-value write path (WriteLiveValues) with exponential backoff. The gateway connection is sourced from ServerHistorian; this section carries only the recorder + outbox knobs:

Key	Default	Notes
Enabled	false	true (with ServerHistorian configured) wires + spawns the recorder
OutboxPath	"" (required when enabled)	Directory holding the FasterLog segment + commit files. In production set an absolute path on durable storage
CommitMode	PerEntry	PerEntry = fsync before the append returns (no loss window); Periodic = batched commits every CommitIntervalMs. ⚠️ PerEntry now means one fsync per ingest burst, not per value — the recorder appends a burst through AppendBatchAsync, which enqueues every entry then issues a single group commit; the call still does not return until that commit lands
CommitIntervalMs	100	Periodic-mode commit cadence; required > 0 only under Periodic
DrainBatchSize	1024	Entries peeked + written per drain pass. Raised from 64: the peek is an in-memory scan and the pass is bounded-parallel (not sequential), so a wide batch no longer costs tags × latency — while 64 drew down ≤ 64 entries per DrainIntervalSeconds, letting the outbox fall permanently behind during any gateway hiccup
DrainIntervalSeconds	2	Steady drain cadence (and post-success reschedule)
DrainConcurrency	8	Historian tags one drain pass writes concurrently (a batch is grouped per tag; each group is its own gateway round-trip), so a sequential tags × latency pass becomes ⌈tags / concurrency⌉ × latency with outstanding round-trips bounded. 1 = fully sequential. The ack is unchanged — all-or-nothing after every group reports, and the #538 quarantine still keys on "another group wrote this pass"
Capacity	500000	Max un-acked outbox entries before drop-oldest; 0 = unbounded. Non-zero by default since #538 — the disk backstop for a historian refusing every write, the one case the per-tag quarantine deliberately cannot resolve
MaxWriteAttempts	5	#538: consecutive failed drain passes one tag may accumulate — counted only across passes where other tags wrote successfully — before its queued entries are abandoned so the FIFO head can advance
MinBackoffSeconds	1	Initial retry backoff after a failed drain pass
MaxBackoffSeconds	30	Cap on the exponential retry backoff
CaptureWhilePrimaryUnreachable	true	#672: a standby stood down for its mesh Primary (#537) captures as soon as that Primary is reported UNREACHABLE, instead of after auto-down — crash-failover gap ~30 s ⇒ ~10–12 s, at the cost of duplicate windows (see #537/#672 below). false = pre-#672

Tag auto-provisioning (IHistorianProvisioning EnsureTags hook)

AddressSpaceApplier.Apply() fires a non-blocking, fire-and-forget IHistorianProvisioning.EnsureTagsAsync hook for added historized value tags — and, since #600.1, for changed raw tags that flip to historized or change their resolved historian name or DataType — the gateway-backed GatewayTagProvisioner calls the gateway's EnsureTags so a brand-new historized tag exists in the historian before the recorder's WriteLiveValues land. The hook is wrapped so a faulted/synchronously-throwing provisioner can never block or fail a deploy. Gateway backend: non-numeric (String/DateTime/Reference) data types are skipped (not provisioned), the recorder likewise drops + meters non-numeric values, and continuous historization is numeric-analog only (UInt16→UInt4 is a documented fallback). The SqlServer backend provisions and records every type (see "Pluggable backends" above).

The provisioner is registered by the Host's AddHistorianProvisioning (Runtime.ServiceCollectionExtensions), built via GatewayHistorian.CreateProvisioner, and gated on the same ServerHistorian:Enabled flag as the read path — WithOtOpcUaRuntimeActors resolves IHistorianProvisioning and passes it into the applier. When the section is disabled the applier binds the no-op NullHistorianProvisioning and provisioning does not run. Every dispatch logs a tally (dispatched/requested/ensured/skipped/failed); dispatched=N with requested=0 flags the dormant no-op. The API key must carry historian:tags:write for EnsureTags. (PR #423 shipped GatewayTagProvisioner but left this wiring out, so provisioning was dormant; restored 2026-06-27.) DataType mapping (issue #441): EnsureTags is an upsert (create-or-update, not skip-if-exists). HistorianTypeMapper maps Boolean → Int2 (Boolean historizes as a small 0/1 integer) because the historian's Int1 analog-tag creation path is server-degenerate — EnsureTags(Int1) stores an unusable stub (live-probed; gateway-side resolution in HistorianGateway #11 / PR #16). Both Float and Int2 are supported analog types, so a pre-existing Float Boolean tag retypes cleanly to Int2 on its next deploy — see the "Boolean historization maps to Int2" section in docs/Historian.md.

Gateway-side prerequisites

The target HistorianGateway OtOpcUa points at must run with:

• RuntimeDb:Enabled=true — enables the WriteLiveValues SQL live path (continuous historization).
• RuntimeDb:EventReadsEnabled=true — enables ReadEvents from Runtime.dbo.Events (alarm-history reads).
• RuntimeDb:EventReadMaxRows=0 (its default, unlimited) — or larger than the reverse-events walk's biggest read (2·MaxTieClusterOverfetch + 4). The cap truncates silently, so a walked-back reverse events read (#691.4) would lose the newest events of a segment it cut.
• An API key carrying scopes historian:read, historian:write, historian:tags:write.

Migration note (deployments upgrading from the Wonderware backend)

The ServerHistorian section changed shape. Rename the old Wonderware keys and supply the key via env:

Old (Wonderware) key	New (gateway) key
ServerHistorian:Host + :Port	ServerHistorian:Endpoint (https://host:5222)
ServerHistorian:SharedSecret	ServerHistorian:ApiKey (env ServerHistorian__ApiKey)
ServerHistorian:ServerCertThumbprint	ServerHistorian:CaCertificatePath (+ UseTls / AllowUntrustedServerCertificate)

The AlarmHistorian section's old Wonderware connection keys (Host/Port/UseTls/ServerCertThumbprint/SharedSecret) were pruned — remove them; the SQLite knobs are retained and the downstream connection now comes from ServerHistorian. See docs/Historian.md for the full guide.

Live-validation gate — PASSED (was "KNOWN LIMITATION 1")

No longer a limitation. The cutover was live-validated against the real gateway on wonder-sql-vd03: read ✅, write-persist ✅, alarm-send ✅, alarm-readback skipped as a confirmed protocol limitation (the captured CM_EVENT wire never carries SourceName, so the gateway cannot populate Source_Object — not a fixable bug). Evidence: docs/plans/2026-06-27-otopcua-historian-followups.md.

Kept here because it is the recipe to re-run the gate after any gateway or backend change (VPN to wonder-sql-vd03, gateway running the prerequisites above):

export HISTGW_GATEWAY_ENDPOINT=https://wonder-sql-vd03:5222   # absolute gateway URI; absent ⇒ all live tests skip
export HISTGW_GATEWAY_APIKEY=histgw_<id>_<secret>             # must carry historian:read + historian:write (+ tags:write) scopes
export HISTGW_TEST_TAG=<existing-tag>                         # read round-trip
export HISTGW_WRITE_SANDBOX_TAG=<writable-float-tag>          # e.g. HistGW.LiveTest.Sandbox — write round-trip (EnsureTags + write)
export HISTGW_BOOL_SANDBOX_TAG=<writable-bool-tag>            # e.g. HistGW.LiveTest.BoolSandbox — Boolean→Int2 EnsureTags leg + retype probe (issue #441)
export HISTGW_ALARM_SOURCE=<source-name>                      # alarm SendEvent → ReadEvents round-trip
dotnet test --filter "Category=LiveIntegration"

The live suite skips cleanly when these env vars are absent (safe to run offline on macOS). It is the gate the operator runs on the VPN before trusting the cutover.

Continuous-historization value capture — wired and live-gated (was "KNOWN LIMITATION 2"; #491 CLOSED 2026-08-04)

The ContinuousHistorizationRecorder is fully wired (actor + FasterLog outbox + gateway value-writer + meters). It is spawned with an empty initial ref set (Array.Empty<string>() in WithOtOpcUaRuntimeActors) because the deployed address space — and thus the historized-ref set — is built later at deploy time, not at actor-spawn time. The ref-feed gap is now closed: on every deploy the AddressSpaceApplier.FeedHistorizedRefs hook computes the added/removed historized-ref delta and posts it through ActorHistorizedTagSubscriptionSink → the recorder's UpdateHistorizedRefs message, and the recorder keeps the full set and re-registers its dependency-mux interest. So the recorder converges to the deployed set of historized value tags from the first deploy onward (the feed is a non-blocking, deploy-safe Tell; a faulting feed can never break a deploy).

The live gate RAN 2026-08-01 against a real HistorianGateway + a real AVEVA Historian, and it found a production-fatal defect — see docs/plans/2026-08-01-continuous-historization-live-gate.md. Three things to know:

• ⚠️ ConfigureAwait(false) in an Akka handler destroys Self. OnValueChangedAsync touched Self after awaiting IHistorizationOutbox.AppendAsync(...).ConfigureAwait(false), which discards Akka's ActorTaskScheduler synchronization context — so ActorBase.Context was gone and it threw NotSupportedException on the FIRST value. The throw reached supervision, the actor restarted, and because production seeds an empty ref set the restart discarded every historized ref: one value captured, then silence. Fixed by capturing IActorRef self = Self before the await (the idiom the GetStatus handler already used for Sender).
• ⚠️ A fake that completes synchronously hides this entire class of bug, and 13 tests passed because of one. ValueTask.CompletedTask never yields, so the continuation ran inline with the context intact. Worse, the obvious fix to the fake — await Task.Yield() — is also vacuous: Yield() posts back to the captured context, so the caller still resumes on the actor thread. The fake must complete on a pool thread (await Task.Delay(1).ConfigureAwait(false)), as a real per-entry fsync does. Only reverting the production fix exposed that the first regression test proved nothing.
• Both pair nodes historized the same value — the recorder had no Primary gate, unlike the alarm store-and-forward sink, so a redundant pair sent duplicate live values per historized tag. Filed as #537 and fixed — see the paragraph below.

Gate COMPLETE — #491 CLOSED 2026-08-04. Outbox append, drain, WriteLiveValues, restart-convergence, EnsureTags provisioning (ensured=2 failed=0) and read-back all ✅. The read-back leg ran against the licensed wonder-sql-vd03 with the VPN up: two registers carrying distinct non-zero constants (8224, 120) each read back as its own value — a 0-valued source would have proved nothing (the #536 trap).

Both gate findings are now CLOSED and separately live-gated — #537 (Primary gate) and #538 (poison tag); evidence in the same plan doc. Two things to know before touching either.

• #538 — the discriminator is "did another tag write this pass?", NOT a failure count. DrainOnceAsync writes every per-tag group and acks all-or-nothing (deliberate: RemoveAsync truncates through an id, so partial acking could drop older unwritten entries). One permanently-failing tag therefore pinned the FIFO head while every healthy tag was re-submitted on every pass, forever — duplicate samples, an outbox that never truncated (observed 8.3 MB; signature = the gateway's ok and error counters climbing together). The drain now quarantines a tag after MaxWriteAttempts consecutive failures, but ⚠️ only counts a failure on a pass where some other tag succeeded — otherwise an ordinary gateway restart (where everything fails) would discard the entire backlog, converting a duplicate-samples bug into a lost-history one. A count-only quarantine passes the poison-tag test and fails nothing else, so the load-bearing test is the falsification control (A_backend_that_fails_every_tag_abandons_nothing). Abandonment is also deferred until every failing tag is over budget, so nothing is discarded on a pass that could not have advanced the head anyway.
• #537 — it gates the CAPTURE, not the drain, and that is the opposite of the alarm sink. The alarm queue is in LocalDb and replicates, so the Secondary holds the Primary's rows and gating the drain loses nothing. The historization outbox is a node-local FasterLog replicating to nobody: gating only its drain leaves the standby appending forever to a queue no one empties (unbounded disk, then a flood of stale duplicates on promotion). ⚠️ And the peer that matters is one recording the SAME refs, not one holding the same rows — a node stands down only for a Primary in its own per-cluster mesh, and a node with no cluster-scoped role never stands down at all. Under a fleet-wide election the elected Primary is usually running another cluster's configuration, and deferring to it ends this cluster's historization silently (the drain reports success on an empty outbox forever) — the same scope mismatch that once suspended the alarm drain on every node of the rig at once. An unknown role captures. New verdict on RedundancyRoleView behind IHistorizationRoleView, registered unconditionally on driver nodes (NOT alongside the alarm IRedundancyRoleView, which only exists when that section is enabled — the #511 registration accident).
  • #672 — an UNREACHABLE recording Primary does not count (crash-failover gap). A SIGKILLed Primary stays a member (unreachable, not downed) for 10 s failure detection + 15 s auto-down, and the redundancy singleton died with it, so the standby's cached snapshot kept naming it Primary and nobody captured for ~30 s. DriverHostActor now holds its own ReachabilityEvent (+MemberUp) subscription — not inside StartPrimaryVerdictPublishing, which early-returns with no primary-role source — handled in all three behaviours, caches the last snapshot, and on a reachability change re-derives and republishes only the historization verdict with unreachable peers excluded. Gap ⇒ the failure-detector window (~10–12 s). Gated by ContinuousHistorization:CaptureWhilePrimaryUnreachable (default true). ⚠️ The alarm drain and ShouldServiceAsPrimary/PublishPrimaryVerdict must NOT adopt this (a false allow there is a dual claimant, not a duplicate) — the tests assert both stay put. ⚠️ A removed member deliberately stays marked unreachable (only MemberUp at that address clears it): un-marking it would re-derive from a cached snapshot still naming it Primary and close the gate on a dead node. Cost: duplicates while a live Primary is merely unreachable and under a real partition — idempotent on SqlServer/QuestDB only for source-timestamped drivers (Modbus stamps each node's own poll time ⇒ duplicate rows on every backend); Gateway stores duplicates.

Re-confirmed live in 2026-06 and since fixed (#600.1): EnsureTags provisioning used to fire only for newly-ADDED historized tags, so flipping an existing tag's historianTagname (or historization, or DataType) provisioned nothing (2026-06-27-historian-gateway-integration-issues.md Issue 5). It now also fires for those changed raw tags.