Files
lmxopcua/docs/Historian.md
T
Joseph Doherty e7fbe97759 fix(historian): gate historization capture on the Primary, quarantine a poison tag (#537, #538)
Both defects were found by #491's live gate against a real HistorianGateway and filed rather than
fixed at the time, because each one's fix is a design decision rather than a patch.

#537 — both nodes of a redundant pair captured every value, so the historian received duplicate
live samples for every historized tag (measured: the gateway's write counter advanced twice per
value and halved when one central was stopped).

The recorder now reads a capture gate per value, fed from the new
PrimaryGatePolicy.ShouldRecordHistorizedValues through IHistorizationRoleView — a third verdict on
the same RedundancyRoleView object, registered unconditionally on driver nodes rather than
alongside the alarm section's IRedundancyRoleView (which only exists when that section is enabled;
piggybacking is the #511 registration accident).

Two things are deliberate and easy to get backwards:

  * It gates the CAPTURE, not the drain — the opposite of the alarm sink. That queue is in LocalDb
    and replicates, so the Secondary already holds the Primary's rows. The historization outbox is
    a node-local FasterLog replicating to nobody, so a drain-only gate would leave the standby
    appending forever to a queue no one empties: unbounded disk, then a flood of stale duplicates
    the moment it is promoted. Entries a node already captured are still drained by that node.

  * The peer it stands down for must RECORD THE SAME REFS, not merely hold the same rows — a
    Primary in this node's own per-cluster mesh. A node carrying no cluster-scoped role cannot
    establish that and never stands down: under a fleet-wide election the elected Primary is
    usually running a different cluster's configuration, and deferring to it would end this
    cluster's historization silently, with the drain reporting success on an empty outbox forever.
    That is the scope mismatch that once suspended the alarm drain on every node of the rig.

An unknown role captures, matching the alarm drain — a duplicate sample is recoverable, a hole in
the trend is not.

#538 — the drain writes every per-tag group and then acks all-or-nothing (deliberate: RemoveAsync
truncates THROUGH an id, so partial acking could drop older unwritten entries). One permanently
failing tag therefore pinned the FIFO head while every healthy tag in the batch was re-submitted
to the historian on every pass, indefinitely — duplicate samples and an outbox that never
truncated, observed live at 8.3 MB from a single unprovisionable tag.

The drain now tracks consecutive failures per historian tag and abandons that tag's entries once
it exceeds MaxWriteAttempts (new, default 5), so the head advances. The counter only advances on a
pass where some OTHER tag wrote successfully: that is what separates "this tag is poison" from
"the backend is down". A count-only quarantine looks identical on the poison-tag case and silently
discards the whole backlog during an ordinary gateway restart, turning a duplicate-samples bug
into a lost-history one. Abandonment is also deferred until every failing tag is over budget, so
nothing is discarded on a pass that could not have advanced the head anyway. Each abandonment logs
a Warning naming the tag; the total rides RecorderStatus.AbandonedEntries.

Capacity also gains a non-zero default (500,000 entries) as the disk backstop for the one case the
quarantine deliberately cannot resolve — a historian refusing every write, where retaining is
correct. Set 0 to restore unbounded retention.

Every new test was falsified by reverting the production change it guards, and each one reddens
alone:

  * drop the backend-reachable evidence -> only A_backend_that_fails_every_tag_abandons_nothing
  * never abandon -> only A_permanently_failing_tag_is_abandoned_so_healthy_tags_stop_being_rewritten
  * latch the capture gate at construction -> only The_capture_gate_is_re_read_per_value...
  * treat any Primary as a scope peer -> only the two scope-mismatch cases

Full suite green. (Four OpcUaClient.Browser failures during the run were the opc-plc fixture being
down on the shared Docker host; brought it back up and they pass.)

Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo
2026-08-04 12:12:16 -04:00

36 KiB
Raw Blame History

Historian — server-side OPC UA HistoryRead (Phase C)

Phase C wires server-side OPC UA HistoryRead for authored equipment tags flagged historized. The feature is driver-agnostic: any equipment tag (Galaxy, Modbus, OpcUaClient, or any other driver) can be marked historized; the server dispatches all history reads to the registered IHistorianDataSource — the HistorianGateway read client (GatewayHistorianDataSource, talking gRPC to the external ZB.MOM.WW.HistorianGateway via the ZB.MOM.WW.HistorianGateway.Client package). No EF migration is required; the historian flag rides in the existing schemaless TagConfig JSON blob alongside the Phase B alarm object. (The bespoke Wonderware TCP sidecar backend this replaced was retired — see drivers/Historian.Wonderware.md.)

Design reference: docs/plans/2026-06-14-galaxy-phase-c-historian-design.md.


Historized TagConfig schema

A tag is historized by setting the Historize this tag checkbox and optional Historian tagname (override) textbox in the Tag modal on the /uns equipment page Tags tab. These controls work for all drivers — typed editors (Modbus, S7, OpcUaClient, etc.) and the raw-JSON textarea (Galaxy) alike. The controls merge isHistorized / historianTagname into the existing TagConfig JSON blob via the TagHistorizeConfig helper, preserving all other keys byte-stable. Drivers that still use the raw-JSON editor (Galaxy) can also add the fields directly in the textarea.

Fields

Field Type Required Description
isHistorized bool yes Marks the tag historized. Materialises the OPC UA node with Historizing=true and the HistoryRead AccessLevel bit.
historianTagname string no Explicit historian tagname to query. When absent or empty, defaults to the tag's driver FullName (the TagConfig.FullName).

Examples

Default historian tagname (uses FullName = TestMachine_002.TestFloat):

{"FullName":"TestMachine_002.TestFloat","isHistorized":true}

Explicit historian tagname override:

{"FullName":"TestMachine_002.TestFloat","isHistorized":true,"historianTagname":"Plant.Line1.Flow"}

A tag that is both historized and a native alarm:

{"FullName":"TestMachine_002.HiAlarm","isHistorized":true,"alarm":{"alarmType":"OffNormalAlarm","severity":700}}

ServerHistorian configuration

The ServerHistorian section in appsettings.json controls the historian read path. Enabled defaults to false; when disabled, the server registers NullHistorianDataSource and all HistoryRead calls on historized nodes return GoodNoData (empty, not an error).

{
  "ServerHistorian": {
    "Enabled": false,
    "Endpoint": "",
    "ApiKey": "",
    "UseTls": true,
    "AllowUntrustedServerCertificate": false,
    "CaCertificatePath": null,
    "CallTimeout": "00:00:30",
    "MaxTieClusterOverfetch": 65536,
    "HistoryReadBatchParallelism": 4,
    "MaxConcurrentHistoryReadBatches": 16,
    "HistoryReadDeadline": "00:01:00"
  }
}
Key Type Default Description
Enabled bool false Enable the live GatewayHistorianDataSource. false → NullHistorianDataSource (empty reads).
Endpoint string "" Absolute gateway URI, e.g. https://host:5222. Scheme selects transport (https:// = TLS, http:// = h2c plaintext). Required when Enabled.
ApiKey string "" The gateway peppered-HMAC key (histgw_<id>_<secret>) sent as Authorization: Bearer. Required when Enabled. Supply via env ServerHistorian__ApiKey.
UseTls bool true Connect over TLS; must match the Endpoint scheme.
AllowUntrustedServerCertificate bool false Accept a self-signed / untrusted server certificate (dev / on-prem only).
CaCertificatePath string|null null PEM CA file pinning the gateway's TLS chain. Null/empty uses the OS trust store.
CallTimeout TimeSpan 00:00:30 Per-call deadline applied to each unary gateway read. A read for a tag the historian does not yet know about — e.g. a freshly-historized tag that has not been provisioned (see Tag auto-provisioning) — can run to this full deadline before the gateway errors, so an unprovisioned-tag HistoryRead can block for ~30 s. Lower this to fail faster, at the cost of truncating legitimately long reads.
MaxTieClusterOverfetch int 65536 Maximum samples the server will fetch in one shot to page through a tie cluster (multiple samples sharing one SourceTimestamp). A cluster larger than this ceiling fails BadHistoryOperationUnsupported. Raise to handle abnormally large tie clusters; the default covers all normal-data cases.
HistoryReadBatchParallelism int 4 Max historized nodes served concurrently within one HistoryRead batch (arch-review 03/S3). The SDK's HistoryRead* override surface is synchronous, so each arm bridges once at its boundary; this turns per-node serving from sequential (N × CallTimeout worst case) into ⌈N / P⌉ × CallTimeout. Also caps the gateway-load multiplication a single client can command. ≤ 1 serves sequentially.
MaxConcurrentHistoryReadBatches int 16 Process-wide limit on HistoryRead batches served concurrently (arch-review 03/S3). A flood of history clients degrades to BadTooManyOperations rather than exhausting the SDK request-thread pool. With HistoryReadBatchParallelism this caps total in-flight gateway reads at MaxConcurrentHistoryReadBatches × HistoryReadBatchParallelism (default 16 × 4 = 64). A batch that cannot get a slot within min(HistoryReadDeadline, 5 s) fails every handle BadTooManyOperations.
HistoryReadDeadline TimeSpan 00:01:00 Per-request wall-clock deadline for a HistoryRead batch (arch-review 03/S3). A node still in flight at the deadline surfaces BadTimeout, so one request can never hold an SDK request thread longer than this regardless of node count. Non-positive = unbounded (a Validate() warning).

Do not commit ApiKey to appsettings.json. Set it via the environment variable ServerHistorian__ApiKey, a secrets store, or a deployment-time overlay. The checked-in default is always empty.

Gateway-side prerequisites. The target gateway must run RuntimeDb:Enabled=true (continuous WriteLiveValues) + RuntimeDb:EventReadsEnabled=true (alarm-history ReadEvents), and the API key must carry the scopes historian:read, historian:write, historian:tags:write.

Migration from the Wonderware backend. Rename the old keys: Host/Port → Endpoint (https://host:5222); SharedSecret → ApiKey (env ServerHistorian__ApiKey); ServerCertThumbprint → CaCertificatePath (+ UseTls / AllowUntrustedServerCertificate).

The ServerHistorian section is independent of the AlarmHistorian section (the alarm write path) and the ContinuousHistorization section (driver-value capture). All three target the same gateway — but only ServerHistorian carries the connection (endpoint/key/TLS); the other two source it from there.

Continuous-historization value quality is server-stamped. The gateway's WriteLiveValues SQL path discards the quality the recorder sends (a server-managed column; 0.2.0 Contracts C-002), so persisted quality is stamped gateway-side regardless of what the ContinuousHistorizationRecorder provides (it hardcodes OPC-DA Good, 192). Capturing real per-node quality will not persist until the gateway honors the field — see GatewayHistorianValueWriter's remarks (archreview 06/C-7).

✅ Continuous value capture is LIVE-PROVEN END-TO-END (#491 CLOSED — gate run 2026-08-01, completed 2026-08-04; see docs/plans/2026-08-01-continuous-historization-live-gate.md). Against a real ZB.MOM.WW.HistorianGateway talking to the real AVEVA Historian on wonder-sql-vd03: deploy → EnsureTags provisioning → dependency-mux value delta → FasterLog outbox append → drain → WriteLiveValues → value read back from the historian, plus restart convergence (the recorder re-registers the deployed ref set after a process restart, with no new deployment).

The read-back used two registers carrying distinct non-zero constants (8224, 120), each of which read back as its own value — a 0-valued source cannot distinguish a correct value from a wrong one.

⚠️ The gate found a production-fatal defect: the recorder threw NotSupportedException on the first value it ever appended, because it touched Self after an await …ConfigureAwait(false), which discards Akka's ActorContext. Supervision restarted the actor, and since the recorder is seeded with an empty ref set, the restart discarded every historized ref — historization stopped after one value. Fixed; pinned by a regression test whose outbox fake completes on a pool thread (a fake returning ValueTask.CompletedTask, or even await Task.Yield(), cannot reproduce it).

The gate also found the two defects below, both since fixed.

Related, separately open: EnsureTags provisioning fires only for newly-added historized tags (AddressSpaceApplier iterates the added set), so flipping an existing tag to "isHistorized": true never provisions it — see docs/plans/2026-06-27-historian-gateway-integration-issues.md Issue 5.


A poison tag no longer pins the outbox (#538)

DrainOnceAsync peeks a batch, writes every per-tag group in it, and then acks all-or-nothing — because the outbox's RemoveAsync truncates the FIFO head through the acked id, so acking one tag's entries could silently drop older, unwritten entries belonging to another. That much is deliberate and unchanged.

What was wrong is what happened when a tag failed permanently: the head never advanced, so every healthy tag in the batch was re-submitted to the historian on every drain pass, indefinitely — duplicate samples at identical timestamps, and an outbox that never truncated (observed live at 8.3 MB from a single unprovisionable tag, with the gateway's ok and error counters climbing together — the signature of this bug).

The drain now tracks consecutive failures per historian tag and abandons a tag's queued entries once it exceeds ContinuousHistorization:MaxWriteAttempts (default 5), letting the head advance. Two properties matter:

  • A pass where NOTHING wrote abandons nothing. The counter only advances on a pass where at least one other tag wrote successfully — that is what distinguishes "this tag is poison" from "the backend is down". A count-only quarantine looks identical on the poison-tag case and silently discards the whole backlog during an ordinary gateway restart, turning a duplicate-samples bug into a lost-history one. Pinned by a falsification-control test (A_backend_that_fails_every_tag_abandons_nothing) that goes red when the evidence check is removed while the poison-tag test stays green.
  • A tag over budget waits for the others. Abandonment is deferred until every failing tag in the batch is over budget, so a tag is never discarded on a pass that could not have advanced the head anyway.

Each abandonment logs a Warning naming the tag and the discarded sample count, and the total rides RecorderStatus.AbandonedEntries. The most common cause is a tag the historian has no definition for — easy to reach, because EnsureTags provisioning runs only for newly-added historized tags, so a changed historianTagname is never provisioned (Issue 5 above).

ContinuousHistorization:Capacity now also defaults to 500,000 entries rather than unbounded. It is the disk backstop for the one case the quarantine deliberately cannot resolve — a historian refusing every write, where retaining is correct. Set 0 to restore unbounded retention.

Only the recording Primary captures (#537)

Both nodes of a redundant pair serve the same address space and see the same dependency-mux fan-out, so an ungated recorder wrote every sample twice (measured: the gateway's write counter advanced twice per value with both centrals up, and halved when one was stopped).

The recorder now reads a capture gate per value, fed from PrimaryGatePolicy.ShouldRecordHistorizedValues via IHistorizationRoleView — the same shape as the alarm store-and-forward sink's drainGate, with two deliberate differences:

  • It gates the CAPTURE, not the drain. The alarm queue lives in LocalDb and replicates, so the Secondary already holds the Primary's rows and gating the drain loses nothing. The historization outbox is a node-local FasterLog replicating to nobody: gating only its drain would leave the standby appending forever to a queue no one empties — unbounded disk, and a flood of stale duplicates the moment it is promoted. Entries a node already captured are still drained by that node, so a demotion loses nothing recorded.
  • The relevant peer is one that records the same refs, not one that holds the same rows. A node stands down only for a Primary in its own per-cluster mesh. A node carrying no cluster-scoped role cannot establish that, so it never stands down: under a fleet-wide election the elected Primary is usually running a different cluster's configuration, and deferring to it would end this cluster's historization silently — the drain would report success on an empty outbox forever. That is the same scope mismatch that once suspended the alarm drain on every node of the docker-dev rig at once.

An unknown role captures, matching the alarm drain: a duplicate sample is recoverable, a hole in the trend is not. The residual cost is a capture gap across a failover, bounded by how long the newly promoted node takes to see its first redundancy snapshot — inherent to any non-duplicating design. Suppressed values are counted in RecorderStatus.SuppressedNotPrimary, and each gate transition logs once.


Tag auto-provisioning (EnsureTags)

When a deploy adds a historized tag, the server auto-ensures it exists in the historian via the gateway's EnsureTags before the first value or HistoryRead lands. AddressSpaceApplier.Apply() fires a non-blocking, fire-and-forget IHistorianProvisioning.EnsureTagsAsync for each added historized value tag (with the resolved historian name — the historianTagname override when set, else the driver FullName); the hook is wrapped so a faulted, slow, or throwing provisioner can never block or fail a deploy. Non-numeric (String/DateTime/Reference) types are skipped, not provisioned.

Wiring + gating. The concrete provisioner is the gateway-backed GatewayTagProvisioner, registered by the Host's AddHistorianProvisioning (GatewayHistorian.CreateProvisioner) and gated on the same ServerHistorian:Enabled flag as the read path — both target the same single gateway. When the section is disabled (or absent) the applier resolves the no-op NullHistorianProvisioning and provisioning does not run (so on a read-only-historian deployment, ensure tags exist by some other means). The Host requires the API key to carry the historian:tags:write scope for EnsureTags to succeed.

Observability. Every dispatch logs an Information tally — historian provisioning completed (dispatched=N, requested=…, ensured=…, skipped=…, failed=…). A line showing dispatched=N but requested=0 indicates the no-op provisioner ran (provisioning is not wired for this node), which is the signal to check that ServerHistorian:Enabled=true.

Why this matters for HistoryRead latency. Until a tag is provisioned, the historian does not know it, so a HistoryRead on it runs to the full CallTimeout (~30 s) before erroring. Auto-provisioning on deploy keeps freshly-historized tags from hitting that slow path. See the CallTimeout row above.

Boolean historization maps to Int2 (issue #441)

EnsureTags is contract-documented (HistorianGateway 0.2.0+) as an upsert — "creates or updates historian tags according to the given definitions" — that maps per-tag onto AVEVA's HistorianClient.EnsureTagAsync(definition) -> bool. It is not idempotent (not routed through the idempotent retry pipeline) and not skip-if-exists.

HistorianTypeMapper maps DriverDataType.Boolean → HistorianDataType.Int2 and Boolean values historize as a small 0/1 integer. Int2 is used rather than the semantically-exact Int1 because the historian's Int1 analog-tag creation path is server-degenerate — EnsureTags(Int1) stores an unusable stub, so a Boolean tag mapped to Int1 never becomes historizable. This is an evidence-based decision from a live probe against the deployed 2023 R2 historian; the root cause and gateway-side resolution are tracked in issue #441 and HistorianGateway #11 / PR #16 (which documents Int1 as an explicit, evidence-based exclusion). The value-write path is unaffected: the recorder sends every value as a double on WriteLiveValues regardless of tag type, so a Boolean 0.0/1.0 round-trips cleanly into the Int2 tag.

Observed live (2026-07-13, real 0.2.0 gateway → real AVEVA historian on wonder-sql-vd03, probed via grpcurl — archreview R2-06 T12):

  • EnsureTags is a real create-or-update upsert — re-EnsureTags of an existing tag at the same type returns Success=true.
  • An in-place retype between SUPPORTED types is ACCEPTED — Float→Double on an existing tag returns Success=true.
  • A type sweep confirmed Int2/UInt2/Int4/UInt4/Float/Double all provision Success=true; only Int1 fails (Success=false, Error="ProtocolEvidenceMissingException" — the vendored client's "no capture evidence for this type" guard → gRPC Unimplemented), both for a fresh tag and as a retype target. Mapping Boolean to Int2 sidesteps that gap entirely.

Prerequisite for provisioning to land at all: the provisioner must stamp a positive StorageRateMs — a 0 made EnsureTags throw ArgumentOutOfRangeException and fail every provision (fixed in GatewayTagProvisioner, PR #439).

Migration note for pre-existing Boolean tags. A Boolean tag historian-side as Float (e.g. from the 0.1.0-era default) is asked, on its next deploy, to upsert to Int2. Both are supported analog types, so the retype is accepted. Provisioning is fire-and-forget and never blocks/fails a deploy; a per-tag rejection (if any) surfaces only as the tally's failed count, and values written to a still-Float tag remain readable (Float supersets the 0/1 range) pending reconcile. A bulk retype sweep is not done in code.


HistoryRead behavior

Read variants

The server supports all four OPC UA HistoryRead variants:

Variant Node type CLI --aggregate
Raw Historized variable (omit --aggregate)
Processed Historized variable Average, Minimum, Maximum, Total, Count
AtTime Historized variable n/a (client supplies exact timestamps)
Events Equipment folder (event notifier) n/a

Variable nodes (historized tags) serve Raw, Processed, and AtTime history. Historizing=true and AccessLevels.HistoryRead are set at materialization so any compliant OPC UA client can discover historized capability from the node's attributes.

v3 Batch 4 — dual-namespace registration. A historized raw tag surfaces as two variable nodes: the Raw-namespace node (ns=Raw, s=<RawPath>) and, for each referencing equipment, a UNS-namespace node (ns=UNS, s=<Area>/<Line>/<Equipment>/<EffectiveName>). Both NodeIds register the SAME historian tagname (the raw tag's FullName / historianTagname), so HistoryRead against either NodeId resolves to the same tagname and returns the same series. The historization intent is single-sourced — the mux's HistorizedTagRef set stays keyed by RawPath (no doubles), and continuous historization / EnsureTags provisioning run once per raw tag regardless of how many equipment reference it.

Equipment-folder event-notifier nodes serve Event history. Every equipment folder that owns at least one alarm condition is already an event notifier; the server registers a sourceName (the equipment id) for each such folder and maps event history reads to the HistorianGateway using that source. (Alarm-history ReadEvents requires the gateway running RuntimeDb:EventReadsEnabled=true.) Event-field projection supports the standard BaseEventType select clauses — EventId, SourceName, Time, ReceiveTime, Message, and Severity; an unsupported select operand returns a null field (spec-conformant).

OpcUaClient driver — upstream passthrough for all four variants

The OpcUaClient driver's IHistoryProvider implementation forwards all four history-read variants (Raw, Processed, AtTime, and Events) to its upstream OPC UA server. For the Events variant it sends a fixed canonical BaseEventType EventFilter selecting the standard six fields (EventId, SourceName, Time, ReceiveTime, Message, Severity) and maps the upstream HistoryEvent onto HistoricalEvent — the same six-field projection the OtOpcUa node-manager itself projects when serving event history. This is a driver-level capability: the OpcUaClient driver acts as a passthrough to whatever historian the upstream server exposes, and is independent of the single server-side IHistorianDataSource backend (GatewayHistorianDataSource / NullHistorianDataSource) that the OtOpcUa node-manager dispatches HistoryRead to for tags on other drivers (Galaxy, Modbus, S7, etc.).

Graceful degradation

Situation HistoryRead status
Historized node, historian configured and reachable, results found Good
Historized node, historian configured, time range empty GoodNoData
Historized node, historian NOT configured (Enabled=false / Null source) GoodNoData (empty)
Non-historized node BadHistoryOperationUnsupported
Backend timeout or exception BadHistoryOperationUnsupported per node; other nodes in the same batch are unaffected

A historized node with no historian configured never returns an error status — it returns empty. This means a deployment can author and publish historized tags before the historian gateway is provisioned, without producing error spikes in connected clients.

Continuation-point paging (Raw)

HistoryRead-Raw is paged server-side. The historian backend is single-shot (it returns up to NumValuesPerNode samples with no continuation point of its own), so the server synthesises paging time-based:

  • A page that returns exactly NumValuesPerNode samples (with NumValuesPerNode > 0) MAY have more behind it, so the server stores a resume cursor and returns an opaque continuation point (16 bytes). The client hands it back to fetch the next page.
  • A short page (fewer than the cap) is the last page — no continuation point.
  • NumValuesPerNode == 0 ("all values, no limit") is never paged; the whole window returns in one shot.
  • The resume cursor is tie-safe: the next page resumes from the last returned sample's SourceTimestamp inclusive and drops the boundary samples already emitted, so samples sharing the boundary timestamp are neither duplicated nor skipped.

Oversized tie clusters — within-timestamp paging. When more samples share one SourceTimestamp than the current page cap, the server detects that the cursor has stalled on a tie cluster (the last returned timestamp equals the resume timestamp). It then over-fetches the entire cluster at that single timestamp up to a bounded ceiling controlled by ServerHistorian:MaxTieClusterOverfetch (default 65 536), then serves the cluster NumValuesPerNode samples at a time across successive pages, advancing the cursor one tick past the timestamp once the cluster is fully drained.

Short pages within a cluster still carry a continuation point. A within-cluster page that returns fewer than NumValuesPerNode samples (because the cluster happened to be smaller than the cap, or is the final partial batch) is not the last page if the cluster itself has not been fully emitted — the server retains the continuation point so the client can drain the remainder. Only when the cluster is exhausted and the cursor has advanced past the timestamp does the short-page rule apply.

Cluster larger than MaxTieClusterOverfetch. If the over-fetch itself reaches the ceiling without spanning the full cluster, the node fails loudly with BadHistoryOperationUnsupported and the tag + timestamp + ceiling are logged. Remedies: raise MaxTieClusterOverfetch (or NumValuesPerNode) to cover the full cluster, or investigate the data anomaly (raw samples normally carry strictly increasing distinct timestamps).

For a single tag's raw history a tie cluster larger than the default 65 536 is a severe data anomaly. The ceiling exists to bound server-side memory on pathological data, not to cap normal operation.

Continuation points are bound to the OPC UA session (the SDK's ServerConfiguration.MaxHistoryContinuationPoints cap, default 100, with oldest-eviction; points are disposed when the session closes). Resuming an unknown / evicted / released point returns BadContinuationPointInvalid. releaseContinuationPoints drops the stored cursors without reading data.

Total aggregate

The OPC UA Total aggregate is supported over the HistorianGateway backend. The gateway exposes a native Integral retrieval mode, so Total maps straight to it (HistoryAggregateType.Total → RetrievalMode.Integral) — no client-side scaling. (This replaces the retired Wonderware path, which had no Total column and derived it client-side as time-weighted Average × interval-seconds.) Count is likewise a native gateway mode. Bucket status codes and timestamps are preserved unchanged; empty / null buckets surface as BadNoData.

Known limitations

  • Processed and AtTime are single-shot (no continuation points). Unlike Raw, neither ReadProcessedDetails nor ReadAtTimeDetails carries a client count cap (NumValuesPerNode): the Processed bucket count is deterministic (window / interval) and AtTime returns exactly one sample per requested timestamp, so the single-shot backend returns the complete result in one read and there is no "full page ⇒ maybe more" signal to page on. Returning the full result with no continuation point is spec-conformant.
  • No modified-value history (HistoryReadModified). Requests for modified values return BadHistoryOperationUnsupported. This is infra-gated, not a server-code gap: the HistorianGateway backend (GatewayHistorianDataSource) exposes only a current-value read path — there is no modified/edited-history surface to source the data from. The server-side override is in place (it cleanly rejects modified reads per node) and IsReadModified is honoured; serving real modified-value history is unblocked only once the gateway grows a modified-read RPC. Until then, rejecting is the correct, spec-conformant behaviour.

Redundancy and authorization

History reads are served from any node — there is no Primary gate. Authorization is the standard OPC UA HistoryRead permission enforced by the SDK through the AccessLevels.HistoryRead bit set at materialization. A session without sufficient permissions receives BadUserAccessDenied from the SDK before the dispatch reaches the historian.


Authoring workflow

  1. Open the equipment's Tags tab on /uns/equipment/{id}.
  2. Create or edit the tag.
    • For typed-editor drivers (Modbus, S7, OpcUaClient, etc.): check the Historize this tag checkbox and, if needed, fill in the Historian tagname (override) textbox.
    • For raw-JSON editors (Galaxy): you can check the same first-class checkboxes (they appear below the JSON textarea), or add "isHistorized":true (and optionally "historianTagname":"...") directly in the textarea.
  3. Save and publish. The server rebuilds its address space; the node materialises with Historizing=true and the HistoryRead AccessLevel bit.
  4. Confirm with Client.CLI read that the node's Status is Good and that the value is updating. Then issue a historyread to verify the historian connection returns data.

Native-alarm historian opt-out (alarm.historizeToAveva)

A tag carrying a native "alarm" object has a separate historian opt-out for its alarm transitions (distinct from tag-value historization). On the Tag modal, check or uncheck Historize to AVEVA in the alarm section. This maps to alarm.historizeToAveva (bool?) in the TagConfig JSON:

  • Absent or true (default) — the alarm's transitions are written to AVEVA Historian via HistorianAdapterActor.
  • false — the durable AVEVA write is suppressed for this alarm's transitions. The live /alerts feed and OPC UA condition events are unaffected.

The gate is applied in HistorianAdapterActor using is not false semantics, matching the scripted-alarm HistorizeToAveva posture. See ScriptedAlarms.md §Native driver alarms and AlarmTracking.md §Historian write-back for the full alarm-historian routing.


Client.CLI historyread examples

The historyread command reads historical data from any node. Supply start and end times in ISO 8601 UTC form. See docs/Client.CLI.md for the full flag reference.

v3 Batch 4 NodeIds. A historized tag is readable through either of its two NodeIds — the Raw-namespace node (s=<RawPath>, e.g. Plant/Modbus/dev1/Speed) or a referencing equipment's UNS-namespace node (s=<Area>/<Line>/<Equipment>/<EffectiveName>). Both register the same historian tagname, so the returned series is identical. The ns=N index is assigned at connect time per the namespace URI (https://zb.com/otopcua/raw / https://zb.com/otopcua/uns) — resolve it from the server's NamespaceArray rather than hard-coding it. The examples below show a Raw-namespace read (ns=2 illustrative).

# Raw history for a historized tag via its Raw-namespace NodeId (last 24 hours by default)
otopcua-cli historyread \
  -u opc.tcp://localhost:4840/OtOpcUa \
  -n "ns=2;s=Plant/Modbus/dev1/Speed" \
  --start "2026-06-13T00:00:00Z" --end "2026-06-14T00:00:00Z"

# Same series via a referencing equipment's UNS-namespace NodeId, limited to 100 values
otopcua-cli historyread \
  -u opc.tcp://localhost:4840/OtOpcUa \
  -n "ns=3;s=filling/line1/station1/Speed" \
  --start "2026-06-13T00:00:00Z" --end "2026-06-14T00:00:00Z" \
  --max 100

# 1-hour average aggregate
otopcua-cli historyread \
  -u opc.tcp://localhost:4840/OtOpcUa \
  -n "ns=2;s=Plant/Modbus/dev1/Speed" \
  --start "2026-06-13T00:00:00Z" --end "2026-06-14T00:00:00Z" \
  --aggregate Average --interval 3600000

# Authenticated read (ReadOnly role or higher required)
otopcua-cli historyread \
  -u opc.tcp://localhost:4840/OtOpcUa \
  -n "ns=2;s=Plant/Modbus/dev1/Speed" \
  --start "2026-06-13T00:00:00Z" --end "2026-06-14T00:00:00Z" \
  -U reader -P password

Supported --aggregate values: Average, Minimum, Maximum, Count, Start, End, StandardDeviation (aliases: avg, min, max, stddev/stdev, first, last). --interval is the processing interval in milliseconds (default 3600000 = 1 hour). The Total aggregate is served by the server's OPC UA HistoryRead endpoint for any conformant client (derived as time-weighted Average × interval-seconds — see "Total aggregate derivation" above), but is not exposed by this bundled CLI.


Live /run gate

The live read gate requires a reachable ZB.MOM.WW.HistorianGateway (VPN to wonder-sql-vd03) with the AVEVA Historian behind it healthy. Set ServerHistorian:Enabled=true with the correct Endpoint (https://host:5222) and supply ServerHistorian__ApiKey via the environment, then deploy and publish at least one historized Galaxy tag. The gate is operator-driven — it is not part of the local docker-dev rig. The gateway-backed driver also ships an env-gated live suite (Category=LiveIntegration); see the HISTGW_GATEWAY_ENDPOINT / HISTGW_GATEWAY_APIKEY / HISTGW_TEST_TAG / HISTGW_WRITE_SANDBOX_TAG / HISTGW_BOOL_SANDBOX_TAG / HISTGW_ALARM_SOURCE env vars (it skips cleanly when they are absent). HISTGW_BOOL_SANDBOX_TAG (e.g. HistGW.LiveTest.BoolSandbox, a dedicated writable Boolean tag — never the shared Float write sandbox) drives the Boolean historization leg (EnsureTags_boolean_sandbox_provisions_as_Int2) and the retype probe (EnsureTags_type_change_outcome_is_observable) that back the Boolean historization maps to Int2 decision. Running the full suite against the gateway (all six tests, none skipped) also discharges 06/U-2's "full documented run" for the historian suite.

See AlarmHistorian.md for the alarm write path and ServiceHosting.md for the (external) HistorianGateway deployment.



Cross-driver TagConfig key reference

The TagConfig JSON blob is the single extension surface for per-tag server-side behaviours across all drivers. Keys from multiple phases coexist in the same blob; unknown keys are preserved byte-stable on round-trip by all typed editors.

Historian + alarm keys (Phase B / Phase C)

Key Type Phase Description
isHistorized bool C Marks the tag historized (HistoryRead + Historizing=true).
historianTagname string C Explicit historian tagname override (defaults to FullName).
alarm object B Native alarm definition (alarmType, severity, historizeToAveva, …).

Array keys (Phase 4c)

Key Type Phase Description
isArray bool 4c When true and arrayLength >= 1, materialises the node as a 1-D array (ValueRank=OneDimension). Absent or false → scalar.
arrayLength uint (≥1) 4c Element count; sets ArrayDimensions. Required when isArray: true.

Cross-driver array coverage matrix

Driver Read mechanism Live-verify status
Modbus Contiguous FC03/FC04 block; String/BitInRegister modes supported Mac-verifiable (sim 10.100.0.35:5020)
S7 ReadBytesAsync block + per-element decode loop Unit-proven (fixture down)
AB CIP libplctag native array read (atomic + UDT member arrays) Unit-proven (fixture down)
AB Legacy PCCC multi-element file read via libplctag (cap 256 elements) Unit-proven (fixture down)
TwinCAT ADS native array symbol read Unit-proven (fixture down)

Array writes, multi-dimensional arrays (ValueRank>1), and per-element historization are out of scope — see Uns.md §Array tags.


Closed backlog §2 items (Phase 4 / Phase 4b)

The following items from the backlog §2 (archived at docs/plans/2026-06-18-stillpending-backlog-audit.md) were closed by earlier phases and are recorded here so future audits don't re-flag them.

Item Closed by Commit
Modbus Int64 / UInt64 OPC UA node DataType correction Phase 4 bd8fee61
HistoryAggregateType.Total — Total aggregate Phase 4 (derived client-side as time-weighted Average × interval-seconds) 5e27b5f7
Historian poison alarm-event indefinite retry — dead-letter cap (task #437) Phase 4 fcb38014

See also