From 163f57b6bc52f4678af111fd50c1e56dc16b4a2c Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Wed, 17 Jun 2026 20:35:34 -0400 Subject: [PATCH] docs: within-timestamp tie-cluster paging + AbCip/TwinCAT UDT member discovery --- docs/Historian.md | 38 ++++++++++++++++++++++++++----------- docs/drivers/AbCip.md | 37 ++++++++++++++++++++++++++++++++++++ docs/drivers/TwinCAT.md | 42 +++++++++++++++++++++++++++++++++++++++++ 3 files changed, 106 insertions(+), 11 deletions(-) diff --git a/docs/Historian.md b/docs/Historian.md index 612e92bb..be658b8a 100644 --- a/docs/Historian.md +++ b/docs/Historian.md @@ -64,7 +64,8 @@ and all HistoryRead calls on historized nodes return `GoodNoData` (empty, not an "Port": 32569, "UseTls": false, "ServerCertThumbprint": "", - "SharedSecret": "" + "SharedSecret": "", + "MaxTieClusterOverfetch": 65536 } } ``` @@ -77,6 +78,7 @@ and all HistoryRead calls on historized nodes return `GoodNoData` (empty, not an | `UseTls` | bool | `false` | Wrap the TCP connection in TLS. | | `ServerCertThumbprint` | string | — | Optional SHA-1 thumbprint to pin the sidecar's TLS certificate. Leave empty for CA-chain validation. | | `SharedSecret` | string | — | Shared secret token the sidecar expects on every connection. Required when `Enabled`. | +| `MaxTieClusterOverfetch` | int | `65536` | Maximum samples the server will fetch in one shot to page through a tie cluster (multiple samples sharing one `SourceTimestamp`). A cluster larger than this ceiling fails `BadHistoryOperationUnsupported`. Raise to handle abnormally large tie clusters; the default covers all normal-data cases. | > **Do not commit `SharedSecret` to `appsettings.json`.** Set it via an environment variable, > a secrets store, or a deployment-time overlay. The checked-in default is always empty. @@ -141,16 +143,30 @@ paging time-based: SourceTimestamp *inclusive* and drops the boundary samples already emitted, so samples sharing the boundary timestamp are neither duplicated nor skipped. -> **Paging limitation — oversized tie clusters.** The tie-safe cursor is a `(timestamp, skip)` -> pair, and the single-shot backend only accepts `(start, end, cap)` — it cannot skip. So if **more -> samples share one `SourceTimestamp` than `NumValuesPerNode`** (a tie cluster larger than the page -> cap), the cursor cannot advance past that timestamp: every resume re-reads the same first `cap` -> ties. Rather than silently truncate the read to `GoodNoData` (which would permanently drop the -> un-emitted ties), the resume read fails that node **loudly** with -> `BadHistoryOperationUnsupported` and logs the tag + timestamp + cap. The operator's remedy is to -> re-issue the read with a larger `NumValuesPerNode`. For a single tag's raw history this is a data -> anomaly (raw samples normally carry strictly increasing distinct timestamps); a fully cursor-based -> fix that pages *within* a single timestamp is a possible follow-up. +> **Oversized tie clusters — within-timestamp paging.** When more samples share one +> `SourceTimestamp` than the current page cap, the server detects that the cursor has stalled on +> a tie cluster (the last returned timestamp equals the resume timestamp). It then **over-fetches +> the entire cluster** at that single timestamp up to a bounded ceiling controlled by +> `ServerHistorian:MaxTieClusterOverfetch` (default **65 536**), then serves the cluster +> `NumValuesPerNode` samples at a time across successive pages, advancing the cursor one tick past +> the timestamp once the cluster is fully drained. +> +> **Short pages within a cluster still carry a continuation point.** A within-cluster page that +> returns fewer than `NumValuesPerNode` samples (because the cluster happened to be smaller than +> the cap, or is the final partial batch) is not the last page if the cluster itself has not been +> fully emitted — the server retains the continuation point so the client can drain the remainder. +> Only when the cluster is exhausted and the cursor has advanced past the timestamp does the +> short-page rule apply. +> +> **Cluster larger than `MaxTieClusterOverfetch`.** If the over-fetch itself reaches the ceiling +> without spanning the full cluster, the node fails **loudly** with +> `BadHistoryOperationUnsupported` and the tag + timestamp + ceiling are logged. Remedies: raise +> `MaxTieClusterOverfetch` (or `NumValuesPerNode`) to cover the full cluster, or investigate the +> data anomaly (raw samples normally carry strictly increasing distinct timestamps). +> +> For a single tag's raw history a tie cluster larger than the default 65 536 is a severe data +> anomaly. The ceiling exists to bound server-side memory on pathological data, not to cap normal +> operation. Continuation points are bound to the OPC UA session (the SDK's `ServerConfiguration.MaxHistoryContinuationPoints` cap, default 100, with oldest-eviction; points diff --git a/docs/drivers/AbCip.md b/docs/drivers/AbCip.md index 6b844d03..3724670d 100644 --- a/docs/drivers/AbCip.md +++ b/docs/drivers/AbCip.md @@ -119,6 +119,43 @@ integration fixture is down during normal dev (Docker host fixture currently off See [Uns.md §Array tags](../Uns.md#array-tags-1-d) for the cross-driver coverage matrix and the UI authoring flow. +## Controller-discovered UDT member variables + +When `EnableControllerBrowse` is set, the driver walks the controller's `@tags` symbol table and +surfaces each UDT-typed tag's **atomic member leaves** as individually addressable OPC UA +Variables — in addition to the UDT container tag itself. + +### What is discovered + +- **Top-level UDT members** — the Template Object for each controller-browse UDT is fetched via + its template instance ID; each atomic member (BOOL, SINT, INT, DINT, REAL, …) becomes a + separate Variable node under the UDT's `Discovered/` sub-folder, addressable by its member + path (e.g. `MyUdt.Temperature`, `MyUdt.Flags`). Top-level atomic member discovery is + **functional in production**. +- **Nested struct members** — when a UDT member is itself a struct, the driver walks into it up + to a **depth cap of 8 levels**. Nested-struct expansion is a **documented deferral in + production**: the Template Object member block carries no nested template ID, so the + sub-shape cannot be re-fetched from the controller at discovery time. Nested struct leaves are + never mis-emitted — they are simply dropped (the parent member is omitted from discovery if it + is not atomic and its sub-shape is unavailable). Use pre-declared `Members` for nested structs + that must be individually addressed. + +### Bare-container reads + +Reading the UDT container tag itself (without a member suffix) returns +`BadNotSupported` — address the atomic member leaves instead. + +### Member writes + +UDT member **writes** discovered via controller browse are **deferred** in this release. +Pre-declared `Members` with `Writable: true` retain full read/write support as before. + +### Configuration + +No new configuration keys are required. Set `EnableControllerBrowse: true` in the +`AbCipDriverOptions` JSON to enable the `@tags` walk; UDT member expansion is automatic for all +UDT-typed tags found in the walk. + ## Operational Notes - **Native heap is invisible to the GC.** `GetMemoryFootprint()` reports CLR allocations only; libplctag's native `Tag` heap does not show up there. Watch whole-process RSS, and use `ReinitializeAsync` (tears down + re-creates every device's libplctag handles) as the remediation for native-heap growth. diff --git a/docs/drivers/TwinCAT.md b/docs/drivers/TwinCAT.md index 9f7e2998..e1e308ec 100644 --- a/docs/drivers/TwinCAT.md +++ b/docs/drivers/TwinCAT.md @@ -154,6 +154,48 @@ on PLC re-download (unchanged semantics from scalar nodes). See [Uns.md §Array tags](../Uns.md#array-tags-1-d) for the cross-driver coverage matrix and the UI authoring flow. +## Controller-discovered struct/UDT/FB member variables + +When `EnableControllerBrowse` is set, `BrowseSymbolsAsync` walks the device's symbol tree +recursively and expands **Struct / UDT / Function-Block typed symbols** into their **atomic +member leaves** as individually addressable OPC UA Variables, surfaced under the `Discovered/` +sub-folder. + +### What is discovered + +- Each symbol of a struct, UDT, or FB type is recursively expanded. The walk follows the ADS + symbol tree's `SubSymbols` collection. +- **Full instance paths** are preserved for every leaf (e.g. `MAIN.Motor.Speed`, + `GVL.Drive1.Status.Running`). +- The recursion is **depth-capped at 8 levels** to guard against pathological or circular-like + layouts. +- **Unsupported leaves** (types the driver cannot map to an OPC UA data type) are silently + dropped rather than surfaced as error nodes. + +### Pre-declared Structure-typed tags + +Pre-declared `Tags` with a `DataType` of `Structure` are **still rejected** — the driver cannot +read an opaque struct blob without knowing its member layout. For structs that must be +individually addressed, use `EnableControllerBrowse` to let the driver discover the member +layout from the controller's symbol table, rather than pre-declaring the container tag. + +### Member writes + +Discovered struct/UDT/FB member **writes** are **deferred** in this release. Scalar atomic +members can be written when pre-declared with `Writable: true`; the controller-browse expansion +path does not yet wire the write channel. + +### Live caveat — Flat-mode vs VirtualTree-mode browse + +The ADS client's `BrowseSymbolsAsync` may return symbols in **Flat mode** (a flat list of all +instance paths, with no `SubSymbols` populated) on some TC3 runtime configurations, in which +case the recursive struct expansion produces no sub-symbols and the walk yields only top-level +symbols. If a live TC3 browse does not populate `SubSymbols` on struct-typed symbols, the +alternative is **VirtualTree mode** (which returns a hierarchical symbol tree with `SubSymbols` +populated). VirtualTree-mode browse is an unverified follow-up; if Flat-mode struct expansion +produces empty `Discovered/` sub-folders for struct symbols, switch to VirtualTree mode via the +driver configuration. + ## Further reading - [`docs/v2/driver-specs.md §6`](../v2/driver-specs.md) — full per-field spec and