docs(historian): pluggable backends, SQL Server backend, HistoryUpdate, outbox v2, spec corrections
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
This commit is contained in:
Joseph Doherty
@@ -250,6 +250,15 @@ otopcua-cli historyupdate -u ... -n "..." --mode delete-at-time --time 2026-01-1
|
||||
| `--type` | `double` (default), `int64`, `bool`, `string`, `datetime` - how `--value` is parsed |
|
||||
| `--start` / `--end` | Range for `delete-raw` (both required; `--end` later than `--start`) |
|
||||
|
||||
Expected per-item codes on the SQL Server historian backend: `insert` ⇒ `GoodEntryInserted` (repeat ⇒ `BadEntryExists`), `replace` ⇒ `GoodEntryReplaced`
|
||||
(`BadNoEntryExists` if nothing is stored at that time), `update` ⇒ either Good code, `delete-at-time` ⇒ `Good` / `BadNoEntryExists`; `delete-raw` reports one
|
||||
node-level code (`Good`, or `BadNoData` if the range held nothing). A node that is not historized, or a server on the Gateway historian backend (no
|
||||
updater, so no `HistoryWrite` access bit), answers `BadHistoryOperationUnsupported`; a `NaN`/`Infinity` double is `BadInvalidArgument`. The session's API key
|
||||
(`-U apikey -P otua_<keyId>_<secret>`) needs the `HistoryUpdate` permission — via a `Cluster`-scoped grant when `Security:Acl:Enabled` is off, or a scoped
|
||||
grant on the node when it is on; a key with only `WriteOperate` gets `BadUserAccessDenied`. See [Historian.md §HistoryUpdate](Historian.md#historyupdate) and
|
||||
[security.md](security.md#historyupdate-historian-writes). `historyread --aggregate` only knows the seven names in the table below; the SQL backend's other
|
||||
aggregates (Total, Range, Delta, PercentGood, …) need a different client.
|
||||
|
||||
`delete-raw` is sent with `IsDeleteModified=false`. Invalid or inconsistent options (unknown mode, unpaired
|
||||
value/time, unparsable value, options that do not apply to the mode) fail before any connection is made.
|
||||
|
||||
|
||||
+26
-4
@@ -377,9 +377,12 @@ The Galaxy/MxAccess connection settings are **not an `appsettings` section.** Th
|
||||
|
||||
> The `OTOPCUA_GALAXY_*` environment variables that v1's in-process `Galaxy.Host` consumed **no longer live in this repo** — they moved into the separately-installed mxaccessgw gateway's own config (see the v1 archive pointer in `docs/README.md` and the Galaxy overview at [`docs/drivers/Galaxy.md`](drivers/Galaxy.md)). The only Galaxy connection secret this repo touches is the gateway API key via `ApiKeySecretRef` above.
|
||||
|
||||
### Historian config (HistorianGateway)
|
||||
### Historian config (`ServerHistorian:Backend` — HistorianGateway or SQL Server)
|
||||
|
||||
The historian backend is the external **`ZB.MOM.WW.HistorianGateway`** sidecar, consumed as the
|
||||
The historian backend is **pluggable**, selected by `ServerHistorian:Backend` (case-insensitive; default `Gateway`;
|
||||
unknown value ⇒ host start fails listing the allowed values): `Gateway` is the external **`ZB.MOM.WW.HistorianGateway`**
|
||||
sidecar below, `SqlServer` is a plain SQL Server database (see [`ServerHistorian:SqlServer`](#serverhistoriansqlserver-backendsqlserver)).
|
||||
The default backend is the gateway, consumed as the
|
||||
`ZB.MOM.WW.HistorianGateway.Client` gRPC package (the retired Wonderware TCP sidecar is documented at
|
||||
[`docs/drivers/Historian.Wonderware.md`](drivers/Historian.Wonderware.md)). The OtOpcUa host reads three
|
||||
appsettings sections — `ServerHistorian` (read path + gateway connection), `ContinuousHistorization`
|
||||
@@ -394,9 +397,26 @@ key must carry the scopes `historian:read`, `historian:write`, `historian:tags:w
|
||||
[`docs/Historian.md`](Historian.md) for the full key reference, the migration note (old Wonderware keys →
|
||||
gateway keys), and the deployment prerequisites.
|
||||
|
||||
#### `ServerHistorian:SqlServer` (`Backend=SqlServer`)
|
||||
|
||||
Read only when `ServerHistorian:Backend` is `SqlServer`; the gateway keys (`Endpoint`, `ApiKey`, `UseTls`, …) are then not required.
|
||||
Validated at host start (driver-role nodes) when `ServerHistorian:Enabled` **or** `AlarmHistorian:Enabled` is true.
|
||||
|
||||
| Key | Default | Validation / meaning |
|
||||
|---|---|---|
|
||||
| `ServerHistorian:Backend` | `Gateway` | `Gateway` or `SqlServer`; anything else fails host start (checked even when the historian is disabled). |
|
||||
| `ServerHistorian:SqlServer:ConnectionString` | `""` | **Required.** Historian database connection string (not the ConfigDb). Supply via `${secret:OT-HISTORIAN-SQL}` or env `ServerHistorian__SqlServer__ConnectionString`; the database must already exist. |
|
||||
| `ServerHistorian:SqlServer:Schema` | `hist` | Must match `^[A-Za-z_][A-Za-z0-9_]{0,127}$` (spliced into DDL/DML). |
|
||||
| `ServerHistorian:SqlServer:CommandTimeout` | `00:00:30` | Must be `> 0`; rounded up to whole seconds. |
|
||||
| `ServerHistorian:SqlServer:AutoCreateSchema` | `true` | `true` applies the embedded schema script under an app-lock on first use; `false` only verifies `SchemaVersion >= 1` and fails every historian call (health `LastError`, no crash) if behind. |
|
||||
|
||||
HistoryUpdate (SqlServer only) additionally needs `ServerHistorian:Enabled=true` and a key whose principal holds the
|
||||
`HistoryUpdate` permission — see [`docs/security.md`](security.md#historyupdate-historian-writes). Full semantics, schema, aggregates and the
|
||||
docker-dev recipe: [`docs/Historian.md`](Historian.md#sql-server-backend).
|
||||
|
||||
#### `ContinuousHistorization` (recorder + outbox knobs)
|
||||
|
||||
Only the knobs; the gateway connection comes from `ServerHistorian`. Full descriptions in
|
||||
Only the knobs; the backend connection comes from `ServerHistorian`. Outbox records are format v2 — **drain or delete `OutboxPath` before downgrading a node** to a pre-v2 build (see [`docs/Historian.md` §Outbox format v2 and rollback](Historian.md#outbox-format-v2-and-rollback)); undecodable records are skipped and counted by the gauge `otopcua.historian.outbox.unreadable`. Full descriptions in
|
||||
[`docs/Historian.md` §ContinuousHistorization configuration](Historian.md#continuoushistorization-configuration).
|
||||
|
||||
| Key | Default | Meaning |
|
||||
@@ -498,7 +518,9 @@ historian secret this repo reads from the environment is the gateway API key:
|
||||
|
||||
| Variable | Effect / default |
|
||||
|---|---|
|
||||
| `ServerHistorian__ApiKey` | The HistorianGateway peppered-HMAC key (`histgw_<id>_<secret>`) sent as `Authorization: Bearer`. Supply via environment — **never commit**. Required when `ServerHistorian:Enabled=true`. |
|
||||
| `ServerHistorian__ApiKey` | The HistorianGateway peppered-HMAC key (`histgw_<id>_<secret>`) sent as `Authorization: Bearer`. Supply via environment — **never commit**. Required when `ServerHistorian:Enabled=true` **and `Backend=Gateway`**. |
|
||||
| `ServerHistorian__Backend` | `Gateway` (default) or `SqlServer`. |
|
||||
| `ServerHistorian__SqlServer__ConnectionString` | The SQL Server historian connection string. Required when `Backend=SqlServer` and the historian (or alarm history) is enabled. Supply via environment / `${secret:}` — **never commit**. |
|
||||
|
||||
### Driver integration-test / fixture sim endpoints
|
||||
|
||||
|
||||
+376
-31
@@ -1,4 +1,4 @@
|
||||
# Historian — server-side OPC UA HistoryRead (Phase C)
|
||||
# Historian — server-side OPC UA HistoryRead / HistoryUpdate (pluggable backends)
|
||||
|
||||
Phase C wires server-side OPC UA **HistoryRead** for authored equipment tags flagged
|
||||
historized. The feature is driver-agnostic: any equipment tag (Galaxy, Modbus, OpcUaClient,
|
||||
@@ -10,7 +10,14 @@ the existing schemaless `TagConfig` JSON blob alongside the Phase B `alarm` obje
|
||||
Wonderware TCP sidecar backend this replaced was retired — see
|
||||
[drivers/Historian.Wonderware.md](drivers/Historian.Wonderware.md).)
|
||||
|
||||
Design reference: [docs/plans/2026-06-14-galaxy-phase-c-historian-design.md](plans/2026-06-14-galaxy-phase-c-historian-design.md).
|
||||
> **The backend is pluggable (2026-10).** `ServerHistorian:Backend` selects `Gateway` (the default — everything
|
||||
> described below as "the gateway") or `SqlServer` (a plain SQL Server database the server reads and writes
|
||||
> itself, which also supports OPC UA **HistoryUpdate**). See [Backend selection](#backend-selection-serverhistorianbackend),
|
||||
> [SQL Server backend](#sql-server-backend) and [HistoryUpdate](#historyupdate). Wherever this page says
|
||||
> "the historian" without qualification it means the selected backend; sections that only apply to one backend say so.
|
||||
|
||||
Design references: [docs/plans/2026-06-14-galaxy-phase-c-historian-design.md](plans/2026-06-14-galaxy-phase-c-historian-design.md)
|
||||
(Phase C) and [docs/plans/2026-10-09-pluggable-historian-design.md](plans/2026-10-09-pluggable-historian-design.md) (pluggable backends).
|
||||
|
||||
---
|
||||
|
||||
@@ -53,6 +60,34 @@ A tag that is both historized and a native alarm:
|
||||
|
||||
---
|
||||
|
||||
## Backend selection (`ServerHistorian:Backend`)
|
||||
|
||||
`ServerHistorian:Backend` (case-insensitive, trimmed; default `Gateway`) picks the provider
|
||||
(`IHistorianBackendProvider`) that builds all five historian seams — the HistoryRead data source, the continuous-
|
||||
historization value writer, the alarm-history writer, the tag provisioner and (optionally) the HistoryUpdate
|
||||
updater. `HistorianBackendRegistry` in the Host owns the name table. An unknown name fails host start with an
|
||||
`OptionsValidationException` that lists the allowed values (`Gateway`, `SqlServer`), regardless of whether anything
|
||||
consumes the historian. The existing enable gates (`ServerHistorian:Enabled`, `AlarmHistorian:Enabled`,
|
||||
`ContinuousHistorization:Enabled`) are unchanged — they simply get their objects from the selected provider.
|
||||
|
||||
| | `Gateway` (default) | `SqlServer` |
|
||||
|---|---|---|
|
||||
| Storage | AVEVA Historian via the HistorianGateway sidecar | `{Schema}` tables in a SQL Server database you point it at |
|
||||
| Value kinds recorded | `Double`, `Int64`, `Boolean` (numeric analog only) | all: `Double`, `Int64`, `Boolean`, `String`, `DateTime`, plus `Empty` (a Bad sample with no value) |
|
||||
| Status code stored | Sent as DA quality but **discarded** by the gateway (C-002); AVEVA stores every sample Good | Stored bit-for-bit (`StatusCode` int column) |
|
||||
| Comms-loss / driver-Bad publishes | Dropped, counted in `DroppedUnsupportedKind` | Stored as an `Empty` outage-marker sample carrying the Bad code (see [below](#typed-values-real-status-and-outage-markers)) |
|
||||
| Processed aggregates | Average, Minimum, Maximum, Total, Count | The thirteen in the [aggregate table](#aggregates-processed-reads) |
|
||||
| HistoryUpdate | No (`BadHistoryOperationUnsupported`) | Yes (Insert / Replace / Update / DeleteRaw / DeleteAtTime) |
|
||||
| Alarm history | gateway `SendEvent` | `Event` table (insert-if-absent) |
|
||||
| Provisioning (`EnsureTags`) | numeric types only; others skipped | `MERGE` upsert of every request, all types |
|
||||
| Connection | `Endpoint` + `ApiKey` + TLS keys (top-level `ServerHistorian:*`) | `ServerHistorian:SqlServer:*` |
|
||||
|
||||
Recording stays **Primary-only** (#537) and **at-least-once** with the #538 per-tag quarantine on both backends;
|
||||
HistoryRead and HistoryUpdate are served from any node (the database is shared). The historian connection string is
|
||||
**not** a ConfigDb string, and a driver-only node may carry it without breaking the Phase 4 "no ConfigDb" posture.
|
||||
|
||||
---
|
||||
|
||||
## ServerHistorian configuration
|
||||
|
||||
The `ServerHistorian` section in `appsettings.json` controls the historian read path.
|
||||
@@ -63,6 +98,7 @@ and all HistoryRead calls on historized nodes return `GoodNoData` (empty, not an
|
||||
{
|
||||
"ServerHistorian": {
|
||||
"Enabled": false,
|
||||
"Backend": "Gateway",
|
||||
"Endpoint": "",
|
||||
"ApiKey": "",
|
||||
"UseTls": true,
|
||||
@@ -72,24 +108,51 @@ and all HistoryRead calls on historized nodes return `GoodNoData` (empty, not an
|
||||
"MaxTieClusterOverfetch": 65536,
|
||||
"HistoryReadBatchParallelism": 4,
|
||||
"MaxConcurrentHistoryReadBatches": 16,
|
||||
"HistoryReadDeadline": "00:01:00"
|
||||
"HistoryReadDeadline": "00:01:00",
|
||||
"SqlServer": {
|
||||
"ConnectionString": "",
|
||||
"Schema": "hist",
|
||||
"CommandTimeout": "00:00:30",
|
||||
"AutoCreateSchema": true
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
| Key | Type | Default | Description |
|
||||
|---|---|---|---|
|
||||
| `Enabled` | bool | `false` | Enable the live `GatewayHistorianDataSource`. `false` → `NullHistorianDataSource` (empty reads). |
|
||||
| `Endpoint` | string | `""` | Absolute gateway URI, e.g. `https://host:5222`. Scheme selects transport (`https://` = TLS, `http://` = h2c plaintext). Required when `Enabled`. |
|
||||
| `ApiKey` | string | `""` | The gateway peppered-HMAC key (`histgw_<id>_<secret>`) sent as `Authorization: Bearer`. Required when `Enabled`. **Supply via env `ServerHistorian__ApiKey`.** |
|
||||
| `UseTls` | bool | `true` | Connect over TLS; must match the `Endpoint` scheme. |
|
||||
| `AllowUntrustedServerCertificate` | bool | `false` | Accept a self-signed / untrusted server certificate (dev / on-prem only). |
|
||||
| `CaCertificatePath` | string\|null | `null` | PEM CA file pinning the gateway's TLS chain. Null/empty uses the OS trust store. |
|
||||
| `CallTimeout` | TimeSpan | `00:00:30` | Per-call deadline applied to each unary gateway read. A read for a tag the historian does not yet know about — e.g. a freshly-historized tag that has **not** been provisioned (see [Tag auto-provisioning](#tag-auto-provisioning-ensuretags)) — can run to this **full** deadline before the gateway errors, so an unprovisioned-tag HistoryRead can block for ~30 s. Lower this to fail faster, at the cost of truncating legitimately long reads. |
|
||||
| `MaxTieClusterOverfetch` | int | `65536` | Maximum samples the server will fetch in one shot to page through a tie cluster (multiple samples sharing one `SourceTimestamp`). A cluster larger than this ceiling fails `BadHistoryOperationUnsupported`. Raise to handle abnormally large tie clusters; the default covers all normal-data cases. |
|
||||
| `Enabled` | bool | `false` | Enable the selected backend's read client. `false` → `NullHistorianDataSource` (empty reads) and no HistoryUpdate. |
|
||||
| `Backend` | string | `Gateway` | `Gateway` or `SqlServer` (case-insensitive). Unknown ⇒ host start fails. See [Backend selection](#backend-selection-serverhistorianbackend). |
|
||||
| `Endpoint` | string | `""` | **Gateway backend only.** Absolute gateway URI, e.g. `https://host:5222`. Scheme selects transport (`https://` = TLS, `http://` = h2c plaintext). Required when `Enabled`. |
|
||||
| `ApiKey` | string | `""` | **Gateway backend only.** The gateway peppered-HMAC key (`histgw_<id>_<secret>`) sent as `Authorization: Bearer`. Required when `Enabled`. **Supply via env `ServerHistorian__ApiKey`.** |
|
||||
| `UseTls` | bool | `true` | **Gateway backend only.** Connect over TLS; must match the `Endpoint` scheme. |
|
||||
| `AllowUntrustedServerCertificate` | bool | `false` | **Gateway backend only.** Accept a self-signed / untrusted server certificate (dev / on-prem only). |
|
||||
| `CaCertificatePath` | string\|null | `null` | **Gateway backend only.** PEM CA file pinning the gateway's TLS chain. Null/empty uses the OS trust store. |
|
||||
| `CallTimeout` | TimeSpan | `00:00:30` | **Gateway backend only.** Per-call deadline applied to each unary gateway read. A read for a tag the historian does not yet know about — e.g. a freshly-historized tag that has **not** been provisioned (see [Tag auto-provisioning](#tag-auto-provisioning-ensuretags)) — can run to this **full** deadline before the gateway errors, so an unprovisioned-tag HistoryRead can block for ~30 s. Lower this to fail faster, at the cost of truncating legitimately long reads. |
|
||||
| `MaxTieClusterOverfetch` | int | `65536` | (Node-manager paging; all backends — a `SqlServer` PK rules out tie clusters, so it is moot there.) Maximum samples the server will fetch in one shot to page through a tie cluster (multiple samples sharing one `SourceTimestamp`). A cluster larger than this ceiling fails `BadHistoryOperationUnsupported`. Raise to handle abnormally large tie clusters; the default covers all normal-data cases. |
|
||||
| `HistoryReadBatchParallelism` | int | `4` | Max historized nodes served **concurrently** within one HistoryRead batch (arch-review 03/S3). The SDK's `HistoryRead*` override surface is synchronous, so each arm bridges once at its boundary; this turns per-node serving from sequential (`N × CallTimeout` worst case) into `⌈N / P⌉ × CallTimeout`. Also caps the gateway-load multiplication a single client can command. `≤ 1` serves sequentially. |
|
||||
| `MaxConcurrentHistoryReadBatches` | int | `16` | Process-wide limit on HistoryRead batches served **concurrently** (arch-review 03/S3). A flood of history clients degrades to `BadTooManyOperations` rather than exhausting the SDK request-thread pool. With `HistoryReadBatchParallelism` this caps total in-flight gateway reads at `MaxConcurrentHistoryReadBatches × HistoryReadBatchParallelism` (default 16 × 4 = 64). A batch that cannot get a slot within `min(HistoryReadDeadline, 5 s)` fails every handle `BadTooManyOperations`. |
|
||||
| `HistoryReadDeadline` | TimeSpan | `00:01:00` | Per-request wall-clock deadline for a HistoryRead batch (arch-review 03/S3). A node still in flight at the deadline surfaces `BadTimeout`, so one request can never hold an SDK request thread longer than this regardless of node count. Non-positive = unbounded (a `Validate()` warning). |
|
||||
| `MaxConcurrentHistoryReadBatches` | int | `16` | Process-wide limit on HistoryRead (and HistoryUpdate) batches served **concurrently** (arch-review 03/S3). A flood of history clients degrades to `BadTooManyOperations` rather than exhausting the SDK request-thread pool. With `HistoryReadBatchParallelism` this caps total in-flight gateway reads at `MaxConcurrentHistoryReadBatches × HistoryReadBatchParallelism` (default 16 × 4 = 64). A batch that cannot get a slot within `min(HistoryReadDeadline, 5 s)` fails every handle `BadTooManyOperations`. |
|
||||
| `HistoryReadDeadline` | TimeSpan | `00:01:00` | Per-request wall-clock deadline for a HistoryRead (and HistoryUpdate) batch (arch-review 03/S3). A node still in flight at the deadline surfaces `BadTimeout`, so one request can never hold an SDK request thread longer than this regardless of node count. Non-positive = unbounded (a `Validate()` warning). |
|
||||
|
||||
**`ServerHistorian:SqlServer`** (read only when `Backend=SqlServer`; bound by `HistorianBackendRegistry.BindSqlServer`):
|
||||
|
||||
| Key | Type | Default | Validation | Description |
|
||||
|---|---|---|---|---|
|
||||
| `SqlServer:ConnectionString` | string | `""` | **Required** (non-blank) | ADO.NET connection string for the historian database. Never the ConfigDb. Supply via `${secret:OT-HISTORIAN-SQL}` / environment (`ServerHistorian__SqlServer__ConnectionString`) — never commit a real one. The database must already exist; `AutoCreateSchema` creates the *schema and tables*, not the database. |
|
||||
| `SqlServer:Schema` | string | `hist` | Must match `^[A-Za-z_][A-Za-z0-9_]{0,127}$` (it is spliced into DDL/DML) | SQL schema that holds `SchemaVersion`, `Tag`, `Sample`, `Event` and the `SampleRow`/`TimeRow` table types. |
|
||||
| `SqlServer:CommandTimeout` | TimeSpan | `00:00:30` | Must be `> 0` | Per-command timeout, rounded up to whole seconds (minimum 1 s). The SQL backend does **not** use `CallTimeout`. |
|
||||
| `SqlServer:AutoCreateSchema` | bool | `true` | — | `true`: on first use apply any missing embedded script under a `sp_getapplock` (`OtOpcUaHistorianSchema`), so concurrent nodes serialise. `false`: only verify `SchemaVersion >= 1`; if behind, **every** historian call fails with a clear error naming the missing version (surfaced in the read client's health `LastError`, never a host crash). A failed bootstrap is retried on the next call, not latched. |
|
||||
|
||||
**Validation rules** (`ServerHistorianOptionsValidator`, driver-role nodes only; admin-only nodes never build historian singletons):
|
||||
|
||||
- `Backend` must name a registered backend — checked **always**, with no I/O, even when nothing is enabled.
|
||||
- Everything else is **consumer-gated**: it runs only when `ServerHistorian:Enabled` **or** `AlarmHistorian:Enabled` is true
|
||||
(the alarm sink sources its connection from this section even when `ServerHistorian` is off).
|
||||
- `Backend=Gateway`: `Endpoint` must be an absolute http(s) URI, `ApiKey` non-empty, and `UseTls` must agree with the endpoint scheme.
|
||||
- `Backend=SqlServer`: the three `SqlServer` rules above. `Endpoint`/`ApiKey`/`UseTls` are **not** required.
|
||||
- ⚠️ `Backend=SqlServer` with a malformed `Schema` throws `ArgumentException` while the provider is built, so it stops
|
||||
host start even when no consumer is enabled (the validator is consumer-gated, provider construction is not).
|
||||
- The alarm-only warning at startup (`AlarmHistorian` on, `ServerHistorian` off) names the selected backend.
|
||||
|
||||
> **Do not commit `ApiKey` to `appsettings.json`.** Set it via the environment variable
|
||||
> `ServerHistorian__ApiKey`, a secrets store, or a deployment-time overlay. The checked-in default is
|
||||
@@ -108,11 +171,14 @@ write path) and the `ContinuousHistorization` section (driver-value capture). Al
|
||||
**same** gateway — but only `ServerHistorian` carries the connection (endpoint/key/TLS); the other two
|
||||
source it from there.
|
||||
|
||||
> **Continuous-historization value quality is server-stamped.** The gateway's `WriteLiveValues` SQL path
|
||||
> **discards** the quality the recorder sends (a server-managed column; 0.2.0 Contracts C-002), so persisted
|
||||
> quality is stamped gateway-side regardless of what the `ContinuousHistorizationRecorder` provides (it
|
||||
> hardcodes OPC-DA Good, `192`). Capturing real per-node quality will not persist until the gateway honors
|
||||
> the field — see `GatewayHistorianValueWriter`'s remarks (archreview 06/C-7).
|
||||
> **Continuous-historization value quality on the Gateway backend is still server-stamped.** The recorder now
|
||||
> carries each sample's **real** OPC UA status (it used to hard-code Good `192`, so a Bad sample was historized as
|
||||
> Good). `GatewayHistorianValueWriter` maps that status to an OPC-DA quality byte with
|
||||
> `GatewayQualityMapper.ToDaQuality` (the inverse of the read-side table; unknown codes fall back by severity:
|
||||
> Good ⇒ 192, Uncertain ⇒ 64, Bad ⇒ 0). But the gateway's `WriteLiveValues` SQL path **discards** the quality
|
||||
> it is sent (a server-managed column; 0.2.0 Contracts C-002), so **AVEVA still stores every recorded value as
|
||||
> Good** — a pre-existing limitation of that backend, now documented rather than fixed (archreview 06/C-7). The
|
||||
> `SqlServer` backend stores the real status.
|
||||
|
||||
> ✅ **Continuous value capture is LIVE-PROVEN END-TO-END
|
||||
> ([#491](https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/491) CLOSED — gate run 2026-08-01,
|
||||
@@ -145,12 +211,13 @@ source it from there.
|
||||
## ContinuousHistorization configuration
|
||||
|
||||
`ContinuousHistorizationOptions` binds the `ContinuousHistorization` section. It carries **only** the
|
||||
recorder + outbox knobs — the gateway connection (endpoint / key / TLS) is sourced from
|
||||
`ServerHistorian` (above), never restated here.
|
||||
recorder + outbox knobs — the backend connection (gateway endpoint / key / TLS, or the SQL Server connection
|
||||
string) is sourced from `ServerHistorian` (above), never restated here. The outbox is a node-local FasterLog
|
||||
and is registered the same way for every backend.
|
||||
|
||||
| Key | Type | Default | Description |
|
||||
|---|---|---|---|
|
||||
| `Enabled` | bool | `false` | `true` **and** a configured `ServerHistorian` wires the FasterLog outbox + gateway value-writer and spawns the `ContinuousHistorizationRecorder`. `false` → no recorder; driver tag values are not historized. |
|
||||
| `Enabled` | bool | `false` | `true` **and** a configured `ServerHistorian` wires the FasterLog outbox + the selected backend's value writer and spawns the `ContinuousHistorizationRecorder`. `false` → no recorder; driver tag values are not historized. |
|
||||
| `OutboxPath` | string | `""` | **Directory** holding the FasterLog segment + commit files. Required when `Enabled`. In production set an **absolute** path on durable storage — a relative path resolves against the host's working directory. |
|
||||
| `CommitMode` | string | `PerEntry` | `PerEntry` fsyncs before the append returns (no loss window); `Periodic` batches commits onto a background timer every `CommitIntervalMs`. Parsed case-insensitively; an unrecognized value falls back to `PerEntry`. **See the batch-commit note below — `PerEntry` is one fsync per ingest *burst*, not per value.** |
|
||||
| `CommitIntervalMs` | int | `100` | Periodic-mode commit cadence (ms); must be `> 0` under `Periodic`, ignored under `PerEntry`. |
|
||||
@@ -181,6 +248,50 @@ recorder + outbox knobs — the gateway connection (endpoint / key / TLS) is sou
|
||||
> lands, raising `DrainBatchSize` widens each hop's payload but not the hop count — which is why the
|
||||
> concurrency window, not the batch width, is what governs drain throughput today.
|
||||
|
||||
### Typed values, real status and outage markers
|
||||
|
||||
The recorder no longer assumes "a numeric Good sample". Each mux delta is coerced to a typed `HistorianValue`
|
||||
(`Double` / `Int64` / `Boolean` / `String` / `DateTime` / `Empty`; a `ulong` above `long.MaxValue` becomes `Double`;
|
||||
arrays and anything else are dropped and counted) and appended with the **real** status the driver published.
|
||||
A value whose kind is not in the selected backend's `SupportedKinds` is dropped at capture and counted in
|
||||
`RecorderStatus.DroppedUnsupportedKind` (the field was `DroppedNonNumeric` before the rename; `Empty` is supported
|
||||
by `SqlServer`, not by `Gateway`).
|
||||
|
||||
⚠️ **Comms loss is historized — as an outage marker (design §9.10).** `DriverHostActor.FanCommsLossToDependencyMux`
|
||||
(#481) fans `(null, BadNoCommunication)` for every value ref of a comms-lost driver to the dependency mux, and the
|
||||
recorder cannot tell that from a driver's own Bad publish. So:
|
||||
|
||||
- **`SqlServer`:** each disconnect stores **one `Empty` / `BadNoCommunication` (`0x80310000`) sample per historized
|
||||
tag**. A driver's own Bad publish is stored the same way (`Empty` kind plus its own code — e.g. Modbus
|
||||
`BadCommunicationError` `0x80050000`). This is deliberate: raw reads show the gap, and aggregates and AtTime
|
||||
interpolation do not bridge it. Cost: one extra row per tag per outage.
|
||||
- **`Gateway`:** the marker is `Empty`, which that backend does not support, so it is dropped and counted in
|
||||
`DroppedUnsupportedKind`.
|
||||
|
||||
### Outbox format v2 and rollback
|
||||
|
||||
Outbox entries are written in format **v2**: `"HV2\0"` magic + version byte `2` + id + tag + value kind + a
|
||||
kind-specific payload + the 32-bit status + timestamp. Entries left in an outbox by an older build (the fixed
|
||||
38-bytes-plus-tag layout, always quality 192) are still **read** — as a `Double` with status `Good`. A record that
|
||||
matches neither layout is **skipped and counted, never thrown**, so a corrupt record cannot jam the drain (this also
|
||||
holds during the FasterLog recovery scan at start-up); the lifetime count is the gauge
|
||||
**`otopcua.historian.outbox.unreadable`** (`IHistorizationOutbox.UnreadableCount`). A growing value means
|
||||
undecodable records are being skipped.
|
||||
|
||||
⚠️ **Rollback: drain or delete the outbox before downgrading.** A node running a pre-v2 build cannot read v2
|
||||
records — its outbox deserializes every record in its constructor and fails to start on the first v2 one. Before
|
||||
rolling a node back, let the outbox drain to zero (`otopcua.historization.outbox.depth`) or stop the node and
|
||||
delete the directory named by `ContinuousHistorization:OutboxPath` (accepting the loss of anything un-drained).
|
||||
Upgrading needs no action.
|
||||
|
||||
**Delivery semantics on `SqlServer`.** Writes are *insert-if-absent* on `(TagId, SourceTimeUtc)`: the outbox's
|
||||
at-least-once redelivery (and both pair nodes writing the same sample in a boot window) converges on one row, and the
|
||||
first-stored sample wins. **Duplicate timestamps inside one batch collapse to the first.** A `NaN` / `±Infinity` `Double`
|
||||
cannot be stored in a SQL `float`, so the **value writer stores it as an `Empty` sample keeping its status** — the tag
|
||||
is not quarantined for one bad reading (HistoryUpdate behaves differently: see [below](#historyupdate)). A duplicate-key
|
||||
error (2627/2601) rolls the whole statement back, so the writer re-runs the statement (at most twice) instead of
|
||||
treating the error as success.
|
||||
|
||||
---
|
||||
|
||||
## A poison tag no longer pins the outbox (#538)
|
||||
@@ -257,8 +368,11 @@ tag (with the resolved historian name — the `historianTagname` override when s
|
||||
`FullName`); the hook is wrapped so a faulted, slow, or throwing provisioner can **never** block or fail a
|
||||
deploy. Non-numeric (`String`/`DateTime`/`Reference`) types are skipped, not provisioned.
|
||||
|
||||
**Wiring + gating.** The concrete provisioner is the gateway-backed `GatewayTagProvisioner`, registered by
|
||||
the Host's `AddHistorianProvisioning` (`GatewayHistorian.CreateProvisioner`) and gated on the **same**
|
||||
**Wiring + gating.** The concrete provisioner comes from the selected backend's provider (`CreateProvisioning`):
|
||||
the gateway-backed `GatewayTagProvisioner`, or — for `SqlServer` — `SqlServerTagProvisioner` (one `MERGE … WITH (HOLDLOCK)`
|
||||
upsert per chunk of 400 tags; every data type is provisionable so `skipped` is always 0; a blank or over-400-character
|
||||
name fails alone; a null unit/description in a request keeps the stored value). It is registered by the Host's
|
||||
`AddHistorianProvisioning` and gated on the **same**
|
||||
`ServerHistorian:Enabled` flag as the read path — both target the same single gateway. When the section is
|
||||
disabled (or absent) the applier resolves the no-op `NullHistorianProvisioning` and **provisioning does not
|
||||
run** (so on a read-only-historian deployment, ensure tags exist by some other means). The Host requires the
|
||||
@@ -275,13 +389,15 @@ for this node), which is the signal to check that `ServerHistorian:Enabled=true`
|
||||
|
||||
### Deferred types and the UInt16 fallback (#600 item 4)
|
||||
|
||||
Historization is **numeric-analog only** in v1. `HistorianTypeMapper` classifies exactly nine types as
|
||||
> **Gateway backend only.** The `SqlServer` backend provisions and records every type (`String` and `DateTime` included).
|
||||
|
||||
Historization on the **Gateway** backend is **numeric-analog only**. `HistorianTypeMapper` classifies exactly nine types as
|
||||
historizable (`Boolean`, `Int16/32/64`, `UInt16/32/64`, `Float32`, `Float64`):
|
||||
|
||||
- **Deferred: `String`, `DateTime`, `Reference`.** The historian's SQL analog write path cannot carry them.
|
||||
Provisioning skips them (counted in the `skipped=` tally, via `IsHistorizable`), the mapper throws
|
||||
`NotSupportedException` for them, and `ContinuousHistorizationRecorder` drops their values and counts them
|
||||
in `DroppedNonNumeric`. A tag authored `isHistorized` with one of these types therefore silently records
|
||||
in `DroppedUnsupportedKind` (renamed from `DroppedNonNumeric`). A tag authored `isHistorized` with one of these types therefore silently records
|
||||
nothing; HistoryRead on it returns whatever the historian holds (typically `GoodNoData`).
|
||||
- **Fallback: `UInt16 -> Uint4`.** The historian's `UInt2` write path is unproven upstream, so UInt16 tags
|
||||
are provisioned as the wider `UInt4`. Values are unaffected; only the historian-side storage type differs.
|
||||
@@ -339,7 +455,7 @@ The server supports all four OPC UA HistoryRead variants:
|
||||
| Variant | Node type | CLI `--aggregate` |
|
||||
|---|---|---|
|
||||
| **Raw** | Historized variable | (omit `--aggregate`) |
|
||||
| **Processed** | Historized variable | `Average`, `Minimum`, `Maximum`, `Total`, `Count` |
|
||||
| **Processed** | Historized variable | `Average`, `Minimum`, `Maximum`, `Count`, `Start`, `End`, `StandardDeviation` (the CLI's own list; `SqlServer` serves thirteen aggregates — see the [aggregate table](#aggregates-processed-reads)) |
|
||||
| **AtTime** | Historized variable | n/a (client supplies exact timestamps) |
|
||||
| **Events** | Equipment folder (event notifier) | n/a |
|
||||
|
||||
@@ -384,6 +500,7 @@ node without `isHistorized` has no HistoryRead access bit and answers
|
||||
| Historized node, historian NOT configured (`Enabled=false` / Null source) | `GoodNoData` (empty) |
|
||||
| Non-historized node | `BadHistoryOperationUnsupported` |
|
||||
| Backend timeout or exception | `BadHistoryOperationUnsupported` per node; other nodes in the same batch are unaffected |
|
||||
| `Backend=SqlServer`, `AutoCreateSchema=false`, schema missing/behind | every read/write fails with a clear error naming the missing schema version (health `LastError`); reads answer `BadHistoryOperationUnsupported` per node; no host crash |
|
||||
|
||||
A historized node with no historian configured never returns an error status — it returns
|
||||
empty. This means a deployment can author and publish historized tags before the historian
|
||||
@@ -450,7 +567,9 @@ buckets surface as `BadNoData`.
|
||||
`HistoryAggregateType` carries the wider Part 13 set (Interpolative, TimeAverage, Start, End, Delta, Range,
|
||||
Min/MaxActualTime, Start/EndBound, DeltaBounds, DurationGood/Bad, PercentGood/Bad, WorstQuality,
|
||||
NumberOfTransitions, AnnotationCount, StandardDeviationPopulation, VariancePopulation); `OtOpcUaNodeManager.MapAggregate`
|
||||
maps each standard aggregate NodeId. The **HistorianGateway** backend only has retrieval modes for the original
|
||||
maps each standard aggregate NodeId. The **`SqlServer`** backend serves the thirteen listed in the
|
||||
[aggregate table](#aggregates-processed-reads); everything else (e.g. `Interpolative`, `TimeAverage`,
|
||||
`StandardDeviationPopulation`) is `BadAggregateNotSupported` for that node. The **HistorianGateway** backend only has retrieval modes for the original
|
||||
five (Average, Minimum, Maximum, Total, Count): for any other member `AggregateModeMapper` throws
|
||||
`HistoryAggregateNotSupportedException`, which the HistoryRead path turns into `BadAggregateNotSupported` for that
|
||||
node only (not a generic failure, and not counted as a backend fault). The OPC UA Client driver forwards every
|
||||
@@ -474,10 +593,236 @@ member upstream. A NodeId outside the mapped set (e.g. `StandardDeviationSample`
|
||||
|
||||
### Redundancy and authorization
|
||||
|
||||
History reads are served from any node — there is no Primary gate. Authorization is the
|
||||
standard OPC UA `HistoryRead` permission enforced by the SDK through the `AccessLevels.HistoryRead`
|
||||
bit set at materialization. A session without sufficient permissions receives
|
||||
`BadUserAccessDenied` from the SDK before the dispatch reaches the historian.
|
||||
History reads are served from any node — there is no Primary gate. The `HistoryRead` AccessLevel bit set at
|
||||
materialization admits a node to the read dispatch. With `Security:Acl:Enabled=false` (the default) HistoryRead is
|
||||
**ungated** for any admitted session; with it `true`, the scoped `HistoryRead` grant decides each node
|
||||
(see [security.md](security.md#data-plane-authorization)). **HistoryUpdate is always gated** — see [HistoryUpdate](#historyupdate).
|
||||
|
||||
---
|
||||
|
||||
---
|
||||
|
||||
## SQL Server backend
|
||||
|
||||
Selected with `ServerHistorian:Backend=SqlServer`. Implemented in `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer`
|
||||
(the project does not reference the Gateway project). All five seams share one connection helper, one schema
|
||||
bootstrapper and one tag catalog, so the schema is ensured once per process and tag ids are cached once.
|
||||
|
||||
### Schema
|
||||
|
||||
Created by the embedded idempotent script `Schema/001_initial.sql` (mirrored at
|
||||
[`docs/historian/sqlserver/001_initial.sql`](historian/sqlserver/001_initial.sql)); `$(Schema)` is the configured schema.
|
||||
|
||||
| Object | Purpose |
|
||||
|---|---|
|
||||
| `SchemaVersion(Version, AppliedUtc)` | Applied-script ledger. Required version: **1**. |
|
||||
| `Tag(TagId int identity, Name nvarchar(400) unique, DataType tinyint, EngineeringUnit, Description, CreatedUtc, UpdatedUtc)` | One row per historian tagname. `DataType` is a `DriverDataType`. |
|
||||
| `Sample(TagId, SourceTimeUtc datetime2(7), ValueKind tinyint, NumValue float, IntValue bigint, StrValue nvarchar(max), StatusCode int, ServerTimeUtc)` | **PK `(TagId, SourceTimeUtc)`** — one value per timestamp (Part 11), which makes redelivery idempotent and rules out tie clusters. `StatusCode` is the uint status stored bit-for-bit. |
|
||||
| `Event(EventId nvarchar(64) PK nonclustered, SourceName, AlarmId, AlarmName, AlarmTypeName, Severity, EventKind, Message, UserName, Comment, EventTimeUtc, ReceivedTimeUtc)` | Alarm history. Clustered on `(EventTimeUtc, EventId)`; extra index `(SourceName, EventTimeUtc)`. |
|
||||
| `SampleRow`, `TimeRow` table types | TVPs for batched sample writes/merges, AtTime and delete-at-time. |
|
||||
|
||||
Column mapping per `ValueKind`: `Double` → `NumValue`; `Int64` → `IntValue` + `NumValue` (as a double, for aggregates);
|
||||
`Boolean` → `IntValue` 0/1 + `NumValue` 0/1; `String` → `StrValue`; `DateTime` → `IntValue` (UTC ticks); `Empty` → all NULL.
|
||||
`ValueKind` codes: Empty 0, Double 1, Int64 2, Boolean 3, String 4, DateTime 5. Timestamps are bound as `datetime2(7)`
|
||||
everywhere (full 100 ns precision — never a `DataTable`, which would round to 3.33 ms).
|
||||
|
||||
**Applying the script by hand** (needed when `AutoCreateSchema=false`): create the database, then
|
||||
`sqlcmd -S <server> -U <user> -P <pw> -d <db> -I -v Schema=hist -i docs/historian/sqlserver/001_initial.sql`
|
||||
(`-v Schema=` supplies the `$(Schema)` placeholder; verified against the docker-dev SQL container). Re-running is a no-op.
|
||||
|
||||
### Writes
|
||||
|
||||
- **Live values** (`IHistorianValueWriter`): resolve `TagId` through a cache, creating the `Tag` row on a miss
|
||||
(`MERGE … WITH (HOLDLOCK)`; `DataType` from the first non-Empty value's kind, provisioning corrects it later), then
|
||||
`INSERT … SELECT … WHERE NOT EXISTS` through the `SampleRow` TVP. A null timestamp is stamped now;
|
||||
`ServerTimeUtc = SYSUTCDATETIME()`. It returns `false` (never throws) on a SQL failure so the recorder's retry/quarantine
|
||||
owns it. Semantics: [delivery semantics](#outbox-format-v2-and-rollback).
|
||||
- **Alarm events** (`IAlarmHistorianWriter`): per event, insert-if-absent keyed by a deterministic `EventId` = lowercase-hex
|
||||
SHA-256 over the length-prefixed fields `EquipmentPath, AlarmId, EventKind, TimestampUtc.Ticks, User, Comment, Message`.
|
||||
Outcomes: `Ack` on insert or already-present (including a concurrent writer winning a 2627/2601 race); `RetryPlease` on
|
||||
transient SQL errors (connection, timeout, deadlock 1205 …), bootstrap failures, cancellation and any unclassified error;
|
||||
`PermanentFail` (dead-letter) only for data errors SQL will reject forever (truncation 2628/8152, conversion, null, range, constraint).
|
||||
`SourceName` is the event's `EquipmentPath`; `Severity` is the numeric Part 9 value (Low 200, Medium 500, High 700, Critical 900).
|
||||
- **Tag names** are `nvarchar(400)`. `EnsureTags` fails a longer name individually, but the live-value writer's tag-create
|
||||
path does **not** reject one (the parameter silently truncates) — keep historian tagnames under 400 characters.
|
||||
|
||||
### Raw reads
|
||||
|
||||
`ReadRawAsync` returns every sample as stored — outage markers (`Empty` + Bad status) included.
|
||||
|
||||
| Request | Window | Order |
|
||||
|---|---|---|
|
||||
| `start < end` | `[start, end)` | ascending |
|
||||
| `end < start` (Part 11 reverse) | `(end, start]` | descending |
|
||||
| `start == end` | the sample exactly at that time | — |
|
||||
|
||||
`NumValuesPerNode == 0` is unbounded (larger values clamp to `int.MaxValue`). An unknown tag is an empty result (`GoodNoData`);
|
||||
reads **never create** tag rows. The node manager's paging is unchanged; because the PK rules out tie clusters, resuming
|
||||
"from the last timestamp inclusive and trimming it" is exact.
|
||||
|
||||
### AtTime reads
|
||||
|
||||
One statement per call: the distinct requested times go in a `TimeRow` TVP and each picks up its previous-or-equal sample `p` and
|
||||
next sample `n` via two `OUTER APPLY`s. A *usable* sample has severity Good or Uncertain **and** a value (an `Empty` sample is an
|
||||
outage marker and counts as Bad whatever its code). Per requested time `T` (the result is stamped `T`):
|
||||
|
||||
| Case | Result |
|
||||
|---|---|
|
||||
| `p` at exactly `T` | the stored value and status, unchanged |
|
||||
| no `p` (before the first sample), or unknown tag | `BadNoData`, null |
|
||||
| **`p` is Bad** (`T` lies inside an outage) | **null value carrying `p`'s Bad status code** (info bits stripped) — never better than its source |
|
||||
| `p` usable, **`n` Bad** | **`p`'s value held** (not interpolated into the outage), `UncertainDataSubNormal` + Interpolated |
|
||||
| `Double` / `Int64` with a usable `n` | linear interpolation of `NumValue`; `Int64` is computed **through `double`** and rounded to the nearest integer (ties away from zero), keeping the `Int64` type. Status `Good` + Interpolated when both neighbours are Good, else `UncertainDataSubNormal` + Interpolated |
|
||||
| `Boolean` / `String` / `DateTime`, **or `T` after the last sample** | **stepped**: `p`'s value with `p`'s status + Interpolated |
|
||||
|
||||
⚠️ Because `Int64` interpolation goes through `double`, magnitudes beyond 2^53 lose precision.
|
||||
|
||||
### Events
|
||||
|
||||
`[start, end)` by `EventTimeUtc`; **a reversed window (`end < start`) is swapped and returned ascending**. A null `sourceName` means all
|
||||
sources, otherwise `SourceName = @source`. Ordered `(EventTimeUtc, EventId)` ascending. A bounded read fetches `maxEvents + 1` rows and
|
||||
returns a non-null continuation point when the extra row exists; the node manager does not currently issue or resume event
|
||||
continuation points, so a client-capped read returns only the first page.
|
||||
|
||||
### Aggregates (Processed reads)
|
||||
|
||||
> **These are simple per-sample definitions, not full OPC UA Part 13.** There is no time-weighting, no interpolated interval-boundary
|
||||
> value and no partial-interval status. They are pushed down to T-SQL and suit trend-style consumption. Any `HistoryAggregateType` not
|
||||
> in the table throws `HistoryAggregateNotSupportedException` ⇒ `BadAggregateNotSupported` for that node (e.g. `Interpolative`,
|
||||
> `TimeAverage`, `StartBound`, `DurationGood`, `StandardDeviationPopulation`).
|
||||
|
||||
**Bucketing.** One result per interval of `[start, end)`; bucket *i* = `[start + i·interval, start + (i+1)·interval)`, start-inclusive /
|
||||
end-exclusive to the tick, the last bucket truncated at `end`. `interval <= 0` ⇒ a single interval `[start, end)`; `start == end` ⇒ no
|
||||
intervals. The bucket index is computed in **100 ns ticks** (not milliseconds): `DATEDIFF_BIG(microsecond, @start, t) × 10` plus a
|
||||
sub-microsecond correction from the two tick digits, divided by the interval in ticks. (`DATEDIFF` counts unit *boundaries crossed*, so a
|
||||
millisecond formulation misplaces samples near a bucket edge, and a nanosecond one overflows past ≈292 years.) A window of more than
|
||||
**100 000 intervals** is refused before any I/O and reported `BadHistoryOperationUnsupported` for that node. **Reverse windows**
|
||||
(`end < start`) are swapped, computed forward **anchored at the earlier time**, and returned in **descending** bucket order. Every returned
|
||||
status carries the historian *Calculated* info bit, `BadNoData` included.
|
||||
|
||||
**Sample classes.** *Usable* = severity Good or Uncertain **and** a stored value (not `Empty`). *Good* = usable with severity Good.
|
||||
*Bad* = not usable.
|
||||
|
||||
| Aggregate | Computed over | Value timestamp |
|
||||
|---|---|---|
|
||||
| `Average`, `Minimum`, `Maximum`, `Total` | `AVG` / `MIN` / `MAX` / `SUM` of `NumValue` over usable samples | bucket start |
|
||||
| `Count` | number of usable samples (`Int32`) | bucket start |
|
||||
| `Range` | `Maximum − Minimum` | bucket start |
|
||||
| `Start`, `End` | first / last usable sample by time (keeps the sample's type) | bucket start |
|
||||
| `Delta` | `End − Start` (numeric samples) | bucket start |
|
||||
| `MinimumActualTime`, `MaximumActualTime` | the usable sample with the smallest / largest `NumValue`; ties ⇒ earliest | **the sample's own time** |
|
||||
| `PercentGood` | `100 × (Good samples) / (all samples)` | bucket start |
|
||||
| `PercentBad` | `100 × (Bad samples) / (all samples)` | bucket start |
|
||||
|
||||
Status and edge rules:
|
||||
|
||||
- An **empty interval** (no samples) ⇒ `BadNoData`. **`Count` — and every value aggregate — over an interval whose samples are all Bad ⇒ `BadNoData`**, not `0`.
|
||||
- Value aggregates are `Good` when every sample in the bucket is Good and `UncertainDataSubNormal` when the bucket also holds any Bad **or Uncertain** sample.
|
||||
- `Count`, `Start`, `End`, `PercentGood`, `PercentBad` accept any value kind; the other aggregates (and `Delta`) on a bucket that holds a usable `String`/`DateTime` sample ⇒ `BadAggregateInvalidInputs` for that interval.
|
||||
- **`PercentGood` / `PercentBad` and Uncertain samples:** Uncertain samples are counted in the denominator but are **neither Good nor Bad**, so `PercentGood + PercentBad < 100` whenever a bucket holds Uncertain samples. Both report status `Good` for any non-empty bucket.
|
||||
- Everything except `Count` and `Start`/`End` is a `double`; `Int64` samples aggregate as doubles and `Boolean` as 0/1.
|
||||
|
||||
### Health
|
||||
|
||||
The read client reports `GetHealthSnapshot()` (query/failure counters, consecutive failures, `LastError` — e.g. a missing schema under
|
||||
`AutoCreateSchema=false`). A SQL error on a read surfaces as `BadHistoryOperationUnsupported` for **that node only**; a cancelled read
|
||||
(deadline, client abort) is not counted as a backend fault.
|
||||
|
||||
### Docker-dev recipe
|
||||
|
||||
The docker-dev `sql` container (`otopcua-dev-sql-1`, host port `14330`, `sa` with the committed dev password from
|
||||
`docker-dev/docker-compose.yml`) already hosts the ConfigDb; give the historian its own database on the same instance.
|
||||
|
||||
```bash
|
||||
# 1. Create the database (the backend creates the schema + tables on first use, not the database).
|
||||
sqlcmd -S localhost,14330 -U sa -P 'OtOpcUa!Dev123' -C -I -Q "CREATE DATABASE OtOpcUaHistorian"
|
||||
|
||||
# 2. Point the central nodes at it. Add to the `environment:` of central-1 AND central-2 in docker-dev/docker-compose.yml
|
||||
# (or a compose override), rebuild the images (the rig image goes stale), then `up -d`:
|
||||
# ServerHistorian__Enabled: "true"
|
||||
# ServerHistorian__Backend: "SqlServer"
|
||||
# ServerHistorian__SqlServer__ConnectionString: "Server=sql,1433;Database=OtOpcUaHistorian;User Id=sa;Password=OtOpcUa!Dev123;TrustServerCertificate=True"
|
||||
# ContinuousHistorization__Enabled: "true"
|
||||
# ContinuousHistorization__OutboxPath: "/app/data/historization-outbox"
|
||||
# (ServerHistorian__Endpoint / __ApiKey are not used by this backend.)
|
||||
|
||||
# 3. Mark a rig tag historized ("isHistorized": true in its TagConfig), deploy, then watch rows land:
|
||||
sqlcmd -S localhost,14330 -U sa -P 'OtOpcUa!Dev123' -C -I -d OtOpcUaHistorian \
|
||||
-Q "SELECT t.Name, COUNT(*) n, MAX(s.SourceTimeUtc) last FROM hist.Sample s JOIN hist.Tag t ON t.TagId=s.TagId GROUP BY t.Name"
|
||||
```
|
||||
|
||||
Only the Primary of a pair records, so the row count advances once per published value, not twice. Read it back with
|
||||
`Client.CLI historyread` and rewrite it with `Client.CLI historyupdate` ([Client.CLI.md](Client.CLI.md)). The password above is the
|
||||
committed docker-dev credential, not a secret; use `${secret:…}` / environment for anything real.
|
||||
|
||||
### Integration tests
|
||||
|
||||
`tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests` (bootstrap incl. two concurrent bootstrappers, every kind
|
||||
round-trip, redelivery, raw / AtTime / aggregate / event reads, every update result code) is gated on the environment variable
|
||||
**`HISTORIAN_SQL_TEST_CONNECTION`** — a connection string with `CREATE`/`DROP DATABASE` rights. Absent ⇒ every test skips. Present ⇒ each
|
||||
run creates a throwaway `OtOpcUaHistorian_T_<guid>` database and drops it on dispose.
|
||||
|
||||
```bash
|
||||
export HISTORIAN_SQL_TEST_CONNECTION='Server=localhost,14330;User Id=sa;Password=OtOpcUa!Dev123;TrustServerCertificate=True;Encrypt=False'
|
||||
dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## HistoryUpdate
|
||||
|
||||
OPC UA **HistoryUpdate** lets a client insert, replace, update and delete stored raw history. It is available only when the selected
|
||||
backend provides an updater — **`SqlServer` does; `Gateway` does not**, and an updater is registered only when `ServerHistorian:Enabled=true`.
|
||||
Updates go straight to the backend (not through the outbox), are **not Primary-gated** (the database is shared), and share the HistoryRead
|
||||
batch limiter and `HistoryReadDeadline`.
|
||||
|
||||
### Operations
|
||||
|
||||
| OPC UA request | Server override | Per-item results (`OperationResults`) |
|
||||
|---|---|---|
|
||||
| `UpdateDataDetails`, `PerformInsertReplace = Insert` | `HistoryUpdateData` | `GoodEntryInserted`; `BadEntryExists` if a sample already exists at that timestamp |
|
||||
| …`Replace` | `HistoryUpdateData` | `GoodEntryReplaced`; `BadNoEntryExists` if none exists |
|
||||
| …`Update` | `HistoryUpdateData` | `GoodEntryInserted` or `GoodEntryReplaced` (upsert) |
|
||||
| `DeleteRawModifiedDetails` (`IsDeleteModified=false`) | `HistoryDeleteRawModified` | one node-level code: `Good`; `BadNoData` if `[start, end)` held nothing |
|
||||
| `DeleteAtTimeDetails` | `HistoryDeleteAtTime` | per time: `Good`; `BadNoEntryExists` |
|
||||
|
||||
Each backend call is one transaction (`MERGE … OUTPUT $action` against the TVP). The node-level `StatusCode` stays `Good` when item results are
|
||||
mixed (Part 11 reports item outcomes in `OperationResults`); it is Bad only for whole-node failures.
|
||||
|
||||
**Per-item rejections** (answered without touching SQL; other items still proceed): a null item ⇒ `BadInvalidArgument`;
|
||||
`SourceTimestamp == DateTime.MinValue` ⇒ `BadInvalidTimestamp`; a value the historian cannot type (array, unsupported CLR type) ⇒
|
||||
`BadTypeMismatch`; and a **`NaN` / `±Infinity` `Double` ⇒ `BadInvalidArgument`** — it cannot live in a SQL `float`, and unlike the live-value
|
||||
writer (which stores it as `Empty`) an operator's explicit update is rejected rather than silently rewritten. A caller-supplied status code is stored as given.
|
||||
|
||||
**Duplicate timestamps inside one request** (compared at 100 ns): *Insert* — the first occurrence is applied, later ones are `BadEntryExists`;
|
||||
*Replace* / *Update* — the last one wins and every earlier occurrence reports the winner's code; *DeleteAtTime* — the first reports the real
|
||||
outcome, later ones `BadNoEntryExists`. **Unknown tag:** Insert / Update create the tag row; Replace and the deletes report `BadNoEntryExists`
|
||||
(DeleteRaw: `BadNoData`) without creating anything.
|
||||
|
||||
**Unsupported ⇒ `BadHistoryOperationUnsupported`** (per node): the node is not historized; the backend has no updater (`Gateway`, or
|
||||
`ServerHistorian:Enabled=false`); `PerformUpdateType.Remove`; `DeleteRawModifiedDetails.IsDeleteModified=true` (no modified-value history is
|
||||
kept); and the out-of-scope detail types — **event** HistoryUpdate (`UpdateEventDetails` / `DeleteEventDetails`) and `UpdateStructureData` —
|
||||
which are not overridden, so the SDK base rejects them. A backend exception is `BadInternalError` for that node and a deadline is `BadTimeout`.
|
||||
No `AuditHistoryUpdateEvent` is emitted.
|
||||
|
||||
### The `HistoryWrite` AccessLevel bit
|
||||
|
||||
The SDK base admits a node to HistoryUpdate only if its `AccessLevel` carries `HistoryWrite`. The server ORs that bit in **only when the node is
|
||||
historized *and* an updater is wired at materialise time** (the raw node and every UNS mirror; the surgical `UpdateTagAttributes` path recomputes
|
||||
it the same way). `OtOpcUaServerHostedService.StartAsync` therefore wires the updater **before** installing the sink, and nodes materialised
|
||||
earlier are never retro-updated. With `Backend=Gateway` the bit is never set.
|
||||
|
||||
### Authorization
|
||||
|
||||
HistoryUpdate is gated in **both** ACL modes (rewriting recorded history was never ungated). `WriteOperate` is deliberately **not** reused.
|
||||
|
||||
| `Security:Acl:Enabled` | Decision |
|
||||
|---|---|
|
||||
| `false` (default) | The session's API key is re-checked first (`AuthorizeKeyUsable`, Gitea #563 — an expired or disabled key loses its activation-time roles), then the session must carry the **`HistoryUpdate` role**. `ApiKeyInterimRoleMapper` confers it **only from a `Cluster`-scoped grant** carrying `NodePermissions.HistoryUpdate`; narrower-scoped grants confer nothing (the same under-grant rule as `WriteOperate`). Denied ⇒ `BadUserAccessDenied`. A `WriteOperate`-only key is denied. |
|
||||
| `true` | `DataPlaneAuthorizer.Authorize(identity, OpcUaOperation.HistoryUpdate, nodeId, realm)` — the scoped `NodeAcl` grant decides per node and **replaces** the role check (never chained), so a session with no roles succeeds on the strength of a scoped grant. Denied ⇒ `BadUserAccessDenied` for that node. |
|
||||
|
||||
`NodePermissions.HistoryUpdate` (bit 12) is authorable in the AdminUI ACL editor and is **not** part of any bundle
|
||||
(`ReadOnly` / `Operator` / `Engineer` / `Admin`) — it must be granted explicitly. See [security.md](security.md#data-plane-authorization).
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -11,7 +11,7 @@ OtOpcUa is a multi-driver OPC UA server. The Core (`ZB.MOM.WW.OtOpcUa.Core` + `C
|
||||
- `IAlarmSource` — driver-emitted OPC UA A&C events
|
||||
- `IRediscoverable` — driver-initiated address-space rebuild notifications
|
||||
- `ISkippedTagReporter` — authored tags the driver could not build into its poll table (see [Skipped tags](#skipped-tags) below)
|
||||
- `IHistorianDataSource` — server-side historian read backend registration (the HistorianGateway backend) (drivers carry no history capability; Gitea #655)
|
||||
- `IHistorianDataSource` — server-side historian read backend registration (the selected `ServerHistorian:Backend` — HistorianGateway or SQL Server) (drivers carry no history capability; Gitea #655)
|
||||
|
||||
Each driver opts into only the capabilities it supports. Every async capability call at the Server dispatch layer goes through `CapabilityInvoker` (`Core/Resilience/CapabilityInvoker.cs`), which wraps it in a Polly pipeline keyed on `(DriverInstanceId, HostName, DriverCapability)`. The `OTOPCUA0001` analyzer enforces the wrap at build time. Drivers themselves never depend on Polly; they just implement the capability interface and let the Core wrap it.
|
||||
|
||||
@@ -75,7 +75,8 @@ Three properties worth knowing:
|
||||
| [MTConnect](MTConnect.md) | `Driver.MTConnect` (+ `.Contracts`) | — | Hand-rolled `System.Xml.Linq` HTTP/XML client against a vendor-neutral MTConnect Agent (`/probe`, `/current`, `/sample`) — no TrakHound dependency (dropped; see the driver doc) | IDriver, ITagDiscovery, IReadable, ISubscribable, IHostConnectivityProbe, IRediscoverable | Read-only by design (no `IWritable`). Production data plane is entirely `ISubscribable`/`OnDataChange` — `IReadable` is CLI/test-only. Browse is free via the Wave-0 universal discovery browser, no bespoke browser project. `IRediscoverable`/`IHostConnectivityProbe` have no server-side consumer yet — a pre-existing fleet-wide gap, not MTConnect-specific |
|
||||
| [Sql](Sql.md) | `Driver.Sql` (+ `.Browser`, `.Contracts`) | A | `Microsoft.Data.SqlClient` behind an `ISqlDialect` seam (`SqlServerDialect` shipped; Postgres/ODBC deferred) | IDriver, ITagDiscovery, IReadable, ISubscribable, IHostConnectivityProbe | Read-only DB-poll driver with a bespoke schema browser. **Credentials are never persisted** — the deploy gate rejects a stored `connectionString` (#498) and catalog identifiers are validated against the live catalog's own spelling (#496). `SupportsOnlineDiscovery => false` by design (the bespoke browser wins). `SqlTagModel.Query` is deferred to P3 and rejected end-to-end by parser, validator, planner and editor. Merged `4ad54037` |
|
||||
| [Calculation](Calculation.md) | `Driver.Calculation` | A | none — computes from other tags' RawPaths | IDriver, IDependencyConsumer, ISubscribable, IReadable | **Pseudo-driver**, not a wire protocol: signal-level tags computed from other tags via a script, with deploy-time `scriptId`-existence + Tarjan cycle gates. Its probe is parse-only (no backend to reach). `runTimeoutMs` is authorable through `CalculationDriverForm` (the "no typed config form" gap — G-1 — was closed 2026-07-27) |
|
||||
| [Historian.Gateway](../Historian.md) | `Driver.Historian.Gateway` | — | `ZB.MOM.WW.HistorianGateway.Client` gRPC (`historian_gateway.v1`) | IHistorianDataSource (server-side read backend) + alarm `SendEvent` writer + `WriteLiveValues` recorder + `IHistorianProvisioning` | Not a tag driver — the sole historian backend. Registers `GatewayHistorianDataSource : IHistorianDataSource` for HistoryRead and serves alarm-write + continuous historization through the gateway. No `IDriver`/`ITagDiscovery` surface. (The retired Wonderware sidecar backend it replaced is documented at [Historian.Wonderware.md](Historian.Wonderware.md).) |
|
||||
| [Historian.Gateway](../Historian.md) | `Driver.Historian.Gateway` | — | `ZB.MOM.WW.HistorianGateway.Client` gRPC (`historian_gateway.v1`) | IHistorianDataSource (server-side read backend) + alarm `SendEvent` writer + `WriteLiveValues` recorder + `IHistorianProvisioning` | Not a tag driver — the default historian backend (the other is `Historian.SqlServer`, below). Registers `GatewayHistorianDataSource : IHistorianDataSource` for HistoryRead and serves alarm-write + continuous historization through the gateway. No `IDriver`/`ITagDiscovery` surface. (The retired Wonderware sidecar backend it replaced is documented at [Historian.Wonderware.md](Historian.Wonderware.md).) |
|
||||
| [Historian.SqlServer](../Historian.md#sql-server-backend) | `Driver.Historian.SqlServer` | — | `Microsoft.Data.SqlClient` (a plain SQL Server database, schema `hist` by default) | `IHistorianDataSource` + `IHistorianValueWriter` + `IAlarmHistorianWriter` + `IHistorianProvisioning` + `IHistorianDataUpdater` | Not a tag driver — a pluggable historian backend selected with `ServerHistorian:Backend=SqlServer`. Stores typed values with their real status, serves Raw / AtTime / Events / 13 push-down aggregates, and is the only backend that supports OPC UA **HistoryUpdate**. No `IDriver`/`ITagDiscovery` surface. |
|
||||
|
||||
## Per-driver documentation
|
||||
|
||||
@@ -99,8 +100,8 @@ Three properties worth knowing:
|
||||
data-plane precedence rules, and a non-trivial known-limitations list need explaining up front:
|
||||
- [MTConnect.md](MTConnect.md) — Agent config, `FullName`==DataItem `id`, `UNAVAILABLE`→`BadNoCommunication`, CONDITION-as-`String`, browse-via-universal-browser, fixture recipe, deferred write-back
|
||||
|
||||
- **Historian.Gateway** (server-side historian backend, not a tag driver) is documented in the main guide:
|
||||
- [../Historian.md](../Historian.md) — HistorianGateway backend: read-path registration, HistoryRead dispatch, alarm store-and-forward (`SendEvent`), continuous historization (`WriteLiveValues`), `EnsureTags` provisioning, config keys, deployment prerequisites. (The retired Wonderware sidecar backend it replaced: [Historian.Wonderware.md](Historian.Wonderware.md).)
|
||||
- **Historian.Gateway** and **Historian.SqlServer** (server-side historian backends, not tag drivers) are documented in the main guide:
|
||||
- [../Historian.md](../Historian.md) — backend selection (`ServerHistorian:Backend`), the SQL Server backend and HistoryUpdate, plus the HistorianGateway backend: read-path registration, HistoryRead dispatch, alarm store-and-forward (`SendEvent`), continuous historization (`WriteLiveValues`), `EnsureTags` provisioning, config keys, deployment prerequisites. (The retired Wonderware sidecar backend it replaced: [Historian.Wonderware.md](Historian.Wonderware.md).)
|
||||
|
||||
- The full per-field spec (capability surface, config schema, addressing, data-type maps, connection settings, quirks for every driver) lives in [docs/v2/driver-specs.md](../v2/driver-specs.md). The overview pages above are the short path and are the current reference; that file is the v2-era per-field spec (see its status banner for which drivers it covers).
|
||||
|
||||
@@ -121,7 +122,7 @@ Each driver has a dedicated fixture doc that lays out what the integration / uni
|
||||
|
||||
## Related cross-driver docs
|
||||
|
||||
- [HistoricalDataAccess.md](../v1/HistoricalDataAccess.md) — v1 archive of the retired per-driver history design (aggregate mapping, continuation points). History is now served only by the server-side `IHistorianDataSource` (HistorianGateway); no driver implements a history capability, and nodes without `isHistorized` return `BadHistoryOperationUnsupported`.
|
||||
- [HistoricalDataAccess.md](../v1/HistoricalDataAccess.md) — v1 archive of the retired per-driver history design (aggregate mapping, continuation points). History is now served only by the server-side `IHistorianDataSource` (the selected historian backend: HistorianGateway or SQL Server); no driver implements a history capability, and nodes without `isHistorized` return `BadHistoryOperationUnsupported`.
|
||||
- [AlarmTracking.md](../AlarmTracking.md) — `IAlarmSource` event model and filtering. Implemented by Galaxy (native MxAccess alarms, working end-to-end), OPC UA Client, AB CIP, and FOCAS; AB Legacy, Modbus, S7, TwinCAT, and MQTT have no alarm source.
|
||||
- [Subscriptions.md](../v1/Subscriptions.md) — how the Server multiplexes subscriptions onto `ISubscribable.OnDataChange`.
|
||||
- [docs/v2/driver-stability.md](../v2/driver-stability.md) — historical design record of the A / B / C tier model. Only `DriverTier` + the `CapabilityPolicy` defaults per tier × capability (`DriverResilienceOptions.GetTierDefaults`) are live; its `MemoryTracking` / recycle / out-of-process isolation machinery was deleted (#522) and every driver runs Tier A.
|
||||
|
||||
@@ -80,7 +80,9 @@ It lives in **Runtime** because the seams it returns span `Core.Abstractions` an
|
||||
- `ServerHistorianOptionsValidator` runs its three gateway checks (Endpoint, ApiKey, UseTls⇔scheme) only when `Backend == Gateway`; backend-specific options are validated there too, consumer-gated (`ServerHistorian:Enabled || AlarmHistorian:Enabled`), never at registry resolution.
|
||||
- New `SqlServerHistorianOptions` validation: `ConnectionString` required, `Schema` a valid SQL identifier
|
||||
(`^[A-Za-z_][A-Za-z0-9_]{0,127}$` — it is spliced into DDL/DML), `CommandTimeout > 0`.
|
||||
- Validators remain driver-block-only (admin-only nodes never build historian singletons).
|
||||
- Validators remain driver-block-only (admin-only nodes never build historian singletons). *As built:* an invalid `SqlServer:Schema`
|
||||
also throws `ArgumentException` when the provider is constructed, so `Backend=SqlServer` + a malformed schema stops host start even
|
||||
when no consumer is enabled.
|
||||
- The `Program.cs` alarm-only warning names the selected backend instead of "gateway".
|
||||
- The historian connection string is **not** a ConfigDb string; driver-only nodes may carry it (Phase 4 intact).
|
||||
|
||||
@@ -160,14 +162,14 @@ CREATE TABLE {s}.Sample (
|
||||
CONSTRAINT FK_Sample_Tag FOREIGN KEY (TagId) REFERENCES {s}.Tag(TagId));
|
||||
|
||||
CREATE TABLE {s}.Event (
|
||||
EventId nvarchar(64) NOT NULL PRIMARY KEY, -- deterministic SHA-256 hex of the event payload
|
||||
EventId nvarchar(64) NOT NULL CONSTRAINT PK_Event PRIMARY KEY NONCLUSTERED, -- deterministic SHA-256 hex of the event payload
|
||||
SourceName nvarchar(400) NOT NULL, -- AlarmHistorianEvent.EquipmentPath
|
||||
AlarmId nvarchar(400) NOT NULL, AlarmName nvarchar(400) NOT NULL, AlarmTypeName nvarchar(100) NOT NULL,
|
||||
Severity int NOT NULL, EventKind nvarchar(50) NOT NULL, Message nvarchar(max) NULL,
|
||||
UserName nvarchar(200) NULL, Comment nvarchar(max) NULL,
|
||||
EventTimeUtc datetime2(7) NOT NULL, ReceivedTimeUtc datetime2(7) NOT NULL);
|
||||
CREATE CLUSTERED INDEX CX_Event_Time ON {s}.Event (EventTimeUtc, EventId); -- as built (§9.6); the sketched IX_Event_Time is replaced by this
|
||||
CREATE INDEX IX_Event_Source_Time ON {s}.Event (SourceName, EventTimeUtc);
|
||||
CREATE INDEX IX_Event_Time ON {s}.Event (EventTimeUtc);
|
||||
|
||||
CREATE TYPE {s}.SampleRow AS TABLE (
|
||||
SourceTimeUtc datetime2(7) NOT NULL PRIMARY KEY, ValueKind tinyint NOT NULL,
|
||||
@@ -196,7 +198,10 @@ next call (not latched).
|
||||
`ConcurrentDictionary<string,int>` cache, creating the tag row on a miss (`MERGE` on Name; DataType from
|
||||
the first value's kind, provisioning later corrects it). Insert via `{s}.SampleRow` TVP with
|
||||
`INSERT … SELECT … WHERE NOT EXISTS`. Returns `true` on success, `false` on a SQL failure (the recorder's
|
||||
retry/quarantine owns retry). `ServerTimeUtc = SYSUTCDATETIME()`.
|
||||
retry/quarantine owns retry). `ServerTimeUtc = SYSUTCDATETIME()`. *As built:* duplicate timestamps within one batch
|
||||
collapse to the **first**; a `NaN`/`±Inf` `Double` is stored as `Empty` keeping its status (it cannot live in a SQL
|
||||
`float`); a 2627/2601 duplicate-key error rolls the statement back, so the statement is re-run (≤ 2 retries) rather
|
||||
than treated as success.
|
||||
- `IHistorianProvisioning.EnsureTagsAsync`: `MERGE` upsert of every request (all types — no numeric
|
||||
filter on this backend). Per-request failures are counted, never thrown.
|
||||
- `IAlarmHistorianWriter.WriteBatchAsync`: per event insert-if-absent; outcome `Ack` on insert or
|
||||
@@ -211,20 +216,25 @@ next call (not latched).
|
||||
tie clusters cannot occur (PK).
|
||||
- **AtTime:** one statement, `{s}.TimeRow` TVP + two `OUTER APPLY`s (previous-or-equal, next). Exact ⇒ stored
|
||||
value + status. Otherwise Double/Int64 linearly interpolated (status `Good` with the historian
|
||||
*Interpolated* info bits), Boolean/String/DateTime stepped from the previous
|
||||
sample (status keeps the previous sample's code + Interpolated bit); no previous sample ⇒ `BadNoData`.
|
||||
*Interpolated* info bits **when both neighbours are Good, else `UncertainDataSubNormal` + Interpolated**; Int64 is
|
||||
interpolated through `double` and rounded, ties away from zero), Boolean/String/DateTime — and any time **after the last
|
||||
sample** — stepped from the previous sample (status keeps the previous sample's code + Interpolated bit); no previous
|
||||
sample ⇒ `BadNoData`.
|
||||
If the **previous** neighbour is Bad, the requested time lies inside an outage: the result is a null value
|
||||
carrying that neighbour's Bad code (never better than its source, per the quality contract; corrected after the
|
||||
T10 review). If the previous neighbour is usable and the **next** is Bad, the previous value is held (stepped,
|
||||
not interpolated into the outage) with `UncertainDataSubNormal`.
|
||||
- **Events:** `TOP(@maxEvents)` from `{s}.Event` filtered by `SourceName = @source` (null ⇒ all) and
|
||||
`EventTimeUtc` range, ordered ascending. `HistoricalEvent(EventId, SourceName, EventTimeUtc,
|
||||
`EventTimeUtc` range, ordered ascending (a reversed window is swapped and still returned ascending). `HistoricalEvent(EventId, SourceName, EventTimeUtc,
|
||||
ReceivedTimeUtc, Message, (ushort)Severity)`.
|
||||
- **Processed (push-down):** bucket `b = DATEDIFF_BIG(millisecond, @start, SourceTimeUtc) / @intervalMs`,
|
||||
one row per bucket start `@start + b × interval` for every interval in `[start, end)` (empty intervals
|
||||
emitted as `BadNoData`). Usable sample = status severity bits Good (`00`) or Uncertain (`01`). Every result
|
||||
carries the historian *Calculated* info bits. An interval with any Bad
|
||||
sample but ≥1 usable ⇒ `UncertainDataSubNormal`; no usable ⇒ `BadNoData`.
|
||||
- **Processed (push-down):** bucket `b = elapsedTicks / @intervalTicks` in **100 ns ticks** (*corrected after T10: the original
|
||||
sketch used `DATEDIFF_BIG(millisecond, …) / @intervalMs`; `DATEDIFF` counts boundaries crossed, so a ms formulation
|
||||
misplaces samples near a bucket edge and a ns one overflows past ≈292 years. As built:
|
||||
`DATEDIFF_BIG(microsecond, @start, t) × 10` plus a sub-microsecond tick correction*), one row per bucket start `@start + b × interval` for every interval in `[start, end)` (empty intervals
|
||||
emitted as `BadNoData`). Usable sample = status severity bits Good (`00`) or Uncertain (`01`) **and a stored value** (an `Empty`
|
||||
outage marker is Bad whatever its code). Every result carries the historian *Calculated* info bits. An interval with any
|
||||
**non-Good (Bad or Uncertain)** sample but ≥1 usable ⇒ `UncertainDataSubNormal`; no usable ⇒ `BadNoData` (so `Count` over an
|
||||
all-Bad interval is `BadNoData`, not 0). A reversed window is swapped, anchored at the earlier time, and returned descending.
|
||||
|
||||
| Aggregate | SQL over usable samples in the bucket | Value timestamp |
|
||||
|---|---|---|
|
||||
@@ -233,7 +243,7 @@ next call (not latched).
|
||||
| Delta | End − Start | bucket start |
|
||||
| Range | Max − Min | bucket start |
|
||||
| MinimumActualTime / MaximumActualTime | min / max value, ties ⇒ earliest | the sample's time |
|
||||
| PercentGood / PercentBad | `100 × count(good or bad) / count(*)` over ALL samples | bucket start |
|
||||
| PercentGood / PercentBad | `100 × count(Good) / count(*)` and `100 × count(Bad) / count(*)` over ALL samples; **Uncertain samples are neither** (so the two can sum to < 100) | bucket start |
|
||||
|
||||
Count / Start / End / PercentGood / PercentBad accept any kind; the rest on a String/DateTime tag ⇒
|
||||
`BadAggregateInvalidInputs` per interval. Any other `HistoryAggregateType` ⇒ throw
|
||||
@@ -262,7 +272,11 @@ public interface IHistorianDataUpdater
|
||||
| DeleteAtTime | per time: `Good`; `BadNoEntryExists` |
|
||||
|
||||
Each call is one transaction (`MERGE … OUTPUT $action` against the TVP). A value with a null timestamp ⇒
|
||||
`BadInvalidTimestamp` for that item. Update/Insert on an unknown tag creates the tag row (same as live
|
||||
`BadInvalidTimestamp` for that item. *As built:* a `NaN`/`±Inf` `Double` ⇒ `BadInvalidArgument` (rejected, not stored as `Empty`
|
||||
as the live-value writer does); duplicate timestamps in one request — Insert: first applies, later `BadEntryExists`; Replace/Update:
|
||||
last wins, earlier ones share its code; DeleteAtTime: first reports, later `BadNoEntryExists`. The node manager answers a null item
|
||||
`BadInvalidArgument`, `MinValue` `BadInvalidTimestamp`, an untypable value `BadTypeMismatch`, a backend throw `BadInternalError`
|
||||
and a deadline `BadTimeout` (a failure is not "unsupported"). Update/Insert on an unknown tag creates the tag row (same as live
|
||||
writes); Replace/Delete on an unknown tag ⇒ `BadNoEntryExists` / `BadNoData`.
|
||||
|
||||
**Status codes.** Every status this backend produces is a named `const uint` in an internal
|
||||
|
||||
+13
-2
@@ -416,7 +416,9 @@ write and each Part 9 method call first asks `DataPlaneAuthorizer.AuthorizeKeyUs
|
||||
`AclGate: denied … key … is no longer enabled or has expired`), checked *before* the "no gateway ⇒
|
||||
`BadNotWritable`" availability answer. Things to know:
|
||||
|
||||
- **Only these two gates.** Read, Browse, Subscribe and HistoryRead stay ungated with the flag off —
|
||||
- **Only these two gates — plus HistoryUpdate.** (HistoryUpdate, added with the pluggable historian, is a third gated
|
||||
operation in both modes; see [HistoryUpdate](#historyupdate-historian-writes). It re-checks the key the same way.)
|
||||
Read, Browse, Subscribe and HistoryRead stay ungated with the flag off —
|
||||
that is the flag-off posture, and an expired key's session can still read and keep its subscriptions.
|
||||
Turn `Security:Acl:Enabled` on to have expiry reach those too.
|
||||
- **Anonymous is unaffected.** It carries no key id, so there is nothing to expire; its roles stay what
|
||||
@@ -484,6 +486,7 @@ the seams consult. Same dark-switch discipline as `MeshTransport`, `ConfigSource
|
||||
| Alarm Confirm | role `AlarmAck` | `AlarmConfirm` |
|
||||
| Alarm OneShotShelve / TimedShelve / Unshelve | role `AlarmAck` | `AlarmShelve` |
|
||||
| HistoryRead (all four variants) | **nothing** | `HistoryRead` on each node's scope, decided **per node in the batch** |
|
||||
| HistoryUpdate (insert / replace / update / delete) | role `HistoryUpdate` (from a `Cluster`-scoped grant) + key re-check | `HistoryUpdate` on that node's scope, decided **per node in the batch** |
|
||||
| Read (every attribute, not just Value) | nothing | `Read` on that node's scope, decided per node in the batch |
|
||||
| Browse / TranslateBrowsePath | nothing | `Browse` on the source node, **and** on each returned reference |
|
||||
| CreateMonitoredItems / SubscribeToEvents | nothing | `Subscribe` on that node's scope, decided per item |
|
||||
@@ -872,7 +875,15 @@ public enum NodePermissions : int
|
||||
|
||||
The three Write tiers map to Galaxy's v1 `SecurityClassification` — `FreeAccess`/`Operate` → `WriteOperate`, `Tune` → `WriteTune`, `Configure` → `WriteConfigure`. `SecuredWrite` / `VerifiedWrite` / `ViewOnly` classifications remain read-only from OPC UA regardless of grant.
|
||||
|
||||
`HistoryUpdate` (bit 12) is a **separate** permission from `HistoryRead` so a read-only grant cannot also authorize OPC UA HistoryUpdate (annotation / insert / delete) operations. `TriePermissionEvaluator` maps `OpcUaOperation.HistoryUpdate` to this bit, closing the read⇒update hole that existed when the only history permission was `HistoryRead`. Note: `HistoryUpdate` is **not** included in any composite bundle (`ReadOnly` / `Operator` / `Engineer` / `Admin`) because the HistoryUpdate service surface (the actual insert/replace/delete backend RPC) is not yet implemented — a client calling HistoryUpdate still receives the SDK's default reject regardless of the grant.
|
||||
`HistoryUpdate` (bit 12) is a **separate** permission from `HistoryRead` so a read-only grant cannot also authorize OPC UA HistoryUpdate operations. `TriePermissionEvaluator` maps `OpcUaOperation.HistoryUpdate` to this bit. It is **not** included in any composite bundle (`ReadOnly` / `Operator` / `Engineer` / `Admin`) — it must be granted explicitly, and it is offered as its own checkbox in the AdminUI ACL editor (`AclEdit.razor`). The HistoryUpdate service itself is now implemented (SQL Server historian backend only) — see the next section.
|
||||
|
||||
### HistoryUpdate (historian writes)
|
||||
|
||||
HistoryUpdate rewrites recorded history, so unlike HistoryRead it is gated **whatever `Security:Acl:Enabled` says**, and `WriteOperate` (live writes) never implies it.
|
||||
|
||||
- **Flag off (default):** the session's API key is re-checked on the call (`DataPlaneAuthorizer.AuthorizeKeyUsable`, Gitea #563 — an expired or disabled key loses its activation-time roles), then the session must carry the **`HistoryUpdate` role** (`OpcUaDataPlaneRoles.HistoryUpdate`). `ApiKeyInterimRoleMapper` confers it **only from a `Cluster`-scoped grant** carrying `NodePermissions.HistoryUpdate` — narrower-scoped grants confer nothing, for the same reason as `WriteOperate` (honouring a flag without honouring its scope would widen it). A key holding only `WriteOperate` is denied `BadUserAccessDenied`.
|
||||
- **Flag on:** `DataPlaneAuthorizer.Authorize(identity, OpcUaOperation.HistoryUpdate, nodeId, realm)` decides per node and **replaces** the role check (never chained) — a session with no roles succeeds on the strength of a scoped `HistoryUpdate` grant, and a `WriteOperate` grant does not help.
|
||||
- **Availability is separate from authorization.** A node accepts HistoryUpdate only if it is historized **and** the selected historian backend has an updater when the node materialises (the `HistoryWrite` AccessLevel bit). `Backend=Gateway` has none, so it answers `BadHistoryOperationUnsupported` for every node whatever the grant. Operations, result codes and unsupported cases: [`docs/Historian.md` §HistoryUpdate](Historian.md#historyupdate).
|
||||
|
||||
### Evaluator — `PermissionTrie`
|
||||
|
||||
|
||||
Reference in New Issue
Block a user