Merge master into feat/677-aggregates

# Conflicts:
#	docs/Historian.md
This commit is contained in:
Joseph Doherty
2026-10-11 11:30:46 -04:00
66 changed files with 2054 additions and 154 deletions
+1 -1
View File
@@ -1348,7 +1348,7 @@ historian tagname, which is the tag's driver `FullName`.
- ⚠️ **Reverse processed reads are Part 13/SDK-anchored (#677, client-visible):** every aggregate on SqlServer/QuestDB now anchors a reverse read at its later time — window `(end, start]`, partial interval earliest, each stamped with its later edge (was: earlier-anchored, stamped earlier edge); Partial also marks the interval the data starts inside (not Average); a non-default client `AggregateConfiguration` ⇒ `BadAggregateConfigurationRejected`; Gateway provisioning sends `IntegralDivisor = 1`. SDK-parity tests: `SdkAggregateParityTests`. `docs/Historian.md` §Aggregates. - ⚠️ **Reverse processed reads are Part 13/SDK-anchored (#677, client-visible):** every aggregate on SqlServer/QuestDB now anchors a reverse read at its later time — window `(end, start]`, partial interval earliest, each stamped with its later edge (was: earlier-anchored, stamped earlier edge); Partial also marks the interval the data starts inside (not Average); a non-default client `AggregateConfiguration` ⇒ `BadAggregateConfigurationRejected`; Gateway provisioning sends `IntegralDivisor = 1`. SDK-parity tests: `SdkAggregateParityTests`. `docs/Historian.md` §Aggregates.
- **Real status fix:** the recorder used to stamp every sample Good (`192`); it now carries the driver's status. ⚠️ On the **Gateway** backend this changes nothing stored: the gateway discards quality (C-002), so AVEVA still stores Good. - **Real status fix:** the recorder used to stamp every sample Good (`192`); it now carries the driver's status. ⚠️ On the **Gateway** backend this changes nothing stored: the gateway discards quality (C-002), so AVEVA still stores Good.
- **Comms loss is historized as an outage marker** (`Empty` + `BadNoCommunication`, one per tag per outage) on SqlServer; dropped (`DroppedUnsupportedKind`) on Gateway. Outbox records are **format v2** — drain or delete `ContinuousHistorization:OutboxPath` before downgrading a node. - **Comms loss is historized as an outage marker** (`Empty` + `BadNoCommunication`, one per tag per outage) on SqlServer; dropped (`DroppedUnsupportedKind`) on Gateway. Outbox records are **format v2** — drain or delete `ContinuousHistorization:OutboxPath` before downgrading a node.
- **HistoryUpdate** (SqlServer only; Gateway ⇒ `BadHistoryOperationUnsupported`): Insert/Replace/Update/DeleteRaw/DeleteAtTime, gated in both ACL modes — flag off needs the `HistoryUpdate` role from a **Cluster-scoped** grant (+ key re-check), flag on uses `DataPlaneAuthorizer`'s HistoryUpdate operation; `WriteOperate` never implies it. The `HistoryWrite` AccessLevel bit is set only when an updater is wired at materialise time. `docs/Historian.md` §HistoryUpdate, `docs/security.md`. - **HistoryUpdate** (SqlServer + QuestDb; Gateway ⇒ `BadHistoryOperationUnsupported`): Insert/Replace/Update/DeleteRaw/DeleteAtTime, **Primary-only** since #674 (off-Primary ⇒ `BadInvalidState`, after authorization, same `IDriverPrimaryRoleSource` verdict as the native-ack gate; reads ungated), items with a `SourceTimestamp` past now + `ServerHistorian:HistoryUpdateMaxFutureSkew` (default 1 day, ≤0 disables) ⇒ `BadOutOfRange` in the node manager (every backend), gated in both ACL modes — flag off needs the `HistoryUpdate` role from a **Cluster-scoped** grant (+ key re-check), flag on uses `DataPlaneAuthorizer`'s HistoryUpdate operation; `WriteOperate` never implies it. The `HistoryWrite` AccessLevel bit is set only when an updater is wired at materialise time. `docs/Historian.md` §HistoryUpdate, `docs/security.md`. ⚠️ **SqlServer schema v1 is accepted (#674)** with `AutoCreateSchema=false`: writes/reads/events/retention work; modified reads + HistoryUpdate ⇒ `BadHistoryOperationUnsupported` (nothing sent), retention skips `SampleModified`, one Warning per boot (`SqlServerSchemaBootstrapper.SupportsModifiedTrail`; `RequiredVersion` = 1, `ModifiedTrailVersion` = 2). QuestDB: keep `cairo.ttl.use.wall.clock=true`; a QuestDB trail INSERT retried after a client timeout duplicates (no DEDUP key).
- **Modified-value history (#668, Part 11):** every applied HistoryUpdate item leaves a trail row (prior value, principal from the session's API key / `anonymous`, `HistoryModificationType`, time) — SqlServer `hist.SampleModified` (**schema v2**, written by `OUTPUT … INTO` in the same statement ⇒ atomic; ⚠️ no FK/CHECK/trigger on it, and `AutoCreateSchema=false` installs must run `002_sample_modified.sql` before upgrading), QuestDB `{p}sample_modified` (append-only, written **before** the change ⇒ a crash leaves an orphan trail row, never an untracked change). `IsReadModified=true` HistoryRead serves it as `HistoryModifiedData` via `IHistorianDataSource.ReadModifiedAsync` (default ⇒ `NotSupported` ⇒ `BadHistoryOperationUnsupported` on Gateway/Null), paged on `(SourceTime, ModifiedId)`, gated by plain HistoryRead. Live writes and retention never write it; retention/TTL purge it on source time. `IHistorianDataUpdater` methods take a `HistoryUpdateContext`. CLI `historyread --modified`. `docs/Historian.md` §Modified-value history. - **Modified-value history (#668, Part 11):** every applied HistoryUpdate item leaves a trail row (prior value, principal from the session's API key / `anonymous`, `HistoryModificationType`, time) — SqlServer `hist.SampleModified` (**schema v2**, written by `OUTPUT … INTO` in the same statement ⇒ atomic; ⚠️ no FK/CHECK/trigger on it, and `AutoCreateSchema=false` installs must run `002_sample_modified.sql` before upgrading), QuestDB `{p}sample_modified` (append-only, written **before** the change ⇒ a crash leaves an orphan trail row, never an untracked change). `IsReadModified=true` HistoryRead serves it as `HistoryModifiedData` via `IHistorianDataSource.ReadModifiedAsync` (default ⇒ `NotSupported` ⇒ `BadHistoryOperationUnsupported` on Gateway/Null), paged on `(SourceTime, ModifiedId)`, gated by plain HistoryRead. Live writes and retention never write it; retention/TTL purge it on source time. `IHistorianDataUpdater` methods take a `HistoryUpdateContext`. CLI `historyread --modified`. `docs/Historian.md` §Modified-value history.
### Read path (`ServerHistorian` section) ### Read path (`ServerHistorian` section)
+3 -1
View File
@@ -207,7 +207,9 @@ The `-v` drops the SQL volume; remove it to keep ConfigDb state across restarts.
The full single-mesh stack (`central-1`/`central-2` + the four site nodes) can OOM-kill `central-1` on a loaded host. Two settings in the compose file guard against that: The full single-mesh stack (`central-1`/`central-2` + the four site nodes) can OOM-kill `central-1` on a loaded host. Two settings in the compose file guard against that:
- **EF Core + ASP.NET Core logs are pinned to `Warning`** on every host node (`Serilog__MinimumLevel__Override__Microsoft.EntityFrameworkCore` / `…Microsoft.AspNetCore` = `Warning`). The host logs via Serilog (`AddZbSerilog` → `ReadFrom.Configuration`), and in `Development` the default level is `Debug` — without these overrides every Deployment-poll emits an `Executed DbCommand` / `SELECT … FROM [Deployment]` line, flooding the Serilog pipeline and starving the Akka cluster heartbeat thread. Application + Akka log levels are left untouched, so this only silences the per-poll SQL chatter. To temporarily restore the SQL log flood for debugging, drop those two env vars (or set them back to `Information`) on the node you're inspecting. - **EF Core + ASP.NET Core logs are pinned to `Warning`** on every host node (`Serilog__MinimumLevel__Override__Microsoft.EntityFrameworkCore` / `…Microsoft.AspNetCore` = `Warning`). The host logs via Serilog (`AddZbSerilog` → `ReadFrom.Configuration`), and a `Development` environment would default to `Debug` (`appsettings.Development.json`) — the rig does NOT set `ASPNETCORE_ENVIRONMENT`, so nodes actually run at the base `Information` default, but these pins keep the SQL chatter out should the rig ever run as Development; without them a Debug/Information-level EF pipeline emits an `Executed DbCommand` / `SELECT … FROM [Deployment]` line, flooding the Serilog pipeline and starving the Akka cluster heartbeat thread. Application + Akka log levels are left untouched, so this only silences the per-poll SQL chatter. To temporarily restore the SQL log flood for debugging, drop those two env vars (or set them back to `Information`) on the node you're inspecting.
To raise Debug logging for chosen categories on `central-1`/`central-2` (e.g. `ZB.MOM.WW.OtOpcUa.Runtime`), use the overlay: `docker compose -f docker-dev/docker-compose.yml -f docker-dev/docker-compose.debug.yml up -d --no-deps central-1 central-2` — it sets `Serilog__MinimumLevel__Override__<Namespace>=Debug`; edit its category list to taste.
- **Each host node has `mem_limit: 1g`** (`mem_reservation: 512m`). A quiet solo `central-1` measures ~357 MiB; the limit leaves headroom for the deploy/UI load and per-cluster driver subscriptions that push a fully-loaded node higher. The limit/reservation live on the `&otopcua-host` anchor, so all six host services inherit them; `sql`, `traefik`, and the one-shot `migrator`/`cluster-seed` are left unbounded. - **Each host node has `mem_limit: 1g`** (`mem_reservation: 512m`). A quiet solo `central-1` measures ~357 MiB; the limit leaves headroom for the deploy/UI load and per-cluster driver subscriptions that push a fully-loaded node higher. The limit/reservation live on the `&otopcua-host` anchor, so all six host services inherit them; `sql`, `traefik`, and the one-shot `migrator`/`cluster-seed` are left unbounded.
The full six-node host stack therefore needs roughly **6 GiB** of Docker Desktop VM memory just for the host nodes (plus SQL Server's own footprint on top). On a constrained host, either raise the Docker Desktop VM memory or run fewer host services (e.g. just `central-1` + `central-2`, or a single central node) rather than the full mesh. The full six-node host stack therefore needs roughly **6 GiB** of Docker Desktop VM memory just for the host nodes (plus SQL Server's own footprint on top). On a constrained host, either raise the Docker Desktop VM memory or run fewer host services (e.g. just `central-1` + `central-2`, or a single central node) rather than the full mesh.
+26
View File
@@ -0,0 +1,26 @@
# docker-dev/docker-compose.debug.yml — optional overlay: raise Serilog levels per category on the central pair.
#
# The rig nodes run with ASPNETCORE_ENVIRONMENT unset (Production), so there is NO appsettings.Development.json and the
# Serilog default level is Information (the base appsettings.json sets none). Debug is therefore OFF by default; this
# overlay turns it on for chosen namespaces only, via Serilog:MinimumLevel:Override:<Namespace> (env: double underscore
# for ':', and the namespace's own dots stay dots).
#
# Up (recreates only the two central nodes with the extra env):
# docker compose -f docker-dev/docker-compose.yml -f docker-dev/docker-compose.debug.yml up -d --no-deps central-1 central-2
# Back to normal: re-run `up -d --no-deps central-1 central-2` with the base file alone.
#
# Edit the list below to the categories you care about. Useful ones:
# ZB.MOM.WW.OtOpcUa.Runtime actors: DriverHostActor, OpcUaPublishActor, redundancy, deploys
# ZB.MOM.WW.OtOpcUa.Runtime.Drivers.DriverHostActor one actor only (e.g. the #672 historization verdict logs)
# ZB.MOM.WW.OtOpcUa.Driver.OpcUaClient a single driver
# Opc.Ua OPC Foundation stack (very chatty)
# Akka cluster/remoting internals (chatty; base is Information)
# Leave Microsoft.EntityFrameworkCore / Microsoft.AspNetCore at Warning (see README "Resource limits & dev logging"):
# a Debug SQL flood starves the Akka heartbeat thread.
services:
central-1:
environment:
Serilog__MinimumLevel__Override__ZB.MOM.WW.OtOpcUa.Runtime: "Debug"
central-2:
environment:
Serilog__MinimumLevel__Override__ZB.MOM.WW.OtOpcUa.Runtime: "Debug"
+14 -3
View File
@@ -57,6 +57,11 @@ otopcua-cli write -u opc.tcp://localhost:4840 -n "ns=2;s=MyNode" -v 42 -U operat
### Failover ### Failover
**Mapped ports and internal hostnames.** Endpoint discovery rewrites the host *and port* of the endpoint the server
advertises to the URL you passed (when the scheme matches), so connecting through a port mapping works even though the
server advertises its internal address (for example the docker-dev rig: `-u opc.tcp://localhost:4841` reaches the
node mapped to 4841, although every container advertises `:4840`).
When `-F` is provided, the shared service tries the primary URL first, then each failover URL in order. For long-running commands (`subscribe`, `alarms`), the service monitors the session via keep-alive and automatically reconnects to the next available server on failure. When `-F` is provided, the shared service tries the primary URL first, then each failover URL in order. For long-running commands (`subscribe`, `alarms`), the service monitors the session via keep-alive and automatically reconnects to the next available server on failure.
```bash ```bash
@@ -229,7 +234,7 @@ continuation points up to `--max`. A server on the `Gateway` backend (no modific
| `--start` | Start time in ISO 8601 UTC format, e.g. `2026-01-15T08:00:00Z` (default: 24 hours ago) | | `--start` | Start time in ISO 8601 UTC format, e.g. `2026-01-15T08:00:00Z` (default: 24 hours ago) |
| `--end` | End time in ISO 8601 UTC format, e.g. `2026-01-15T09:00:00Z` (default: now) | | `--end` | End time in ISO 8601 UTC format, e.g. `2026-01-15T09:00:00Z` (default: now) |
| `--max` | Maximum number of values (default: 1000) | | `--max` | Maximum number of values (default: 1000) |
| `--aggregate` | Aggregate function: Average, Minimum, Maximum, Count, Start, End, StandardDeviation | | `--aggregate` | Aggregate function: Average, Minimum, Maximum, Count, Start, End, StandardDeviation, TimeAverage, TimeAverage2, Total, Total2, Interpolative |
| `--interval` | Processing interval in milliseconds for aggregates (default: 3600000) | | `--interval` | Processing interval in milliseconds for aggregates (default: 3600000) |
| `--modified` | Read modified-value history (Part 11 `IsReadModified`) instead of raw samples | | `--modified` | Read modified-value history (Part 11 `IsReadModified`) instead of raw samples |
@@ -270,8 +275,9 @@ node-level code (`Good`, or `BadNoData` if the range held nothing). A node that
updater, so no `HistoryWrite` access bit), answers `BadHistoryOperationUnsupported`; a `NaN`/`Infinity` double is `BadInvalidArgument`. The session's API key updater, so no `HistoryWrite` access bit), answers `BadHistoryOperationUnsupported`; a `NaN`/`Infinity` double is `BadInvalidArgument`. The session's API key
(`-U apikey -P otua_<keyId>_<secret>`) needs the `HistoryUpdate` permission — via a `Cluster`-scoped grant when `Security:Acl:Enabled` is off, or a scoped (`-U apikey -P otua_<keyId>_<secret>`) needs the `HistoryUpdate` permission — via a `Cluster`-scoped grant when `Security:Acl:Enabled` is off, or a scoped
grant on the node when it is on; a key with only `WriteOperate` gets `BadUserAccessDenied`. See [Historian.md §HistoryUpdate](Historian.md#historyupdate) and grant on the node when it is on; a key with only `WriteOperate` gets `BadUserAccessDenied`. See [Historian.md §HistoryUpdate](Historian.md#historyupdate) and
[security.md](security.md#historyupdate-historian-writes). `historyread --aggregate` only knows the seven names in the table below; the SQL backend's other [security.md](security.md#historyupdate-historian-writes). `historyread --aggregate` only knows the twelve names in the table below; the SQL backend's other
aggregates (Total, Range, Delta, PercentGood, …) need a different client. aggregates (Range, Delta, PercentGood, …) need a different client. A refused history read (for example
`BadHistoryOperationUnsupported`) prints one error line and exits non-zero; it does not dump a stack trace.
`delete-raw` is sent with `IsDeleteModified=false`. Invalid or inconsistent options (unknown mode, unpaired `delete-raw` is sent with `IsDeleteModified=false`. Invalid or inconsistent options (unknown mode, unpaired
value/time, unparsable value, options that do not apply to the mode) fail before any connection is made. value/time, unparsable value, options that do not apply to the mode) fail before any connection is made.
@@ -287,6 +293,11 @@ value/time, unparsable value, options that do not apply to the mode) fail before
| `Start` | `first` | `AggregateFunction_Start` | | `Start` | `first` | `AggregateFunction_Start` |
| `End` | `last` | `AggregateFunction_End` | | `End` | `last` | `AggregateFunction_End` |
| `StandardDeviation` | `stddev`, `stdev` | `AggregateFunction_StandardDeviationPopulation` | | `StandardDeviation` | `stddev`, `stdev` | `AggregateFunction_StandardDeviationPopulation` |
| `TimeAverage` | `timeavg` | `AggregateFunction_TimeAverage` |
| `TimeAverage2` | `timeavg2` | `AggregateFunction_TimeAverage2` |
| `Total` | | `AggregateFunction_Total` |
| `Total2` | | `AggregateFunction_Total2` |
| `Interpolative` | `interp` | `AggregateFunction_Interpolative` |
### alarms ### alarms
+1 -1
View File
@@ -206,7 +206,7 @@ All times are in UTC. Invalid input turns red on blur.
| Option | Description | | Option | Description |
|--------|-------------| |--------|-------------|
| Aggregate | Raw (default), Average, Minimum, Maximum, Count, Start, End, Standard Deviation | | Aggregate | Raw (default), Average, Minimum, Maximum, Count, Start, End, Standard Deviation, Time Average, Time Average 2, Total, Total 2, Interpolative |
| Interval (ms) | Processing interval for aggregate queries (shown only for aggregates) | | Interval (ms) | Processing interval for aggregate queries (shown only for aggregates) |
| Max Values | Maximum number of raw values to return (default 1000) | | Max Values | Maximum number of raw values to return (default 1000) |
+26 -5
View File
@@ -22,14 +22,30 @@ Two related concerns get their own dedicated pages and are **only summarised + l
The Host (`Program.cs`) loads `appsettings.json`, then overlays a **per-role** file chosen from the cluster roles: The Host (`Program.cs`) loads `appsettings.json`, then overlays a **per-role** file chosen from the cluster roles:
- A single role → `appsettings.{role}.json` (e.g. `appsettings.driver.json`, `appsettings.admin.json`). - A single plain role → `appsettings.{role}.json` (e.g. `appsettings.driver.json`, `appsettings.admin.json`).
- Both roles → `appsettings.admin-driver.json` (roles joined with `-`, ordinal-sorted). - Both roles → `appsettings.admin-driver.json` (plain roles joined with `-`, ordinal-sorted).
- `cluster-{ClusterId}` roles are **ignored** when naming the overlay (#678.6) — `[driver, cluster-SITE-A]` loads `appsettings.driver.json`.
- `appsettings.{ASPNETCORE_ENVIRONMENT}.json` (e.g. `appsettings.Development.json`) is layered on by the host builder. - `appsettings.{ASPNETCORE_ENVIRONMENT}.json` (e.g. `appsettings.Development.json`) is layered on by the host builder.
All role overlays are **optional** — the base `appsettings.json` plus the Options-class C# defaults are enough to boot. The roles themselves come from `EffectiveRoles.Resolve` (`ZB.MOM.WW.OtOpcUa.Cluster`) — `Cluster:Roles`, falling back to `OTOPCUA_ROLES` when that is empty (see [`ServiceHosting.md`](ServiceHosting.md) and the table below). All role overlays are **optional** — the base `appsettings.json` plus the Options-class C# defaults are enough to boot. The roles themselves come from `EffectiveRoles.Resolve` (`ZB.MOM.WW.OtOpcUa.Cluster`) — `Cluster:Roles`, falling back to `OTOPCUA_ROLES` when that is empty (see [`ServiceHosting.md`](ServiceHosting.md) and the table below).
The checked-in `appsettings*.json` files are deliberately thin: they carry only `Serilog` and the `Security:Ldap` overlay. Everything else (`OpcUa`, `Cluster`, `ConnectionStrings`/`ConfigDb`) binds from the Options-class defaults documented below unless an operator adds the section explicitly or supplies the corresponding environment variable. The checked-in `appsettings*.json` files are deliberately thin: they carry only `Serilog` and the `Security:Ldap` overlay. Everything else (`OpcUa`, `Cluster`, `ConnectionStrings`/`ConfigDb`) binds from the Options-class defaults documented below unless an operator adds the section explicitly or supplies the corresponding environment variable.
<a id="role-overlay-audit"></a>
#### Role overlay audit (#678.6)
Until #678.6 the overlay never loaded on a node carrying a `cluster-*` role (the cluster role leaked into the filename). Fixing that **activates** the shipped overlays on those nodes (the docker-dev rig and any per-cluster-mesh production node), so each key was audited against base `appsettings.json` / the Options defaults. Env vars and command-line args are re-appended after the overlay, so any value a deployment sets explicitly (the rig sets every `Security__Ldap__*` key) still wins.
| Key | Overlay(s) | Base / default | Effect on activation |
|---|---|---|---|
| `Serilog:MinimumLevel:Default` = `Information` | all three | Serilog default `Information` | no-op (in `Development` it outranks `appsettings.Development.json`'s `Debug`, by design) |
| `…Override:Akka` = `Information` | all three | inherits `Default` (`Information`) | no-op |
| `…Override:Microsoft.AspNetCore` = `Warning` | `admin`, `admin-driver` | unset (`Information`) | quieter ASP.NET Core logs; log-level only, matches what the rig already pins |
| `…Override:Opc.Ua` = `Information` | `admin-driver`, `driver` | inherits `Default` | no-op. **`driver` shipped `Debug` — changed to `Information` in #678.6**: activating a Debug OPC UA stack trace on every production driver node would flood Serilog (the same pipeline starvation the rig's EF/AspNetCore pins guard against). Raise it per node with `Serilog__MinimumLevel__Override__Opc.Ua=Debug`. |
| `Security:Ldap:DevStubMode` = `false` | all three | `false` | no-op; the overlay's point is that it outranks a developer's `appsettings.Development.json` (`true`) |
| `Security:Ldap:Transport` = `Ldaps` | all three | `LdapOptions.Transport` default `None` | **behaviour-changing, kept deliberately** (documented secure production default). A cluster-role node that configures `Security:Ldap:Server`/`Port` for a plaintext LDAP but never sets `Transport` would now dial LDAPS and fail AdminUI sign-in; set `Security__Ldap__Transport` (the rig already sets `None` explicitly). Data-plane auth is API-key based and unaffected; driver-only nodes have no AdminUI. |
--- ---
## `appsettings` sections ## `appsettings` sections
@@ -38,7 +54,7 @@ The checked-in `appsettings*.json` files are deliberately thin: they carry only
- **Purpose:** logging. Console + rolling daily file sink, layered with the shared `ZB.MOM.WW.Telemetry` enrichers (`AddZbSerilog` in `Program.cs`). - **Purpose:** logging. Console + rolling daily file sink, layered with the shared `ZB.MOM.WW.Telemetry` enrichers (`AddZbSerilog` in `Program.cs`).
- **Where bound:** `builder.AddZbSerilog(...)` reads `Serilog` from configuration (`ReadFrom.Configuration`). - **Where bound:** `builder.AddZbSerilog(...)` reads `Serilog` from configuration (`ReadFrom.Configuration`).
- **Checked-in shape** (`appsettings.json`): `Using` = `[ "Serilog.Sinks.Console", "Serilog.Sinks.File" ]`, `WriteTo` = a `Console` sink and a `File` sink (`path: logs/otopcua-.log`, `rollingInterval: Day`). Role overlays add `MinimumLevel` / `Override` blocks (e.g. `Opc.Ua: Debug`, `Akka: Information`). - **Checked-in shape** (`appsettings.json`): `Using` = `[ "Serilog.Sinks.Console", "Serilog.Sinks.File" ]`, `WriteTo` = a `Console` sink and a `File` sink (`path: logs/otopcua-.log`, `rollingInterval: Day`). Role overlays add `MinimumLevel` / `Override` blocks (e.g. `Opc.Ua: Information`, `Akka: Information`).
### `OpcUa` ### `OpcUa`
@@ -410,7 +426,7 @@ Validated at host start (driver-role nodes) when `ServerHistorian:Enabled` **or*
| `ServerHistorian:SqlServer:ConnectionString` | `""` | **Required.** Historian database connection string (not the ConfigDb). Supply via `${secret:OT-HISTORIAN-SQL}` or env `ServerHistorian__SqlServer__ConnectionString`; the database must already exist. | | `ServerHistorian:SqlServer:ConnectionString` | `""` | **Required.** Historian database connection string (not the ConfigDb). Supply via `${secret:OT-HISTORIAN-SQL}` or env `ServerHistorian__SqlServer__ConnectionString`; the database must already exist. |
| `ServerHistorian:SqlServer:Schema` | `hist` | Must match `^[A-Za-z_][A-Za-z0-9_]{0,127}$` (spliced into DDL/DML). | | `ServerHistorian:SqlServer:Schema` | `hist` | Must match `^[A-Za-z_][A-Za-z0-9_]{0,127}$` (spliced into DDL/DML). |
| `ServerHistorian:SqlServer:CommandTimeout` | `00:00:30` | Must be `> 0`; rounded up to whole seconds. | | `ServerHistorian:SqlServer:CommandTimeout` | `00:00:30` | Must be `> 0`; rounded up to whole seconds. |
| `ServerHistorian:SqlServer:AutoCreateSchema` | `true` | `true` applies the embedded schema script under an app-lock on first use; `false` only verifies `SchemaVersion >= 1` and fails every historian call (health `LastError`, no crash) if behind. | | `ServerHistorian:SqlServer:AutoCreateSchema` | `true` | `true` applies the embedded schema scripts under an app-lock on first use (upgrading a v1 schema to v2); `false` only verifies `SchemaVersion >= 1` and fails every historian call (health `LastError`, no crash) if behind. A verified **v1** schema (no `002_sample_modified.sql`) is served with modified-value reads and HistoryUpdate refused `BadHistoryOperationUnsupported`, retention skipping `SampleModified`, and one Warning per boot (#674). |
| `ServerHistorian:SqlServer:RetentionDays` | unset | Days of history kept (source time), `>= 1` when set; unset keeps everything. Enables the Primary-gated purge of `hist.Sample`/`hist.Event` (see [Historian.md](Historian.md#retention-retentiondays)). | | `ServerHistorian:SqlServer:RetentionDays` | unset | Days of history kept (source time), `>= 1` when set; unset keeps everything. Enables the Primary-gated purge of `hist.Sample`/`hist.Event` (see [Historian.md](Historian.md#retention-retentiondays)). |
| `ServerHistorian:SqlServer:PurgeInterval` | `01:00:00` | Purge cadence; must be `> 0`. | | `ServerHistorian:SqlServer:PurgeInterval` | `01:00:00` | Purge cadence; must be `> 0`. |
| `ServerHistorian:SqlServer:PurgeBatchSize` | `4000` | Rows per `DELETE TOP (n)` batch; `>= 1`. Keep under ~5000 to avoid lock escalation. | | `ServerHistorian:SqlServer:PurgeBatchSize` | `4000` | Rows per `DELETE TOP (n)` batch; `>= 1`. Keep under ~5000 to avoid lock escalation. |
@@ -430,7 +446,12 @@ when `ServerHistorian:Enabled` **or** `AlarmHistorian:Enabled` is true (`QuestDb
| `ServerHistorian:QuestDb:VisibilityTimeout` | `00:00:05` | Must be `>= 0`. How long a write waits for its rows to become visible to queries (WAL apply); `0` disables the wait. A timeout only logs a Warning. | | `ServerHistorian:QuestDb:VisibilityTimeout` | `00:00:05` | Must be `>= 0`. How long a write waits for its rows to become visible to queries (WAL apply); `0` disables the wait. A timeout only logs a Warning. |
HistoryUpdate (SqlServer and QuestDb only; QuestDb deletes are soft — tombstones) additionally needs `ServerHistorian:Enabled=true` and a key HistoryUpdate (SqlServer and QuestDb only; QuestDb deletes are soft — tombstones) additionally needs `ServerHistorian:Enabled=true` and a key
whose principal holds the `HistoryUpdate` permission — see [`docs/security.md`](security.md#historyupdate-historian-writes). Full semantics, whose principal holds the `HistoryUpdate` permission — see [`docs/security.md`](security.md#historyupdate-historian-writes). It is served
only on the redundancy **Primary** (`BadInvalidState` elsewhere, #674) and is bounded by one backend-independent key:
| Key | Default | Meaning |
|---|---|---|
| `ServerHistorian:HistoryUpdateMaxFutureSkew` | `1.00:00:00` (1 day) | TimeSpan. A HistoryUpdate (Insert / Replace / Update) item whose `SourceTimestamp` is later than server-clock now + this is refused `BadOutOfRange` before any backend sees it (#674 — a far-future typo could otherwise age out a QuestDB TTL table or poison the history). Applies to every backend; the live recorder is not bounded. Non-positive disables the bound. | Full semantics,
schema, aggregates and the docker-dev recipe: [`docs/Historian.md`](Historian.md#sql-server-backend) (SqlServer) and schema, aggregates and the docker-dev recipe: [`docs/Historian.md`](Historian.md#sql-server-backend) (SqlServer) and
[`docs/Historian.md`](Historian.md#questdb-backend) (QuestDb: the µs timestamp rules, pre-1970 limits, TTL and the docker-dev overlay). [`docs/Historian.md`](Historian.md#questdb-backend) (QuestDb: the µs timestamp rules, pre-1970 limits, TTL and the docker-dev overlay).
+86 -19
View File
@@ -85,7 +85,8 @@ consumes the historian. The existing enable gates (`ServerHistorian:Enabled`, `A
| Connection | `Endpoint` + `ApiKey` + TLS keys (top-level `ServerHistorian:*`) | `ServerHistorian:SqlServer:*` | `ServerHistorian:QuestDb:*` | | Connection | `Endpoint` + `ApiKey` + TLS keys (top-level `ServerHistorian:*`) | `ServerHistorian:SqlServer:*` | `ServerHistorian:QuestDb:*` |
Recording stays **Primary-only** (#537) and **at-least-once** with the #538 per-tag quarantine on every backend; Recording stays **Primary-only** (#537) and **at-least-once** with the #538 per-tag quarantine on every backend;
HistoryRead and HistoryUpdate are served from any node (the database is shared). The historian connection string is HistoryRead is served from any node (the database is shared). HistoryUpdate is **Primary-only** since #674
(`BadInvalidState` on any other node — see [HistoryUpdate](#primary-only-674)). The historian connection string is
**not** a ConfigDb string, and a driver-only node may carry it without breaking the Phase 4 "no ConfigDb" posture. **not** a ConfigDb string, and a driver-only node may carry it without breaking the Phase 4 "no ConfigDb" posture.
--- ---
@@ -145,6 +146,7 @@ and all HistoryRead calls on historized nodes return `GoodNoData` (empty, not an
| `MaxConcurrentHistoryReadBatches` | int | `16` | Process-wide limit on HistoryRead (and HistoryUpdate) batches served **concurrently** (arch-review 03/S3). A flood of history clients degrades to `BadTooManyOperations` rather than exhausting the SDK request-thread pool. With `HistoryReadBatchParallelism` this caps total in-flight gateway reads at `MaxConcurrentHistoryReadBatches × HistoryReadBatchParallelism` (default 16 × 4 = 64). A batch that cannot get a slot within `min(HistoryReadDeadline, 5 s)` fails every handle `BadTooManyOperations`. | | `MaxConcurrentHistoryReadBatches` | int | `16` | Process-wide limit on HistoryRead (and HistoryUpdate) batches served **concurrently** (arch-review 03/S3). A flood of history clients degrades to `BadTooManyOperations` rather than exhausting the SDK request-thread pool. With `HistoryReadBatchParallelism` this caps total in-flight gateway reads at `MaxConcurrentHistoryReadBatches × HistoryReadBatchParallelism` (default 16 × 4 = 64). A batch that cannot get a slot within `min(HistoryReadDeadline, 5 s)` fails every handle `BadTooManyOperations`. |
| `MaxTimeWeightedRawRows` | int | `1000000` | `SqlServer` / `QuestDb` only (#671). The most raw in-window samples one [time-weighted aggregate](#time-weighted-aggregates-part-13-671) read (TimeAverage, TimeAverage2, **Total**, Total2, Interpolative) may stream into the server-side evaluator — that family is **not** pushed down to SQL. The window query is capped at this + 1 rows; one row past the cap fails **that node** with `BadTooManyOperations` (not counted as a backend fault) instead of holding a pooled connection until `HistoryReadDeadline` and returning `BadTimeout`. Bound rows outside the window do not count; per-sample aggregates are not capped. 1 000 000 ≈ 11.5 days of a 1 Hz tag. Non-positive = unbounded (a `Validate()` warning). | | `MaxTimeWeightedRawRows` | int | `1000000` | `SqlServer` / `QuestDb` only (#671). The most raw in-window samples one [time-weighted aggregate](#time-weighted-aggregates-part-13-671) read (TimeAverage, TimeAverage2, **Total**, Total2, Interpolative) may stream into the server-side evaluator — that family is **not** pushed down to SQL. The window query is capped at this + 1 rows; one row past the cap fails **that node** with `BadTooManyOperations` (not counted as a backend fault) instead of holding a pooled connection until `HistoryReadDeadline` and returning `BadTimeout`. Bound rows outside the window do not count; per-sample aggregates are not capped. 1 000 000 ≈ 11.5 days of a 1 Hz tag. Non-positive = unbounded (a `Validate()` warning). |
| `HistoryReadDeadline` | TimeSpan | `00:01:00` | Per-request wall-clock deadline for a HistoryRead (and HistoryUpdate) batch (arch-review 03/S3). A node still in flight at the deadline surfaces `BadTimeout`, so one request can never hold an SDK request thread longer than this regardless of node count. Non-positive = unbounded (a `Validate()` warning). | | `HistoryReadDeadline` | TimeSpan | `00:01:00` | Per-request wall-clock deadline for a HistoryRead (and HistoryUpdate) batch (arch-review 03/S3). A node still in flight at the deadline surfaces `BadTimeout`, so one request can never hold an SDK request thread longer than this regardless of node count. Non-positive = unbounded (a `Validate()` warning). |
| `HistoryUpdateMaxFutureSkew` | TimeSpan | `1.00:00:00` | #674. How far past the server clock a HistoryUpdate (Insert / Replace / Update) item's `SourceTimestamp` may lie; an item beyond `now + skew` is refused **`BadOutOfRange`** by the node manager before any backend sees it (every backend). Guards against a far-future typo (a 2062 timestamp). The live recorder is not bounded. Non-positive disables the bound (no warning — a deliberate choice). See [HistoryUpdate](#historyupdate). |
**`ServerHistorian:SqlServer`** (read only when `Backend=SqlServer`; bound by `HistorianBackendRegistry.BindSqlServer`): **`ServerHistorian:SqlServer`** (read only when `Backend=SqlServer`; bound by `HistorianBackendRegistry.BindSqlServer`):
@@ -630,13 +632,31 @@ says is complete carries no continuation point. `NumValuesPerNode == 0` stays un
- The cursor is `(source, next start, end, boundary skip, cap)` in the same session-bound store as Raw (flagged as an events - The cursor is `(source, next start, end, boundary skip, cap)` in the same session-bound store as Raw (flagged as an events
cursor — a Raw point presented to an events read, or the reverse, is `BadContinuationPointInvalid`). The next page reads from cursor — a Raw point presented to an events read, or the reverse, is `BadContinuationPointInvalid`). The next page reads from
the last event's `EventTimeUtc` **inclusive** and drops the events at that timestamp already emitted. the last event's `EventTimeUtc` **inclusive** and drops the events at that timestamp already emitted — and asks the source for
that many more rows than the cap, so every page short of the last is a FULL `NumValuesPerNode` page (#675; resumed pages used
to come back short by the skip).
- A tie cluster larger than the cap is paged within by over-fetching the half-open window `[T, T + resolution)` up to - A tie cluster larger than the cap is paged within by over-fetching the half-open window `[T, T + resolution)` up to
`MaxTieClusterOverfetch`; beyond that bound the node answers `BadHistoryOperationUnsupported` (raise the bound). `MaxTieClusterOverfetch`; beyond that bound the node answers `BadHistoryOperationUnsupported` (raise the bound).
- **Tie exactness depends on the backend.** SqlServer and QuestDB order by `(EventTimeUtc, EventId)`, so the skip count is exact. - **Tie exactness depends on the backend.** SqlServer and QuestDB order by `(EventTimeUtc, EventId)`, so the skip count is exact.
The Gateway backend has no stable intra-timestamp order (and only a truncation flag), so ties at a page boundary are The Gateway backend has no stable intra-timestamp order (and only a truncation flag), so ties at a page boundary are
best-effort there. best-effort there.
- Reverse event reads (`start > end`) are not paged server-side; no continuation point is issued for them. - **Reverse event reads page newest-first (#675).** `StartTime > EndTime` reads the Part 11 reverse window `(EndTime, StartTime]`
with events in DESCENDING `(EventTimeUtc, EventId)` order, so a cap keeps the **latest** N and a continuation point is issued
when the source flags more. The direction is stored **in the cursor**: a reverse point resumes reverse (and a forward one
forward) whatever the resume request's details say. Reverse tie clusters are over-fetched forward and sliced from the newest
end, exactly as Raw does. Before #675 a capped reverse read silently returned the EARLIEST N, ascending, Good, with no
continuation point.
- **End-only events reads read backward (#675).** `StartTime` unset (`DateTime.MinValue`) with an `EndTime` and
`NumValuesPerNode > 0` pages backward from `EndTime` over `(MinValue, EndTime]` — the same rule as Raw. Start-only reads still
page forward to an open upper bound (`DateTime.MaxValue`, UTC). Without a cap an end-only read keeps its forward
`[MinValue, EndTime)` meaning.
- **Backends.** The server calls `IHistorianDataSource.ReadEventsAsync(…, bool reverse, …)`. SqlServer and QuestDB serve reverse
natively (`… > lo AND … <= hi ORDER BY time DESC, id DESC`). **Gateway and Null refuse a reverse read** (the interface default
throws `NotSupportedException`) and the node answers `BadHistoryOperationUnsupported` — the gateway streams ascending only and
rejects `end < start`, and draining the whole window to keep its tail would be exactly the unbounded read the cap prevents.
- **Gateway open upper bound.** The `DateTime.MaxValue` (UTC) bound of a start-only forward read is pinned offline through the
published client's proto `Timestamp` conversion (captured by an in-process gRPC server) and the FILETIME encoding
(`GatewayEventBoundsTests`). What a live AVEVA Historian does with a year-9999 bound is still unverified (needs the VPN).
- No session on the request (in-process callers) degrades to single-shot, as for Raw. - No session on the request (in-process callers) degrades to single-shot, as for Raw.
### Modified-value history (Part 11) ### Modified-value history (Part 11)
@@ -686,6 +706,25 @@ modification time and the Part 11 `HistoryUpdateType`. A HistoryReadRaw with `Is
within one node the order is strict. within one node the order is strict.
- **Retention.** The trail ages out on the same **source-time** cutoff as the samples it describes: `SqlServer` purges `hist.SampleModified` - **Retention.** The trail ages out on the same **source-time** cutoff as the samples it describes: `SqlServer` purges `hist.SampleModified`
per tag alongside `hist.Sample`; QuestDB puts the same TTL on `{p}sample_modified`. per tag alongside `hist.Sample`; QuestDB puts the same TTL on `{p}sample_modified`.
- **SqlServer schema v1 (#674).** With `AutoCreateSchema=false` a database still at schema **version 1** (script `002_sample_modified.sql` not
applied) keeps historizing — live writes, raw/processed/at-time reads, alarm events and retention all work — but has no trail: modified reads
and every HistoryUpdate item answer **`BadHistoryOperationUnsupported`** with nothing sent to SQL, retention skips `SampleModified`, and the
node logs **one Warning per boot**. Apply `002_sample_modified.sql` and restart the node to enable them (the verdict is taken once per
process). `AutoCreateSchema=true` (the default) still upgrades v1 to v2 in place. See [Schema](#schema).
- **QuestDB retry duplicates (#674).** `{p}sample_modified` is append-only with **no DEDUP key** (a key would collapse genuinely distinct
modifications of one sample), and each trail row gets a fresh `modified_id`. So a HistoryUpdate a client **retries** after a timeout — when the
first attempt had in fact landed — records a second trail row for the same change (and, for Insert, the retry is `BadEntryExists`). The
sample itself is idempotent (dedup upsert); only the audit trail can show the duplicate. `SqlServer` has the same exposure only if the client
retries a committed request; there is no request id to dedupe on in either backend.
- **Cross-node ordering (QuestDB).** The per-tag lock is per process; the [Primary-only](#primary-only-674) HistoryUpdate gate (#674) is what keeps
two nodes of a pair from recording concurrent updates of one tag (and so a wrong prior value). Two **separate** pairs sharing one QuestDB can
still race; that is unsupported.
- ⚠️ **Who can read the trail (#674.4).** Modified reads are ordinary HistoryReads: with `Security:Acl:Enabled=false` (the default) **any admitted
session, including Anonymous, can read who rewrote history** (`UserName` is the principal). Turn the ACL flag on, or set
`OpcUa:AllowAnonymous=false`, if the principal names are sensitive (see [security.md](security.md)).
- **Not implemented (#674.5):** `IsDeleteModified` (deleting trail entries — `BadHistoryOperationUnsupported`); recording an **Update** (upsert) as
a split `Insert` / `Replace` — it is recorded as `Update` either way; and **`AuditHistoryUpdateEvent`** (no audit event is emitted for a history
change — the trail is the only record).
```bash ```bash
otopcua-cli historyread -u opc.tcp://localhost:4840 -U apikey -P otua_<keyId>_<secret> \ otopcua-cli historyread -u opc.tcp://localhost:4840 -U apikey -P otua_<keyId>_<secret> \
@@ -758,7 +797,7 @@ The bootstrapper applies every script above the recorded version, in order, unde
| Object | Purpose | | Object | Purpose |
|---|---| |---|---|
| `SchemaVersion(Version, AppliedUtc)` | Applied-script ledger. Required version: **2** (since #668). | | `SchemaVersion(Version, AppliedUtc)` | Applied-script ledger. Required version: **1**; version **2** adds the modification trail (#668). A v1 database is served with the trail features off (#674 — see below). |
| `Tag(TagId int identity, Name nvarchar(400) unique, DataType tinyint, EngineeringUnit, Description, CreatedUtc, UpdatedUtc)` | One row per historian tagname. `DataType` is a `DriverDataType`. | | `Tag(TagId int identity, Name nvarchar(400) unique, DataType tinyint, EngineeringUnit, Description, CreatedUtc, UpdatedUtc)` | One row per historian tagname. `DataType` is a `DriverDataType`. |
| `Sample(TagId, SourceTimeUtc datetime2(7), ValueKind tinyint, NumValue float, IntValue bigint, StrValue nvarchar(max), StatusCode int, ServerTimeUtc)` | **PK `(TagId, SourceTimeUtc)`** — one value per timestamp (Part 11), which makes redelivery idempotent and rules out tie clusters. `StatusCode` is the uint status stored bit-for-bit. | | `Sample(TagId, SourceTimeUtc datetime2(7), ValueKind tinyint, NumValue float, IntValue bigint, StrValue nvarchar(max), StatusCode int, ServerTimeUtc)` | **PK `(TagId, SourceTimeUtc)`** — one value per timestamp (Part 11), which makes redelivery idempotent and rules out tie clusters. `StatusCode` is the uint status stored bit-for-bit. |
| `Event(EventId nvarchar(64) PK nonclustered, SourceName, AlarmId, AlarmName, AlarmTypeName, Severity, EventKind, Message, UserName, Comment, EventTimeUtc, ReceivedTimeUtc)` | Alarm history. Clustered on `(EventTimeUtc, EventId)`; extra index `(SourceName, EventTimeUtc)`. | | `Event(EventId nvarchar(64) PK nonclustered, SourceName, AlarmId, AlarmName, AlarmTypeName, Severity, EventKind, Message, UserName, Comment, EventTimeUtc, ReceivedTimeUtc)` | Alarm history. Clustered on `(EventTimeUtc, EventId)`; extra index `(SourceName, EventTimeUtc)`. |
@@ -774,11 +813,14 @@ everywhere (full 100 ns precision — never a `DataTable`, which would round to
`sqlcmd -S <server> -U <user> -P <pw> -d <db> -I -v Schema=hist -i docs/historian/sqlserver/001_initial.sql`, then the same with `sqlcmd -S <server> -U <user> -P <pw> -d <db> -I -v Schema=hist -i docs/historian/sqlserver/001_initial.sql`, then the same with
`002_sample_modified.sql` (`-v Schema=` supplies the `$(Schema)` placeholder; verified against the docker-dev SQL container). Re-running is a no-op. `002_sample_modified.sql` (`-v Schema=` supplies the `$(Schema)` placeholder; verified against the docker-dev SQL container). Re-running is a no-op.
⚠️ **Upgrading an existing v1 historian (#668).** A build with this change requires schema **version 2**. With `AutoCreateSchema=true` ⚠️ **Upgrading an existing v1 historian (#668, #674).** With `AutoCreateSchema=true` (the default) the first historian call after the
(the default) the first historian call after the upgrade applies `002_sample_modified.sql` in place (it only creates `SampleModified`; existing upgrade applies `002_sample_modified.sql` in place (it only creates `SampleModified`; existing samples are untouched). With
samples are untouched). With `AutoCreateSchema=false` **every historian call fails** (`missing version 2 (found 1)`, in health `LastError`) `AutoCreateSchema=false` a **v1 database is accepted** (#674 — between #668 and #674 it refused every historian call, continuous-historization
until an operator runs `002_sample_modified.sql` by hand as above — do that before rolling the new build out. Modifications made before the drains included): writes, reads, events and retention work, while modified-value reads and HistoryUpdate report
upgrade have no trail; the trail starts with the first HistoryUpdate after it. `BadHistoryOperationUnsupported` without touching SQL, retention skips `SampleModified`, and one Warning per boot names the missing script.
Run `002_sample_modified.sql` by hand as above and **restart the node** to enable them (the version is read once per process). Only a schema
below version 1 (`missing version 1 (found 0)`) still fails every call. Modifications made before the upgrade have no trail; the trail starts
with the first HistoryUpdate after it.
### Writes ### Writes
@@ -843,8 +885,10 @@ result never has both.
### Events ### Events
`[start, end)` by `EventTimeUtc`; **a reversed window (`end < start`) is swapped and returned ascending**. A null `sourceName` means all `[start, end)` by `EventTimeUtc`; through the five-argument read **a reversed window (`end < start`) is swapped and returned
sources, otherwise `SourceName = @source`. Ordered `(EventTimeUtc, EventId)` ascending. A bounded read fetches `maxEvents + 1` rows and ascending**. The direction-aware overload with `reverse: true` (what the node manager calls for a reverse read, #675) reads
`(lo, hi]` ordered `(EventTimeUtc, EventId)` **descending** — the clustered index scanned backwards. A null `sourceName` means all
sources, otherwise `SourceName = @source`. Forward is ordered `(EventTimeUtc, EventId)` ascending. A bounded read fetches `maxEvents + 1` rows and
returns a non-null continuation point when the extra row exists; the node manager turns that flag into an OPC UA continuation returns a non-null continuation point when the extra row exists; the node manager turns that flag into an OPC UA continuation
point (see [Continuation-point paging (Events)](#continuation-point-paging-events-669)). point (see [Continuation-point paging (Events)](#continuation-point-paging-events-669)).
@@ -1042,6 +1086,8 @@ equivalent TTL (`QuestDb:RetentionDays`); the Gateway backend's retention is the
timeout 0) lets one purger run and the others skip that tick (Debug log). timeout 0) lets one purger run and the others skip that tick (Debug log).
- **Observability.** Counter `otopcua.historian.retention.purged{table=Sample|SampleModified|Event}`; a failed pass logs a Warning and retries next - **Observability.** Counter `otopcua.historian.retention.purged{table=Sample|SampleModified|Event}`; a failed pass logs a Warning and retries next
interval — it never affects capture or reads. interval — it never affects capture or reads.
- **Schema v1 (#674).** On a v1 schema (no `SampleModified`, `AutoCreateSchema=false`) the trail purge is skipped and omitted from the
pass results; samples and events purge normally.
- **Not done: partition switching.** For very large installs (billions of rows) a partitioned `hist.Sample` with sliding-window - **Not done: partition switching.** For very large installs (billions of rows) a partitioned `hist.Sample` with sliding-window
`SWITCH`/drop is cheaper than row deletes; the purger is deliberately simple batched deletes. Another table is added by one more `SWITCH`/drop is cheaper than row deletes; the purger is deliberately simple batched deletes. Another table is added by one more
`DeleteLoopAsync` call in `SqlServerHistorianRetention.PurgeAsync` (as `hist.SampleModified` was, #668). `DeleteLoopAsync` call in `SqlServerHistorianRetention.PurgeAsync` (as `hist.SampleModified` was, #668).
@@ -1222,11 +1268,12 @@ Every query filters `NOT deleted`; every value goes through `StoredSample`, so t
from the last timestamp ± `TimestampResolutionTicks`. from the last timestamp ± `TimestampResolutionTicks`.
- **AtTime:** previous-or-equal and next neighbour of each time by `LIMIT 1` seeks, `UNION ALL`-ed, **50 times per statement** (QuestDB 10.0.1 refuses - **AtTime:** previous-or-equal and next neighbour of each time by `LIMIT 1` seeks, `UNION ALL`-ed, **50 times per statement** (QuestDB 10.0.1 refuses
more than 126 branches). `AtTimeEvaluator` decides each value, as on `SqlServer`. more than 126 branches). `AtTimeEvaluator` decides each value, as on `SqlServer`.
- **Events:** `[start, end)` by `ts`, ordered `(ts, event_id)`, `maxEvents + 1` fetched to produce the continuation point. - **Events:** `[start, end)` by `ts`, ordered `(ts, event_id)`, `maxEvents + 1` fetched to produce the continuation point; a
reverse read (#675) is `(lo, hi]` ordered `(ts, event_id)` descending.
- **Processed:** one query per node; bucket index `(cast(ts AS LONG) * 10 - bucketBaseUnixTicks) / intervalTicks` (never `SAMPLE BY`, whose alignment - **Processed:** one query per node; bucket index `(cast(ts AS LONG) * 10 - bucketBaseUnixTicks) / intervalTicks` (never `SAMPLE BY`, whose alignment
differs), so bucket edges are identical to `SqlServer` (forward start-inclusive / end-exclusive with the partial bucket last; reverse anchored at the differs), so bucket edges are identical to `SqlServer` (forward start-inclusive / end-exclusive with the partial bucket last; reverse anchored at the
later time, later-edge-inclusive, partial bucket earliest — #677; interval 0 ⇒ one bucket, 100,000-bucket cap). A second `LIMIT 1` statement fetches later time, later-edge-inclusive, partial bucket earliest — #677; interval 0 ⇒ one bucket, 100,000-bucket cap). On a forward read of any aggregate
the tag's earliest live sample for the start-of-data Partial bit. but Average, a second `LIMIT 1` statement (capped at the window end) fetches the tag's earliest live sample for the start-of-data Partial bit.
Min/max times use `row_number()` windows with the earliest-timestamp tie-break. Min/max times use `row_number()` windows with the earliest-timestamp tie-break.
### Retention (TTL) ### Retention (TTL)
@@ -1240,7 +1287,14 @@ expired day partitions itself, tombstones included.
re-read to verify; a very large `RetentionDays` that QuestDB normalises into MONTH/YEAR is re-`ALTER`ed on every boot (harmless). re-read to verify; a very large `RetentionDays` that QuestDB normalises into MONTH/YEAR is re-`ALTER`ed on every boot (harmless).
- ⚠️ **TTL is enforced only on write commits.** QuestDB checks for expired partitions when a write commits to the table, so **a quiet table never - ⚠️ **TTL is enforced only on write commits.** QuestDB checks for expired partitions when a write commits to the table, so **a quiet table never
drops anything**: partitions older than the TTL linger until the next write lands. Do not read a stale-looking partition as a broken TTL. drops anything**: partitions older than the TTL linger until the next write lands. Do not read a stale-looking partition as a broken TTL.
- Expiry is judged against the wall clock, so a future-dated sample does not expire current data. - **Expiry is judged against the wall clock** (`min(max timestamp, now)`), so a future-dated sample does not expire current data.
⚠️ **Keep `cairo.ttl.use.wall.clock` at its default, `true`** (#674). It is what QuestDB documents as protecting a TTL table from a
future-dated row; with it `false` the TTL would be measured from the table's newest timestamp and one far-future row (a 2062 typo) could
drop every older partition — including `{p}sample_modified`, the record of who changed history. Evidence (QuestDB 10.0.1, 2026-10-11): the
default is `true`, and a 2066 row inserted into a 30-day-TTL table evicted nothing inside the TTL, while a partition genuinely older than
30 days was evicted by that same commit — pinned by `TtlWallClockTests`. (The same run with the flag `false` also kept the in-TTL
partitions, so the test pins the outcome rather than isolating the flag; do not rely on that.) The server-side
`ServerHistorian:HistoryUpdateMaxFutureSkew` bound (default 1 day) is the primary guard: a HistoryUpdate cannot write a far-future row.
- The TTL is visible in table metadata: `SELECT table_name, ttlValue, ttlUnit FROM tables() WHERE starts_with(table_name, 'hist_')`. - The TTL is visible in table metadata: `SELECT table_name, ttlValue, ttlUnit FROM tables() WHERE starts_with(table_name, 'hist_')`.
### Docker-dev recipe ### Docker-dev recipe
@@ -1281,8 +1335,8 @@ dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.Integration
OPC UA **HistoryUpdate** lets a client insert, replace, update and delete stored raw history. It is available only when the selected OPC UA **HistoryUpdate** lets a client insert, replace, update and delete stored raw history. It is available only when the selected
backend provides an updater — **`SqlServer` and `QuestDb` do; `Gateway` does not**, and an updater is registered only when `ServerHistorian:Enabled=true`. backend provides an updater — **`SqlServer` and `QuestDb` do; `Gateway` does not**, and an updater is registered only when `ServerHistorian:Enabled=true`.
Updates go straight to the backend (not through the outbox), are **not Primary-gated** (the database is shared), and share the HistoryRead Updates go straight to the backend (not through the outbox), are **Primary-only** (#674, [below](#primary-only-674)), and share the
batch limiter and `HistoryReadDeadline`. HistoryRead batch limiter and `HistoryReadDeadline`.
### Operations ### Operations
@@ -1300,7 +1354,8 @@ caller's principal, in the [modification trail](#modified-value-history-part-11)
mixed (Part 11 reports item outcomes in `OperationResults`); it is Bad only for whole-node failures. mixed (Part 11 reports item outcomes in `OperationResults`); it is Bad only for whole-node failures.
**Per-item rejections** (answered without touching SQL; other items still proceed): a null item ⇒ `BadInvalidArgument`; **Per-item rejections** (answered without touching SQL; other items still proceed): a null item ⇒ `BadInvalidArgument`;
`SourceTimestamp == DateTime.MinValue` ⇒ `BadInvalidTimestamp`; a value the historian cannot type (array, unsupported CLR type) ⇒ `SourceTimestamp == DateTime.MinValue` ⇒ `BadInvalidTimestamp`; a **`SourceTimestamp` later than `now + ServerHistorian:HistoryUpdateMaxFutureSkew`**
(default 1 day, server clock; #674) ⇒ **`BadOutOfRange`** — on every backend, never sent; a value the historian cannot type (array, unsupported CLR type) ⇒
`BadTypeMismatch`; a **`NaN` / `±Infinity` `Double` ⇒ `BadInvalidArgument`** — it cannot live in a SQL `float`, and unlike the live-value `BadTypeMismatch`; a **`NaN` / `±Infinity` `Double` ⇒ `BadInvalidArgument`** — it cannot live in a SQL `float`, and unlike the live-value
writer (which stores it as `Empty`) an operator's explicit update is rejected rather than silently rewritten; and a **null value with a writer (which stores it as `Empty`) an operator's explicit update is rejected rather than silently rewritten; and a **null value with a
Good or Uncertain status ⇒ `BadInvalidArgument`** — an `Empty` sample is always Bad, so a null value is accepted only with a Bad status Good or Uncertain status ⇒ `BadInvalidArgument`** — an `Empty` sample is always Bad, so a null value is accepted only with a Bad status
@@ -1323,7 +1378,19 @@ outcome, later ones `BadNoEntryExists`. **Unknown tag:** Insert / Update create
`ServerHistorian:Enabled=false`); `PerformUpdateType.Remove`; `DeleteRawModifiedDetails.IsDeleteModified=true` (the `ServerHistorian:Enabled=false`); `PerformUpdateType.Remove`; `DeleteRawModifiedDetails.IsDeleteModified=true` (the
[modification trail](#modified-value-history-part-11) is an audit record, aged out only by retention); and the out-of-scope detail types — **event** HistoryUpdate (`UpdateEventDetails` / `DeleteEventDetails`) and `UpdateStructureData` — [modification trail](#modified-value-history-part-11) is an audit record, aged out only by retention); and the out-of-scope detail types — **event** HistoryUpdate (`UpdateEventDetails` / `DeleteEventDetails`) and `UpdateStructureData` —
which are not overridden, so the SDK base rejects them. A backend exception is `BadInternalError` for that node and a deadline is `BadTimeout`. which are not overridden, so the SDK base rejects them. A backend exception is `BadInternalError` for that node and a deadline is `BadTimeout`.
No `AuditHistoryUpdateEvent` is emitted. On a `SqlServer` database still at schema v1 every item is `BadHistoryOperationUnsupported` ([#674](#modified-value-history-part-11)).
No `AuditHistoryUpdateEvent` is emitted (#674.5, not implemented).
### Primary-only (#674)
HistoryUpdate is served only by the node holding the redundancy **Primary** data-plane role — the same verdict
(`IDriverPrimaryRoleSource`, published by `DriverHostActor`) that gates native-alarm acknowledges (#597). On any other node every node of
the request that **passed authorization** is refused **`BadInvalidState`** (node-level code, with a message pointing at the Primary) and
nothing reaches the backend. Authorization runs first, so an unauthorized caller learns `BadUserAccessDenied`, not the node's role.
HistoryRead is **not** gated. Why: rewriting history is a single-writer operation — both backends capture the prior value for the
[modification trail](#modified-value-history-part-11) under a per-tag, **per-process** lock (QuestDB) or statement (SqlServer), so two pair
nodes updating one tag concurrently could record a wrong prior value. A single-node deployment (no redundancy manager) is unaffected. Clients
should send HistoryUpdate to the server advertising the higher `ServiceLevel`.
### The `HistoryWrite` AccessLevel bit ### The `HistoryWrite` AccessLevel bit
+2 -2
View File
@@ -32,13 +32,13 @@ Single-node dev: `OTOPCUA_ROLES=admin,driver` (or the `Cluster:Roles` equivalent
### Per-role configuration overlays ### Per-role configuration overlays
`Program.cs` builds a role suffix by joining the resolved `EffectiveRoles.Gossiped` roles **alphabetically** with `-` and loads `appsettings.{roleSuffix}.json` as an optional overlay on top of base `appsettings.json`. A plain role (`admin`/`driver`/`dev`) is always lower-case in `Gossiped` regardless of how the operator spelled it, so the overlay filename for those is stable (`appsettings.driver.json`); if a `cluster-{ClusterId}` role is also configured, its suffix carries into the filename **verbatim** (e.g. `Cluster:Roles: ["driver", "cluster-SITE-A"]` loads `appsettings.cluster-SITE-A-driver.json`), since `Gossiped` preserves that spelling by design (see [Configuration.md § `Roles`](Configuration.md#appsettings-sections)). Three overlays ship in `src/Server/ZB.MOM.WW.OtOpcUa.Host/`: `Program.cs` builds a role suffix (`RoleOverlay.Suffix`) by joining the resolved `EffectiveRoles.Gossiped` **plain** roles (`admin`/`driver`/`dev`) alphabetically with `-` and loads `appsettings.{roleSuffix}.json` as an optional overlay on top of base `appsettings.json`. A plain role is always lower-case in `Gossiped` regardless of how the operator spelled it, so the overlay filename is stable (`appsettings.driver.json`). **`cluster-{ClusterId}` roles are NOT part of the filename** (#678.6): they are mesh-membership tags, so `Cluster:Roles: ["driver", "cluster-SITE-A"]` loads `appsettings.driver.json` and `["admin", "driver", "cluster-MAIN"]` loads `appsettings.admin-driver.json`. (Before #678.6 the cluster role was folded into the name, producing e.g. `appsettings.admin-cluster-MAIN-driver.json`, which never exists — so on every node with a cluster role no shipped overlay ever loaded.) Three overlays ship in `src/Server/ZB.MOM.WW.OtOpcUa.Host/`:
- `appsettings.admin.json` — admin-only nodes - `appsettings.admin.json` — admin-only nodes
- `appsettings.driver.json` — driver-only nodes - `appsettings.driver.json` — driver-only nodes
- `appsettings.admin-driver.json` — fused single-node dev / small deployments - `appsettings.admin-driver.json` — fused single-node dev / small deployments
All three carry Serilog log-level overrides + `Security:Ldap:DevStubMode = false`. Configuration loading order (lowest to highest precedence) is: All three carry Serilog log-level overrides + `Security:Ldap:DevStubMode = false` + `Security:Ldap:Transport = Ldaps` (audit table: [Configuration.md § Role overlays](Configuration.md#role-overlay-audit)). Configuration loading order (lowest to highest precedence) is:
`appsettings.json` < `appsettings.{Environment}.json` < `appsettings.{role}.json` < environment variables < command-line args `appsettings.json` < `appsettings.{Environment}.json` < `appsettings.{role}.json` < environment variables < command-line args
+1
View File
@@ -884,6 +884,7 @@ HistoryUpdate rewrites recorded history, so unlike HistoryRead it is gated **wha
- **Flag off (default):** the session's API key is re-checked on the call (`DataPlaneAuthorizer.AuthorizeKeyUsable`, Gitea #563 — an expired or disabled key loses its activation-time roles), then the session must carry the **`HistoryUpdate` role** (`OpcUaDataPlaneRoles.HistoryUpdate`). `ApiKeyInterimRoleMapper` confers it **only from a `Cluster`-scoped grant** carrying `NodePermissions.HistoryUpdate` — narrower-scoped grants confer nothing, for the same reason as `WriteOperate` (honouring a flag without honouring its scope would widen it). A key holding only `WriteOperate` is denied `BadUserAccessDenied`. - **Flag off (default):** the session's API key is re-checked on the call (`DataPlaneAuthorizer.AuthorizeKeyUsable`, Gitea #563 — an expired or disabled key loses its activation-time roles), then the session must carry the **`HistoryUpdate` role** (`OpcUaDataPlaneRoles.HistoryUpdate`). `ApiKeyInterimRoleMapper` confers it **only from a `Cluster`-scoped grant** carrying `NodePermissions.HistoryUpdate` — narrower-scoped grants confer nothing, for the same reason as `WriteOperate` (honouring a flag without honouring its scope would widen it). A key holding only `WriteOperate` is denied `BadUserAccessDenied`.
- **Flag on:** `DataPlaneAuthorizer.Authorize(identity, OpcUaOperation.HistoryUpdate, nodeId, realm)` decides per node and **replaces** the role check (never chained) — a session with no roles succeeds on the strength of a scoped `HistoryUpdate` grant, and a `WriteOperate` grant does not help. - **Flag on:** `DataPlaneAuthorizer.Authorize(identity, OpcUaOperation.HistoryUpdate, nodeId, realm)` decides per node and **replaces** the role check (never chained) — a session with no roles succeeds on the strength of a scoped `HistoryUpdate` grant, and a `WriteOperate` grant does not help.
- **Availability is separate from authorization.** A node accepts HistoryUpdate only if it is historized **and** the selected historian backend has an updater when the node materialises (the `HistoryWrite` AccessLevel bit). `Backend=Gateway` has none, so it answers `BadHistoryOperationUnsupported` for every node whatever the grant. Operations, result codes and unsupported cases: [`docs/Historian.md` §HistoryUpdate](Historian.md#historyupdate). - **Availability is separate from authorization.** A node accepts HistoryUpdate only if it is historized **and** the selected historian backend has an updater when the node materialises (the `HistoryWrite` AccessLevel bit). `Backend=Gateway` has none, so it answers `BadHistoryOperationUnsupported` for every node whatever the grant. Operations, result codes and unsupported cases: [`docs/Historian.md` §HistoryUpdate](Historian.md#historyupdate).
- **Primary-only, after authorization (Gitea #674).** An authorized HistoryUpdate on a node that is not the redundancy Primary is refused `BadInvalidState` (the native-alarm-ack verdict, #597); an unauthorized one is told `BadUserAccessDenied` first, so the refusal never reveals the node's role to a caller who may not update.
- **Every applied change is attributed (Part 11 modified-value history, Gitea #668).** The session's principal (the API key's - **Every applied change is attributed (Part 11 modified-value history, Gitea #668).** The session's principal (the API key's
`NodeAcl` principal, or the reserved `anonymous`) and key id are passed to the backend with every HistoryUpdate and recorded in the `NodeAcl` principal, or the reserved `anonymous`) and key id are passed to the backend with every HistoryUpdate and recorded in the
modification trail next to the prior value; a modified-value HistoryRead (`IsReadModified = true`) returns it as modification trail next to the prior value; a modified-value HistoryRead (`IsReadModified = true`) returns it as
@@ -47,7 +47,7 @@ public class HistoryReadCommand : CommandBase
/// <summary> /// <summary>
/// Gets the optional aggregate name used when the operator wants processed history instead of raw samples. /// Gets the optional aggregate name used when the operator wants processed history instead of raw samples.
/// </summary> /// </summary>
[CommandOption("aggregate", Description = "Aggregate function: Average, Minimum, Maximum, Count, Start, End, StandardDeviation")] [CommandOption("aggregate", Description = "Aggregate function: Average, Minimum, Maximum, Count, Start, End, StandardDeviation, TimeAverage, TimeAverage2, Total, Total2, Interpolative")]
public string? Aggregate { get; init; } public string? Aggregate { get; init; }
/// <summary> /// <summary>
@@ -183,6 +183,12 @@ public class HistoryReadCommand : CommandBase
await console.Output.WriteLineAsync(); await console.Output.WriteLineAsync();
await console.Output.WriteLineAsync($"{values.Count} values returned."); await console.Output.WriteLineAsync($"{values.Count} values returned.");
} }
catch (ServiceResultException ex) when (ex.InnerException is null)
{
// e.g. HistoryReadStatus.ThrowIfRefused: a refusal is an operator-facing outcome, not a crash.
// A wrapped failure (certificate validation, identity rejection on connect) keeps its full detail.
throw new CommandException(ex.Message);
}
finally finally
{ {
if (service != null) if (service != null)
@@ -204,8 +210,13 @@ public class HistoryReadCommand : CommandBase
"start" or "first" => AggregateType.Start, "start" or "first" => AggregateType.Start,
"end" or "last" => AggregateType.End, "end" or "last" => AggregateType.End,
"standarddeviation" or "stddev" or "stdev" => AggregateType.StandardDeviation, "standarddeviation" or "stddev" or "stdev" => AggregateType.StandardDeviation,
"timeaverage" or "timeavg" => AggregateType.TimeAverage,
"timeaverage2" or "timeavg2" => AggregateType.TimeAverage2,
"total" => AggregateType.Total,
"total2" => AggregateType.Total2,
"interpolative" or "interp" => AggregateType.Interpolative,
_ => throw new ArgumentException( _ => throw new ArgumentException(
$"Unknown aggregate: '{name}'. Supported: Average, Minimum, Maximum, Count, Start, End, StandardDeviation") $"Unknown aggregate: '{name}'. Supported: Average, Minimum, Maximum, Count, Start, End, StandardDeviation, TimeAverage, TimeAverage2, Total, Total2, Interpolative")
}; };
} }
} }
@@ -85,17 +85,22 @@ internal static class EndpointSelector
$"No endpoint found with security mode '{requestedMode}'. Available endpoints: {available}"); $"No endpoint found with security mode '{requestedMode}'. Available endpoints: {available}");
} }
// Rewrite endpoint URL hostname to match user-supplied hostname. Necessary // Rewrite the endpoint URL host AND port to the user-supplied ones (as the SDK's
// when the OPC UA server returns a discovery URL using a different hostname // CoreClientUtils.SelectEndpoint does). A server behind a port mapping or internal DNS
// (e.g. internal DNS name) than the one the operator routed to. // advertises its own host:port (docker-dev containers all say :4840); following that would
// open the session on a different server and fail the secure-channel certificate check.
// Only when the scheme matches: a different scheme (https vs opc.tcp) is a different transport.
var serverUri = new Uri(best.EndpointUrl); var serverUri = new Uri(best.EndpointUrl);
var requestedUri = new Uri(endpointUrl); var requestedUri = new Uri(endpointUrl);
if (serverUri.Host != requestedUri.Host) // A URL typed without a port (opc.tcp has no registered default, so Port is -1) keeps the advertised port.
var port = requestedUri.Port > 0 ? requestedUri.Port : serverUri.Port;
if (string.Equals(serverUri.Scheme, requestedUri.Scheme, StringComparison.OrdinalIgnoreCase)
&& (serverUri.Host != requestedUri.Host || serverUri.Port != port))
{ {
var builder = new UriBuilder(best.EndpointUrl) { Host = requestedUri.Host }; var builder = new UriBuilder(best.EndpointUrl) { Host = requestedUri.Host, Port = port };
best.EndpointUrl = builder.ToString(); best.EndpointUrl = builder.ToString();
Logger.Debug("Rewrote endpoint host from {ServerHost} to {RequestedHost}", serverUri.Host, Logger.Debug("Rewrote endpoint from {ServerHost}:{ServerPort} to {RequestedHost}:{RequestedPort}",
requestedUri.Host); serverUri.Host, serverUri.Port, requestedUri.Host, port);
} }
return best; return best;
@@ -24,6 +24,11 @@ public static class AggregateTypeMapper
AggregateType.Start => ObjectIds.AggregateFunction_Start, AggregateType.Start => ObjectIds.AggregateFunction_Start,
AggregateType.End => ObjectIds.AggregateFunction_End, AggregateType.End => ObjectIds.AggregateFunction_End,
AggregateType.StandardDeviation => ObjectIds.AggregateFunction_StandardDeviationPopulation, AggregateType.StandardDeviation => ObjectIds.AggregateFunction_StandardDeviationPopulation,
AggregateType.TimeAverage => ObjectIds.AggregateFunction_TimeAverage,
AggregateType.TimeAverage2 => ObjectIds.AggregateFunction_TimeAverage2,
AggregateType.Total => ObjectIds.AggregateFunction_Total,
AggregateType.Total2 => ObjectIds.AggregateFunction_Total2,
AggregateType.Interpolative => ObjectIds.AggregateFunction_Interpolative,
_ => throw new ArgumentOutOfRangeException(nameof(aggregate), aggregate, "Unknown AggregateType value.") _ => throw new ArgumentOutOfRangeException(nameof(aggregate), aggregate, "Unknown AggregateType value.")
}; };
} }
@@ -24,5 +24,20 @@ public enum AggregateType
End, End,
/// <summary>Population standard deviation of values in the interval.</summary> /// <summary>Population standard deviation of values in the interval.</summary>
StandardDeviation StandardDeviation,
/// <summary>Time-weighted average, interpolated bounds (Part 13 TimeAverage).</summary>
TimeAverage,
/// <summary>Time-weighted average, stepped bounds (Part 13 TimeAverage2).</summary>
TimeAverage2,
/// <summary>Time-weighted integral over the interval (Part 13 Total).</summary>
Total,
/// <summary>Time-weighted integral, stepped bounds (Part 13 Total2).</summary>
Total2,
/// <summary>Interpolated value at the interval start (Part 13 Interpolative).</summary>
Interpolative
} }
@@ -53,7 +53,12 @@ public partial class HistoryViewModel : ObservableObject
AggregateType.Count, AggregateType.Count,
AggregateType.Start, AggregateType.Start,
AggregateType.End, AggregateType.End,
AggregateType.StandardDeviation AggregateType.StandardDeviation,
AggregateType.TimeAverage,
AggregateType.TimeAverage2,
AggregateType.Total,
AggregateType.Total2,
AggregateType.Interpolative
]; ];
/// <summary>Gets a value indicating whether an aggregate read is selected.</summary> /// <summary>Gets a value indicating whether an aggregate read is selected.</summary>
@@ -88,7 +88,7 @@ public sealed class HistoryReadLimitExceededException : InvalidOperationExceptio
} }
/// <summary> /// <summary>
/// One row returned by <see cref="IHistorianDataSource.ReadEventsAsync"/> — a historical /// One row returned by <see cref="IHistorianDataSource.ReadEventsAsync(string?, DateTime, DateTime, int, CancellationToken)"/> — a historical
/// alarm/event record, not the OPC UA live-event stream. Fields match the minimum set the /// alarm/event record, not the OPC UA live-event stream. Fields match the minimum set the
/// Server needs to populate a <c>HistoryEventFieldList</c> for HistoryReadEvents responses. /// Server needs to populate a <c>HistoryEventFieldList</c> for HistoryReadEvents responses.
/// </summary> /// </summary>
@@ -106,8 +106,8 @@ public sealed record HistoricalEvent(
string? Message, string? Message,
ushort Severity); ushort Severity);
/// <summary>Result of a <see cref="IHistorianDataSource.ReadEventsAsync"/> call.</summary> /// <summary>Result of a <see cref="IHistorianDataSource.ReadEventsAsync(string?, DateTime, DateTime, int, CancellationToken)"/> call.</summary>
/// <param name="Events">Events in chronological order by <c>EventTimeUtc</c>.</param> /// <param name="Events">Events ordered by <c>EventTimeUtc</c> in the read's direction — ascending, or descending for a reverse read (#675).</param>
/// <param name="ContinuationPoint">Opaque token for the next call when more events are available; null when complete.</param> /// <param name="ContinuationPoint">Opaque token for the next call when more events are available; null when complete.</param>
public sealed record HistoricalEventsResult( public sealed record HistoricalEventsResult(
IReadOnlyList<HistoricalEvent> Events, IReadOnlyList<HistoricalEvent> Events,
@@ -79,7 +79,7 @@ public interface IHistorianDataSource : IDisposable
/// <c>uint maxValuesPerNode</c> because their OPC UA HistoryRead surface has no /// <c>uint maxValuesPerNode</c> because their OPC UA HistoryRead surface has no
/// equivalent "use default" sentinel. /// equivalent "use default" sentinel.
/// ///
/// This surface declares <see cref="ReadAtTimeAsync"/> and <see cref="ReadEventsAsync"/> /// This surface declares <see cref="ReadAtTimeAsync"/> and <see cref="ReadEventsAsync(string?, DateTime, DateTime, int, CancellationToken)"/>
/// as required members — a server-side historian owns the full read surface. /// as required members — a server-side historian owns the full read surface.
/// </remarks> /// </remarks>
/// <param name="sourceName">The source name to filter events, or null to return events from all sources.</param> /// <param name="sourceName">The source name to filter events, or null to return events from all sources.</param>
@@ -101,6 +101,46 @@ public interface IHistorianDataSource : IDisposable
int maxEvents, int maxEvents,
CancellationToken cancellationToken); CancellationToken cancellationToken);
/// <summary>
/// Read historical alarm / event records in a chosen direction — the overload the server's HistoryReadEvents
/// paging calls (Gitea #675).
/// </summary>
/// <remarks>
/// <para>
/// The bounds may be given in either order; let <c>lo</c>/<c>hi</c> be the earlier/later of them. Forward
/// (<paramref name="reverse"/> <c>false</c>) is exactly <see cref="ReadEventsAsync(string?, DateTime, DateTime, int, CancellationToken)"/>:
/// <c>[lo, hi)</c> ascending by <c>(EventTimeUtc, EventId)</c>. Reverse is the Part 11 reverse window
/// <c>(lo, hi]</c> in DESCENDING <c>(EventTimeUtc, EventId)</c> order, so a cap keeps the LATEST
/// <paramref name="maxEvents"/> events — the mirror of forward, with the same tie-breaker reversed, so a
/// resumed reverse page that re-reads its boundary timestamp sees the boundary ties in the same order.
/// <paramref name="maxEvents"/> and the continuation-point contract are as for the forward read.
/// </para>
/// <para>
/// The default forwards a forward read to the five-argument overload and throws
/// <see cref="NotSupportedException"/> for a reverse one — a backend that cannot read newest-first (Gateway,
/// Null) — which the server reports as <c>BadHistoryOperationUnsupported</c> rather than returning the
/// earliest events in ascending order as if they were the answer.
/// </para>
/// </remarks>
/// <param name="sourceName">The source name to filter events, or null to return events from all sources.</param>
/// <param name="startUtc">One bound of the window (UTC).</param>
/// <param name="endUtc">The other bound of the window (UTC).</param>
/// <param name="maxEvents">The cap; non-positive uses the backend default (see the five-argument overload).</param>
/// <param name="reverse"><c>true</c> to read newest-first over <c>(lo, hi]</c>.</param>
/// <param name="cancellationToken">Cancellation.</param>
/// <returns>The matching events in the read's direction, with a continuation point if truncated.</returns>
/// <exception cref="NotSupportedException">A reverse read on a backend that cannot read newest-first.</exception>
Task<HistoricalEventsResult> ReadEventsAsync(
string? sourceName,
DateTime startUtc,
DateTime endUtc,
int maxEvents,
bool reverse,
CancellationToken cancellationToken) =>
reverse
? throw new NotSupportedException("This historian backend cannot read events newest-first (reverse).")
: ReadEventsAsync(sourceName, startUtc, endUtc, maxEvents, cancellationToken);
/// <summary> /// <summary>
/// Point-in-time health snapshot for diagnostics and dashboards. Pure /// Point-in-time health snapshot for diagnostics and dashboards. Pure
/// observation; never blocks on backend I/O. /// observation; never blocks on backend I/O.
@@ -125,6 +125,21 @@ public sealed class GatewayHistorianDataSource : IHistorianDataSource, IAsyncDis
} }
} }
/// <inheritdoc />
/// <remarks>
/// A reverse (newest-first) read is refused with <see cref="NotSupportedException"/> — the server reports
/// <c>BadHistoryOperationUnsupported</c> (Gitea #675). The gateway's <c>ReadEvents</c> streams ascending only and
/// rejects <c>end &lt; start</c>, so the only way to serve "the latest N" would be to drain the whole window and
/// keep its tail, an unbounded read the cap exists to prevent. Nothing is sent.
/// </remarks>
public Task<HistoricalEventsResult> ReadEventsAsync(
string? sourceName, DateTime startUtc, DateTime endUtc, int maxEvents, bool reverse,
CancellationToken cancellationToken) =>
reverse
? throw new NotSupportedException(
"The HistorianGateway backend reads events oldest-first only; a reverse (newest-first) events read is not supported.")
: ReadEventsAsync(sourceName, startUtc, endUtc, maxEvents, cancellationToken);
/// <inheritdoc /> /// <inheritdoc />
public async Task<HistoricalEventsResult> ReadEventsAsync( public async Task<HistoricalEventsResult> ReadEventsAsync(
string? sourceName, DateTime startUtc, DateTime endUtc, int maxEvents, string? sourceName, DateTime startUtc, DateTime endUtc, int maxEvents,
@@ -361,8 +361,18 @@ public sealed class QuestDbHistorianDataSource : IHistorianDataSource
/// <c>maxEvents + 1</c> rows are fetched; when the extra row arrives it is dropped and the continuation point is /// <c>maxEvents + 1</c> rows are fetched; when the extra row arrives it is dropped and the continuation point is
/// the UTF-8 of <see cref="EventContinuation.Encode"/> for that first not-returned event. /// the UTF-8 of <see cref="EventContinuation.Encode"/> for that first not-returned event.
/// </remarks> /// </remarks>
public Task<HistoricalEventsResult> ReadEventsAsync(
string? sourceName, DateTime startUtc, DateTime endUtc, int maxEvents, CancellationToken cancellationToken) =>
ReadEventsAsync(sourceName, startUtc, endUtc, maxEvents, reverse: false, cancellationToken);
/// <inheritdoc />
/// <remarks>
/// Forward is the five-argument read. Reverse (Gitea #675) is the (rounded) <c>(lo, hi]</c> ordered by
/// <c>(ts, event_id)</c> DESCENDING, so <c>LIMIT</c> keeps the latest events; truncation and the continuation point
/// work as forward.
/// </remarks>
public async Task<HistoricalEventsResult> ReadEventsAsync( public async Task<HistoricalEventsResult> ReadEventsAsync(
string? sourceName, DateTime startUtc, DateTime endUtc, int maxEvents, CancellationToken cancellationToken) string? sourceName, DateTime startUtc, DateTime endUtc, int maxEvents, bool reverse, CancellationToken cancellationToken)
{ {
// No stored event can carry a source longer than the limit, so an over-long filter matches nothing. // No stored event can carry a source longer than the limit, so an over-long filter matches nothing.
if (sourceName is not null && sourceName.Length > HistorianTagNames.MaxLength) if (sourceName is not null && sourceName.Length > HistorianTagNames.MaxLength)
@@ -376,9 +386,9 @@ public sealed class QuestDbHistorianDataSource : IHistorianDataSource
var bounded = maxEvents > 0 && maxEvents < int.MaxValue; var bounded = maxEvents > 0 && maxEvents < int.MaxValue;
var sql = var sql =
$"SELECT event_id, source, ts, received_utc, message, severity FROM {_conn.EventTable} " + $"SELECT event_id, source, ts, received_utc, message, severity FROM {_conn.EventTable} " +
"WHERE ts >= $1 AND ts < $2" + (reverse ? "WHERE ts > $1 AND ts <= $2" : "WHERE ts >= $1 AND ts < $2") +
(sourceName is null ? "" : " AND source = $3") + (sourceName is null ? "" : " AND source = $3") +
" ORDER BY ts ASC, event_id ASC" + (reverse ? " ORDER BY ts DESC, event_id DESC" : " ORDER BY ts ASC, event_id ASC") +
(bounded ? (sourceName is null ? " LIMIT $3" : " LIMIT $4") : ""); (bounded ? (sourceName is null ? " LIMIT $3" : " LIMIT $4") : "");
await using var conn = await _conn.OpenAsync(cancellationToken).ConfigureAwait(false); await using var conn = await _conn.OpenAsync(cancellationToken).ConfigureAwait(false);
@@ -137,7 +137,10 @@ internal sealed class QuestDbSchemaBootstrapper
// The ALTER is a WAL transaction: tables() keeps the old value until it applies, so do NOT re-read to verify. // The ALTER is a WAL transaction: tables() keeps the old value until it applies, so do NOT re-read to verify.
await using (var alter = _conn.CreateCommand(conn, QuestDbTables.AlterTtl(table, _options.RetentionDays))) await using (var alter = _conn.CreateCommand(conn, QuestDbTables.AlterTtl(table, _options.RetentionDays)))
await alter.ExecuteNonQueryAsync(ct).ConfigureAwait(false); await alter.ExecuteNonQueryAsync(ct).ConfigureAwait(false);
_log.LogInformation("Applied TTL {Days} day(s) to QuestDB table {Table}.", _options.RetentionDays, table); if (_options.RetentionDays is { } days)
_log.LogInformation("Applied TTL {Days} day(s) to QuestDB table {Table}.", days, table);
else
_log.LogInformation("Cleared TTL on QuestDB table {Table}.", table);
} }
} }
} }
@@ -27,4 +27,10 @@ public static class HistorianStatusCodes
/// status, or a tag name longer than <see cref="HistorianTagNames.MaxLength"/>. /// status, or a tag name longer than <see cref="HistorianTagNames.MaxLength"/>.
/// </summary> /// </summary>
public const uint BadInvalidArgument = 0x80AB0000u; public const uint BadInvalidArgument = 0x80AB0000u;
/// <summary>
/// A modified-value read or HistoryUpdate item against a SQL Server schema still at version 1 (no Part 11
/// modification trail, Gitea #674): refused without touching the database.
/// </summary>
public const uint BadHistoryOperationUnsupported = 0x80720000u;
} }
@@ -154,6 +154,12 @@ public sealed class SqlServerHistorianDataSource : IHistorianDataSource
var bounded = maxValues > 0 && maxValues < int.MaxValue; var bounded = maxValues > 0 && maxValues < int.MaxValue;
try try
{ {
// #674: a v1 schema has no SampleModified. NotSupportedException is the interface's "no trail" signal (the
// node manager reports BadHistoryOperationUnsupported); it is not a backend failure and sends no query.
await _bootstrap.EnsureAsync(cancellationToken).ConfigureAwait(false);
if (!_bootstrap.SupportsModifiedTrail)
throw new NotSupportedException(
$"Historian schema [{_conn.Schema}] is below version {SqlServerSchemaBootstrapper.ModifiedTrailVersion}: no modified-value history.");
var values = new List<HistoryModifiedValue>(); var values = new List<HistoryModifiedValue>();
var hasMore = false; var hasMore = false;
var tagId = await ResolveAsync(fullReference, cancellationToken).ConfigureAwait(false); var tagId = await ResolveAsync(fullReference, cancellationToken).ConfigureAwait(false);
@@ -219,7 +225,7 @@ public sealed class SqlServerHistorianDataSource : IHistorianDataSource
RecordSuccess(); RecordSuccess();
return new HistoryModifiedReadResult(values, hasMore); return new HistoryModifiedReadResult(values, hasMore);
} }
catch (Exception ex) catch (Exception ex) when (ex is not NotSupportedException)
{ {
RecordFailure(ex); RecordFailure(ex);
throw; throw;
@@ -314,8 +320,18 @@ public sealed class SqlServerHistorianDataSource : IHistorianDataSource
/// of <c>"{EventTimeUtc.Ticks}|{EventId}"</c> of that first not-returned event (opaque to callers; only /// of <c>"{EventTimeUtc.Ticks}|{EventId}"</c> of that first not-returned event (opaque to callers; only
/// its presence is contractual). /// its presence is contractual).
/// </remarks> /// </remarks>
public Task<HistoricalEventsResult> ReadEventsAsync(
string? sourceName, DateTime startUtc, DateTime endUtc, int maxEvents, CancellationToken cancellationToken) =>
ReadEventsAsync(sourceName, startUtc, endUtc, maxEvents, reverse: false, cancellationToken);
/// <inheritdoc />
/// <remarks>
/// Forward is the five-argument read. Reverse (Gitea #675) is <c>(lo, hi]</c> ordered by
/// <c>(EventTimeUtc, EventId)</c> DESCENDING — the clustered index scanned backwards — so <c>TOP</c> keeps the latest
/// events; truncation and the continuation point work as forward.
/// </remarks>
public async Task<HistoricalEventsResult> ReadEventsAsync( public async Task<HistoricalEventsResult> ReadEventsAsync(
string? sourceName, DateTime startUtc, DateTime endUtc, int maxEvents, CancellationToken cancellationToken) string? sourceName, DateTime startUtc, DateTime endUtc, int maxEvents, bool reverse, CancellationToken cancellationToken)
{ {
// No stored event can carry a source longer than the column (the alarm writer dead-letters it rather than // No stored event can carry a source longer than the column (the alarm writer dead-letters it rather than
// truncating), so an over-long filter matches nothing — never a truncated prefix of someone else's source. // truncating), so an over-long filter matches nothing — never a truncated prefix of someone else's source.
@@ -328,9 +344,10 @@ public sealed class SqlServerHistorianDataSource : IHistorianDataSource
var bounded = maxEvents > 0 && maxEvents < int.MaxValue; var bounded = maxEvents > 0 && maxEvents < int.MaxValue;
var sql = var sql =
$"SELECT {(bounded ? "TOP (@take) " : "")}EventId, SourceName, EventTimeUtc, ReceivedTimeUtc, Message, Severity " + $"SELECT {(bounded ? "TOP (@take) " : "")}EventId, SourceName, EventTimeUtc, ReceivedTimeUtc, Message, Severity " +
$"FROM {_conn.Q("Event")} WHERE EventTimeUtc >= @lo AND EventTimeUtc < @hi" + $"FROM {_conn.Q("Event")} WHERE " +
(reverse ? "EventTimeUtc > @lo AND EventTimeUtc <= @hi" : "EventTimeUtc >= @lo AND EventTimeUtc < @hi") +
(sourceName is null ? "" : " AND SourceName = @source") + (sourceName is null ? "" : " AND SourceName = @source") +
" ORDER BY EventTimeUtc ASC, EventId ASC"; (reverse ? " ORDER BY EventTimeUtc DESC, EventId DESC" : " ORDER BY EventTimeUtc ASC, EventId ASC");
await using var conn = await _conn.OpenAsync(cancellationToken).ConfigureAwait(false); await using var conn = await _conn.OpenAsync(cancellationToken).ConfigureAwait(false);
await using var cmd = _conn.CreateCommand(conn, sql); await using var cmd = _conn.CreateCommand(conn, sql);
@@ -27,21 +27,44 @@ namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer;
/// value from <c>deleted.*</c> (NULL for an insert), the caller's <see cref="HistoryUpdateContext"/>, the /// value from <c>deleted.*</c> (NULL for an insert), the caller's <see cref="HistoryUpdateContext"/>, the
/// <see cref="HistoryModificationType"/> and <c>SYSUTCDATETIME()</c>. One statement means the sample change and its /// <see cref="HistoryModificationType"/> and <c>SYSUTCDATETIME()</c>. One statement means the sample change and its
/// audit row commit or roll back together: a failed update leaves no audit row.</para> /// audit row commit or roll back together: a failed update leaves no audit row.</para>
/// <para><b>Schema v1 (Gitea #674).</b> Without the trail (<see cref="SqlServerSchemaBootstrapper.SupportsModifiedTrail"/>
/// false) a change could not be audited, so every item (and DeleteRaw's single result) reports
/// <c>BadHistoryOperationUnsupported</c> and nothing is sent to the database beyond the one-time schema check.</para>
/// </summary> /// </summary>
internal sealed class SqlServerHistorianDataUpdater : IHistorianDataUpdater internal sealed class SqlServerHistorianDataUpdater : IHistorianDataUpdater
{ {
private readonly SqlHistorianConnection _conn; private readonly SqlHistorianConnection _conn;
private readonly SqlServerTagCatalog _tags; private readonly SqlServerTagCatalog _tags;
private readonly SqlServerSchemaBootstrapper _bootstrap;
/// <summary>Creates the updater.</summary> /// <summary>Creates the updater.</summary>
/// <param name="conn">Connection helper.</param> /// <param name="conn">Connection helper.</param>
/// <param name="bootstrap">Schema bootstrapper — decides whether the modification trail exists (#674).</param>
/// <param name="tags">Tag catalog (runs the schema bootstrap itself).</param> /// <param name="tags">Tag catalog (runs the schema bootstrap itself).</param>
public SqlServerHistorianDataUpdater(SqlHistorianConnection conn, SqlServerTagCatalog tags) public SqlServerHistorianDataUpdater(SqlHistorianConnection conn, SqlServerSchemaBootstrapper bootstrap,
SqlServerTagCatalog tags)
{ {
_conn = conn; _conn = conn;
_bootstrap = bootstrap;
_tags = tags; _tags = tags;
} }
/// <summary>#674: ensures the schema and reports whether the Part 11 trail exists (no SQL after the first success).</summary>
/// <param name="ct">Cancellation.</param>
/// <returns>True when the modification trail is available.</returns>
private async Task<bool> TrailAvailableAsync(CancellationToken ct)
{
await _bootstrap.EnsureAsync(ct).ConfigureAwait(false);
return _bootstrap.SupportsModifiedTrail;
}
private static uint[] Unsupported(int count)
{
var codes = new uint[count];
Array.Fill(codes, HistorianStatusCodes.BadHistoryOperationUnsupported);
return codes;
}
/// <inheritdoc/> /// <inheritdoc/>
public async Task<IReadOnlyList<uint>> UpdateDataAsync(string tag, HistoryUpdateMode mode, public async Task<IReadOnlyList<uint>> UpdateDataAsync(string tag, HistoryUpdateMode mode,
IReadOnlyList<HistorizationValue> values, HistoryUpdateContext context, CancellationToken ct) IReadOnlyList<HistorizationValue> values, HistoryUpdateContext context, CancellationToken ct)
@@ -49,6 +72,7 @@ internal sealed class SqlServerHistorianDataUpdater : IHistorianDataUpdater
ArgumentNullException.ThrowIfNull(tag); ArgumentNullException.ThrowIfNull(tag);
ArgumentNullException.ThrowIfNull(values); ArgumentNullException.ThrowIfNull(values);
ArgumentNullException.ThrowIfNull(context); ArgumentNullException.ThrowIfNull(context);
if (!await TrailAvailableAsync(ct).ConfigureAwait(false)) return Unsupported(values.Count);
var plan = HistoryUpdatePlanner.Plan(tag, mode, values); var plan = HistoryUpdatePlanner.Plan(tag, mode, values);
if (plan.Winners.Count == 0) return HistoryUpdatePlanner.Complete(plan, mode, new Dictionary<long, UpsertOutcome>()); if (plan.Winners.Count == 0) return HistoryUpdatePlanner.Complete(plan, mode, new Dictionary<long, UpsertOutcome>());
@@ -68,6 +92,7 @@ internal sealed class SqlServerHistorianDataUpdater : IHistorianDataUpdater
{ {
ArgumentNullException.ThrowIfNull(tag); ArgumentNullException.ThrowIfNull(tag);
ArgumentNullException.ThrowIfNull(context); ArgumentNullException.ThrowIfNull(context);
if (!await TrailAvailableAsync(ct).ConfigureAwait(false)) return HistorianStatusCodes.BadHistoryOperationUnsupported;
if (!HistorianTagNames.IsValid(tag)) return HistorianStatusCodes.BadInvalidArgument; if (!HistorianTagNames.IsValid(tag)) return HistorianStatusCodes.BadInvalidArgument;
var tagId = await _tags.TryGetTagIdAsync(tag, ct).ConfigureAwait(false); var tagId = await _tags.TryGetTagIdAsync(tag, ct).ConfigureAwait(false);
if (tagId is null) return HistoryUpdatePlanner.DeleteRawResult(tagKnown: false, 0); if (tagId is null) return HistoryUpdatePlanner.DeleteRawResult(tagKnown: false, 0);
@@ -95,6 +120,7 @@ internal sealed class SqlServerHistorianDataUpdater : IHistorianDataUpdater
ArgumentNullException.ThrowIfNull(tag); ArgumentNullException.ThrowIfNull(tag);
ArgumentNullException.ThrowIfNull(timesUtc); ArgumentNullException.ThrowIfNull(timesUtc);
ArgumentNullException.ThrowIfNull(context); ArgumentNullException.ThrowIfNull(context);
if (!await TrailAvailableAsync(ct).ConfigureAwait(false)) return Unsupported(timesUtc.Count);
if (!HistorianTagNames.IsValid(tag)) if (!HistorianTagNames.IsValid(tag))
{ {
var refused = new uint[timesUtc.Count]; var refused = new uint[timesUtc.Count];
@@ -18,8 +18,10 @@ public sealed partial class SqlServerHistorianOptions
/// <summary> /// <summary>
/// <c>true</c> creates the schema and applies missing scripts on first use; <c>false</c> only verifies /// <c>true</c> creates the schema and applies missing scripts on first use; <c>false</c> only verifies
/// that the required schema version (<c>SqlServerSchemaBootstrapper.RequiredVersion</c>, currently 2) exists and /// that the base schema version (<c>SqlServerSchemaBootstrapper.RequiredVersion</c>, 1) exists and fails every
/// fails every historian call with a clear error otherwise. /// historian call with a clear error otherwise. A verified v1 schema (no Part 11 modification trail, script 002
/// not applied) is served with modified reads and HistoryUpdate refused <c>BadHistoryOperationUnsupported</c>
/// and one Warning per boot (Gitea #674).
/// </summary> /// </summary>
public bool AutoCreateSchema { get; set; } = true; public bool AutoCreateSchema { get; set; } = true;
@@ -99,7 +99,7 @@ public sealed class SqlServerHistorianProvider : IHistorianBackendProvider
public IHistorianDataUpdater? CreateUpdater(IServiceProvider sp) public IHistorianDataUpdater? CreateUpdater(IServiceProvider sp)
{ {
var s = Get(sp); var s = Get(sp);
return new SqlServerHistorianDataUpdater(s.Conn, s.Catalog); return new SqlServerHistorianDataUpdater(s.Conn, s.Bootstrap, s.Catalog);
} }
/// <inheritdoc /> /// <inheritdoc />
@@ -11,7 +11,8 @@ namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer;
/// <c>DELETE TOP(n)</c> batches (under the lock-escalation threshold). Samples are purged per tag so each /// <c>DELETE TOP(n)</c> batches (under the lock-escalation threshold). Samples are purged per tag so each
/// delete is a seek on the clustered key <c>(TagId, SourceTimeUtc)</c>; events seek on <c>CX_Event_Time</c>. /// delete is a seek on the clustered key <c>(TagId, SourceTimeUtc)</c>; events seek on <c>CX_Event_Time</c>.
/// A session <c>sp_getapplock</c> (timeout 0) serialises purgers that share one database; a loser skips. /// A session <c>sp_getapplock</c> (timeout 0) serialises purgers that share one database; a loser skips.
/// To purge another table later, add it to <see cref="PurgeAsync"/>. /// To purge another table later, add it to <see cref="PurgeAsync"/>. On a v1 schema (no Part 11 trail, Gitea #674)
/// the <c>SampleModified</c> purge is skipped and absent from the results.
/// </summary> /// </summary>
internal sealed class SqlServerHistorianRetention : IHistorianRetention internal sealed class SqlServerHistorianRetention : IHistorianRetention
{ {
@@ -42,6 +43,9 @@ internal sealed class SqlServerHistorianRetention : IHistorianRetention
/// <summary>Test seam for <see cref="SqlConnection.ClearPool"/>.</summary> /// <summary>Test seam for <see cref="SqlConnection.ClearPool"/>.</summary>
internal Action<SqlConnection> ClearPool { get; set; } = SqlConnection.ClearPool; internal Action<SqlConnection> ClearPool { get; set; } = SqlConnection.ClearPool;
/// <summary>Test seam: the release statement. Tests point it at a failing statement to exercise the pool-clear fallback.</summary>
internal string ReleaseSql { get; set; } = "EXEC sp_releaseapplock @Resource = @res, @LockOwner = 'Session';";
public async Task<IReadOnlyList<RetentionPurgeResult>> PurgeAsync(CancellationToken ct) public async Task<IReadOnlyList<RetentionPurgeResult>> PurgeAsync(CancellationToken ct)
{ {
await _bootstrap.EnsureAsync(ct).ConfigureAwait(false); await _bootstrap.EnsureAsync(ct).ConfigureAwait(false);
@@ -69,6 +73,8 @@ internal sealed class SqlServerHistorianRetention : IHistorianRetention
return []; return [];
} }
// #674: a v1 schema has no SampleModified table — skip its purge rather than fail the whole pass.
var trail = _bootstrap.SupportsModifiedTrail;
long samples = 0, modified = 0; long samples = 0, modified = 0;
var tagIds = new List<int>(); var tagIds = new List<int>();
await using (var cmd = _conn.CreateCommand(conn, $"SELECT TagId FROM {_conn.Q("Tag")}")) await using (var cmd = _conn.CreateCommand(conn, $"SELECT TagId FROM {_conn.Q("Tag")}"))
@@ -82,21 +88,30 @@ internal sealed class SqlServerHistorianRetention : IHistorianRetention
cutoff, id, ct).ConfigureAwait(false); cutoff, id, ct).ConfigureAwait(false);
// The Part 11 modification trail (#668) ages out on the SAME source-time cutoff as the samples it // The Part 11 modification trail (#668) ages out on the SAME source-time cutoff as the samples it
// describes; a purge never writes to it (no trigger, no OUTPUT INTO). // describes; a purge never writes to it (no trigger, no OUTPUT INTO).
modified += await DeleteLoopAsync(conn, if (trail)
$"DELETE TOP (@n) FROM {_conn.Q("SampleModified")} WHERE TagId = @id AND SourceTimeUtc < @cutoff", {
cutoff, id, ct).ConfigureAwait(false); modified += await DeleteLoopAsync(conn,
$"DELETE TOP (@n) FROM {_conn.Q("SampleModified")} WHERE TagId = @id AND SourceTimeUtc < @cutoff",
cutoff, id, ct).ConfigureAwait(false);
}
} }
var events = await DeleteLoopAsync(conn, var events = await DeleteLoopAsync(conn,
$"DELETE TOP (@n) FROM {_conn.Q("Event")} WHERE EventTimeUtc < @cutoff", $"DELETE TOP (@n) FROM {_conn.Q("Event")} WHERE EventTimeUtc < @cutoff",
cutoff, null, ct).ConfigureAwait(false); cutoff, null, ct).ConfigureAwait(false);
return return trail
[ ?
new RetentionPurgeResult("Sample", samples), [
new RetentionPurgeResult("SampleModified", modified), new RetentionPurgeResult("Sample", samples),
new RetentionPurgeResult("Event", events), new RetentionPurgeResult("SampleModified", modified),
]; new RetentionPurgeResult("Event", events),
]
:
[
new RetentionPurgeResult("Sample", samples),
new RetentionPurgeResult("Event", events),
];
} }
finally finally
{ {
@@ -137,8 +152,7 @@ internal sealed class SqlServerHistorianRetention : IHistorianRetention
{ {
try try
{ {
await using var cmd = _conn.CreateCommand(conn, await using var cmd = _conn.CreateCommand(conn, ReleaseSql);
"EXEC sp_releaseapplock @Resource = @res, @LockOwner = 'Session';");
cmd.Parameters.Add(new SqlParameter("@res", SqlDbType.NVarChar, 255) { Value = LockResource }); cmd.Parameters.Add(new SqlParameter("@res", SqlDbType.NVarChar, 255) { Value = LockResource });
await cmd.ExecuteNonQueryAsync(CancellationToken.None).ConfigureAwait(false); await cmd.ExecuteNonQueryAsync(CancellationToken.None).ConfigureAwait(false);
} }
@@ -15,15 +15,27 @@ namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer;
/// SQL, so a missing schema (or a SQL outage) does not serialise every write, read and alarm event behind one /// SQL, so a missing schema (or a SQL outage) does not serialise every write, read and alarm event behind one
/// semaphore and one round-trip each. /// semaphore and one round-trip each.
/// With <c>AutoCreateSchema=false</c> it only verifies <c>MAX(Version) &gt;= </c><see cref="RequiredVersion"/>. /// With <c>AutoCreateSchema=false</c> it only verifies <c>MAX(Version) &gt;= </c><see cref="RequiredVersion"/>.
/// <para><b>Schema v1 is accepted (Gitea #674).</b> A database at version 1 (no <c>SampleModified</c>) still serves
/// live writes, raw / processed / at-time reads, alarm events and retention; only the Part 11 modification trail is
/// unavailable — <see cref="SupportsModifiedTrail"/> is false, modified reads and HistoryUpdate report
/// <c>BadHistoryOperationUnsupported</c> without touching SQL, and one Warning is logged per instance (= per boot: the
/// provider shares one bootstrapper). Before #674 a v1 database refused EVERY historian call, continuous-historization
/// drains included, until script 002 was applied by hand.</para>
/// </summary> /// </summary>
internal sealed partial class SqlServerSchemaBootstrapper internal sealed partial class SqlServerSchemaBootstrapper
{ {
/// <summary> /// <summary>
/// The schema version this build requires: 2 = <c>002_sample_modified.sql</c> (Part 11 modified-value /// The schema version this build requires to serve at all: 1 = <c>001_initial.sql</c>. Gitea #674 lowered it from
/// history, Gitea #668). An existing v1 schema is upgraded in place on the next bootstrap when /// 2 — a missing modification trail disables only the features that need it (<see cref="ModifiedTrailVersion"/>).
/// <c>AutoCreateSchema</c> is on; otherwise the script must be applied by hand.
/// </summary> /// </summary>
public const int RequiredVersion = 2; public const int RequiredVersion = 1;
/// <summary>
/// The schema version that carries the Part 11 modification trail: 2 = <c>002_sample_modified.sql</c> (Gitea #668).
/// An auto-creating bootstrap always upgrades to it; with <c>AutoCreateSchema=false</c> the script must be applied by
/// hand (and the node restarted — the verdict is taken once per instance).
/// </summary>
public const int ModifiedTrailVersion = 2;
private const string LockResource = "OtOpcUaHistorianSchema"; private const string LockResource = "OtOpcUaHistorianSchema";
private const int LockTimeoutMs = 30_000; private const int LockTimeoutMs = 30_000;
@@ -34,6 +46,13 @@ internal sealed partial class SqlServerSchemaBootstrapper
private readonly SemaphoreSlim _gate = new(1, 1); private readonly SemaphoreSlim _gate = new(1, 1);
private readonly BootstrapFailureCache _failure; // guarded by _gate private readonly BootstrapFailureCache _failure; // guarded by _gate
private volatile bool _ready; private volatile bool _ready;
private volatile bool _supportsModifiedTrail;
/// <summary>
/// Whether the schema carries the Part 11 modification trail (version &gt;= <see cref="ModifiedTrailVersion"/>).
/// Meaningful only after <see cref="EnsureAsync"/> has succeeded (false before). Taken once per instance.
/// </summary>
public bool SupportsModifiedTrail => _supportsModifiedTrail;
/// <summary>Creates the bootstrapper.</summary> /// <summary>Creates the bootstrapper.</summary>
/// <param name="conn">Connection helper.</param> /// <param name="conn">Connection helper.</param>
@@ -99,6 +118,13 @@ internal sealed partial class SqlServerSchemaBootstrapper
throw new HistorianSchemaMissingException( throw new HistorianSchemaMissingException(
$"Historian schema [{_conn.Schema}] is missing version {RequiredVersion} (found {v}) and AutoCreateSchema is false. " + $"Historian schema [{_conn.Schema}] is missing version {RequiredVersion} (found {v}) and AutoCreateSchema is false. " +
"Apply the docs/historian/sqlserver/NNN_*.sql scripts above version " + v + " in order, or enable AutoCreateSchema."); "Apply the docs/historian/sqlserver/NNN_*.sql scripts above version " + v + " in order, or enable AutoCreateSchema.");
_supportsModifiedTrail = v >= ModifiedTrailVersion;
if (!_supportsModifiedTrail)
_log.LogWarning("Historian schema [{Schema}] is at version {Version}, below {TrailVersion}, and AutoCreateSchema is false: " +
"the Part 11 modification trail is unavailable, so modified-value history reads and HistoryUpdate report " +
"BadHistoryOperationUnsupported and retention skips SampleModified. Live historization, reads, events and retention " +
"are unaffected. Apply docs/historian/sqlserver/002_sample_modified.sql and restart the node to enable them.",
_conn.Schema, v, ModifiedTrailVersion);
} }
private async Task ApplyAsync(SqlConnection conn, CancellationToken ct) private async Task ApplyAsync(SqlConnection conn, CancellationToken ct)
@@ -0,0 +1,28 @@
using ZB.MOM.WW.OtOpcUa.Cluster;
namespace ZB.MOM.WW.OtOpcUa.Host.Configuration;
/// <summary>
/// Names the per-role appsettings overlay (<c>appsettings.{suffix}.json</c>) for a node.
/// </summary>
/// <remarks>
/// The suffix is built from the node's PLAIN roles only. <c>cluster-{ClusterId}</c> roles are mesh-membership
/// tags (and carry a verbatim, operator-chosen ClusterId), not configuration profiles — folding one into the
/// filename produced names like <c>appsettings.admin-cluster-MAIN-driver.json</c> that never exist, so on every
/// node with a cluster role the shipped overlays silently never loaded.
/// </remarks>
internal static class RoleOverlay
{
/// <summary>
/// The overlay filename suffix: the plain (non-cluster) roles, ordinal-sorted and joined with <c>-</c>, or
/// <see langword="null"/> when the node has none.
/// </summary>
public static string? Suffix(IReadOnlyList<string> gossipedRoles)
{
var plain = gossipedRoles
.Where(r => !string.IsNullOrWhiteSpace(r) && !RoleParser.IsClusterRole(r))
.OrderBy(r => r, StringComparer.Ordinal)
.ToList();
return plain.Count == 0 ? null : string.Join('-', plain);
}
}
@@ -252,13 +252,15 @@ public sealed class OtOpcUaServerHostedService : IHostedService, IAsyncDisposabl
if (_primaryRoleSource is { } primaryRoleSource) if (_primaryRoleSource is { } primaryRoleSource)
{ {
_server.SetNativeAlarmAckPrimaryGate(() => primaryRoleSource.ShouldActAsPrimary); _server.SetNativeAlarmAckPrimaryGate(() => primaryRoleSource.ShouldActAsPrimary);
// Gitea #674: HistoryUpdate is single-writer — only the Primary rewrites history (BadInvalidState elsewhere).
_server.SetHistoryUpdatePrimaryGate(() => primaryRoleSource.ShouldActAsPrimary);
} }
else else
{ {
// Every driver node registers one (AddOtOpcUaRuntime), so this is a wiring regression, not a mode: // Every driver node registers one (AddOtOpcUaRuntime), so this is a wiring regression, not a mode:
// say so rather than let a Secondary silently go back to accepting acks it will revert. // say so rather than let a Secondary silently go back to accepting acks it will revert.
_logger.LogWarning( _logger.LogWarning(
"OtOpcUaServerHostedService: no IDriverPrimaryRoleSource registered — native-alarm acknowledges are accepted on every node regardless of redundancy role"); "OtOpcUaServerHostedService: no IDriverPrimaryRoleSource registered — native-alarm acknowledges and HistoryUpdate are accepted on every node regardless of redundancy role");
} }
// Wire the reverse-path inbound operator-write gateway: a client write to a writable equipment-tag // Wire the reverse-path inbound operator-write gateway: a client write to a writable equipment-tag
@@ -317,6 +319,11 @@ public sealed class OtOpcUaServerHostedService : IHostedService, IAsyncDisposabl
_server.NodeManager.MaxConcurrentHistoryReadBatches = _serverHistorianOptions.MaxConcurrentHistoryReadBatches; _server.NodeManager.MaxConcurrentHistoryReadBatches = _serverHistorianOptions.MaxConcurrentHistoryReadBatches;
_server.NodeManager.HistoryReadDeadline = _serverHistorianOptions.HistoryReadDeadline; _server.NodeManager.HistoryReadDeadline = _serverHistorianOptions.HistoryReadDeadline;
// Gitea #674: HistoryUpdate refuses a SourceTimestamp beyond now + HistoryUpdateMaxFutureSkew (BadOutOfRange),
// judged against the host clock.
_server.NodeManager.HistoryUpdateMaxFutureSkew = _serverHistorianOptions.HistoryUpdateMaxFutureSkew;
_server.NodeManager.Clock = _time;
// ServiceLevel publisher needs IServerInternal — only available after Start. // ServiceLevel publisher needs IServerInternal — only available after Start.
if (_server.CurrentInstance is { } serverInternal) if (_server.CurrentInstance is { } serverInternal)
{ {
@@ -382,6 +389,7 @@ public sealed class OtOpcUaServerHostedService : IHostedService, IAsyncDisposabl
// Clear the native-alarm-ack router so a late ack doesn't Tell a stopping DriverHostActor. // Clear the native-alarm-ack router so a late ack doesn't Tell a stopping DriverHostActor.
_server?.SetNativeAlarmAckRouter(null); _server?.SetNativeAlarmAckRouter(null);
_server?.SetNativeAlarmAckPrimaryGate(null); _server?.SetNativeAlarmAckPrimaryGate(null);
_server?.SetHistoryUpdatePrimaryGate(null);
// Restore the Null historian so a late HistoryRead doesn't hit a disposed read client. // Restore the Null historian so a late HistoryRead doesn't hit a disposed read client.
_server?.SetHistorianDataSource(null); _server?.SetHistorianDataSource(null);
_server?.SetHistorianDataUpdater(null); _server?.SetHistorianDataUpdater(null);
+3 -2
View File
@@ -90,8 +90,9 @@ builder.WebHost.UseStaticWebAssets();
builder.Host.UseWindowsService(options => options.ServiceName = "OtOpcUaHost"); builder.Host.UseWindowsService(options => options.ServiceName = "OtOpcUaHost");
// Per-role appsettings overlay: appsettings.{role}.json (single role) or appsettings.admin-driver.json // Per-role appsettings overlay: appsettings.{role}.json (single role) or appsettings.admin-driver.json
// (both). Optional — base appsettings.json carries enough to boot if these don't exist. // (both). Optional — base appsettings.json carries enough to boot if these don't exist. Built from the
var roleSuffix = roles.Count == 0 ? null : string.Join('-', roles.OrderBy(r => r, StringComparer.Ordinal)); // PLAIN roles only (#678.6): a cluster-{ClusterId} role in the name pointed at a file that never exists.
var roleSuffix = RoleOverlay.Suffix(roles);
if (roleSuffix is not null) if (roleSuffix is not null)
{ {
builder.Configuration.AddJsonFile($"appsettings.{roleSuffix}.json", optional: true, reloadOnChange: true); builder.Configuration.AddJsonFile($"appsettings.{roleSuffix}.json", optional: true, reloadOnChange: true);
@@ -3,7 +3,7 @@
"MinimumLevel": { "MinimumLevel": {
"Default": "Information", "Default": "Information",
"Override": { "Override": {
"Opc.Ua": "Debug", "Opc.Ua": "Information",
"Akka": "Information" "Akka": "Information"
} }
} }
@@ -46,6 +46,11 @@ namespace ZB.MOM.WW.OtOpcUa.OpcUaServer;
/// <see cref="HistoryCursorKind.Modified"/> only: the normalised window start, so a resumed page keeps the original /// <see cref="HistoryCursorKind.Modified"/> only: the normalised window start, so a resumed page keeps the original
/// window and direction (the modified read resumes by key, not by moving the window start). /// window and direction (the modified read resumes by key, not by moving the window start).
/// </param> /// </param>
/// <param name="Reverse">
/// <see cref="HistoryCursorKind.Events"/> only (Gitea #675): the read is newest-first over
/// <c>(EndUtc, NextStartUtc]</c>. A resumed events read takes its direction from here, never from the resume
/// request's details. Raw keeps deriving it from the window (<c>NextStartUtc &gt; EndUtc</c>).
/// </param>
internal sealed record HistoryContinuationState( internal sealed record HistoryContinuationState(
string Tagname, string Tagname,
DateTime NextStartUtc, DateTime NextStartUtc,
@@ -54,7 +59,8 @@ internal sealed record HistoryContinuationState(
uint NumValuesPerNode, uint NumValuesPerNode,
HistoryCursorKind Kind = HistoryCursorKind.Raw, HistoryCursorKind Kind = HistoryCursorKind.Raw,
long LastModifiedId = 0, long LastModifiedId = 0,
DateTime WindowStartUtc = default); DateTime WindowStartUtc = default,
bool Reverse = false);
/// <summary>The HistoryRead kind that issued a <see cref="HistoryContinuationState"/>.</summary> /// <summary>The HistoryRead kind that issued a <see cref="HistoryContinuationState"/>.</summary>
internal enum HistoryCursorKind internal enum HistoryCursorKind
@@ -102,7 +102,7 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
/// <see cref="_alarmConditions"/> / <see cref="_notifierFolders"/>.</summary> /// <see cref="_alarmConditions"/> / <see cref="_notifierFolders"/>.</summary>
private readonly Dictionary<RealmKey, AlarmNotifierWiring> _alarmNotifierWiring = new(); private readonly Dictionary<RealmKey, AlarmNotifierWiring> _alarmNotifierWiring = new();
/// <summary>Phase C: event-notifier folder NodeId-identifier → the event-history source /// <summary>Phase C: event-notifier folder NodeId-identifier → the event-history source
/// name passed to <see cref="IHistorianDataSource.ReadEventsAsync"/>. The equipment-folder NodeId /// name passed to <see cref="IHistorianDataSource.ReadEventsAsync(string?, DateTime, DateTime, int, CancellationToken)"/>. The equipment-folder NodeId
/// identifier IS the equipment id, which IS the sourceName, so key and value are the same string; /// identifier IS the equipment id, which IS the sourceName, so key and value are the same string;
/// the map's presence (not its value) is what makes a folder an event-history source. Populated by /// the map's presence (not its value) is what makes a folder an event-history source. Populated by
/// <see cref="EnsureFolderIsEventNotifier"/> only when a real historian is wired at promotion time, /// <see cref="EnsureFolderIsEventNotifier"/> only when a real historian is wired at promotion time,
@@ -247,6 +247,48 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
/// </summary> /// </summary>
public Func<bool>? NativeAlarmAckPrimaryGate { get; set; } public Func<bool>? NativeAlarmAckPrimaryGate { get; set; }
/// <summary>
/// Gitea #674 — whether this node may serve an OPC UA HistoryUpdate right now: the redundancy data-plane
/// verdict (<c>PrimaryGatePolicy.ShouldServiceAsPrimary</c>, as published to <c>IDriverPrimaryRoleSource</c>) —
/// the same source as <see cref="NativeAlarmAckPrimaryGate"/>.
/// <para>
/// <c>false</c> ⇒ every node of the request that passed authorization is refused <c>BadInvalidState</c>
/// and nothing reaches the updater. Rewriting history is a single-writer operation: the SQL backends'
/// prior-value capture (and QuestDB's per-process per-tag lock) assume one node serialises a tag's updates,
/// so both pair nodes accepting updates concurrently could record a wrong prior value in the modification
/// trail. HistoryRead is NOT gated — reads are served from any node.
/// </para>
/// <para>
/// Evaluated AFTER authorization (a caller who may not update learns nothing about the node's role). It must
/// be a cheap, non-blocking read (the host passes a volatile-backed property). Null (the default) allows — the
/// single-node posture, the same default <c>UnmanagedDriverPrimaryRoleSource</c> answers with.
/// </para>
/// </summary>
public Func<bool>? HistoryUpdatePrimaryGate { get; set; }
/// <summary>
/// Gitea #674 — how far beyond <see cref="Clock"/>'s now a HistoryUpdate (Insert / Replace / Update) item's
/// <c>SourceTimestamp</c> may lie. An item past <c>now + skew</c> is refused <c>BadOutOfRange</c> before it
/// reaches any backend: a far-future row (a 2062 typo) is never a legitimate history edit, and on a backend whose
/// retention is judged against the newest timestamp it could age out every older partition. Mirrors
/// <c>ServerHistorianOptions.HistoryUpdateMaxFutureSkew</c>; the Host sets it at <c>StartAsync</c>. Non-positive
/// disables the bound. Default 1 day. Live recorder writes are not bounded by this.
/// </summary>
public TimeSpan HistoryUpdateMaxFutureSkew { get; set; } = TimeSpan.FromDays(1);
/// <summary>
/// The clock the HistoryUpdate future-skew bound (<see cref="HistoryUpdateMaxFutureSkew"/>) reads. The Host
/// assigns its <see cref="TimeProvider"/> at <c>StartAsync</c>; tests inject a fixed clock. Defaults to
/// <see cref="TimeProvider.System"/>; assigning null restores it.
/// </summary>
public TimeProvider Clock
{
get => _clock;
set => _clock = value ?? TimeProvider.System;
}
private volatile TimeProvider _clock = TimeProvider.System;
private volatile IOpcUaNodeWriteGateway _nodeWriteGateway = NullOpcUaNodeWriteGateway.Instance; private volatile IOpcUaNodeWriteGateway _nodeWriteGateway = NullOpcUaNodeWriteGateway.Instance;
/// <summary> /// <summary>
@@ -5161,6 +5203,7 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
var endUtc = details.EndTime; var endUtc = details.EndTime;
var numValuesPerNode = details.NumValuesPerNode; var numValuesPerNode = details.NumValuesPerNode;
var boundarySkip = 0; var boundarySkip = 0;
bool reverse;
if (resumed) if (resumed)
{ {
@@ -5180,36 +5223,59 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
endUtc = state.EndUtc; endUtc = state.EndUtc;
boundarySkip = state.BoundarySkipCount; boundarySkip = state.BoundarySkipCount;
numValuesPerNode = state.NumValuesPerNode; numValuesPerNode = state.NumValuesPerNode;
// The direction is the CURSOR's (Gitea #675), never re-derived from the resume request's details: a
// reverse read resumes reverse, a forward one forward.
reverse = state.Reverse;
} }
else
{
// Part 11: with one bound unset, an unset EndTime (MinValue) with a StartTime means "page FORWARD from
// StartTime" — normalise it to an open upper bound, or the cursor would store MinValue and the read would
// look reversed. An unset StartTime with an EndTime and a count cap means "page BACKWARD from EndTime"
// (#675, mirroring raw): the reverse window (MinValue, EndTime]. Without a count cap an end-only read
// keeps its historical forward [MinValue, EndTime) meaning.
if (endUtc == DateTime.MinValue && startUtc != DateTime.MinValue)
endUtc = new DateTime(DateTime.MaxValue.Ticks, DateTimeKind.Utc); // Utc kind: a proto Timestamp conversion rejects Unspecified
else if (numValuesPerNode > 0 && startUtc == DateTime.MinValue && endUtc != DateTime.MinValue)
(startUtc, endUtc) = (endUtc, new DateTime(0, DateTimeKind.Utc));
// Part 11: an unset EndTime (MinValue) with a StartTime means "page forward from StartTime". Normalise it // Part 11: StartTime later than EndTime is a reverse read — events newest-first over (end, start].
// to an open upper bound, or the cursor would store MinValue and a backend would swap the reversed reverse = startUtc > endUtc;
// resume window. A resumed read is always forward (the cursor is only ever stored for a forward read), }
// so its direction comes from the cursor, not from the resume request's details.
if (!resumed && endUtc == DateTime.MinValue && startUtc != DateTime.MinValue)
endUtc = new DateTime(DateTime.MaxValue.Ticks, DateTimeKind.Utc); // Utc kind: a proto Timestamp conversion rejects Unspecified
// Reverse event reads (start > end) are not paged: the cursor below is forward-only.
var forward = resumed || startUtc <= endUtc;
// NOT under the node-manager Lock — awaiting the async source is safe here. // NOT under the node-manager Lock — awaiting the async source is safe here.
// NumValuesPerNode==0 means "no limit — return ALL values" per OPC UA Part 4/11, so it is translated // NumValuesPerNode==0 means "no limit — return ALL values" per OPC UA Part 4/11, so it is translated
// to UNBOUNDED (EventMaxEvents) rather than the backend's <=0 default-cap sentinel. // to UNBOUNDED (EventMaxEvents) rather than the backend's <=0 default-cap sentinel. A resumed page re-reads its
// boundary timestamp inclusively and then drops the `boundarySkip` ties already emitted, so it asks for that
// many more rows than the cap — otherwise every resumed page would come back short by the skip.
var fetch = EventMaxEvents(numValuesPerNode);
if (numValuesPerNode > 0 && boundarySkip > 0)
fetch = (int)Math.Min(int.MaxValue - 1L, (long)fetch + boundarySkip);
var sourceResult = await _historianDataSource.ReadEventsAsync( var sourceResult = await _historianDataSource.ReadEventsAsync(
sourceName, startUtc, endUtc, EventMaxEvents(numValuesPerNode), ct).ConfigureAwait(false); sourceName, startUtc, endUtc, fetch, reverse, ct).ConfigureAwait(false);
// Truncation is the SOURCE's word (non-null continuation point) — never inferred from the count, so a // Truncation is the SOURCE's word (non-null continuation point) — never inferred from the count, so a
// page that happens to be exactly cap-sized but complete never emits a point. // page that happens to be exactly cap-sized but complete never emits a point.
var truncated = sourceResult.ContinuationPoint is not null && numValuesPerNode > 0 && forward; var truncated = sourceResult.ContinuationPoint is not null && numValuesPerNode > 0;
IReadOnlyList<HistoricalEvent> events = sourceResult.Events; IReadOnlyList<HistoricalEvent> events = sourceResult.Events;
if (resumed && boundarySkip > 0) if (resumed && boundarySkip > 0)
{ {
// Drop the events at the boundary timestamp already emitted on the prior page. // Drop the events at the boundary timestamp already emitted on the prior page. The resumed read is in
// the cursor's direction, so in both directions those ties lead the page.
var i = 0; var i = 0;
while (i < events.Count && i < boundarySkip && events[i].EventTimeUtc == startUtc) i++; while (i < events.Count && i < boundarySkip && events[i].EventTimeUtc == startUtc) i++;
if (i > 0) events = events.Skip(i).ToList(); if (i > 0) events = events.Skip(i).ToList();
} }
// The skip over-fetch can leave more than the cap (fewer boundary ties came back than were skipped — pruned
// between pages): keep the cap, and the dropped tail means the read continues.
if (numValuesPerNode > 0 && events.Count > numValuesPerNode)
{
events = events.Take((int)Math.Min(int.MaxValue, numValuesPerNode)).ToList();
truncated = true;
}
// The cursor to store, if any. Saved only AFTER the page is projected, so a projection throw can // The cursor to store, if any. Saved only AFTER the page is projected, so a projection throw can
// never orphan a stored cursor. // never orphan a stored cursor.
HistoryContinuationState? pendingCursor = null; HistoryContinuationState? pendingCursor = null;
@@ -5222,10 +5288,14 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
// with an explicit bounded cap and page WITHIN it (same scheme as raw). // with an explicit bounded cap and page WITHIN it (same scheme as raw).
var resolution = Math.Max(1L, _historianDataSource.TimestampResolutionTicks); var resolution = Math.Max(1L, _historianDataSource.TimestampResolutionTicks);
var overfetchCap = (int)Math.Min(int.MaxValue - 1L, (long)Math.Max(1, MaxTieClusterOverfetch) + 1); var overfetchCap = (int)Math.Min(int.MaxValue - 1L, (long)Math.Max(1, MaxTieClusterOverfetch) + 1);
var cluster = (await _historianDataSource.ReadEventsAsync( IReadOnlyList<HistoricalEvent> cluster = (await _historianDataSource.ReadEventsAsync(
sourceName, startUtc, startUtc.AddTicks(resolution), overfetchCap, ct) sourceName, startUtc, startUtc.AddTicks(resolution), overfetchCap, reverse: false, ct)
.ConfigureAwait(false)).Events; .ConfigureAwait(false)).Events;
// The over-fetch reads the cluster FORWARD (ascending EventId), but a reverse read emitted its ties — and
// counted boundarySkip — in descending order. Slice in the read's own direction, as raw does.
if (reverse) cluster = cluster.Reverse().ToList();
if (cluster.Count > MaxTieClusterOverfetch) if (cluster.Count > MaxTieClusterOverfetch)
{ {
_logger.LogError( _logger.LogError(
@@ -5240,7 +5310,7 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
HistoryPaging.SliceTieCluster( HistoryPaging.SliceTieCluster(
cluster.Count, boundarySkip, numValuesPerNode, startUtc, endUtc, cluster.Count, boundarySkip, numValuesPerNode, startUtc, endUtc,
out var sliceStart, out var sliceCount, out var nextStartUtc, out var nextSkip, out var sliceStart, out var sliceCount, out var nextStartUtc, out var nextSkip,
reverse: false, resolutionTicks: resolution); reverse: reverse, resolutionTicks: resolution);
var slice = new List<HistoricalEvent>(sliceCount); var slice = new List<HistoricalEvent>(sliceCount);
for (var i = sliceStart; i < sliceStart + sliceCount; i++) slice.Add(cluster[i]); for (var i = sliceStart; i < sliceStart + sliceCount; i++) slice.Add(cluster[i]);
@@ -5249,17 +5319,20 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
if (nextStartUtc is { } resumeAt) if (nextStartUtc is { } resumeAt)
{ {
pendingCursor = new HistoryContinuationState( pendingCursor = new HistoryContinuationState(
sourceName ?? string.Empty, resumeAt, endUtc, nextSkip, numValuesPerNode, HistoryCursorKind.Events); sourceName ?? string.Empty, resumeAt, endUtc, nextSkip, numValuesPerNode, HistoryCursorKind.Events,
Reverse: reverse);
} }
} }
else if (truncated && events.Count > 0) else if (truncated && events.Count > 0)
{ {
// Cursor from the last event: resume at its time INCLUSIVE, skipping the trailing ties emitted. // Cursor from the last event: resume at its time INCLUSIVE, skipping the trailing ties emitted. In a
// reverse read the last event is the OLDEST on the page, and the resumed (end, T] read starts there.
var boundary = events[^1].EventTimeUtc; var boundary = events[^1].EventTimeUtc;
var skip = 0; var skip = 0;
for (var i = events.Count - 1; i >= 0 && events[i].EventTimeUtc == boundary; i--) skip++; for (var i = events.Count - 1; i >= 0 && events[i].EventTimeUtc == boundary; i--) skip++;
pendingCursor = new HistoryContinuationState( pendingCursor = new HistoryContinuationState(
sourceName ?? string.Empty, boundary, endUtc, skip, numValuesPerNode, HistoryCursorKind.Events); sourceName ?? string.Empty, boundary, endUtc, skip, numValuesPerNode, HistoryCursorKind.Events,
Reverse: reverse);
} }
var historyEvent = ProjectEvents(events, selectClauses); var historyEvent = ProjectEvents(events, selectClauses);
@@ -5280,6 +5353,15 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
errors[handle.Index] = StatusCodes.BadTimeout; errors[handle.Index] = StatusCodes.BadTimeout;
results[handle.Index] = new SdkHistoryReadResult { StatusCode = StatusCodes.BadTimeout }; results[handle.Index] = new SdkHistoryReadResult { StatusCode = StatusCodes.BadTimeout };
} }
catch (NotSupportedException ex)
{
// The backend cannot serve this read's shape — today a reverse (newest-first) read on Gateway / Null
// (#675). A client request the backend refuses, not a backend fault: Warning, not Error.
_logger.LogWarning("OtOpcUaNodeManager: HistoryReadEvents for node {NodeId} is not supported by the historian backend: {Reason}",
handle.NodeId, ex.Message);
errors[handle.Index] = StatusCodes.BadHistoryOperationUnsupported;
results[handle.Index] = new SdkHistoryReadResult { StatusCode = StatusCodes.BadHistoryOperationUnsupported };
}
catch (Exception ex) catch (Exception ex)
{ {
// One node's backend failure must not throw out of the batch — surface Bad for THIS node // One node's backend failure must not throw out of the batch — surface Bad for THIS node
@@ -5373,6 +5455,8 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
List<NodeHandle> nodesToProcess, List<NodeHandle> nodesToProcess,
IDictionary<NodeId, NodeState> cache) IDictionary<NodeId, NodeState> cache)
{ {
// #674: one "now" per request, so every item of the batch is judged against the same bound.
var latest = LatestAcceptedSourceTime();
ServeHistoryUpdate(context, nodesToProcess, results, errors, handle => ServeHistoryUpdate(context, nodesToProcess, results, errors, handle =>
{ {
var details = nodesToUpdate[handle.Index]; var details = nodesToUpdate[handle.Index];
@@ -5386,7 +5470,7 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
} }
var items = (IList<DataValue>?)details.UpdateValues ?? Array.Empty<DataValue>(); var items = (IList<DataValue>?)details.UpdateValues ?? Array.Empty<DataValue>();
return (updater, tagname, who, ct) => UpdateDataAsync(updater, tagname, mode, items, who, ct); return (updater, tagname, who, ct) => UpdateDataAsync(updater, tagname, mode, items, latest, who, ct);
}); });
} }
@@ -5448,22 +5532,40 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
}); });
} }
/// <summary>
/// #674: the latest <c>SourceTimestamp</c> a HistoryUpdate item may carry — <see cref="Clock"/>'s now plus
/// <see cref="HistoryUpdateMaxFutureSkew"/> — or <c>null</c> when the bound is disabled (non-positive skew).
/// Saturates at <see cref="DateTime.MaxValue"/> rather than overflowing for an absurd configured skew.
/// </summary>
/// <returns>The inclusive upper bound, or null for unbounded.</returns>
private DateTime? LatestAcceptedSourceTime()
{
var skew = HistoryUpdateMaxFutureSkew;
if (skew <= TimeSpan.Zero) return null;
var now = _clock.GetUtcNow().UtcDateTime;
return skew.Ticks > DateTime.MaxValue.Ticks - now.Ticks
? new DateTime(DateTime.MaxValue.Ticks, DateTimeKind.Utc)
: now + skew;
}
/// <summary> /// <summary>
/// Applies one node's UpdateDataDetails: items the server can refuse on its own /// Applies one node's UpdateDataDetails: items the server can refuse on its own
/// (<c>SourceTimestamp == DateTime.MinValue</c> ⇒ <c>BadInvalidTimestamp</c>; a value /// (<c>SourceTimestamp == DateTime.MinValue</c> ⇒ <c>BadInvalidTimestamp</c>; a <c>SourceTimestamp</c> later than
/// <see cref="HistorianValue.TryFromObject"/> cannot type ⇒ <c>BadTypeMismatch</c>) are answered locally, /// <paramref name="latest"/> ⇒ <c>BadOutOfRange</c>, #674; a value <see cref="HistorianValue.TryFromObject"/>
/// the rest go to the updater in ONE call, and the per-item codes are stitched back in request order. /// cannot type ⇒ <c>BadTypeMismatch</c>) are answered locally, the rest go to the updater in ONE call, and the
/// per-item codes are stitched back in request order.
/// </summary> /// </summary>
/// <param name="updater">The wired updater.</param> /// <param name="updater">The wired updater.</param>
/// <param name="tagname">The node's historian tagname.</param> /// <param name="tagname">The node's historian tagname.</param>
/// <param name="mode">The update mode.</param> /// <param name="mode">The update mode.</param>
/// <param name="items">The request's values, in order.</param> /// <param name="items">The request's values, in order.</param>
/// <param name="latest">The latest accepted source time (<see cref="LatestAcceptedSourceTime"/>); null = unbounded.</param>
/// <param name="who">The caller, recorded in the modification trail.</param> /// <param name="who">The caller, recorded in the modification trail.</param>
/// <param name="ct">The request deadline.</param> /// <param name="ct">The request deadline.</param>
/// <returns>The node's result, carrying one operation result per item.</returns> /// <returns>The node's result, carrying one operation result per item.</returns>
private static async Task<HistoryUpdateResult> UpdateDataAsync( private static async Task<HistoryUpdateResult> UpdateDataAsync(
IHistorianDataUpdater updater, string tagname, HistoryUpdateMode mode, IList<DataValue> items, IHistorianDataUpdater updater, string tagname, HistoryUpdateMode mode, IList<DataValue> items,
HistoryUpdateContext who, CancellationToken ct) DateTime? latest, HistoryUpdateContext who, CancellationToken ct)
{ {
var codes = new uint[items.Count]; var codes = new uint[items.Count];
var toSend = new List<HistorizationValue>(items.Count); var toSend = new List<HistorizationValue>(items.Count);
@@ -5479,6 +5581,11 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
{ {
codes[i] = StatusCodes.BadInvalidTimestamp; codes[i] = StatusCodes.BadInvalidTimestamp;
} }
else if (latest is { } bound && item.SourceTimestamp > bound)
{
// #674: a far-future history edit is refused before any backend sees it (wire timestamps are UTC).
codes[i] = StatusCodes.BadOutOfRange;
}
else if (!HistorianValue.TryFromObject(item.Value, out var value)) else if (!HistorianValue.TryFromObject(item.Value, out var value))
{ {
codes[i] = StatusCodes.BadTypeMismatch; codes[i] = StatusCodes.BadTypeMismatch;
@@ -5529,6 +5636,21 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
nodesToProcess = AuthorizeHistoryUpdate(context, nodesToProcess, results, errors); nodesToProcess = AuthorizeHistoryUpdate(context, nodesToProcess, results, errors);
if (nodesToProcess.Count == 0) return; if (nodesToProcess.Count == 0) return;
// #674 — PRIMARY GATE, after authorization (a caller who may not update learns nothing about the node's role).
// Off-Primary every authorized node is refused BadInvalidState and nothing reaches the updater.
if (HistoryUpdatePrimaryGate is { } primary && !primary())
{
var refusal = new ServiceResult(StatusCodes.BadInvalidState,
new LocalizedText("This server is not the redundancy Primary; update history on the Primary (the server with the higher ServiceLevel)."));
foreach (var handle in nodesToProcess)
{
errors[handle.Index] = refusal;
results[handle.Index] = new HistoryUpdateResult { StatusCode = StatusCodes.BadInvalidState };
}
return;
}
// Who is rewriting history — recorded on every modification-trail row (Part 11 ModificationInfo.UserName, #668). // Who is rewriting history — recorded on every modification-trail row (Part 11 ModificationInfo.UserName, #668).
var who = HistoryUpdateCallerOf(context); var who = HistoryUpdateCallerOf(context);
@@ -5619,7 +5741,7 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
private static int ClampToInt(uint value) => value > int.MaxValue ? int.MaxValue : (int)value; private static int ClampToInt(uint value) => value > int.MaxValue ? int.MaxValue : (int)value;
/// <summary>Map a HistoryRead-Events <c>NumValuesPerNode</c> request cap onto the /// <summary>Map a HistoryRead-Events <c>NumValuesPerNode</c> request cap onto the
/// <see cref="IHistorianDataSource.ReadEventsAsync"/> <c>maxEvents</c> argument, honouring the OPC UA /// <see cref="IHistorianDataSource.ReadEventsAsync(string?, DateTime, DateTime, int, CancellationToken)"/> <c>maxEvents</c> argument, honouring the OPC UA
/// Part 4/11 semantics that <c>NumValuesPerNode == 0</c> means "no limit — return ALL values". /// Part 4/11 semantics that <c>NumValuesPerNode == 0</c> means "no limit — return ALL values".
/// We translate 0 to UNBOUNDED (<see cref="int.MaxValue"/>) — a very large positive cap — rather than /// We translate 0 to UNBOUNDED (<see cref="int.MaxValue"/>) — a very large positive cap — rather than
/// the backend's <c>maxEvents &lt;= 0</c> "use the default cap" sentinel, so a client asking for the /// the backend's <c>maxEvents &lt;= 0</c> "use the default cap" sentinel, so a client asking for the
@@ -83,6 +83,21 @@ public sealed class OtOpcUaSdkServer : StandardServer
return true; return true;
} }
/// <summary>
/// Gitea #674 — wire the redundancy verdict that decides whether this node serves an OPC UA HistoryUpdate
/// (<see cref="OtOpcUaNodeManager.HistoryUpdatePrimaryGate"/>); on any other node every authorized node of a
/// HistoryUpdate is refused <c>BadInvalidState</c>. Passing <c>null</c> clears it (allow). No-op (returns
/// <c>false</c>) when the node manager has not been created yet (mirrors <see cref="SetNativeAlarmAckPrimaryGate"/>).
/// </summary>
/// <param name="isPrimary">A cheap, non-blocking read of the data-plane Primary verdict; null to allow.</param>
/// <returns><c>true</c> when set on a live node manager; <c>false</c> when no node manager exists yet.</returns>
public bool SetHistoryUpdatePrimaryGate(Func<bool>? isPrimary)
{
if (_otOpcUaNodeManager is null) return false;
_otOpcUaNodeManager.HistoryUpdatePrimaryGate = isPrimary;
return true;
}
/// <summary> /// <summary>
/// Wire the reverse-path gateway for inbound operator writes to writable equipment-tag nodes onto the /// Wire the reverse-path gateway for inbound operator writes to writable equipment-tag nodes onto the
/// created <see cref="OtOpcUaNodeManager"/>. The host calls this after start with an /// created <see cref="OtOpcUaNodeManager"/>. The host calls this after start with an
@@ -122,6 +122,16 @@ public sealed class ServerHistorianOptions
/// </summary> /// </summary>
public int MaxTimeWeightedRawRows { get; init; } = 1_000_000; public int MaxTimeWeightedRawRows { get; init; } = 1_000_000;
/// <summary>
/// Gitea #674: how far into the future an OPC UA HistoryUpdate (Insert / Replace / Update) item's
/// <c>SourceTimestamp</c> may lie, measured from the server clock. An item beyond <c>now + skew</c> is refused
/// <c>BadOutOfRange</c> by the node manager before any backend sees it — a far-future row is never a legitimate
/// history edit, and on a backend that judges retention against the newest timestamp it could age out every
/// older partition. Applies to every backend. The live recorder is not bounded. Non-positive disables the bound.
/// Default 1 day.
/// </summary>
public TimeSpan HistoryUpdateMaxFutureSkew { get; init; } = TimeSpan.FromDays(1);
/// <summary>Returns operator-facing misconfiguration warnings for an <c>Enabled</c> historian /// <summary>Returns operator-facing misconfiguration warnings for an <c>Enabled</c> historian
/// (empty when disabled or correctly configured). Pure — the registration logs each entry. /// (empty when disabled or correctly configured). Pure — the registration logs each entry.
/// <para> /// <para>
@@ -108,6 +108,9 @@ public sealed class FakeOpcUaClientService : IOpcUaClientService
/// </summary> /// </summary>
public Dictionary<string, IReadOnlyList<BrowseResult>> BrowseResultsByParent { get; } = new(); public Dictionary<string, IReadOnlyList<BrowseResult>> BrowseResultsByParent { get; } = new();
/// <summary>Gets or sets an exception thrown by HistoryReadRawAsync (simulates a refused read).</summary>
public Exception? HistoryReadException { get; set; }
/// <summary>Gets or sets the history read result returned by HistoryReadRawAsync and HistoryReadAggregateAsync.</summary> /// <summary>Gets or sets the history read result returned by HistoryReadRawAsync and HistoryReadAggregateAsync.</summary>
public IReadOnlyList<DataValue> HistoryReadResult { get; set; } = new List<DataValue> public IReadOnlyList<DataValue> HistoryReadResult { get; set; } = new List<DataValue>
{ {
@@ -323,6 +326,7 @@ public sealed class FakeOpcUaClientService : IOpcUaClientService
NodeId nodeId, DateTime startTime, DateTime endTime, int maxValues = 1000, CancellationToken ct = default) NodeId nodeId, DateTime startTime, DateTime endTime, int maxValues = 1000, CancellationToken ct = default)
{ {
HistoryReadRawCalls.Add((nodeId, startTime, endTime, maxValues)); HistoryReadRawCalls.Add((nodeId, startTime, endTime, maxValues));
if (HistoryReadException != null) throw HistoryReadException;
return Task.FromResult(HistoryReadResult); return Task.FromResult(HistoryReadResult);
} }
@@ -197,4 +197,53 @@ public class HistoryReadCommandTests
using var console = TestConsoleHelper.CreateConsole(); using var console = TestConsoleHelper.CreateConsole();
await Should.ThrowAsync<CliFx.Exceptions.CommandException>(async () => await command.ExecuteAsync(console)); await Should.ThrowAsync<CliFx.Exceptions.CommandException>(async () => await command.ExecuteAsync(console));
} }
/// <summary>#677.5: the new aggregate names parse (and the aliases).</summary>
[Theory]
[InlineData("TimeAverage", AggregateType.TimeAverage)]
[InlineData("timeaverage2", AggregateType.TimeAverage2)]
[InlineData("Total", AggregateType.Total)]
[InlineData("total2", AggregateType.Total2)]
[InlineData("Interpolative", AggregateType.Interpolative)]
[InlineData("interp", AggregateType.Interpolative)]
public async Task Execute_NewAggregates_Parse(string name, AggregateType expected)
{
var fakeService = new FakeOpcUaClientService();
var command = new HistoryReadCommand(new FakeOpcUaClientServiceFactory(fakeService))
{
Url = "opc.tcp://localhost:4840",
NodeId = "ns=2;s=HistNode",
Aggregate = name,
};
using var console = TestConsoleHelper.CreateConsole();
await command.ExecuteAsync(console);
fakeService.HistoryReadAggregateCalls.ShouldHaveSingleItem().Aggregate.ShouldBe(expected);
}
/// <summary>#674.6: a refused read is a one-line CommandException (non-zero exit), not a stack trace.</summary>
[Fact]
public async Task Execute_RefusedRead_ThrowsCommandException_WithOneLineMessage()
{
var fakeService = new FakeOpcUaClientService
{
HistoryReadException = new ServiceResultException(StatusCodes.BadHistoryOperationUnsupported,
"HistoryRead of ns=2;s=HistNode failed: BadHistoryOperationUnsupported (0x80B40000)."),
};
var command = new HistoryReadCommand(new FakeOpcUaClientServiceFactory(fakeService))
{
Url = "opc.tcp://localhost:4840",
NodeId = "ns=2;s=HistNode",
};
using var console = TestConsoleHelper.CreateConsole();
var ex = await Should.ThrowAsync<CliFx.Exceptions.CommandException>(async () => await command.ExecuteAsync(console));
ex.Message.ShouldContain("BadHistoryOperationUnsupported");
ex.Message.ShouldNotContain("\n");
ex.Message.ShouldNotContain(" at ");
ex.ExitCode.ShouldNotBe(0);
fakeService.DisconnectCalled.ShouldBeTrue();
}
} }
@@ -160,4 +160,67 @@ public class EndpointSelectorTests
Should.Throw<ArgumentException>(() => Should.Throw<ArgumentException>(() =>
EndpointSelector.SelectBest(Array.Empty<EndpointDescription>(), "", MessageSecurityMode.None)); EndpointSelector.SelectBest(Array.Empty<EndpointDescription>(), "", MessageSecurityMode.None));
} }
/// <summary>#678.5: a server advertising its internal port must not redirect the session off the mapped port.</summary>
[Fact]
public void SelectBest_RewritesHostAndPort_WhenSchemeMatches()
{
var endpoints = new[] { Ep(MessageSecurityMode.None, SecurityPolicies.None, "opc.tcp://internal-host:4840/OtOpcUa") };
var best = EndpointSelector.SelectBest(endpoints, "opc.tcp://localhost:4841", MessageSecurityMode.None);
var uri = new Uri(best.EndpointUrl);
uri.Host.ShouldBe("localhost");
uri.Port.ShouldBe(4841);
uri.AbsolutePath.ShouldBe("/OtOpcUa");
}
/// <summary>A different scheme (e.g. https endpoint advertised to an opc.tcp caller) is left untouched.</summary>
[Fact]
public void SelectBest_LeavesEndpointUntouched_WhenSchemeDiffers()
{
var endpoints = new[] { Ep(MessageSecurityMode.None, SecurityPolicies.None, "https://internal-host:4843/x") };
var best = EndpointSelector.SelectBest(endpoints, "opc.tcp://localhost:4841", MessageSecurityMode.None);
new Uri(best.EndpointUrl).Host.ShouldBe("internal-host");
new Uri(best.EndpointUrl).Port.ShouldBe(4843);
}
/// <summary>Identical host, port and scheme: nothing to rewrite.</summary>
[Fact]
public void SelectBest_Unchanged_WhenHostPortAndSchemeMatch()
{
var endpoints = new[] { Ep(MessageSecurityMode.None, SecurityPolicies.None, "opc.tcp://server:4840") };
var best = EndpointSelector.SelectBest(endpoints, "opc.tcp://server:4840", MessageSecurityMode.None);
best.EndpointUrl.ShouldBe("opc.tcp://server:4840");
}
/// <summary>A URL typed without a port keeps the advertised port (opc.tcp has no default; Uri.Port is -1).</summary>
[Fact]
public void SelectBest_KeepsAdvertisedPort_WhenRequestedUrlHasNoPort()
{
var endpoints = new[] { Ep(MessageSecurityMode.None, SecurityPolicies.None, "opc.tcp://internal-host:4840/OtOpcUa") };
var best = EndpointSelector.SelectBest(endpoints, "opc.tcp://localhost", MessageSecurityMode.None);
var uri = new Uri(best.EndpointUrl);
uri.Host.ShouldBe("localhost");
uri.Port.ShouldBe(4840);
}
/// <summary>An IPv6 host is rewritten with its brackets intact.</summary>
[Fact]
public void SelectBest_RewritesIpv6Host()
{
var endpoints = new[] { Ep(MessageSecurityMode.None, SecurityPolicies.None, "opc.tcp://internal-host:4840/OtOpcUa") };
var best = EndpointSelector.SelectBest(endpoints, "opc.tcp://[::1]:4841", MessageSecurityMode.None);
var uri = new Uri(best.EndpointUrl);
uri.Host.ShouldBe("[::1]");
uri.Port.ShouldBe(4841);
}
} }
@@ -18,6 +18,11 @@ public class AggregateTypeMapperTests
[InlineData(AggregateType.Start)] [InlineData(AggregateType.Start)]
[InlineData(AggregateType.End)] [InlineData(AggregateType.End)]
[InlineData(AggregateType.StandardDeviation)] [InlineData(AggregateType.StandardDeviation)]
[InlineData(AggregateType.TimeAverage)]
[InlineData(AggregateType.TimeAverage2)]
[InlineData(AggregateType.Total)]
[InlineData(AggregateType.Total2)]
[InlineData(AggregateType.Interpolative)]
public void ToNodeId_ReturnsNonNullForAllValues(AggregateType aggregate) public void ToNodeId_ReturnsNonNullForAllValues(AggregateType aggregate)
{ {
var nodeId = AggregateTypeMapper.ToNodeId(aggregate); var nodeId = AggregateTypeMapper.ToNodeId(aggregate);
@@ -82,4 +87,15 @@ public class AggregateTypeMapperTests
Should.Throw<ArgumentOutOfRangeException>(() => Should.Throw<ArgumentOutOfRangeException>(() =>
AggregateTypeMapper.ToNodeId((AggregateType)99)); AggregateTypeMapper.ToNodeId((AggregateType)99));
} }
/// <summary>#677.5: the time-weighted Part 13 aggregates map to their ObjectIds.</summary>
[Fact]
public void ToNodeId_TimeWeightedAggregates_MapCorrectly()
{
AggregateTypeMapper.ToNodeId(AggregateType.TimeAverage).ShouldBe(ObjectIds.AggregateFunction_TimeAverage);
AggregateTypeMapper.ToNodeId(AggregateType.TimeAverage2).ShouldBe(ObjectIds.AggregateFunction_TimeAverage2);
AggregateTypeMapper.ToNodeId(AggregateType.Total).ShouldBe(ObjectIds.AggregateFunction_Total);
AggregateTypeMapper.ToNodeId(AggregateType.Total2).ShouldBe(ObjectIds.AggregateFunction_Total2);
AggregateTypeMapper.ToNodeId(AggregateType.Interpolative).ShouldBe(ObjectIds.AggregateFunction_Interpolative);
}
} }
@@ -136,6 +136,6 @@ public class ModelConstructionTests
[Fact] [Fact]
public void AggregateType_Enum_HasExpectedValues() public void AggregateType_Enum_HasExpectedValues()
{ {
Enum.GetValues<AggregateType>().Length.ShouldBe(7); Enum.GetValues<AggregateType>().Length.ShouldBe(12);
} }
} }
@@ -141,7 +141,7 @@ public class HistoryViewModelTests
public void AggregateTypes_ContainsNullForRaw() public void AggregateTypes_ContainsNullForRaw()
{ {
_vm.AggregateTypes.ShouldContain((AggregateType?)null); _vm.AggregateTypes.ShouldContain((AggregateType?)null);
_vm.AggregateTypes.Count.ShouldBe(8); // null + 7 enum values _vm.AggregateTypes.Count.ShouldBe(13); // null + 12 enum values
} }
/// <summary>Verifies that the Clear method resets the view model state.</summary> /// <summary>Verifies that the Clear method resets the view model state.</summary>
@@ -45,9 +45,10 @@ public sealed class IHistorianDataSourceContractTests
[InlineData("ReadEventsAsync", typeof(Task<HistoricalEventsResult>))] [InlineData("ReadEventsAsync", typeof(Task<HistoricalEventsResult>))]
public void ReadMethods_ReturnExpectedTaskShape(string methodName, Type expectedReturnType) public void ReadMethods_ReturnExpectedTaskShape(string methodName, Type expectedReturnType)
{ {
var method = typeof(IHistorianDataSource).GetMethod(methodName); // ReadEventsAsync is overloaded (the #675 direction-aware overload) — every overload must keep the shape.
method.ShouldNotBeNull(); var methods = Overloads(methodName);
method!.ReturnType.ShouldBe(expectedReturnType); methods.ShouldNotBeEmpty();
foreach (var method in methods) method.ReturnType.ShouldBe(expectedReturnType);
} }
/// <summary>Verifies that GetHealthSnapshot is synchronous.</summary> /// <summary>Verifies that GetHealthSnapshot is synchronous.</summary>
@@ -146,11 +147,14 @@ public sealed class IHistorianDataSourceContractTests
[InlineData("ReadEventsAsync", "maxEvents", typeof(int))] [InlineData("ReadEventsAsync", "maxEvents", typeof(int))]
public void HistoryRead_MaxParameter_TypePinned(string methodName, string parameterName, Type expectedType) public void HistoryRead_MaxParameter_TypePinned(string methodName, string parameterName, Type expectedType)
{ {
var method = typeof(IHistorianDataSource).GetMethod(methodName); var methods = Overloads(methodName);
method.ShouldNotBeNull(); methods.ShouldNotBeEmpty();
var parameter = method!.GetParameters().FirstOrDefault(p => p.Name == parameterName); foreach (var method in methods)
parameter.ShouldNotBeNull($"Method {methodName} should expose a '{parameterName}' parameter."); {
parameter!.ParameterType.ShouldBe(expectedType); var parameter = method.GetParameters().FirstOrDefault(p => p.Name == parameterName);
parameter.ShouldNotBeNull($"Method {methodName} should expose a '{parameterName}' parameter.");
parameter!.ParameterType.ShouldBe(expectedType);
}
} }
/// <summary> /// <summary>
@@ -163,13 +167,17 @@ public sealed class IHistorianDataSourceContractTests
[Fact] [Fact]
public void ReadEventsAsync_sentinel_and_continuation_contract_types_pinned() public void ReadEventsAsync_sentinel_and_continuation_contract_types_pinned()
{ {
var method = typeof(IHistorianDataSource).GetMethod("ReadEventsAsync"); var methods = Overloads("ReadEventsAsync");
method.ShouldNotBeNull(); methods.Count.ShouldBe(2, "the five-argument read plus the #675 direction-aware overload");
methods.ShouldContain(m => m.GetParameters().Any(p => p.Name == "reverse" && p.ParameterType == typeof(bool)));
var maxEventsParam = method!.GetParameters().FirstOrDefault(p => p.Name == "maxEvents"); foreach (var method in methods)
maxEventsParam.ShouldNotBeNull("maxEvents parameter must exist for the sentinel contract"); {
maxEventsParam!.ParameterType.ShouldBe(typeof(int), var maxEventsParam = method.GetParameters().FirstOrDefault(p => p.Name == "maxEvents");
"maxEvents is int (not uint) so callers can pass <=0 as a 'use backend default cap' sentinel"); maxEventsParam.ShouldNotBeNull("maxEvents parameter must exist for the sentinel contract");
maxEventsParam!.ParameterType.ShouldBe(typeof(int),
"maxEvents is int (not uint) so callers can pass <=0 as a 'use backend default cap' sentinel");
}
// HistoricalEventsResult must carry a nullable ContinuationPoint so implementations // HistoricalEventsResult must carry a nullable ContinuationPoint so implementations
// can signal 'more results exist' (non-null) vs 'all results returned' (null). // can signal 'more results exist' (non-null) vs 'all results returned' (null).
@@ -179,4 +187,7 @@ public sealed class IHistorianDataSourceContractTests
continuationProp!.PropertyType.ShouldBe(typeof(byte[]), continuationProp!.PropertyType.ShouldBe(typeof(byte[]),
"ContinuationPoint is byte[]? — null means no more pages, non-null means caller must page"); "ContinuationPoint is byte[]? — null means no more pages, non-null means caller must page");
} }
private static List<System.Reflection.MethodInfo> Overloads(string name) =>
typeof(IHistorianDataSource).GetMethods().Where(m => m.Name == name).ToList();
} }
@@ -67,6 +67,20 @@ public sealed class NullHistorianDataSourceTests
result.ContinuationPoint.ShouldBeNull(); result.ContinuationPoint.ShouldBeNull();
} }
/// <summary>
/// #675: the direction-aware overload's default — forward is the five-argument read, reverse is refused
/// (the server answers <c>BadHistoryOperationUnsupported</c>).
/// </summary>
[Fact]
public async Task Direction_aware_ReadEventsAsync_forwards_forward_and_refuses_reverse()
{
IHistorianDataSource source = Source;
(await source.ReadEventsAsync(null, DateTime.UtcNow.AddHours(-1), DateTime.UtcNow, 0, reverse: false, Ct))
.Events.ShouldBeEmpty();
await Should.ThrowAsync<NotSupportedException>(() =>
source.ReadEventsAsync(null, DateTime.UtcNow, DateTime.UtcNow.AddHours(-1), 0, reverse: true, Ct));
}
/// <summary>GetHealthSnapshot reports a fully-disabled (zeroed, disconnected, no-node) shape.</summary> /// <summary>GetHealthSnapshot reports a fully-disabled (zeroed, disconnected, no-node) shape.</summary>
[Fact] [Fact]
public void GetHealthSnapshot_is_disabled() public void GetHealthSnapshot_is_disabled()
@@ -0,0 +1,106 @@
using Shouldly;
using Xunit;
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Conformance;
/// <summary>
/// The direction-aware event read (Gitea #675): forward is <c>[lo, hi)</c> ascending by <c>(EventTimeUtc, EventId)</c>;
/// reverse is the Part 11 reverse window <c>(lo, hi]</c> DESCENDING by the same key, so a capped reverse read returns the
/// LATEST events and flags truncation. Paging a reverse read the way the server does — resume at the last returned
/// time inclusive and drop the boundary ties already emitted — must visit every event exactly once, newest-first,
/// through a same-timestamp tie cluster.
/// </summary>
public abstract class EventReadConformance : ConformanceBase
{
private static readonly CancellationToken Ct = CancellationToken.None;
/// <summary>e0@0, e1@1, then the ties t0..t4@2, then e5@3 and e6@4 (all source "s"), plus o0@2 from another source.</summary>
private async Task<IHistorianDataSource> SeedAsync()
{
await B.SeedEventsAsync(
(At(0), "e0", "s"), (At(1), "e1", "s"),
(At(2), "t0", "s"), (At(2), "t1", "s"), (At(2), "t2", "s"), (At(2), "t3", "s"), (At(2), "t4", "s"),
(At(3), "e5", "s"), (At(4), "e6", "s"),
(At(2), "o0", "other"));
return B.CreateDataSource();
}
private static readonly string[] NewestFirst = ["e6", "e5", "t4", "t3", "t2", "t1", "t0", "e1", "e0"];
[Fact]
public async Task Reverse_read_is_newest_first_by_time_then_EventId_descending()
{
using var src = await SeedAsync();
var r = await src.ReadEventsAsync("s", At(-1), At(10), 0, reverse: true, Ct);
r.Events.Select(e => e.EventId).ShouldBe(NewestFirst);
r.ContinuationPoint.ShouldBeNull();
}
[Fact]
public async Task Reverse_window_is_end_exclusive_start_inclusive_whichever_order_the_bounds_come_in()
{
using var src = await SeedAsync();
// (At(1), At(3)]: e1 at the lower bound is excluded, e5 at the upper bound included.
(await src.ReadEventsAsync("s", At(3), At(1), 0, reverse: true, Ct)).Events.Select(e => e.EventId)
.ShouldBe(["e5", "t4", "t3", "t2", "t1", "t0"]);
(await src.ReadEventsAsync("s", At(1), At(3), 0, reverse: true, Ct)).Events.Select(e => e.EventId)
.ShouldBe(["e5", "t4", "t3", "t2", "t1", "t0"]);
// Forward is unchanged: [At(1), At(3)) ascending.
(await src.ReadEventsAsync("s", At(1), At(3), 0, reverse: false, Ct)).Events.Select(e => e.EventId)
.ShouldBe(["e1", "t0", "t1", "t2", "t3", "t4"]);
}
[Fact]
public async Task A_capped_reverse_read_keeps_the_LATEST_events_and_flags_truncation()
{
using var src = await SeedAsync();
var r = await src.ReadEventsAsync("s", At(10), At(-1), 3, reverse: true, Ct);
r.Events.Select(e => e.EventId).ShouldBe(["e6", "e5", "t4"]);
r.ContinuationPoint.ShouldNotBeNull();
// An exact fit is complete.
(await src.ReadEventsAsync("s", At(10), At(-1), 9, reverse: true, Ct)).ContinuationPoint.ShouldBeNull();
}
[Fact]
public async Task Paging_a_reverse_read_through_a_tie_cluster_visits_every_event_once_newest_first()
{
using var src = await SeedAsync();
foreach (var cap in new[] { 1, 2, 3, 4 })
{
var all = new List<string>();
var lo = At(-1);
var from = At(10);
var skip = 0;
for (var page = 0; ; page++)
{
page.ShouldBeLessThan(40, "paging must terminate");
var r = await src.ReadEventsAsync("s", from, lo, cap + skip, reverse: true, Ct);
var events = r.Events.ToList();
var drop = 0;
while (drop < skip && drop < events.Count && events[drop].EventTimeUtc == from) drop++;
events = events.Skip(drop).ToList();
// The server asks for cap + skip rows (it re-reads the boundary ties it then drops), so a page is never
// emptied by the trim while more remains.
if (r.ContinuationPoint is not null) events.Count.ShouldBe(cap);
all.AddRange(events.Select(e => e.EventId));
if (r.ContinuationPoint is null) break;
var boundary = events[^1].EventTimeUtc;
var ties = events.AsEnumerable().Reverse().TakeWhile(e => e.EventTimeUtc == boundary).Count();
skip = boundary == from ? skip + ties : ties;
from = boundary;
}
all.SequenceEqual(NewestFirst).ShouldBeTrue($"cap {cap}: {string.Join(",", all)}");
}
}
[Fact]
public async Task Reverse_read_with_a_null_source_spans_every_source()
{
using var src = await SeedAsync();
var r = await src.ReadEventsAsync(null, At(2.5), At(1.5), 0, reverse: true, Ct);
r.Events.Select(e => e.EventId).ShouldBe(["t4", "t3", "t2", "t1", "t0", "o0"]);
}
}
@@ -41,6 +41,13 @@ public interface IConformanceBackend : IAsyncDisposable
/// <param name="samples">The rows to store.</param> /// <param name="samples">The rows to store.</param>
Task SeedAsync(string tag, params (DateTime T, HistorianValue V, uint Status)[] samples); Task SeedAsync(string tag, params (DateTime T, HistorianValue V, uint Status)[] samples);
/// <summary>
/// Writes historical events directly in the backend's event table (never through the alarm writer): event id,
/// source and time as given (rounded to the storage resolution), severity 500, received time = event time.
/// </summary>
/// <param name="events">The events to store.</param>
Task SeedEventsAsync(params (DateTime T, string Id, string Source)[] events);
/// <summary> /// <summary>
/// Live (non-deleted) rows for a tag, ascending by time, read directly from storage. The status is returned /// Live (non-deleted) rows for a tag, ascending by time, read directly from storage. The status is returned
/// VERBATIM as stored — there is no read-side clamp (an Empty row stored non-Bad reads back non-Bad here). /// VERBATIM as stored — there is no read-side clamp (an Empty row stored non-Bad reads back non-Bad here).
@@ -71,7 +71,7 @@ public sealed class WireBackendTests
await driver.InitializeAsync("{}", ct); await driver.InitializeAsync("{}", ct);
await WaitFor(() => await WaitFor(() =>
driver.GetDeviceState(DeviceHost) is { FixedTreeCache: not null }, TimeSpan.FromSeconds(5)); driver.GetDeviceState(DeviceHost) is { FixedTreeCache: not null }, WaitTimeout);
var state = driver.GetDeviceState(DeviceHost); var state = driver.GetDeviceState(DeviceHost);
state.ShouldNotBeNull(); state.ShouldNotBeNull();
@@ -82,7 +82,7 @@ public sealed class WireBackendTests
await WaitFor(() => await WaitFor(() =>
state.LastFixedSnapshots.ContainsKey($"{DeviceHost}/Axes/X/AbsolutePosition"), state.LastFixedSnapshots.ContainsKey($"{DeviceHost}/Axes/X/AbsolutePosition"),
TimeSpan.FromSeconds(3)); WaitTimeout);
state.LastFixedSnapshots[$"{DeviceHost}/Axes/X/AbsolutePosition"].ShouldBe(123456); state.LastFixedSnapshots[$"{DeviceHost}/Axes/X/AbsolutePosition"].ShouldBe(123456);
} }
} }
@@ -136,7 +136,7 @@ public sealed class WireBackendTests
await WaitFor(() => await WaitFor(() =>
driver.GetDeviceState(DeviceHost) is { LastProgramInfo: not null }, driver.GetDeviceState(DeviceHost) is { LastProgramInfo: not null },
TimeSpan.FromSeconds(5)); WaitTimeout);
var snapshots = await driver.ReadAsync( var snapshots = await driver.ReadAsync(
[$"{DeviceHost}/Program/Name", [$"{DeviceHost}/Program/Name",
@@ -194,7 +194,7 @@ public sealed class WireBackendTests
{ {
var state = driver.GetDeviceState(DeviceHost); var state = driver.GetDeviceState(DeviceHost);
return state is not null && state.LastTimers.Count == 4; return state is not null && state.LastTimers.Count == 4;
}, TimeSpan.FromSeconds(5)); }, WaitTimeout);
var snapshots = await driver.ReadAsync( var snapshots = await driver.ReadAsync(
[$"{DeviceHost}/Timers/PowerOnSeconds", [$"{DeviceHost}/Timers/PowerOnSeconds",
@@ -265,8 +265,11 @@ public sealed class WireBackendTests
{ {
var state = driver.GetDeviceState(DeviceHost); var state = driver.GetDeviceState(DeviceHost);
return state?.FixedTreeCache is { Capabilities.SpindleLoad: true, Capabilities.SpindleMotorSpeed: true } return state?.FixedTreeCache is { Capabilities.SpindleLoad: true, Capabilities.SpindleMotorSpeed: true }
&& state.LastSpindleLoads.Count >= 2; && state.LastSpindleLoads.Count >= 2
}, TimeSpan.FromSeconds(5)); // Motor speeds are cached after a SECOND wire round-trip (FocasDriver spindle poll), so the loads
// arriving first does not mean the speeds have — waiting on loads alone raced under load (#678.3).
&& state.LastSpindleMotorSpeeds.Count >= 2;
}, WaitTimeout);
var snapshots = await driver.ReadAsync( var snapshots = await driver.ReadAsync(
[$"{DeviceHost}/Spindle/S1/Load", [$"{DeviceHost}/Spindle/S1/Load",
@@ -282,6 +285,10 @@ public sealed class WireBackendTests
} }
} }
/// <summary>Generous: the sim fixture answers fast, but a loaded CI host (e.g. a concurrent docker build) does not.</summary>
private static readonly TimeSpan WaitTimeout = TimeSpan.FromSeconds(15);
/// <summary>Polls <paramref name="pred"/>; FAILS the test with a clear message on timeout rather than returning silently.</summary>
private static async Task WaitFor(Func<bool> pred, TimeSpan timeout) private static async Task WaitFor(Func<bool> pred, TimeSpan timeout)
{ {
var deadline = DateTime.UtcNow + timeout; var deadline = DateTime.UtcNow + timeout;
@@ -290,5 +297,7 @@ public sealed class WireBackendTests
if (pred()) return; if (pred()) return;
await Task.Delay(50); await Task.Delay(50);
} }
if (pred()) return;
Assert.Fail($"Condition not met within {timeout.TotalSeconds:0.#}s waiting on the FOCAS wire backend.");
} }
} }
@@ -0,0 +1,123 @@
using System.Net;
using Google.Protobuf.WellKnownTypes;
using Grpc.Core;
using Microsoft.AspNetCore.Builder;
using Microsoft.AspNetCore.Hosting;
using Microsoft.AspNetCore.Server.Kestrel.Core;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Logging.Abstractions;
using Shouldly;
using Xunit;
using ZB.MOM.WW.HistorianGateway.Contracts.Grpc;
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
using ZB.MOM.WW.OtOpcUa.Runtime.Historian;
namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway.Tests;
/// <summary>
/// Gitea #675: the Gateway backend's event-read bounds and direction, offline.
/// <list type="bullet">
/// <item>A reverse (newest-first) read is refused before anything is sent — the gateway streams ascending only.</item>
/// <item>The open upper bound a start-only forward read uses (<c>DateTime.MaxValue</c>, UTC kind) survives the
/// PUBLISHED client package's proto conversion — captured on the wire by an in-process gRPC server — and the
/// Windows FILETIME encoding the gateway converts it to. The live AVEVA leg (what the historian does with a
/// year-9999 bound) stays open: it needs the VPN.</item>
/// </list>
/// </summary>
public sealed class GatewayEventBoundsTests
{
private static readonly DateTime UtcMax = new(DateTime.MaxValue.Ticks, DateTimeKind.Utc);
private static CancellationToken Ct => TestContext.Current.CancellationToken;
[Fact]
public async Task A_reverse_events_read_is_refused_and_sends_nothing()
{
var fake = new FakeHistorianGatewayClient();
IHistorianDataSource ds = new GatewayHistorianDataSource(fake, NullLogger<GatewayHistorianDataSource>.Instance);
await Should.ThrowAsync<NotSupportedException>(() =>
ds.ReadEventsAsync("Pump1", DateTime.UtcNow, DateTime.UtcNow.AddHours(-1), 10, reverse: true, Ct));
fake.ReadEventsCallCount.ShouldBe(0);
// Forward through the same overload is the five-argument read, unchanged.
await ds.ReadEventsAsync("Pump1", DateTime.UtcNow.AddHours(-1), DateTime.UtcNow, 10, reverse: false, Ct);
fake.ReadEventsCallCount.ShouldBe(1);
}
[Fact]
public void UtcMax_round_trips_through_the_proto_Timestamp_and_FILETIME_encodings()
{
var ts = Timestamp.FromDateTime(UtcMax);
ts.ToDateTime().ShouldBe(UtcMax);
ts.ToDateTime().Kind.ShouldBe(DateTimeKind.Utc);
// The gateway hands its bounds to the historian as FILETIME (100 ns since 1601) — in range up to year 30828.
var fileTime = ts.ToDateTime().ToFileTimeUtc();
DateTime.FromFileTimeUtc(fileTime).ShouldBe(UtcMax);
}
[Fact]
public async Task A_start_only_bound_of_UtcMax_reaches_the_wire_intact_through_the_published_client()
{
var capture = new CapturingReadService();
await using var server = await StartServerAsync(capture);
using var adapter = HistorianGatewayClientAdapter.Create(
new ServerHistorianOptions
{
Enabled = true,
Endpoint = $"http://127.0.0.1:{server.Port}",
ApiKey = "histgw_test_key",
UseTls = false,
},
NullLoggerFactory.Instance);
IHistorianDataSource ds = new GatewayHistorianDataSource(adapter, NullLogger<GatewayHistorianDataSource>.Instance);
var start = new DateTime(2026, 1, 1, 0, 0, 0, DateTimeKind.Utc);
var r = await ds.ReadEventsAsync("Pump1", start, UtcMax, 5, Ct);
r.Events.ShouldBeEmpty();
capture.Request.ShouldNotBeNull();
capture.Request!.StartTime.ToDateTime().ShouldBe(start);
capture.Request.EndTime.ToDateTime().ShouldBe(UtcMax);
DateTime.FromFileTimeUtc(capture.Request.EndTime.ToDateTime().ToFileTimeUtc()).ShouldBe(UtcMax);
}
private sealed class CapturingReadService : HistorianRead.HistorianReadBase
{
public ReadEventsRequest? Request { get; private set; }
public override Task ReadEvents(
ReadEventsRequest request, IServerStreamWriter<HistorianEvent> responseStream, ServerCallContext context)
{
Request = request;
return Task.CompletedTask;
}
}
private sealed class Server(WebApplication app, int port) : IAsyncDisposable
{
public int Port { get; } = port;
public async ValueTask DisposeAsync()
{
await app.StopAsync();
await app.DisposeAsync();
}
}
private static async Task<Server> StartServerAsync(CapturingReadService service)
{
var builder = WebApplication.CreateSlimBuilder();
builder.Logging.ClearProviders();
builder.WebHost.ConfigureKestrel(k =>
k.Listen(IPAddress.Loopback, 0, o => o.Protocols = HttpProtocols.Http2));
builder.Services.AddGrpc();
builder.Services.AddSingleton(service);
var app = builder.Build();
app.MapGrpcService<CapturingReadService>();
await app.StartAsync();
var port = new Uri(app.Urls.First()).Port;
return new Server(app, port);
}
}
@@ -15,12 +15,18 @@
<PackageReference Include="Microsoft.NET.Test.Sdk"/> <PackageReference Include="Microsoft.NET.Test.Sdk"/>
<!-- ServiceCollection + BuildServiceProvider for the CreateDataSource factory test. --> <!-- ServiceCollection + BuildServiceProvider for the CreateDataSource factory test. -->
<PackageReference Include="Microsoft.Extensions.DependencyInjection"/> <PackageReference Include="Microsoft.Extensions.DependencyInjection"/>
<!-- In-process gRPC server capturing the published client's ReadEvents request (#675 bounds pin). -->
<PackageReference Include="Grpc.AspNetCore"/>
<PackageReference Include="xunit.runner.visualstudio"> <PackageReference Include="xunit.runner.visualstudio">
<PrivateAssets>all</PrivateAssets> <PrivateAssets>all</PrivateAssets>
<IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets> <IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
</PackageReference> </PackageReference>
</ItemGroup> </ItemGroup>
<ItemGroup>
<FrameworkReference Include="Microsoft.AspNetCore.App"/>
</ItemGroup>
<ItemGroup> <ItemGroup>
<ProjectReference Include="..\..\..\src\Drivers\ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway\ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway.csproj"/> <ProjectReference Include="..\..\..\src\Drivers\ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway\ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway.csproj"/>
</ItemGroup> </ItemGroup>
@@ -60,3 +60,10 @@ public sealed class QuestDbModifiedValueConformance(QuestDbFixture fx) : Modifie
/// <inheritdoc/> /// <inheritdoc/>
protected override Task<IConformanceBackend> CreateBackendAsync() => QuestDbConformanceBackend.CreateAsync(fx); protected override Task<IConformanceBackend> CreateBackendAsync() => QuestDbConformanceBackend.CreateAsync(fx);
} }
[Collection(QuestDbCollection.Name)]
public sealed class QuestDbEventReadConformance(QuestDbFixture fx) : EventReadConformance
{
/// <inheritdoc/>
protected override Task<IConformanceBackend> CreateBackendAsync() => QuestDbConformanceBackend.CreateAsync(fx);
}
@@ -143,6 +143,27 @@ internal sealed class QuestDbConformanceBackend : IConformanceBackend
await Wal.WaitAsync(Conn.SampleTable, CancellationToken.None); await Wal.WaitAsync(Conn.SampleTable, CancellationToken.None);
} }
/// <inheritdoc/>
public async Task SeedEventsAsync(params (DateTime T, string Id, string Source)[] events)
{
await using (var c = await Conn.OpenAsync(CancellationToken.None))
{
foreach (var (t, id, source) in events)
{
await using var cmd = Conn.CreateCommand(c,
$"INSERT INTO {Conn.EventTable} (ts, event_id, source, severity, message, received_utc) VALUES ($1, $2, $3, $4, $5, $6)");
QuestDbConnection.BindTimestamp(cmd, t);
QuestDbConnection.BindText(cmd, id);
QuestDbConnection.BindText(cmd, source);
cmd.Parameters.Add(new NpgsqlParameter { NpgsqlDbType = NpgsqlDbType.Integer, Value = 500 });
QuestDbConnection.BindText(cmd, null);
QuestDbConnection.BindTimestamp(cmd, t);
await cmd.ExecuteNonQueryAsync();
}
}
await Wal.WaitAsync(Conn.EventTable, CancellationToken.None);
}
/// <summary>Runs a statement against this table set (test-only escape hatch for hand-made rows), then waits for WAL visibility of the sample table.</summary> /// <summary>Runs a statement against this table set (test-only escape hatch for hand-made rows), then waits for WAL visibility of the sample table.</summary>
/// <param name="sql">The statement; parameters are bound positionally by <paramref name="bind"/>.</param> /// <param name="sql">The statement; parameters are bound positionally by <paramref name="bind"/>.</param>
/// <param name="bind">Binds the parameters.</param> /// <param name="bind">Binds the parameters.</param>
@@ -0,0 +1,80 @@
using Shouldly;
using Xunit;
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
using ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common;
namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests;
/// <summary>
/// Gitea #674.1 — pins the QuestDB TTL behaviour the historian's retention relies on: a table TTL is judged against
/// the wall clock (<c>min(max timestamp, now)</c>), so one far-future row — a 2066 typo that reached the backend
/// without going through the node manager's <c>HistoryUpdateMaxFutureSkew</c> bound — cannot age out every older
/// partition, including the Part 11 modification trail (the record of who changed history).
/// </summary>
/// <remarks>
/// <para>
/// <b>Positive control.</b> A partition genuinely older than the TTL (40 days, against a 30-day TTL) is kept until
/// the far-future commit and evicted BY it — proving TTL housekeeping ran on exactly the commit that could have
/// dropped the in-TTL partitions. Without it the "nothing evicted" assertion would also pass against a table
/// whose TTL never ran.
/// </para>
/// <para>
/// ⚠️ Observed on QuestDB 10.0.1 (2026-10-11): the in-TTL partitions also survived with
/// <c>cairo.ttl.use.wall.clock=false</c> (verified effective via <c>SHOW PARAMETERS</c>), so this test does not
/// isolate that flag — it pins the outcome. Keep the flag at its default (<c>true</c>) regardless; the server-side
/// future-skew bound is the primary guard.
/// </para>
/// </remarks>
[Collection(QuestDbCollection.Name)]
public sealed class TtlWallClockTests(QuestDbFixture fx)
{
private static CancellationToken Ct => TestContext.Current.CancellationToken;
private static readonly HistoryUpdateContext Who = new("ttl-test", null);
[Fact]
public async Task A_far_future_row_evicts_only_partitions_older_than_the_ttl_by_the_wall_clock()
{
fx.SkipIfUnavailable();
var o = fx.Options(fx.UniquePrefix(), retentionDays: 30);
var c = new QuestDbConnection(o);
var wal = new QuestDbWalVisibility(c, o.VisibilityTimeout);
var tags = new QuestDbTagCatalog(c, new QuestDbSchemaBootstrapper(c, o), wal);
var updater = new QuestDbHistorianDataUpdater(c, tags, wal);
// Whole seconds (µs-aligned), one row per DAY partition. With the newest row 10 days old, the 40-day partition is
// still inside the TTL measured from the table's max timestamp, so this commit evicts nothing.
var now = DateTime.UtcNow;
var today = new DateTime(now.Year, now.Month, now.Day, now.Hour, now.Minute, now.Second, DateTimeKind.Utc);
DateTime[] older = [today.AddDays(-40), today.AddDays(-20), today.AddDays(-10)];
(await updater.UpdateDataAsync("ttl.tag", HistoryUpdateMode.Insert, older.Select((t, i) => V(t, i)).ToList(), Who, Ct))
.ShouldAllBe(code => code == HistorianStatusCodes.GoodEntryInserted);
(await DaysAsync(c, c.SampleTable)).ShouldBe(older.Select(t => t.Date));
(await DaysAsync(c, c.SampleModifiedTable)).ShouldBe(older.Select(t => t.Date));
// The far-future row, then one more ordinary commit (TTL housekeeping runs on commits).
(await updater.UpdateDataAsync("ttl.tag", HistoryUpdateMode.Insert,
[V(new DateTime(2066, 1, 1, 0, 0, 0, DateTimeKind.Utc), 9)], Who, Ct)).ShouldBe([HistorianStatusCodes.GoodEntryInserted]);
(await updater.UpdateDataAsync("ttl.tag", HistoryUpdateMode.Insert, [V(today, 10)], Who, Ct))
.ShouldBe([HistorianStatusCodes.GoodEntryInserted]);
(await wal.WaitAsync(c.SampleTable, Ct)).ShouldBeTrue();
(await wal.WaitAsync(c.SampleModifiedTable, Ct)).ShouldBeTrue();
// Only the genuinely expired 40-day partition is gone; the 20- and 10-day partitions survive in both tables.
DateTime[] expected = [today.AddDays(-20).Date, today.AddDays(-10).Date, today.Date, new(2066, 1, 1)];
(await DaysAsync(c, c.SampleTable)).ShouldBe(expected);
(await DaysAsync(c, c.SampleModifiedTable)).ShouldBe(expected);
}
private static HistorizationValue V(DateTime t, double v) => new(t, HistorianValue.FromDouble(v), HistorianStatusCodes.Good);
/// <summary>The distinct UTC days holding rows, ascending.</summary>
private static async Task<List<DateTime>> DaysAsync(QuestDbConnection c, string table)
{
await using var conn = await c.OpenAsync(Ct);
await using var cmd = c.CreateCommand(conn, $"SELECT ts FROM {table} ORDER BY ts");
await using var r = await cmd.ExecuteReaderAsync(Ct);
var days = new SortedSet<DateTime>();
while (await r.ReadAsync(Ct)) days.Add(QuestDbConnection.ReadUtc(r, 0).Date);
return days.ToList();
}
}
@@ -14,6 +14,11 @@ public sealed class StatusCodeOracleTests
((uint)Opc.Ua.AggregateBits.Calculated).ShouldBe(HistorianInfoBits.HistorianCalculated); ((uint)Opc.Ua.AggregateBits.Calculated).ShouldBe(HistorianInfoBits.HistorianCalculated);
} }
/// <summary>#674: the v1-schema refusal code is the SDK's BadHistoryOperationUnsupported.</summary>
[Fact]
public void History_operation_unsupported_matches_the_SDK() =>
HistorianStatusCodes.BadHistoryOperationUnsupported.ShouldBe(Opc.Ua.StatusCodes.BadHistoryOperationUnsupported);
[Fact] [Fact]
public void Interpolated_bits_match_the_SDK() public void Interpolated_bits_match_the_SDK()
{ {
@@ -54,7 +54,7 @@ public sealed class ModifiedHistoryAtomicityTests(SqlHistorianFixture fx)
public async Task A_replace_whose_trail_insert_fails_changes_nothing() public async Task A_replace_whose_trail_insert_fails_changes_nothing()
{ {
var db = await ReadTestDb.CreateAsync(fx); var db = await ReadTestDb.CreateAsync(fx);
var u = new SqlServerHistorianDataUpdater(db.Conn, new SqlServerTagCatalog(db.Conn, db.Boot)); var u = new SqlServerHistorianDataUpdater(db.Conn, db.Boot, new SqlServerTagCatalog(db.Conn, db.Boot));
(await u.UpdateDataAsync("t", HistoryUpdateMode.Insert, [V(T1, 1)], Who, CancellationToken.None)) (await u.UpdateDataAsync("t", HistoryUpdateMode.Insert, [V(T1, 1)], Who, CancellationToken.None))
.ShouldBe([HistorianStatusCodes.GoodEntryInserted]); .ShouldBe([HistorianStatusCodes.GoodEntryInserted]);
(await CountTrailAsync(db)).ShouldBe(1); (await CountTrailAsync(db)).ShouldBe(1);
@@ -78,7 +78,7 @@ public sealed class ModifiedHistoryAtomicityTests(SqlHistorianFixture fx)
public async Task Deletes_whose_trail_insert_fails_delete_nothing() public async Task Deletes_whose_trail_insert_fails_delete_nothing()
{ {
var db = await ReadTestDb.CreateAsync(fx); var db = await ReadTestDb.CreateAsync(fx);
var u = new SqlServerHistorianDataUpdater(db.Conn, new SqlServerTagCatalog(db.Conn, db.Boot)); var u = new SqlServerHistorianDataUpdater(db.Conn, db.Boot, new SqlServerTagCatalog(db.Conn, db.Boot));
await u.UpdateDataAsync("t", HistoryUpdateMode.Insert, [V(T1, 1), V(T2, 2)], Who, CancellationToken.None); await u.UpdateDataAsync("t", HistoryUpdateMode.Insert, [V(T1, 1), V(T2, 2)], Who, CancellationToken.None);
await BreakTheTrailAsync(db); await BreakTheTrailAsync(db);
@@ -99,7 +99,7 @@ public sealed class ModifiedHistoryAtomicityTests(SqlHistorianFixture fx)
public async Task An_over_long_principal_is_truncated_never_refused() public async Task An_over_long_principal_is_truncated_never_refused()
{ {
var db = await ReadTestDb.CreateAsync(fx); var db = await ReadTestDb.CreateAsync(fx);
var u = new SqlServerHistorianDataUpdater(db.Conn, new SqlServerTagCatalog(db.Conn, db.Boot)); var u = new SqlServerHistorianDataUpdater(db.Conn, db.Boot, new SqlServerTagCatalog(db.Conn, db.Boot));
var who = new HistoryUpdateContext(new string('p', 500), new string('k', 500)); var who = new HistoryUpdateContext(new string('p', 500), new string('k', 500));
(await u.UpdateDataAsync("t", HistoryUpdateMode.Insert, [V(T1, 1)], who, CancellationToken.None)) (await u.UpdateDataAsync("t", HistoryUpdateMode.Insert, [V(T1, 1)], who, CancellationToken.None))
@@ -178,6 +178,97 @@ public sealed class RetentionTests(SqlHistorianFixture fx)
(await LockIsFreeAsync(s)).ShouldBeTrue(); (await LockIsFreeAsync(s)).ShouldBeTrue();
} }
[Fact]
public async Task Cancelling_mid_command_while_the_delete_is_blocked_still_releases_the_applock()
{
fx.SkipIfUnavailable();
var (c, b, _, s) = Make();
await SeedAsync(b, s);
// A second session holds an exclusive table lock, so the purger's first DELETE blocks server-side.
await using var holder = new SqlConnection(fx.ConnectionString);
await holder.OpenAsync();
await using var tx = (SqlTransaction)await holder.BeginTransactionAsync();
await using (var cmd = holder.CreateCommand())
{
cmd.Transaction = tx;
cmd.CommandText = $"SELECT COUNT(*) FROM [{s}].Sample WITH (TABLOCKX, HOLDLOCK)";
await cmd.ExecuteScalarAsync();
}
short holderSpid;
await using (var cmd = holder.CreateCommand())
{
cmd.Transaction = tx;
cmd.CommandText = "SELECT CAST(@@SPID AS smallint)";
holderSpid = (short)(await cmd.ExecuteScalarAsync())!;
}
using var cts = new CancellationTokenSource();
var purger = new SqlServerHistorianRetention(c, b, TimeSpan.FromHours(1), 30, 4000, time: new FixedTime(Now));
var purge = purger.PurgeAsync(cts.Token);
// Cancel only once the server reports a DELETE actually blocked behind the holder — a fixed delay could
// cancel during bootstrap/applock acquisition on a slow host and never exercise the mid-command path.
var deadline = DateTime.UtcNow.AddSeconds(15);
while (await CountAsync(
$"SELECT COUNT(*) FROM sys.dm_exec_requests WHERE blocking_session_id = {holderSpid} AND command = 'DELETE'") == 0)
{
purge.IsCompleted.ShouldBeFalse("the purge finished without ever blocking on the holder's table lock");
DateTime.UtcNow.ShouldBeLessThan(deadline, "the purge's DELETE never blocked behind the holder");
await Task.Delay(50);
}
cts.Cancel();
// SqlClient surfaces the attention as OperationCanceledException or as a SqlException.
var ex = await Should.ThrowAsync<Exception>(() => purge);
(ex is OperationCanceledException or SqlException).ShouldBeTrue($"unexpected {ex.GetType().Name}: {ex.Message}");
await tx.RollbackAsync();
(await LockIsFreeAsync(s)).ShouldBeTrue();
(await CountAsync($"SELECT COUNT(*) FROM [{s}].Sample")).ShouldBe(21);
}
[Fact]
public async Task A_failing_release_falls_back_to_clearing_the_pool_so_the_applock_is_freed()
{
fx.SkipIfUnavailable();
var (c, b, _, s) = Make();
await SeedAsync(b, s);
var purger = new SqlServerHistorianRetention(c, b, TimeSpan.FromHours(1), 30, 4000, time: new FixedTime(Now))
{
ReleaseSql = "EXEC sp_this_procedure_does_not_exist;",
};
await purger.PurgeAsync(CancellationToken.None);
(await LockIsFreeAsync(s)).ShouldBeTrue();
}
[Fact]
public async Task Falsification_control_a_failing_release_with_a_no_op_ClearPool_leaves_the_applock_held()
{
fx.SkipIfUnavailable();
var (c, b, _, s) = Make();
await SeedAsync(b, s);
var purger = new SqlServerHistorianRetention(c, b, TimeSpan.FromHours(1), 30, 4000, time: new FixedTime(Now))
{
ReleaseSql = "EXEC sp_this_procedure_does_not_exist;",
ClearPool = _ => { },
};
await purger.PurgeAsync(CancellationToken.None);
try
{
// The pooled session still owns the Session-scoped lock: proves the fallback test above is not vacuous.
(await LockIsFreeAsync(s)).ShouldBeFalse();
}
finally
{
SqlConnection.ClearAllPools(); // drop the leaked session so the lock does not outlive the test
}
}
[Fact] [Fact]
public async Task A_throwing_purge_releases_the_applock() public async Task A_throwing_purge_releases_the_applock()
{ {
@@ -32,7 +32,7 @@ public sealed class SchemaBootstrapIntegrationTests(SqlHistorianFixture fx)
var s = SqlHistorianFixture.UniqueSchema(); var s = SqlHistorianFixture.UniqueSchema();
await Boot(Opts(s)).EnsureAsync(CancellationToken.None); await Boot(Opts(s)).EnsureAsync(CancellationToken.None);
await Boot(Opts(s)).EnsureAsync(CancellationToken.None); // fresh instance => re-runs the guarded path await Boot(Opts(s)).EnsureAsync(CancellationToken.None); // fresh instance => re-runs the guarded path
(await ScalarAsync($"SELECT COUNT(*) FROM [{s}].SchemaVersion")).ShouldBe(SqlServerSchemaBootstrapper.RequiredVersion); (await ScalarAsync($"SELECT COUNT(*) FROM [{s}].SchemaVersion")).ShouldBe(SqlServerSchemaBootstrapper.ModifiedTrailVersion);
(await ScalarAsync($"SELECT COUNT(*) FROM sys.indexes WHERE name='CX_Event_Time' AND object_id=OBJECT_ID('[{s}].Event')")).ShouldBe(1); (await ScalarAsync($"SELECT COUNT(*) FROM sys.indexes WHERE name='CX_Event_Time' AND object_id=OBJECT_ID('[{s}].Event')")).ShouldBe(1);
} }
@@ -51,13 +51,14 @@ public sealed class SchemaBootstrapIntegrationTests(SqlHistorianFixture fx)
cmd.CommandText = b; cmd.CommandText = b;
await cmd.ExecuteNonQueryAsync(); await cmd.ExecuteNonQueryAsync();
} }
(await ScalarAsync($"SELECT COUNT(*) FROM [{s}].SchemaVersion")).ShouldBe(SqlServerSchemaBootstrapper.RequiredVersion); (await ScalarAsync($"SELECT COUNT(*) FROM [{s}].SchemaVersion")).ShouldBe(SqlServerSchemaBootstrapper.ModifiedTrailVersion);
} }
/// <summary> /// <summary>
/// #668 upgrade path: a schema left at version 1 (only <c>001_initial.sql</c> applied, as every pre-#668 install is) /// #668 upgrade path: a schema left at version 1 (only <c>001_initial.sql</c> applied, as every pre-#668 install is)
/// is upgraded in place by an auto-creating bootstrap — <c>SampleModified</c> appears, both version rows exist, and /// is upgraded in place by an auto-creating bootstrap — <c>SampleModified</c> appears, both version rows exist, and
/// existing samples are untouched. With <c>AutoCreateSchema=false</c> the same v1 schema is refused. /// existing samples are untouched. With <c>AutoCreateSchema=false</c> the same v1 schema is accepted with the
/// modification trail switched off and left at v1 (#674).
/// </summary> /// </summary>
[Fact] [Fact]
public async Task A_version_1_schema_is_upgraded_to_version_2_in_place() public async Task A_version_1_schema_is_upgraded_to_version_2_in_place()
@@ -81,11 +82,15 @@ public sealed class SchemaBootstrapIntegrationTests(SqlHistorianFixture fx)
} }
(await ScalarAsync($"SELECT COUNT(*) FROM sys.tables WHERE name = 'SampleModified' AND schema_id = SCHEMA_ID('{s}')")).ShouldBe(0); (await ScalarAsync($"SELECT COUNT(*) FROM sys.tables WHERE name = 'SampleModified' AND schema_id = SCHEMA_ID('{s}')")).ShouldBe(0);
var refused = await Should.ThrowAsync<HistorianSchemaMissingException>( // #674: verify-only accepts v1 — with the modification trail switched off — and does not upgrade it.
() => Boot(Opts(s, auto: false)).EnsureAsync(CancellationToken.None)); var verifyOnly = Boot(Opts(s, auto: false));
refused.Message.ShouldContain("missing version 2 (found 1)"); await verifyOnly.EnsureAsync(CancellationToken.None);
verifyOnly.SupportsModifiedTrail.ShouldBeFalse();
(await ScalarAsync($"SELECT MAX(Version) FROM [{s}].SchemaVersion")).ShouldBe(1);
await Boot(Opts(s)).EnsureAsync(CancellationToken.None); var upgrading = Boot(Opts(s));
await upgrading.EnsureAsync(CancellationToken.None);
upgrading.SupportsModifiedTrail.ShouldBeTrue();
(await ScalarAsync($"SELECT COUNT(*) FROM [{s}].SchemaVersion")).ShouldBe(2); (await ScalarAsync($"SELECT COUNT(*) FROM [{s}].SchemaVersion")).ShouldBe(2);
(await ScalarAsync($"SELECT MAX(Version) FROM [{s}].SchemaVersion")).ShouldBe(2); (await ScalarAsync($"SELECT MAX(Version) FROM [{s}].SchemaVersion")).ShouldBe(2);
@@ -101,7 +106,7 @@ public sealed class SchemaBootstrapIntegrationTests(SqlHistorianFixture fx)
var s = SqlHistorianFixture.UniqueSchema(); var s = SqlHistorianFixture.UniqueSchema();
var tasks = Enumerable.Range(0, 4).Select(_ => Task.Run(() => Boot(Opts(s)).EnsureAsync(CancellationToken.None))).ToArray(); var tasks = Enumerable.Range(0, 4).Select(_ => Task.Run(() => Boot(Opts(s)).EnsureAsync(CancellationToken.None))).ToArray();
await Task.WhenAll(tasks); await Task.WhenAll(tasks);
(await ScalarAsync($"SELECT COUNT(*) FROM [{s}].SchemaVersion")).ShouldBe(SqlServerSchemaBootstrapper.RequiredVersion); (await ScalarAsync($"SELECT COUNT(*) FROM [{s}].SchemaVersion")).ShouldBe(SqlServerSchemaBootstrapper.ModifiedTrailVersion);
} }
[Fact] [Fact]
@@ -110,7 +115,7 @@ public sealed class SchemaBootstrapIntegrationTests(SqlHistorianFixture fx)
fx.SkipIfUnavailable(); fx.SkipIfUnavailable();
var ex = await Should.ThrowAsync<HistorianSchemaMissingException>( var ex = await Should.ThrowAsync<HistorianSchemaMissingException>(
() => Boot(Opts(SqlHistorianFixture.UniqueSchema(), auto: false)).EnsureAsync(CancellationToken.None)); () => Boot(Opts(SqlHistorianFixture.UniqueSchema(), auto: false)).EnsureAsync(CancellationToken.None));
ex.Message.ShouldContain("missing version 2"); ex.Message.ShouldContain("missing version 1 (found 0)");
} }
[Fact] [Fact]
@@ -0,0 +1,181 @@
using Microsoft.Data.SqlClient;
using Microsoft.Extensions.Logging;
using Shouldly;
using Xunit;
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
using ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian;
using ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common;
namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests;
/// <summary>
/// Gitea #674: a database left at schema version 1 (no Part 11 <c>SampleModified</c> trail) with
/// <c>AutoCreateSchema=false</c> keeps historizing. Before #674 the bootstrapper required v2 and refused EVERY call —
/// live writes, raw reads, alarm events, retention — until script 002 was applied by hand. Now only the features
/// that need the trail are refused (<c>BadHistoryOperationUnsupported</c>, nothing sent to SQL), and the
/// bootstrapper logs ONE warning for the whole provider. Built through <see cref="SqlServerHistorianProvider"/>, the
/// production wiring, so every seam shares the one bootstrapper the way a node does.
/// </summary>
[Collection(SqlHistorianCollection.Name)]
public sealed class SchemaV1PostureTests(SqlHistorianFixture fx)
{
private static CancellationToken Ct => TestContext.Current.CancellationToken;
private static readonly DateTime Recent = DateTime.UtcNow.AddHours(-1);
private static readonly HistoryUpdateContext Who = new("editor", "k1");
[Fact]
public async Task A_v1_schema_keeps_historizing_and_refuses_only_the_modified_trail_features()
{
fx.SkipIfUnavailable();
var s = SqlHistorianFixture.UniqueSchema();
await ApplyV1Async(s);
var old = DateTime.UtcNow.AddDays(-40);
var loggers = new CapturingLoggerFactory();
var provider = new SqlServerHistorianProvider(new SqlServerHistorianOptions
{
ConnectionString = fx.ConnectionString, Schema = s, AutoCreateSchema = false, RetentionDays = 30,
});
var sp = new LoggerServices(loggers);
// Live writes (the continuous-historization drain) and raw reads work.
var writer = provider.CreateValueWriter(sp);
(await writer.WriteLiveValuesAsync("Plant.Speed",
[new HistorizationValue(old, HistorianValue.FromDouble(1), 0), new HistorizationValue(Recent, HistorianValue.FromDouble(2), 0)],
Ct)).ShouldBeTrue();
var source = provider.CreateDataSource(sp);
var raw = await source.ReadRawAsync("Plant.Speed", Recent.AddMinutes(-1), Recent.AddMinutes(1), 10, Ct);
raw.Samples.Select(v => v.Value).ShouldBe([2d]);
// Alarm events are written.
var alarms = provider.CreateAlarmWriter(sp);
(await alarms.WriteBatchAsync(
[new AlarmHistorianEvent("alarm-1", "Plant/Line/Eq", "HiTemp", "LimitAlarm", AlarmSeverity.High, "Activated", "m", "u", null, Recent)],
Ct)).ShouldBe([HistorianWriteOutcome.Ack]);
(await CountAsync($"SELECT COUNT(*) FROM [{s}].Event")).ShouldBe(1);
// Modified reads: the interface's "no trail" signal (the node manager reports BadHistoryOperationUnsupported),
// not a backend failure — health stays clean.
await Should.ThrowAsync<NotSupportedException>(() =>
source.ReadModifiedAsync("Plant.Speed", Recent.AddDays(-1), Recent.AddDays(1), 10, null, Ct));
source.GetHealthSnapshot().LastError.ShouldBeNull();
// HistoryUpdate: every item refused, and NOTHING reaches SQL — a sent MERGE/DELETE would either change the
// samples or fail on the missing SampleModified target; an Insert on an unknown tag would create its Tag row.
var updater = provider.CreateUpdater(sp)!;
var tagsBefore = await CountAsync($"SELECT COUNT(*) FROM [{s}].Tag");
(await updater.UpdateDataAsync("Plant.Speed", HistoryUpdateMode.Update,
[new HistorizationValue(Recent, HistorianValue.FromDouble(99), 0), new HistorizationValue(Recent.AddSeconds(1), HistorianValue.FromDouble(98), 0)],
Who, Ct)).ShouldBe([HistorianStatusCodes.BadHistoryOperationUnsupported, HistorianStatusCodes.BadHistoryOperationUnsupported]);
(await updater.UpdateDataAsync("Plant.Brand.New", HistoryUpdateMode.Insert,
[new HistorizationValue(Recent, HistorianValue.FromDouble(1), 0)], Who, Ct))
.ShouldBe([HistorianStatusCodes.BadHistoryOperationUnsupported]);
(await updater.DeleteRawAsync("Plant.Speed", Recent.AddMinutes(-1), Recent.AddMinutes(1), Who, Ct))
.ShouldBe(HistorianStatusCodes.BadHistoryOperationUnsupported);
(await updater.DeleteAtTimeAsync("Plant.Speed", [Recent], Who, Ct))
.ShouldBe([HistorianStatusCodes.BadHistoryOperationUnsupported]);
(await CountAsync($"SELECT COUNT(*) FROM [{s}].Tag")).ShouldBe(tagsBefore);
(await source.ReadRawAsync("Plant.Speed", Recent.AddMinutes(-1), Recent.AddMinutes(1), 10, Ct))
.Samples.Select(v => v.Value).ShouldBe([2d]);
// Retention purges samples and skips the trail it does not have.
var purge = await provider.CreateRetention(sp)!.PurgeAsync(Ct);
purge.Select(r => r.Table).ShouldBe(["Sample", "Event"]);
purge.Single(r => r.Table == "Sample").Rows.ShouldBe(1);
(await CountAsync($"SELECT COUNT(*) FROM [{s}].Sample")).ShouldBe(1);
// One warning for the whole provider (= per boot), however many seams and calls touched the bootstrapper.
loggers.Warnings.Count(w => w.Contains("Part 11 modification trail is unavailable")).ShouldBe(1);
(await CountAsync($"SELECT MAX(Version) FROM [{s}].SchemaVersion")).ShouldBe(1); // verify-only: never upgraded
}
/// <summary>
/// #674: <c>AutoCreateSchema=true</c> still upgrades a v1 schema to v2 — the modification trail appears and a
/// HistoryUpdate is recorded in it — and logs no v1 warning.
/// </summary>
[Fact]
public async Task Auto_create_still_upgrades_a_v1_schema_and_enables_the_trail()
{
fx.SkipIfUnavailable();
var s = SqlHistorianFixture.UniqueSchema();
await ApplyV1Async(s);
var loggers = new CapturingLoggerFactory();
var provider = new SqlServerHistorianProvider(new SqlServerHistorianOptions
{
ConnectionString = fx.ConnectionString, Schema = s, AutoCreateSchema = true,
});
var sp = new LoggerServices(loggers);
(await provider.CreateUpdater(sp)!.UpdateDataAsync("Plant.Speed", HistoryUpdateMode.Insert,
[new HistorizationValue(Recent, HistorianValue.FromDouble(5), 0)], Who, Ct))
.ShouldBe([HistorianStatusCodes.GoodEntryInserted]);
(await CountAsync($"SELECT MAX(Version) FROM [{s}].SchemaVersion")).ShouldBe(SqlServerSchemaBootstrapper.ModifiedTrailVersion);
var trail = await provider.CreateDataSource(sp)
.ReadModifiedAsync("Plant.Speed", Recent.AddMinutes(-1), Recent.AddMinutes(1), 10, null, Ct);
trail.Values.ShouldHaveSingleItem().UpdateType.ShouldBe(HistoryModificationType.Insert);
loggers.Warnings.ShouldNotContain(w => w.Contains("Part 11 modification trail is unavailable"));
}
private async Task ApplyV1Async(string schema)
{
var v1 = new StreamReader(typeof(SqlServerSchemaBootstrapper).Assembly.GetManifestResourceStream("Schema.001_initial.sql")!).ReadToEnd();
await using var c = new SqlConnection(fx.ConnectionString);
await c.OpenAsync(Ct);
foreach (var b in SqlServerSchemaBootstrapper.SplitBatches(v1.Replace("$(Schema)", schema)))
{
await using var cmd = c.CreateCommand();
cmd.CommandText = b;
await cmd.ExecuteNonQueryAsync(Ct);
}
}
private async Task<long> CountAsync(string sql)
{
await using var c = new SqlConnection(fx.ConnectionString);
await c.OpenAsync(Ct);
await using var cmd = c.CreateCommand();
cmd.CommandText = sql;
return Convert.ToInt64(await cmd.ExecuteScalarAsync(Ct));
}
/// <summary>The only service the provider resolves is <see cref="ILoggerFactory"/>.</summary>
private sealed class LoggerServices(ILoggerFactory loggers) : IServiceProvider
{
public object? GetService(Type serviceType) => serviceType == typeof(ILoggerFactory) ? loggers : null;
}
/// <summary>Captures every Warning-or-above message from every logger it creates.</summary>
private sealed class CapturingLoggerFactory : ILoggerFactory
{
private readonly List<string> _warnings = [];
public IReadOnlyList<string> Warnings
{
get { lock (_warnings) return _warnings.ToList(); }
}
public ILogger CreateLogger(string categoryName) => new Capturing(this);
public void AddProvider(ILoggerProvider provider)
{
}
public void Dispose()
{
}
private sealed class Capturing(CapturingLoggerFactory owner) : ILogger
{
public IDisposable? BeginScope<TState>(TState state) where TState : notnull => null;
public bool IsEnabled(LogLevel logLevel) => true;
public void Log<TState>(LogLevel logLevel, EventId eventId, TState state, Exception? exception,
Func<TState, Exception?, string> formatter)
{
if (logLevel < LogLevel.Warning) return;
lock (owner._warnings) owner._warnings.Add(formatter(state, exception));
}
}
}
}
@@ -60,3 +60,10 @@ public sealed class SqlServerModifiedValueConformance(SqlHistorianFixture fx) :
/// <inheritdoc/> /// <inheritdoc/>
protected override Task<IConformanceBackend> CreateBackendAsync() => SqlServerConformanceBackend.CreateAsync(fx); protected override Task<IConformanceBackend> CreateBackendAsync() => SqlServerConformanceBackend.CreateAsync(fx);
} }
[Collection(SqlHistorianCollection.Name)]
public sealed class SqlServerEventReadConformance(SqlHistorianFixture fx) : EventReadConformance
{
/// <inheritdoc/>
protected override Task<IConformanceBackend> CreateBackendAsync() => SqlServerConformanceBackend.CreateAsync(fx);
}
@@ -34,12 +34,28 @@ internal sealed class SqlServerConformanceBackend : IConformanceBackend
/// <inheritdoc/> /// <inheritdoc/>
public IHistorianDataUpdater CreateUpdater() => public IHistorianDataUpdater CreateUpdater() =>
new SqlServerHistorianDataUpdater(_db.Conn, new SqlServerTagCatalog(_db.Conn, _db.Boot)); new SqlServerHistorianDataUpdater(_db.Conn, _db.Boot, new SqlServerTagCatalog(_db.Conn, _db.Boot));
/// <inheritdoc/> /// <inheritdoc/>
public async Task SeedAsync(string tag, params (DateTime T, HistorianValue V, uint Status)[] samples) => public async Task SeedAsync(string tag, params (DateTime T, HistorianValue V, uint Status)[] samples) =>
await _db.SeedAsync(tag, samples); await _db.SeedAsync(tag, samples);
/// <inheritdoc/>
public async Task SeedEventsAsync(params (DateTime T, string Id, string Source)[] events)
{
await using var c = await _db.Conn.OpenAsync(CancellationToken.None);
foreach (var (t, id, source) in events)
{
await using var cmd = _db.Conn.CreateCommand(c,
$"INSERT {_db.Conn.Q("Event")} (EventId, SourceName, AlarmId, AlarmName, AlarmTypeName, Severity, EventKind, Message, " +
"UserName, Comment, EventTimeUtc, ReceivedTimeUtc) VALUES (@id, @src, 'a', 'n', 'T', 500, 'Activated', NULL, NULL, NULL, @t, @t)");
cmd.Parameters.AddWithValue("@id", id);
cmd.Parameters.AddWithValue("@src", source);
SqlHistorianConnection.AddDateTime2(cmd, "@t", t);
await cmd.ExecuteNonQueryAsync();
}
}
/// <inheritdoc/> /// <inheritdoc/>
public async Task<IReadOnlyList<(DateTime T, HistorianValue V, uint Status)>> ReadBackAsync(string tag) public async Task<IReadOnlyList<(DateTime T, HistorianValue V, uint Status)>> ReadBackAsync(string tag)
{ {
@@ -32,7 +32,7 @@ public sealed class SchemaScriptTests
/// <summary> /// <summary>
/// #668: schema v2 (the Part 11 modification trail) is embedded, matches its docs copy, records version 2 (which /// #668: schema v2 (the Part 11 modification trail) is embedded, matches its docs copy, records version 2 (which
/// must equal <see cref="SqlServerSchemaBootstrapper.RequiredVersion"/>), and carries none of the constructs an /// must equal <see cref="SqlServerSchemaBootstrapper.ModifiedTrailVersion"/>), and carries none of the constructs an
/// <c>OUTPUT … INTO</c> target forbids (FOREIGN KEY / CHECK / trigger) — any of them would fail every HistoryUpdate. /// <c>OUTPUT … INTO</c> target forbids (FOREIGN KEY / CHECK / trigger) — any of them would fail every HistoryUpdate.
/// </summary> /// </summary>
[Fact] [Fact]
@@ -45,7 +45,8 @@ public sealed class SchemaScriptTests
text.ShouldContain("[SampleModified]"); text.ShouldContain("[SampleModified]");
text.ShouldContain("PRIMARY KEY CLUSTERED (TagId, SourceTimeUtc, ModifiedId)"); text.ShouldContain("PRIMARY KEY CLUSTERED (TagId, SourceTimeUtc, ModifiedId)");
text.ShouldContain("VALUES (2, SYSUTCDATETIME())"); text.ShouldContain("VALUES (2, SYSUTCDATETIME())");
SqlServerSchemaBootstrapper.RequiredVersion.ShouldBe(2); SqlServerSchemaBootstrapper.ModifiedTrailVersion.ShouldBe(2);
SqlServerSchemaBootstrapper.RequiredVersion.ShouldBe(1); // #674: v1 is still served, without the trail
var ddl = string.Join('\n', text.Split('\n').Where(l => !l.TrimStart().StartsWith("--", StringComparison.Ordinal))); var ddl = string.Join('\n', text.Split('\n').Where(l => !l.TrimStart().StartsWith("--", StringComparison.Ordinal)));
ddl.ShouldNotContain("FOREIGN KEY", Case.Insensitive); ddl.ShouldNotContain("FOREIGN KEY", Case.Insensitive);
ddl.ShouldNotContain("CHECK", Case.Insensitive); ddl.ShouldNotContain("CHECK", Case.Insensitive);
@@ -0,0 +1,78 @@
using System.Text.Json;
using Shouldly;
using Xunit;
using ZB.MOM.WW.OtOpcUa.Host.Configuration;
namespace ZB.MOM.WW.OtOpcUa.Host.Tests.Configuration;
/// <summary>
/// #678.6 — the per-role appsettings overlay name is built from plain roles only. A <c>cluster-*</c> role in the
/// name produced a file that never exists, so no overlay ever loaded on a node with a cluster role.
/// </summary>
public sealed class RoleOverlayTests
{
[Fact]
public void A_driver_site_node_loads_the_driver_overlay()
=> RoleOverlay.Suffix(["driver", "cluster-SITE-A"]).ShouldBe("driver");
[Fact]
public void A_central_node_loads_the_admin_driver_overlay()
=> RoleOverlay.Suffix(["admin", "driver", "cluster-MAIN"]).ShouldBe("admin-driver");
[Fact]
public void Plain_roles_are_ordinal_sorted_regardless_of_input_order()
=> RoleOverlay.Suffix(["driver", "cluster-MAIN", "admin"]).ShouldBe("admin-driver");
[Fact]
public void Roles_without_a_cluster_role_keep_the_existing_names()
{
RoleOverlay.Suffix(["admin"]).ShouldBe("admin");
RoleOverlay.Suffix(["driver"]).ShouldBe("driver");
RoleOverlay.Suffix(["admin", "driver"]).ShouldBe("admin-driver");
}
[Fact]
public void A_cluster_role_prefix_is_matched_case_insensitively()
=> RoleOverlay.Suffix(["driver", "CLUSTER-SITE-A"]).ShouldBe("driver");
[Fact]
public void No_plain_role_means_no_overlay()
{
RoleOverlay.Suffix([]).ShouldBeNull();
RoleOverlay.Suffix(["cluster-SITE-A"]).ShouldBeNull();
}
// ---- The shipped overlays: activating them on cluster-role nodes must not change production semantics. ----
private static string HostDir()
{
for (var dir = new DirectoryInfo(AppContext.BaseDirectory); dir is not null; dir = dir.Parent)
{
var candidate = Path.Combine(dir.FullName, "src", "Server", "ZB.MOM.WW.OtOpcUa.Host");
if (File.Exists(Path.Combine(candidate, "appsettings.json"))) return candidate;
}
throw new InvalidOperationException("Host project directory not found above " + AppContext.BaseDirectory);
}
[Theory]
[InlineData("admin")]
[InlineData("driver")]
[InlineData("admin-driver")]
public void Shipped_overlays_carry_only_the_audited_keys(string suffix)
{
using var doc = JsonDocument.Parse(File.ReadAllText(Path.Combine(HostDir(), $"appsettings.{suffix}.json")));
var root = doc.RootElement;
// Only Serilog + Security:Ldap, and in Security:Ldap only the two audited keys.
root.EnumerateObject().Select(p => p.Name).Order().ShouldBe(["Security", "Serilog"]);
var ldap = root.GetProperty("Security").GetProperty("Ldap");
ldap.EnumerateObject().Select(p => p.Name).Order().ShouldBe(["DevStubMode", "Transport"]);
ldap.GetProperty("DevStubMode").GetBoolean().ShouldBeFalse();
// No Debug-level (or lower) category override: activating an overlay must not raise log volume on a node.
foreach (var o in root.GetProperty("Serilog").GetProperty("MinimumLevel").GetProperty("Override").EnumerateObject())
o.Value.GetString().ShouldNotBeOneOf("Debug", "Verbose");
root.GetProperty("Serilog").GetProperty("MinimumLevel").GetProperty("Default").GetString().ShouldBe("Information");
}
}
@@ -54,8 +54,48 @@ public sealed class EventHistoryPagingOverTheWireTests
finally { await rig.DisposeAsync(); } finally { await rig.DisposeAsync(); }
} }
[Fact]
public async Task A_capped_REVERSE_event_read_pages_newest_first_through_the_session_store_exactly_once()
{
// Gitea #675: StartTime later than EndTime. Before the fix the cap kept the EARLIEST 10, ascending, with no
// continuation point.
var ct = TestContext.Current.CancellationToken;
var rig = await StartAsync("EvtPagingWireRev", encrypted: false, ct);
try
{
var events = Enumerable.Range(0, 25)
.Select(i => new HistoricalEvent($"e{i:00}", "src", Epoch.AddSeconds(i / 2), Epoch, "m", 500))
.ToList(); // pairs share a timestamp, so page boundaries cut through ties
rig.Nm.HistorianDataSource = new SqlLikeEventSource(events);
rig.MaterialiseNative();
rig.Enforce(false, [Key()]);
using var session = await rig.ConnectAsync(anonymous: true, ct);
var node = new NodeId(RawDeviceFolder, (ushort)session.NamespaceUris.GetIndex(V3NodeIds.RawNamespaceUri));
var collected = new List<string>();
var sizes = new List<int>();
byte[]? cp = null;
do
{
var (ids, next) = await ReadPageAsync(session, node, cp, ct, reverse: true);
collected.AddRange(ids);
sizes.Add(ids.Count);
cp = next;
sizes.Count.ShouldBeLessThan(30, "paging must terminate");
}
while (cp is not null);
sizes.ShouldBe([10, 10, 5]);
// Newest-first: time descending, ties by EventId descending.
collected.ShouldBe(events.OrderByDescending(e => e.EventTimeUtc)
.ThenByDescending(e => e.EventId, StringComparer.Ordinal).Select(e => e.EventId));
}
finally { await rig.DisposeAsync(); }
}
private static async Task<(List<string> Ids, byte[]? Cp)> ReadPageAsync( private static async Task<(List<string> Ids, byte[]? Cp)> ReadPageAsync(
ISession session, NodeId node, byte[]? cp, CancellationToken ct) ISession session, NodeId node, byte[]? cp, CancellationToken ct, bool reverse = false)
{ {
var filter = new EventFilter(); var filter = new EventFilter();
filter.SelectClauses.Add(new SimpleAttributeOperand(ObjectTypeIds.BaseEventType, new QualifiedName("EventId")) filter.SelectClauses.Add(new SimpleAttributeOperand(ObjectTypeIds.BaseEventType, new QualifiedName("EventId"))
@@ -64,8 +104,8 @@ public sealed class EventHistoryPagingOverTheWireTests
}); });
var details = new ReadEventDetails var details = new ReadEventDetails
{ {
StartTime = Epoch.AddDays(-1), StartTime = reverse ? Epoch.AddDays(1) : Epoch.AddDays(-1),
EndTime = Epoch.AddDays(1), EndTime = reverse ? Epoch.AddDays(-1) : Epoch.AddDays(1),
NumValuesPerNode = 10, NumValuesPerNode = 10,
Filter = filter, Filter = filter,
}; };
@@ -94,6 +134,18 @@ public sealed class EventHistoryPagingOverTheWireTests
return new HistoricalEventsResult(window.Take(cap).ToList(), window.Count > cap ? new byte[] { 1 } : null); return new HistoricalEventsResult(window.Take(cap).ToList(), window.Count > cap ? new byte[] { 1 } : null);
} }
/// <summary>The #675 overload: reverse is <c>(lo, hi]</c> by (time, id) descending.</summary>
public async Task<HistoricalEventsResult> ReadEventsAsync(
string? sourceName, DateTime startUtc, DateTime endUtc, int maxEvents, bool reverse, CancellationToken ct)
{
if (!reverse) return await ReadEventsAsync(sourceName, startUtc, endUtc, maxEvents, ct).ConfigureAwait(false);
await Task.Delay(1, ct).ConfigureAwait(false);
var (lo, hi) = startUtc <= endUtc ? (startUtc, endUtc) : (endUtc, startUtc);
var window = _all.Where(e => e.EventTimeUtc > lo && e.EventTimeUtc <= hi).Reverse().ToList();
var cap = maxEvents <= 0 ? 1000 : maxEvents;
return new HistoricalEventsResult(window.Take(cap).ToList(), window.Count > cap ? new byte[] { 1 } : null);
}
public Task<CoreHistoryReadResult> ReadRawAsync(string r, DateTime s, DateTime e, uint m, CancellationToken ct) => public Task<CoreHistoryReadResult> ReadRawAsync(string r, DateTime s, DateTime e, uint m, CancellationToken ct) =>
NullHistorianDataSource.Instance.ReadRawAsync(r, s, e, m, ct); NullHistorianDataSource.Instance.ReadRawAsync(r, s, e, m, ct);
public Task<CoreHistoryReadResult> ReadProcessedAsync( public Task<CoreHistoryReadResult> ReadProcessedAsync(
@@ -558,6 +558,190 @@ public sealed class HistoryUpdateTests : IDisposable
finally { await host.DisposeAsync(); } finally { await host.DisposeAsync(); }
} }
// ───────────────────────────── #674 future-skew bound ─────────────────────────────
/// <summary>
/// #674: an item whose SourceTimestamp lies beyond <c>now + HistoryUpdateMaxFutureSkew</c> is refused
/// <c>BadOutOfRange</c> by the node manager and never reaches the backend; an item exactly AT the bound and one in
/// the past are sent. Judged against the injected clock, not the wall clock.
/// </summary>
[Fact]
public async Task A_source_time_beyond_the_future_skew_is_bad_out_of_range_and_never_sent()
{
var (host, server) = await BootAsync();
try
{
var nm = server.NodeManager!;
var updater = new FakeUpdater();
nm.HistorianDataUpdater = updater;
nm.Clock = new FakeClock(Start);
nm.HistoryUpdateMaxFutureSkew = TimeSpan.FromHours(1);
WireAcl(nm, enabled: false, NodePermissions.HistoryUpdate);
var nodeId = Historized(nm, AddressSpaceRealm.Raw, RawPath);
var atBound = Start.AddHours(1);
var (results, errors) = Update(nm, KeyIdentity(OpcUaDataPlaneRoles.HistoryUpdate), nodeId,
PerformUpdateType.Update, Value(T1, 1d), Value(atBound.AddTicks(1), 2d), Value(atBound, 3d),
Value(new DateTime(2062, 1, 1, 0, 0, 0, DateTimeKind.Utc), 4d));
errors[0].StatusCode.Code.ShouldBe(StatusCodes.Good);
results[0].OperationResults.Select(c => c.Code).ShouldBe(
[
StatusCodes.GoodEntryInserted, StatusCodes.BadOutOfRange, StatusCodes.GoodEntryInserted,
StatusCodes.BadOutOfRange,
]);
updater.UpdateCalls.ShouldHaveSingleItem().Values.Select(v => v.TimestampUtc).ShouldBe([T1, atBound]);
}
finally { await host.DisposeAsync(); }
}
/// <summary>#674: every mode is bounded, and a request whose items are ALL out of range sends nothing.</summary>
[Theory]
[InlineData(PerformUpdateType.Insert)]
[InlineData(PerformUpdateType.Replace)]
[InlineData(PerformUpdateType.Update)]
public async Task Every_update_mode_is_bounded_and_an_all_refused_request_sends_nothing(PerformUpdateType type)
{
var (host, server) = await BootAsync();
try
{
var nm = server.NodeManager!;
var updater = new FakeUpdater();
nm.HistorianDataUpdater = updater;
nm.Clock = new FakeClock(Start);
WireAcl(nm, enabled: false, NodePermissions.HistoryUpdate);
var nodeId = Historized(nm, AddressSpaceRealm.Raw, RawPath);
// Default skew: one day.
var (results, _) = Update(nm, KeyIdentity(OpcUaDataPlaneRoles.HistoryUpdate), nodeId, type,
Value(Start.AddDays(1).AddSeconds(1), 1d));
results[0].OperationResults.Select(c => c.Code).ShouldBe([StatusCodes.BadOutOfRange]);
updater.UpdateCalls.ShouldBeEmpty();
}
finally { await host.DisposeAsync(); }
}
/// <summary>#674: a non-positive skew disables the bound — a far-future item reaches the backend.</summary>
[Fact]
public async Task A_non_positive_skew_disables_the_bound()
{
var (host, server) = await BootAsync();
try
{
var nm = server.NodeManager!;
var updater = new FakeUpdater();
nm.HistorianDataUpdater = updater;
nm.Clock = new FakeClock(Start);
nm.HistoryUpdateMaxFutureSkew = TimeSpan.Zero;
WireAcl(nm, enabled: false, NodePermissions.HistoryUpdate);
var nodeId = Historized(nm, AddressSpaceRealm.Raw, RawPath);
var farFuture = new DateTime(2062, 1, 1, 0, 0, 0, DateTimeKind.Utc);
var (results, _) = Insert(nm, KeyIdentity(OpcUaDataPlaneRoles.HistoryUpdate), nodeId, Value(farFuture, 1d));
results[0].OperationResults.Select(c => c.Code).ShouldBe([StatusCodes.GoodEntryInserted]);
updater.UpdateCalls.ShouldHaveSingleItem().Values.ShouldHaveSingleItem().TimestampUtc.ShouldBe(farFuture);
}
finally { await host.DisposeAsync(); }
}
// ───────────────────────────── #674 Primary gate ─────────────────────────────
/// <summary>
/// #674: off-Primary, an authorized HistoryUpdate is refused <c>BadInvalidState</c> per node — on every arm
/// (update data, delete raw, delete at time) — and NOTHING reaches the updater.
/// </summary>
[Fact]
public async Task Off_primary_every_arm_is_bad_invalid_state_and_nothing_is_sent()
{
var (host, server) = await BootAsync();
try
{
var nm = server.NodeManager!;
var updater = new FakeUpdater();
nm.HistorianDataUpdater = updater;
var consulted = 0;
nm.HistoryUpdatePrimaryGate = () => { Interlocked.Increment(ref consulted); return false; };
WireAcl(nm, enabled: false, NodePermissions.HistoryUpdate);
var nodeId = Historized(nm, AddressSpaceRealm.Raw, RawPath);
var identity = KeyIdentity(OpcUaDataPlaneRoles.HistoryUpdate);
var (results, errors) = Insert(nm, identity, nodeId, Value(T1, 1d));
var (rawResults, rawErrors) = Call(nm, identity, typeof(DeleteRawModifiedDetails),
new DeleteRawModifiedDetails { NodeId = nodeId, StartTime = T1, EndTime = T2 });
var (atResults, atErrors) = Call(nm, identity, typeof(DeleteAtTimeDetails),
new DeleteAtTimeDetails { NodeId = nodeId, ReqTimes = new DateTimeCollection { T1 } });
foreach (var (r, e) in new[] { (results, errors), (rawResults, rawErrors), (atResults, atErrors) })
{
e[0].StatusCode.Code.ShouldBe(StatusCodes.BadInvalidState);
r[0].StatusCode.Code.ShouldBe(StatusCodes.BadInvalidState);
}
consulted.ShouldBe(3);
updater.UpdateCalls.ShouldBeEmpty();
updater.DeleteRawCalls.ShouldBeEmpty();
updater.DeleteAtTimeCalls.ShouldBeEmpty();
}
finally { await host.DisposeAsync(); }
}
/// <summary>
/// #674: on the Primary the gate allows and the update REACHES the updater — the allow leg that makes the deny
/// leg meaningful (a gate that refused everything, or one never consulted, fails one of the two).
/// </summary>
[Fact]
public async Task On_primary_the_update_reaches_the_updater()
{
var (host, server) = await BootAsync();
try
{
var nm = server.NodeManager!;
var updater = new FakeUpdater();
nm.HistorianDataUpdater = updater;
var consulted = 0;
nm.HistoryUpdatePrimaryGate = () => { Interlocked.Increment(ref consulted); return true; };
WireAcl(nm, enabled: false, NodePermissions.HistoryUpdate);
var nodeId = Historized(nm, AddressSpaceRealm.Raw, RawPath);
var (results, errors) = Insert(nm, KeyIdentity(OpcUaDataPlaneRoles.HistoryUpdate), nodeId, Value(T1, 1d));
errors[0].StatusCode.Code.ShouldBe(StatusCodes.Good);
results[0].OperationResults.Select(c => c.Code).ShouldBe([StatusCodes.GoodEntryInserted]);
updater.UpdateCalls.ShouldHaveSingleItem().Tag.ShouldBe(Tagname);
consulted.ShouldBe(1);
}
finally { await host.DisposeAsync(); }
}
/// <summary>
/// #674: the Primary gate runs AFTER authorization — an unauthorized caller is told <c>BadUserAccessDenied</c>
/// (not the node's role) and the gate is never consulted.
/// </summary>
[Fact]
public async Task The_primary_gate_runs_after_authorization()
{
var (host, server) = await BootAsync();
try
{
var nm = server.NodeManager!;
var updater = new FakeUpdater();
nm.HistorianDataUpdater = updater;
var consulted = 0;
nm.HistoryUpdatePrimaryGate = () => { Interlocked.Increment(ref consulted); return false; };
WireAcl(nm, enabled: false, NodePermissions.WriteOperate);
var nodeId = Historized(nm, AddressSpaceRealm.Raw, RawPath);
var (results, _) = Insert(nm, KeyIdentity(OpcUaDataPlaneRoles.WriteOperate), nodeId, Value(T1, 1d));
results[0].StatusCode.Code.ShouldBe(StatusCodes.BadUserAccessDenied);
consulted.ShouldBe(0);
updater.UpdateCalls.ShouldBeEmpty();
}
finally { await host.DisposeAsync(); }
}
// ───────────────────────────── helpers ───────────────────────────── // ───────────────────────────── helpers ─────────────────────────────
private static DataValue Value(DateTime timestamp, object value) => private static DataValue Value(DateTime timestamp, object value) =>
@@ -199,11 +199,168 @@ public sealed class NodeManagerHistoryReadEventsPagingTests : IDisposable
var mixed = ReadBatch(nm, 4, null, (node, good1[0].Cp), (other, Guid.NewGuid().ToByteArray())); var mixed = ReadBatch(nm, 4, null, (node, good1[0].Cp), (other, Guid.NewGuid().ToByteArray()));
mixed[0].Error.StatusCode.Code.ShouldBe(StatusCodes.Good); mixed[0].Error.StatusCode.Code.ShouldBe(StatusCodes.Good);
mixed[0].Ids.ShouldBe(src.All.Skip(4).Take(3).Select(e => e.EventId)); // A resumed page is FULL (#675): it over-fetches by the boundary skip it then trims.
mixed[0].Ids.ShouldBe(src.All.Skip(4).Take(4).Select(e => e.EventId));
mixed[1].Error.StatusCode.Code.ShouldBe(StatusCodes.BadContinuationPointInvalid); mixed[1].Error.StatusCode.Code.ShouldBe(StatusCodes.BadContinuationPointInvalid);
await host.DisposeAsync(); await host.DisposeAsync();
} }
// --- Gitea #675: reverse (newest-first) event paging ------------------------------------------------------
[Fact]
public async Task Reverse_events_page_newest_first_exactly_once_with_exact_page_counts()
{
var (host, nm, node, src) = await SetupAsync(Distinct(250));
var pages = new List<int>();
var ids = DrainAsync(nm, node, max: 100, start: Late, end: Early, pageSizes: pages);
ids.ShouldBe(src.All.Reverse().Select(e => e.EventId));
pages.ShouldBe([100, 100, 50]);
await host.DisposeAsync();
}
[Fact]
public async Task Reverse_first_page_is_the_LATEST_events_with_a_continuation_point()
{
// The pre-#675 shape: the cap kept the EARLIEST N, ascending, Good, and no continuation point.
var (host, nm, node, _) = await SetupAsync(Distinct(10));
var p1 = ReadEvents(nm, node, max: 3, inboundCp: null, end: Early, start: Late);
p1.Error.StatusCode.Code.ShouldBe(StatusCodes.Good);
p1.Ids.ShouldBe(["e0009", "e0008", "e0007"]);
p1.Cp.ShouldNotBeNull("a capped reverse read has more behind it and must page");
await host.DisposeAsync();
}
[Fact]
public async Task Reverse_tie_at_page_boundary_is_neither_duplicated_nor_skipped()
{
var events = new List<HistoricalEvent>
{
Ev("e0", 0), Ev("e1", 1), Ev("e2", 2), Ev("e3", 2), Ev("e4", 2), Ev("e5", 3),
};
var (host, nm, node, _) = await SetupAsync(events);
// Newest-first, ties in descending EventId: cap 2 cuts the cluster at e4|e3.
DrainAsync(nm, node, max: 2, start: Late, end: Early).ShouldBe(["e5", "e4", "e3", "e2", "e1", "e0"]);
await host.DisposeAsync();
}
[Fact]
public async Task Reverse_tie_cluster_larger_than_the_cap_pages_within_the_cluster()
{
var events = new List<HistoricalEvent> { Ev("a0", 0) };
for (var i = 0; i < 7; i++) events.Add(Ev($"t{i}", 5));
events.Add(Ev("z0", 9));
var (host, nm, node, _) = await SetupAsync(events);
var pages = new List<int>();
DrainAsync(nm, node, max: 3, start: Late, end: Early, pageSizes: pages).ShouldBe(
["z0", "t6", "t5", "t4", "t3", "t2", "t1", "t0", "a0"]);
pages.Sum().ShouldBe(9);
await host.DisposeAsync();
}
[Fact]
public async Task Reverse_tie_cluster_pages_on_a_coarse_resolution_backend()
{
const long res = 10;
var events = new List<HistoricalEvent>();
DateTime At(int slot) => Epoch.AddTicks(slot * res);
events.Add(new("a0", "s", At(0), Epoch, "m", 500));
for (var i = 0; i < 7; i++) events.Add(new($"t{i}", "s", At(3), Epoch, "m", 500));
events.Add(new("z0", "s", At(8), Epoch, "m", 500));
var (host, nm, node, _) = await SetupAsync(events, res);
DrainAsync(nm, node, max: 3, start: Late, end: Early).ShouldBe(
["z0", "t6", "t5", "t4", "t3", "t2", "t1", "t0", "a0"]);
await host.DisposeAsync();
}
[Fact]
public async Task Reverse_tie_cluster_beyond_the_overfetch_bound_fails_loudly()
{
var events = new List<HistoricalEvent>();
for (var i = 0; i < 10; i++) events.Add(Ev($"t{i:00}", 5));
var (host, nm, node, _) = await SetupAsync(events);
nm.MaxTieClusterOverfetch = 4;
var p1 = ReadEvents(nm, node, max: 3, inboundCp: null, end: Early, start: Late);
p1.Cp.ShouldNotBeNull();
ReadEvents(nm, node, max: 3, inboundCp: p1.Cp, end: Early, start: Late).Error.StatusCode.Code
.ShouldBe(StatusCodes.BadHistoryOperationUnsupported);
await host.DisposeAsync();
}
[Fact]
public async Task A_reverse_cursor_resumes_reverse_whatever_the_resume_request_says()
{
// The direction lives in the cursor: a resume carrying FORWARD details still continues newest-first.
var (host, nm, node, _) = await SetupAsync(Distinct(10));
var p1 = ReadEvents(nm, node, max: 4, inboundCp: null, end: Early, start: Late);
p1.Ids.ShouldBe(["e0009", "e0008", "e0007", "e0006"]);
var p2 = ReadEvents(nm, node, max: 4, inboundCp: p1.Cp); // default details: Early -> Late (forward)
p2.Error.StatusCode.Code.ShouldBe(StatusCodes.Good);
p2.Ids.ShouldBe(["e0005", "e0004", "e0003", "e0002"]);
await host.DisposeAsync();
}
[Fact]
public async Task A_forward_cursor_resumes_forward_whatever_the_resume_request_says()
{
var (host, nm, node, _) = await SetupAsync(Distinct(10));
var p1 = ReadEvents(nm, node, max: 4, inboundCp: null);
p1.Ids.ShouldBe(["e0000", "e0001", "e0002", "e0003"]);
var p2 = ReadEvents(nm, node, max: 4, inboundCp: p1.Cp, end: Early, start: Late);
p2.Ids.ShouldBe(["e0004", "e0005", "e0006", "e0007"]);
await host.DisposeAsync();
}
[Fact]
public async Task Reverse_unbounded_read_returns_everything_newest_first_without_paging()
{
var (host, nm, node, src) = await SetupAsync(Distinct(30));
var r = ReadEvents(nm, node, max: 0, inboundCp: null, end: Early, start: Late);
r.Ids.ShouldBe(src.All.Reverse().Select(e => e.EventId));
r.Cp.ShouldBeNull();
await host.DisposeAsync();
}
[Fact]
public async Task End_only_events_read_pages_backward_from_EndTime()
{
// StartTime unset + EndTime + NumValuesPerNode > 0 => backward from EndTime (Part 11; mirrors raw). The reverse
// window is (.., EndTime], so an event exactly AT EndTime is the first one returned.
var (host, nm, node, _) = await SetupAsync(Distinct(25));
var ids = DrainAsync(nm, node, max: 10, start: DateTime.MinValue, end: Epoch.AddSeconds(20));
ids.ShouldBe(Enumerable.Range(0, 21).Reverse().Select(i => $"e{i:0000}"));
await host.DisposeAsync();
}
[Fact]
public async Task A_backend_without_reverse_support_answers_BadHistoryOperationUnsupported_not_the_earliest_events()
{
var (host, nm, node, src) = await SetupAsync(Distinct(10));
nm.HistorianDataSource = new ForwardOnlyEventSource(src);
ReadEvents(nm, node, max: 3, inboundCp: null, end: Early, start: Late).Error.StatusCode.Code
.ShouldBe(StatusCodes.BadHistoryOperationUnsupported);
// Forward reads on the same backend are unaffected.
DrainAsync(nm, node, max: 4).ShouldBe(src.All.Select(e => e.EventId));
await host.DisposeAsync();
}
private static readonly DateTime Early = Epoch.AddDays(-1);
private static readonly DateTime Late = Epoch.AddDays(1);
/// <summary>A backend that implements only the five-argument (forward) event read - Gateway / Null shaped.</summary>
private sealed class ForwardOnlyEventSource(IHistorianDataSource inner) : IHistorianDataSource
{
public Task<HistoricalEventsResult> ReadEventsAsync(string? n, DateTime s, DateTime e, int m, CancellationToken ct) =>
inner.ReadEventsAsync(n, s, e, m, ct);
public Task<HistorianRead> ReadRawAsync(string r, DateTime s, DateTime e, uint m, CancellationToken ct) =>
inner.ReadRawAsync(r, s, e, m, ct);
public Task<HistorianRead> ReadProcessedAsync(string r, DateTime s, DateTime e, TimeSpan i, HistoryAggregateType a, CancellationToken ct) =>
inner.ReadProcessedAsync(r, s, e, i, a, ct);
public Task<HistorianRead> ReadAtTimeAsync(string r, IReadOnlyList<DateTime> t, CancellationToken ct) =>
inner.ReadAtTimeAsync(r, t, ct);
public HistorianHealthSnapshot GetHealthSnapshot() => inner.GetHealthSnapshot();
public void Dispose() { }
}
private sealed class RawSeriesSource : IHistorianDataSource private sealed class RawSeriesSource : IHistorianDataSource
{ {
public Task<HistorianRead> ReadRawAsync(string r, DateTime s, DateTime e, uint m, CancellationToken ct) public Task<HistorianRead> ReadRawAsync(string r, DateTime s, DateTime e, uint m, CancellationToken ct)
@@ -265,16 +422,18 @@ public sealed class NodeManagerHistoryReadEventsPagingTests : IDisposable
return (started.Host, nm, nm.TryGetFolder(equipmentId)!.NodeId, src); return (started.Host, nm, nm.TryGetFolder(equipmentId)!.NodeId, src);
} }
private static List<string> DrainAsync(OtOpcUaNodeManager nm, NodeId node, uint max, DateTime? end = null) private static List<string> DrainAsync(
OtOpcUaNodeManager nm, NodeId node, uint max, DateTime? end = null, DateTime? start = null, List<int>? pageSizes = null)
{ {
var all = new List<string>(); var all = new List<string>();
byte[]? cp = null; byte[]? cp = null;
var pages = 0; var pages = 0;
do do
{ {
var r = ReadEvents(nm, node, max, cp, end); var r = ReadEvents(nm, node, max, cp, end, start);
r.Error.StatusCode.Code.ShouldBe(StatusCodes.Good); r.Error.StatusCode.Code.ShouldBe(StatusCodes.Good);
all.AddRange(r.Ids); all.AddRange(r.Ids);
pageSizes?.Add(r.Ids.Count);
cp = r.Cp; cp = r.Cp;
(++pages).ShouldBeLessThan(60, "paging must terminate"); (++pages).ShouldBeLessThan(60, "paging must terminate");
} }
@@ -283,11 +442,15 @@ public sealed class NodeManagerHistoryReadEventsPagingTests : IDisposable
} }
private static (List<string> Ids, ServiceResult Error, byte[]? Cp) ReadEvents( private static (List<string> Ids, ServiceResult Error, byte[]? Cp) ReadEvents(
OtOpcUaNodeManager nm, NodeId node, uint max, byte[]? inboundCp, DateTime? end = null) => OtOpcUaNodeManager nm, NodeId node, uint max, byte[]? inboundCp, DateTime? end = null, DateTime? start = null) =>
ReadBatch(nm, max, end, (node, inboundCp))[0]; ReadBatch(nm, max, end, start, (node, inboundCp))[0];
private static List<(List<string> Ids, ServiceResult Error, byte[]? Cp)> ReadBatch( private static List<(List<string> Ids, ServiceResult Error, byte[]? Cp)> ReadBatch(
OtOpcUaNodeManager nm, uint max, DateTime? end, params (NodeId Node, byte[]? Cp)[] reads) OtOpcUaNodeManager nm, uint max, DateTime? end, params (NodeId Node, byte[]? Cp)[] reads) =>
ReadBatch(nm, max, end, null, reads);
private static List<(List<string> Ids, ServiceResult Error, byte[]? Cp)> ReadBatch(
OtOpcUaNodeManager nm, uint max, DateTime? end, DateTime? start, params (NodeId Node, byte[]? Cp)[] reads)
{ {
var context = new OperationContext( var context = new OperationContext(
new RequestHeader(), secureChannelContext: null, RequestType.HistoryRead, identity: null); new RequestHeader(), secureChannelContext: null, RequestType.HistoryRead, identity: null);
@@ -298,7 +461,7 @@ public sealed class NodeManagerHistoryReadEventsPagingTests : IDisposable
}); });
var details = new ReadEventDetails var details = new ReadEventDetails
{ {
StartTime = Epoch.AddDays(-1), StartTime = start ?? Epoch.AddDays(-1),
EndTime = end ?? Epoch.AddDays(1), EndTime = end ?? Epoch.AddDays(1),
NumValuesPerNode = max, NumValuesPerNode = max,
Filter = filter, Filter = filter,
@@ -361,6 +524,22 @@ public sealed class NodeManagerHistoryReadEventsPagingTests : IDisposable
window.Take(cap).ToList(), truncated ? new byte[] { 1 } : null)); window.Take(cap).ToList(), truncated ? new byte[] { 1 } : null));
} }
/// <summary>Like the SQL backends' #675 overload: reverse => <c>(lo, hi]</c> newest-first by (time, id) descending.</summary>
public Task<HistoricalEventsResult> ReadEventsAsync(
string? sourceName, DateTime startUtc, DateTime endUtc, int maxEvents, bool reverse, CancellationToken ct)
{
if (!reverse) return ReadEventsAsync(sourceName, startUtc, endUtc, maxEvents, ct);
Calls++;
var (lo, hi) = startUtc <= endUtc ? (startUtc, endUtc) : (endUtc, startUtc);
lo = Trunc(lo);
hi = Trunc(hi);
var window = All.Where(e => e.EventTimeUtc > lo && e.EventTimeUtc <= hi).Reverse().ToList();
var cap = maxEvents <= 0 ? 1000 : maxEvents;
var truncated = window.Count > cap;
return Task.FromResult(new HistoricalEventsResult(
window.Take(cap).ToList(), truncated ? new byte[] { 1 } : null));
}
public Task<HistorianRead> ReadRawAsync(string r, DateTime s, DateTime e, uint m, CancellationToken ct) => public Task<HistorianRead> ReadRawAsync(string r, DateTime s, DateTime e, uint m, CancellationToken ct) =>
NullHistorianDataSource.Instance.ReadRawAsync(r, s, e, m, ct); NullHistorianDataSource.Instance.ReadRawAsync(r, s, e, m, ct);
public Task<HistorianRead> ReadProcessedAsync( public Task<HistorianRead> ReadProcessedAsync(
@@ -203,6 +203,28 @@ public sealed class NodeManagerHistoryReadModifiedTests : IDisposable
await host.DisposeAsync(); await host.DisposeAsync();
} }
/// <summary>
/// #674: the HistoryUpdate Primary gate does not reach HistoryRead — a closed gate neither refuses a read nor is
/// consulted by one (reads are served from any redundancy node), for modified and raw reads alike.
/// </summary>
[Fact]
public async Task The_history_update_primary_gate_is_not_consulted_for_reads()
{
var (host, nm, node, src) = await SetupAsync([Mod(1, 1, 1, HistoryModificationType.Update, "u", Epoch)]);
var consulted = 0;
nm.HistoryUpdatePrimaryGate = () => { Interlocked.Increment(ref consulted); return false; };
var modified = Read(nm, node, Epoch, Epoch.AddHours(1), max: 10, cp: null);
var raw = Read(nm, node, Epoch, Epoch.AddHours(1), max: 10, cp: null, modified: false);
modified.Error.StatusCode.Code.ShouldBe(StatusCodes.Good);
modified.Data!.DataValues.Count.ShouldBe(1);
raw.Error.StatusCode.Code.ShouldNotBe(StatusCodes.BadInvalidState);
src.Calls.ShouldBe(1);
consulted.ShouldBe(0);
await host.DisposeAsync();
}
[Fact] [Fact]
public async Task A_backend_failure_is_isolated_to_its_node() public async Task A_backend_failure_is_isolated_to_its_node()
{ {
@@ -165,6 +165,7 @@ public sealed class NodeManagerHistoryReadTests : IDisposable
StartTime = DateTime.UtcNow.AddHours(-1), StartTime = DateTime.UtcNow.AddHours(-1),
EndTime = DateTime.UtcNow, EndTime = DateTime.UtcNow,
NumValuesPerNode = 10, NumValuesPerNode = 10,
IsReadModified = false, // SDK default is true; this test is about the raw path
}; };
var (_, errors) = InvokeHistoryRead(server, nm, details, nodeId); var (_, errors) = InvokeHistoryRead(server, nm, details, nodeId);
@@ -223,10 +223,30 @@ public sealed class DriverHostActorHistorizationRoleViewTests : RuntimeActorTest
host.Tell(new ClusterEvent.UnreachableMember(MemberAt(Peer672))); host.Tell(new ClusterEvent.UnreachableMember(MemberAt(Peer672)));
AwaitAssert(() => hist.Published.ShouldBe([false, true]), duration: Timeout); AwaitAssert(() => hist.Published.ShouldBe([false, true]), duration: Timeout);
// And self "returning" does not close what the peer's loss opened. // Self "returning" is ignored too, and it does not close what the peer's loss opened. Events are
// processed in order, so the peer's return (which MUST close the gate) is the only thing that may
// appear next: had the self event published anything it would sit between `true` and `false`.
host.Tell(new ClusterEvent.ReachableMember(MemberAt(Self672))); host.Tell(new ClusterEvent.ReachableMember(MemberAt(Self672)));
host.Tell(new ClusterEvent.ReachableMember(MemberAt(Peer672)));
AwaitAssert(() => hist.Published.ShouldBe([false, true, false]), duration: Timeout);
}
/// <summary>
/// A host that starts while its peer is ALREADY unreachable (the <c>InitialStateAsEvents</c> replay
/// delivers the <c>UnreachableMember</c> before the first redundancy snapshot) must still treat that
/// peer as unreachable when the snapshot naming it Primary finally arrives.
/// </summary>
[Fact]
public void A_peer_unreachable_before_the_first_snapshot_still_opens_capture()
{
var hist = new RecordingHistorizationView();
IActorRef host = SpawnHost672(hist, others: null, captureWhilePrimaryUnreachable: true);
host.Tell(new ClusterEvent.UnreachableMember(MemberAt(Peer672)));
host.Tell(Snapshot((Self672, RedundancyRole.Secondary), (Peer672, RedundancyRole.Primary))); host.Tell(Snapshot((Self672, RedundancyRole.Secondary), (Peer672, RedundancyRole.Primary)));
AwaitAssert(() => hist.Published.ShouldBe([false, true, true]), duration: Timeout);
// Exactly one publication, and it is `true`: no spurious `false` from treating the peer as reachable.
AwaitAssert(() => hist.Published.ShouldBe([true]), duration: Timeout);
} }
/// <summary><c>CaptureWhilePrimaryUnreachable = false</c> restores the pre-#672 behaviour.</summary> /// <summary><c>CaptureWhilePrimaryUnreachable = false</c> restores the pre-#672 behaviour.</summary>
@@ -1,3 +1,4 @@
using Microsoft.Extensions.Configuration;
using Xunit; using Xunit;
using ZB.MOM.WW.OtOpcUa.Runtime.Historian; using ZB.MOM.WW.OtOpcUa.Runtime.Historian;
@@ -99,6 +100,29 @@ public sealed class ServerHistorianOptionsTests
Assert.Contains(w, m => m.Contains("MaxTimeWeightedRawRows")); Assert.Contains(w, m => m.Contains("MaxTimeWeightedRawRows"));
} }
/// <summary>
/// #674: the HistoryUpdate future-skew bound defaults to one day and binds from configuration as a TimeSpan;
/// a non-positive value is a deliberate "disabled", not a misconfiguration, so it does not warn.
/// </summary>
[Fact]
public void HistoryUpdateMaxFutureSkew_defaults_to_one_day_binds_and_zero_is_silent()
{
Assert.Equal(TimeSpan.FromDays(1), new ServerHistorianOptions().HistoryUpdateMaxFutureSkew);
var bound = new ConfigurationBuilder()
.AddInMemoryCollection(new Dictionary<string, string?>
{
["ServerHistorian:HistoryUpdateMaxFutureSkew"] = "00:05:00",
})
.Build()
.GetSection(ServerHistorianOptions.SectionName)
.Get<ServerHistorianOptions>();
Assert.Equal(TimeSpan.FromMinutes(5), bound!.HistoryUpdateMaxFutureSkew);
var w = new ServerHistorianOptions
{ Enabled = true, Endpoint = "https://h:5222", ApiKey = "histgw_x_y", HistoryUpdateMaxFutureSkew = TimeSpan.Zero }
.Validate();
Assert.Empty(w);
}
[Fact] [Fact]
public void Disabled_with_nonpositive_HistoryRead_knobs_is_silent() public void Disabled_with_nonpositive_HistoryRead_knobs_is_silent()
{ {