diff --git a/CLAUDE.md b/CLAUDE.md index d2cbe1c4..dbfdb79c 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1348,7 +1348,7 @@ historian tagname, which is the tag's driver `FullName`. - ⚠️ **Reverse processed reads are Part 13/SDK-anchored (#677, client-visible):** every aggregate on SqlServer/QuestDB now anchors a reverse read at its later time — window `(end, start]`, partial interval earliest, each stamped with its later edge (was: earlier-anchored, stamped earlier edge); Partial also marks the interval the data starts inside (not Average); a non-default client `AggregateConfiguration` ⇒ `BadAggregateConfigurationRejected`; Gateway provisioning sends `IntegralDivisor = 1`. SDK-parity tests: `SdkAggregateParityTests`. `docs/Historian.md` §Aggregates. - **Real status fix:** the recorder used to stamp every sample Good (`192`); it now carries the driver's status. ⚠️ On the **Gateway** backend this changes nothing stored: the gateway discards quality (C-002), so AVEVA still stores Good. - **Comms loss is historized as an outage marker** (`Empty` + `BadNoCommunication`, one per tag per outage) on SqlServer; dropped (`DroppedUnsupportedKind`) on Gateway. Outbox records are **format v2** — drain or delete `ContinuousHistorization:OutboxPath` before downgrading a node. -- **HistoryUpdate** (SqlServer only; Gateway ⇒ `BadHistoryOperationUnsupported`): Insert/Replace/Update/DeleteRaw/DeleteAtTime, gated in both ACL modes — flag off needs the `HistoryUpdate` role from a **Cluster-scoped** grant (+ key re-check), flag on uses `DataPlaneAuthorizer`'s HistoryUpdate operation; `WriteOperate` never implies it. The `HistoryWrite` AccessLevel bit is set only when an updater is wired at materialise time. `docs/Historian.md` §HistoryUpdate, `docs/security.md`. +- **HistoryUpdate** (SqlServer + QuestDb; Gateway ⇒ `BadHistoryOperationUnsupported`): Insert/Replace/Update/DeleteRaw/DeleteAtTime, **Primary-only** since #674 (off-Primary ⇒ `BadInvalidState`, after authorization, same `IDriverPrimaryRoleSource` verdict as the native-ack gate; reads ungated), items with a `SourceTimestamp` past now + `ServerHistorian:HistoryUpdateMaxFutureSkew` (default 1 day, ≤0 disables) ⇒ `BadOutOfRange` in the node manager (every backend), gated in both ACL modes — flag off needs the `HistoryUpdate` role from a **Cluster-scoped** grant (+ key re-check), flag on uses `DataPlaneAuthorizer`'s HistoryUpdate operation; `WriteOperate` never implies it. The `HistoryWrite` AccessLevel bit is set only when an updater is wired at materialise time. `docs/Historian.md` §HistoryUpdate, `docs/security.md`. ⚠️ **SqlServer schema v1 is accepted (#674)** with `AutoCreateSchema=false`: writes/reads/events/retention work; modified reads + HistoryUpdate ⇒ `BadHistoryOperationUnsupported` (nothing sent), retention skips `SampleModified`, one Warning per boot (`SqlServerSchemaBootstrapper.SupportsModifiedTrail`; `RequiredVersion` = 1, `ModifiedTrailVersion` = 2). QuestDB: keep `cairo.ttl.use.wall.clock=true`; a QuestDB trail INSERT retried after a client timeout duplicates (no DEDUP key). - **Modified-value history (#668, Part 11):** every applied HistoryUpdate item leaves a trail row (prior value, principal from the session's API key / `anonymous`, `HistoryModificationType`, time) — SqlServer `hist.SampleModified` (**schema v2**, written by `OUTPUT … INTO` in the same statement ⇒ atomic; ⚠️ no FK/CHECK/trigger on it, and `AutoCreateSchema=false` installs must run `002_sample_modified.sql` before upgrading), QuestDB `{p}sample_modified` (append-only, written **before** the change ⇒ a crash leaves an orphan trail row, never an untracked change). `IsReadModified=true` HistoryRead serves it as `HistoryModifiedData` via `IHistorianDataSource.ReadModifiedAsync` (default ⇒ `NotSupported` ⇒ `BadHistoryOperationUnsupported` on Gateway/Null), paged on `(SourceTime, ModifiedId)`, gated by plain HistoryRead. Live writes and retention never write it; retention/TTL purge it on source time. `IHistorianDataUpdater` methods take a `HistoryUpdateContext`. CLI `historyread --modified`. `docs/Historian.md` §Modified-value history. ### Read path (`ServerHistorian` section) diff --git a/docker-dev/README.md b/docker-dev/README.md index 6dbdbcb6..fc2e8957 100644 --- a/docker-dev/README.md +++ b/docker-dev/README.md @@ -207,7 +207,9 @@ The `-v` drops the SQL volume; remove it to keep ConfigDb state across restarts. The full single-mesh stack (`central-1`/`central-2` + the four site nodes) can OOM-kill `central-1` on a loaded host. Two settings in the compose file guard against that: -- **EF Core + ASP.NET Core logs are pinned to `Warning`** on every host node (`Serilog__MinimumLevel__Override__Microsoft.EntityFrameworkCore` / `…Microsoft.AspNetCore` = `Warning`). The host logs via Serilog (`AddZbSerilog` → `ReadFrom.Configuration`), and in `Development` the default level is `Debug` — without these overrides every Deployment-poll emits an `Executed DbCommand` / `SELECT … FROM [Deployment]` line, flooding the Serilog pipeline and starving the Akka cluster heartbeat thread. Application + Akka log levels are left untouched, so this only silences the per-poll SQL chatter. To temporarily restore the SQL log flood for debugging, drop those two env vars (or set them back to `Information`) on the node you're inspecting. +- **EF Core + ASP.NET Core logs are pinned to `Warning`** on every host node (`Serilog__MinimumLevel__Override__Microsoft.EntityFrameworkCore` / `…Microsoft.AspNetCore` = `Warning`). The host logs via Serilog (`AddZbSerilog` → `ReadFrom.Configuration`), and a `Development` environment would default to `Debug` (`appsettings.Development.json`) — the rig does NOT set `ASPNETCORE_ENVIRONMENT`, so nodes actually run at the base `Information` default, but these pins keep the SQL chatter out should the rig ever run as Development; without them a Debug/Information-level EF pipeline emits an `Executed DbCommand` / `SELECT … FROM [Deployment]` line, flooding the Serilog pipeline and starving the Akka cluster heartbeat thread. Application + Akka log levels are left untouched, so this only silences the per-poll SQL chatter. To temporarily restore the SQL log flood for debugging, drop those two env vars (or set them back to `Information`) on the node you're inspecting. + + To raise Debug logging for chosen categories on `central-1`/`central-2` (e.g. `ZB.MOM.WW.OtOpcUa.Runtime`), use the overlay: `docker compose -f docker-dev/docker-compose.yml -f docker-dev/docker-compose.debug.yml up -d --no-deps central-1 central-2` — it sets `Serilog__MinimumLevel__Override__=Debug`; edit its category list to taste. - **Each host node has `mem_limit: 1g`** (`mem_reservation: 512m`). A quiet solo `central-1` measures ~357 MiB; the limit leaves headroom for the deploy/UI load and per-cluster driver subscriptions that push a fully-loaded node higher. The limit/reservation live on the `&otopcua-host` anchor, so all six host services inherit them; `sql`, `traefik`, and the one-shot `migrator`/`cluster-seed` are left unbounded. The full six-node host stack therefore needs roughly **6 GiB** of Docker Desktop VM memory just for the host nodes (plus SQL Server's own footprint on top). On a constrained host, either raise the Docker Desktop VM memory or run fewer host services (e.g. just `central-1` + `central-2`, or a single central node) rather than the full mesh. diff --git a/docker-dev/docker-compose.debug.yml b/docker-dev/docker-compose.debug.yml new file mode 100644 index 00000000..6f196d80 --- /dev/null +++ b/docker-dev/docker-compose.debug.yml @@ -0,0 +1,26 @@ +# docker-dev/docker-compose.debug.yml — optional overlay: raise Serilog levels per category on the central pair. +# +# The rig nodes run with ASPNETCORE_ENVIRONMENT unset (Production), so there is NO appsettings.Development.json and the +# Serilog default level is Information (the base appsettings.json sets none). Debug is therefore OFF by default; this +# overlay turns it on for chosen namespaces only, via Serilog:MinimumLevel:Override: (env: double underscore +# for ':', and the namespace's own dots stay dots). +# +# Up (recreates only the two central nodes with the extra env): +# docker compose -f docker-dev/docker-compose.yml -f docker-dev/docker-compose.debug.yml up -d --no-deps central-1 central-2 +# Back to normal: re-run `up -d --no-deps central-1 central-2` with the base file alone. +# +# Edit the list below to the categories you care about. Useful ones: +# ZB.MOM.WW.OtOpcUa.Runtime actors: DriverHostActor, OpcUaPublishActor, redundancy, deploys +# ZB.MOM.WW.OtOpcUa.Runtime.Drivers.DriverHostActor one actor only (e.g. the #672 historization verdict logs) +# ZB.MOM.WW.OtOpcUa.Driver.OpcUaClient a single driver +# Opc.Ua OPC Foundation stack (very chatty) +# Akka cluster/remoting internals (chatty; base is Information) +# Leave Microsoft.EntityFrameworkCore / Microsoft.AspNetCore at Warning (see README "Resource limits & dev logging"): +# a Debug SQL flood starves the Akka heartbeat thread. +services: + central-1: + environment: + Serilog__MinimumLevel__Override__ZB.MOM.WW.OtOpcUa.Runtime: "Debug" + central-2: + environment: + Serilog__MinimumLevel__Override__ZB.MOM.WW.OtOpcUa.Runtime: "Debug" diff --git a/docs/Client.CLI.md b/docs/Client.CLI.md index 3729023e..76d1b212 100644 --- a/docs/Client.CLI.md +++ b/docs/Client.CLI.md @@ -57,6 +57,11 @@ otopcua-cli write -u opc.tcp://localhost:4840 -n "ns=2;s=MyNode" -v 42 -U operat ### Failover +**Mapped ports and internal hostnames.** Endpoint discovery rewrites the host *and port* of the endpoint the server +advertises to the URL you passed (when the scheme matches), so connecting through a port mapping works even though the +server advertises its internal address (for example the docker-dev rig: `-u opc.tcp://localhost:4841` reaches the +node mapped to 4841, although every container advertises `:4840`). + When `-F` is provided, the shared service tries the primary URL first, then each failover URL in order. For long-running commands (`subscribe`, `alarms`), the service monitors the session via keep-alive and automatically reconnects to the next available server on failure. ```bash @@ -229,7 +234,7 @@ continuation points up to `--max`. A server on the `Gateway` backend (no modific | `--start` | Start time in ISO 8601 UTC format, e.g. `2026-01-15T08:00:00Z` (default: 24 hours ago) | | `--end` | End time in ISO 8601 UTC format, e.g. `2026-01-15T09:00:00Z` (default: now) | | `--max` | Maximum number of values (default: 1000) | -| `--aggregate` | Aggregate function: Average, Minimum, Maximum, Count, Start, End, StandardDeviation | +| `--aggregate` | Aggregate function: Average, Minimum, Maximum, Count, Start, End, StandardDeviation, TimeAverage, TimeAverage2, Total, Total2, Interpolative | | `--interval` | Processing interval in milliseconds for aggregates (default: 3600000) | | `--modified` | Read modified-value history (Part 11 `IsReadModified`) instead of raw samples | @@ -270,8 +275,9 @@ node-level code (`Good`, or `BadNoData` if the range held nothing). A node that updater, so no `HistoryWrite` access bit), answers `BadHistoryOperationUnsupported`; a `NaN`/`Infinity` double is `BadInvalidArgument`. The session's API key (`-U apikey -P otua__`) needs the `HistoryUpdate` permission — via a `Cluster`-scoped grant when `Security:Acl:Enabled` is off, or a scoped grant on the node when it is on; a key with only `WriteOperate` gets `BadUserAccessDenied`. See [Historian.md §HistoryUpdate](Historian.md#historyupdate) and -[security.md](security.md#historyupdate-historian-writes). `historyread --aggregate` only knows the seven names in the table below; the SQL backend's other -aggregates (Total, Range, Delta, PercentGood, …) need a different client. +[security.md](security.md#historyupdate-historian-writes). `historyread --aggregate` only knows the twelve names in the table below; the SQL backend's other +aggregates (Range, Delta, PercentGood, …) need a different client. A refused history read (for example +`BadHistoryOperationUnsupported`) prints one error line and exits non-zero; it does not dump a stack trace. `delete-raw` is sent with `IsDeleteModified=false`. Invalid or inconsistent options (unknown mode, unpaired value/time, unparsable value, options that do not apply to the mode) fail before any connection is made. @@ -287,6 +293,11 @@ value/time, unparsable value, options that do not apply to the mode) fail before | `Start` | `first` | `AggregateFunction_Start` | | `End` | `last` | `AggregateFunction_End` | | `StandardDeviation` | `stddev`, `stdev` | `AggregateFunction_StandardDeviationPopulation` | +| `TimeAverage` | `timeavg` | `AggregateFunction_TimeAverage` | +| `TimeAverage2` | `timeavg2` | `AggregateFunction_TimeAverage2` | +| `Total` | | `AggregateFunction_Total` | +| `Total2` | | `AggregateFunction_Total2` | +| `Interpolative` | `interp` | `AggregateFunction_Interpolative` | ### alarms diff --git a/docs/Client.UI.md b/docs/Client.UI.md index 02497456..8a235fac 100644 --- a/docs/Client.UI.md +++ b/docs/Client.UI.md @@ -206,7 +206,7 @@ All times are in UTC. Invalid input turns red on blur. | Option | Description | |--------|-------------| -| Aggregate | Raw (default), Average, Minimum, Maximum, Count, Start, End, Standard Deviation | +| Aggregate | Raw (default), Average, Minimum, Maximum, Count, Start, End, Standard Deviation, Time Average, Time Average 2, Total, Total 2, Interpolative | | Interval (ms) | Processing interval for aggregate queries (shown only for aggregates) | | Max Values | Maximum number of raw values to return (default 1000) | diff --git a/docs/Configuration.md b/docs/Configuration.md index 6e0bb3fb..90505323 100644 --- a/docs/Configuration.md +++ b/docs/Configuration.md @@ -22,14 +22,30 @@ Two related concerns get their own dedicated pages and are **only summarised + l The Host (`Program.cs`) loads `appsettings.json`, then overlays a **per-role** file chosen from the cluster roles: -- A single role → `appsettings.{role}.json` (e.g. `appsettings.driver.json`, `appsettings.admin.json`). -- Both roles → `appsettings.admin-driver.json` (roles joined with `-`, ordinal-sorted). +- A single plain role → `appsettings.{role}.json` (e.g. `appsettings.driver.json`, `appsettings.admin.json`). +- Both roles → `appsettings.admin-driver.json` (plain roles joined with `-`, ordinal-sorted). +- `cluster-{ClusterId}` roles are **ignored** when naming the overlay (#678.6) — `[driver, cluster-SITE-A]` loads `appsettings.driver.json`. - `appsettings.{ASPNETCORE_ENVIRONMENT}.json` (e.g. `appsettings.Development.json`) is layered on by the host builder. All role overlays are **optional** — the base `appsettings.json` plus the Options-class C# defaults are enough to boot. The roles themselves come from `EffectiveRoles.Resolve` (`ZB.MOM.WW.OtOpcUa.Cluster`) — `Cluster:Roles`, falling back to `OTOPCUA_ROLES` when that is empty (see [`ServiceHosting.md`](ServiceHosting.md) and the table below). The checked-in `appsettings*.json` files are deliberately thin: they carry only `Serilog` and the `Security:Ldap` overlay. Everything else (`OpcUa`, `Cluster`, `ConnectionStrings`/`ConfigDb`) binds from the Options-class defaults documented below unless an operator adds the section explicitly or supplies the corresponding environment variable. + +#### Role overlay audit (#678.6) + +Until #678.6 the overlay never loaded on a node carrying a `cluster-*` role (the cluster role leaked into the filename). Fixing that **activates** the shipped overlays on those nodes (the docker-dev rig and any per-cluster-mesh production node), so each key was audited against base `appsettings.json` / the Options defaults. Env vars and command-line args are re-appended after the overlay, so any value a deployment sets explicitly (the rig sets every `Security__Ldap__*` key) still wins. + +| Key | Overlay(s) | Base / default | Effect on activation | +|---|---|---|---| +| `Serilog:MinimumLevel:Default` = `Information` | all three | Serilog default `Information` | no-op (in `Development` it outranks `appsettings.Development.json`'s `Debug`, by design) | +| `…Override:Akka` = `Information` | all three | inherits `Default` (`Information`) | no-op | +| `…Override:Microsoft.AspNetCore` = `Warning` | `admin`, `admin-driver` | unset (`Information`) | quieter ASP.NET Core logs; log-level only, matches what the rig already pins | +| `…Override:Opc.Ua` = `Information` | `admin-driver`, `driver` | inherits `Default` | no-op. **`driver` shipped `Debug` — changed to `Information` in #678.6**: activating a Debug OPC UA stack trace on every production driver node would flood Serilog (the same pipeline starvation the rig's EF/AspNetCore pins guard against). Raise it per node with `Serilog__MinimumLevel__Override__Opc.Ua=Debug`. | +| `Security:Ldap:DevStubMode` = `false` | all three | `false` | no-op; the overlay's point is that it outranks a developer's `appsettings.Development.json` (`true`) | +| `Security:Ldap:Transport` = `Ldaps` | all three | `LdapOptions.Transport` default `None` | **behaviour-changing, kept deliberately** (documented secure production default). A cluster-role node that configures `Security:Ldap:Server`/`Port` for a plaintext LDAP but never sets `Transport` would now dial LDAPS and fail AdminUI sign-in; set `Security__Ldap__Transport` (the rig already sets `None` explicitly). Data-plane auth is API-key based and unaffected; driver-only nodes have no AdminUI. | + + --- ## `appsettings` sections @@ -38,7 +54,7 @@ The checked-in `appsettings*.json` files are deliberately thin: they carry only - **Purpose:** logging. Console + rolling daily file sink, layered with the shared `ZB.MOM.WW.Telemetry` enrichers (`AddZbSerilog` in `Program.cs`). - **Where bound:** `builder.AddZbSerilog(...)` reads `Serilog` from configuration (`ReadFrom.Configuration`). -- **Checked-in shape** (`appsettings.json`): `Using` = `[ "Serilog.Sinks.Console", "Serilog.Sinks.File" ]`, `WriteTo` = a `Console` sink and a `File` sink (`path: logs/otopcua-.log`, `rollingInterval: Day`). Role overlays add `MinimumLevel` / `Override` blocks (e.g. `Opc.Ua: Debug`, `Akka: Information`). +- **Checked-in shape** (`appsettings.json`): `Using` = `[ "Serilog.Sinks.Console", "Serilog.Sinks.File" ]`, `WriteTo` = a `Console` sink and a `File` sink (`path: logs/otopcua-.log`, `rollingInterval: Day`). Role overlays add `MinimumLevel` / `Override` blocks (e.g. `Opc.Ua: Information`, `Akka: Information`). ### `OpcUa` @@ -410,7 +426,7 @@ Validated at host start (driver-role nodes) when `ServerHistorian:Enabled` **or* | `ServerHistorian:SqlServer:ConnectionString` | `""` | **Required.** Historian database connection string (not the ConfigDb). Supply via `${secret:OT-HISTORIAN-SQL}` or env `ServerHistorian__SqlServer__ConnectionString`; the database must already exist. | | `ServerHistorian:SqlServer:Schema` | `hist` | Must match `^[A-Za-z_][A-Za-z0-9_]{0,127}$` (spliced into DDL/DML). | | `ServerHistorian:SqlServer:CommandTimeout` | `00:00:30` | Must be `> 0`; rounded up to whole seconds. | -| `ServerHistorian:SqlServer:AutoCreateSchema` | `true` | `true` applies the embedded schema script under an app-lock on first use; `false` only verifies `SchemaVersion >= 1` and fails every historian call (health `LastError`, no crash) if behind. | +| `ServerHistorian:SqlServer:AutoCreateSchema` | `true` | `true` applies the embedded schema scripts under an app-lock on first use (upgrading a v1 schema to v2); `false` only verifies `SchemaVersion >= 1` and fails every historian call (health `LastError`, no crash) if behind. A verified **v1** schema (no `002_sample_modified.sql`) is served with modified-value reads and HistoryUpdate refused `BadHistoryOperationUnsupported`, retention skipping `SampleModified`, and one Warning per boot (#674). | | `ServerHistorian:SqlServer:RetentionDays` | unset | Days of history kept (source time), `>= 1` when set; unset keeps everything. Enables the Primary-gated purge of `hist.Sample`/`hist.Event` (see [Historian.md](Historian.md#retention-retentiondays)). | | `ServerHistorian:SqlServer:PurgeInterval` | `01:00:00` | Purge cadence; must be `> 0`. | | `ServerHistorian:SqlServer:PurgeBatchSize` | `4000` | Rows per `DELETE TOP (n)` batch; `>= 1`. Keep under ~5000 to avoid lock escalation. | @@ -430,7 +446,12 @@ when `ServerHistorian:Enabled` **or** `AlarmHistorian:Enabled` is true (`QuestDb | `ServerHistorian:QuestDb:VisibilityTimeout` | `00:00:05` | Must be `>= 0`. How long a write waits for its rows to become visible to queries (WAL apply); `0` disables the wait. A timeout only logs a Warning. | HistoryUpdate (SqlServer and QuestDb only; QuestDb deletes are soft — tombstones) additionally needs `ServerHistorian:Enabled=true` and a key -whose principal holds the `HistoryUpdate` permission — see [`docs/security.md`](security.md#historyupdate-historian-writes). Full semantics, +whose principal holds the `HistoryUpdate` permission — see [`docs/security.md`](security.md#historyupdate-historian-writes). It is served +only on the redundancy **Primary** (`BadInvalidState` elsewhere, #674) and is bounded by one backend-independent key: + +| Key | Default | Meaning | +|---|---|---| +| `ServerHistorian:HistoryUpdateMaxFutureSkew` | `1.00:00:00` (1 day) | TimeSpan. A HistoryUpdate (Insert / Replace / Update) item whose `SourceTimestamp` is later than server-clock now + this is refused `BadOutOfRange` before any backend sees it (#674 — a far-future typo could otherwise age out a QuestDB TTL table or poison the history). Applies to every backend; the live recorder is not bounded. Non-positive disables the bound. | Full semantics, schema, aggregates and the docker-dev recipe: [`docs/Historian.md`](Historian.md#sql-server-backend) (SqlServer) and [`docs/Historian.md`](Historian.md#questdb-backend) (QuestDb: the µs timestamp rules, pre-1970 limits, TTL and the docker-dev overlay). diff --git a/docs/Historian.md b/docs/Historian.md index eea9544d..203cb4ba 100644 --- a/docs/Historian.md +++ b/docs/Historian.md @@ -85,7 +85,8 @@ consumes the historian. The existing enable gates (`ServerHistorian:Enabled`, `A | Connection | `Endpoint` + `ApiKey` + TLS keys (top-level `ServerHistorian:*`) | `ServerHistorian:SqlServer:*` | `ServerHistorian:QuestDb:*` | Recording stays **Primary-only** (#537) and **at-least-once** with the #538 per-tag quarantine on every backend; -HistoryRead and HistoryUpdate are served from any node (the database is shared). The historian connection string is +HistoryRead is served from any node (the database is shared). HistoryUpdate is **Primary-only** since #674 +(`BadInvalidState` on any other node — see [HistoryUpdate](#primary-only-674)). The historian connection string is **not** a ConfigDb string, and a driver-only node may carry it without breaking the Phase 4 "no ConfigDb" posture. --- @@ -145,6 +146,7 @@ and all HistoryRead calls on historized nodes return `GoodNoData` (empty, not an | `MaxConcurrentHistoryReadBatches` | int | `16` | Process-wide limit on HistoryRead (and HistoryUpdate) batches served **concurrently** (arch-review 03/S3). A flood of history clients degrades to `BadTooManyOperations` rather than exhausting the SDK request-thread pool. With `HistoryReadBatchParallelism` this caps total in-flight gateway reads at `MaxConcurrentHistoryReadBatches × HistoryReadBatchParallelism` (default 16 × 4 = 64). A batch that cannot get a slot within `min(HistoryReadDeadline, 5 s)` fails every handle `BadTooManyOperations`. | | `MaxTimeWeightedRawRows` | int | `1000000` | `SqlServer` / `QuestDb` only (#671). The most raw in-window samples one [time-weighted aggregate](#time-weighted-aggregates-part-13-671) read (TimeAverage, TimeAverage2, **Total**, Total2, Interpolative) may stream into the server-side evaluator — that family is **not** pushed down to SQL. The window query is capped at this + 1 rows; one row past the cap fails **that node** with `BadTooManyOperations` (not counted as a backend fault) instead of holding a pooled connection until `HistoryReadDeadline` and returning `BadTimeout`. Bound rows outside the window do not count; per-sample aggregates are not capped. 1 000 000 ≈ 11.5 days of a 1 Hz tag. Non-positive = unbounded (a `Validate()` warning). | | `HistoryReadDeadline` | TimeSpan | `00:01:00` | Per-request wall-clock deadline for a HistoryRead (and HistoryUpdate) batch (arch-review 03/S3). A node still in flight at the deadline surfaces `BadTimeout`, so one request can never hold an SDK request thread longer than this regardless of node count. Non-positive = unbounded (a `Validate()` warning). | +| `HistoryUpdateMaxFutureSkew` | TimeSpan | `1.00:00:00` | #674. How far past the server clock a HistoryUpdate (Insert / Replace / Update) item's `SourceTimestamp` may lie; an item beyond `now + skew` is refused **`BadOutOfRange`** by the node manager before any backend sees it (every backend). Guards against a far-future typo (a 2062 timestamp). The live recorder is not bounded. Non-positive disables the bound (no warning — a deliberate choice). See [HistoryUpdate](#historyupdate). | **`ServerHistorian:SqlServer`** (read only when `Backend=SqlServer`; bound by `HistorianBackendRegistry.BindSqlServer`): @@ -630,13 +632,31 @@ says is complete carries no continuation point. `NumValuesPerNode == 0` stays un - The cursor is `(source, next start, end, boundary skip, cap)` in the same session-bound store as Raw (flagged as an events cursor — a Raw point presented to an events read, or the reverse, is `BadContinuationPointInvalid`). The next page reads from - the last event's `EventTimeUtc` **inclusive** and drops the events at that timestamp already emitted. + the last event's `EventTimeUtc` **inclusive** and drops the events at that timestamp already emitted — and asks the source for + that many more rows than the cap, so every page short of the last is a FULL `NumValuesPerNode` page (#675; resumed pages used + to come back short by the skip). - A tie cluster larger than the cap is paged within by over-fetching the half-open window `[T, T + resolution)` up to `MaxTieClusterOverfetch`; beyond that bound the node answers `BadHistoryOperationUnsupported` (raise the bound). - **Tie exactness depends on the backend.** SqlServer and QuestDB order by `(EventTimeUtc, EventId)`, so the skip count is exact. The Gateway backend has no stable intra-timestamp order (and only a truncation flag), so ties at a page boundary are best-effort there. -- Reverse event reads (`start > end`) are not paged server-side; no continuation point is issued for them. +- **Reverse event reads page newest-first (#675).** `StartTime > EndTime` reads the Part 11 reverse window `(EndTime, StartTime]` + with events in DESCENDING `(EventTimeUtc, EventId)` order, so a cap keeps the **latest** N and a continuation point is issued + when the source flags more. The direction is stored **in the cursor**: a reverse point resumes reverse (and a forward one + forward) whatever the resume request's details say. Reverse tie clusters are over-fetched forward and sliced from the newest + end, exactly as Raw does. Before #675 a capped reverse read silently returned the EARLIEST N, ascending, Good, with no + continuation point. +- **End-only events reads read backward (#675).** `StartTime` unset (`DateTime.MinValue`) with an `EndTime` and + `NumValuesPerNode > 0` pages backward from `EndTime` over `(MinValue, EndTime]` — the same rule as Raw. Start-only reads still + page forward to an open upper bound (`DateTime.MaxValue`, UTC). Without a cap an end-only read keeps its forward + `[MinValue, EndTime)` meaning. +- **Backends.** The server calls `IHistorianDataSource.ReadEventsAsync(…, bool reverse, …)`. SqlServer and QuestDB serve reverse + natively (`… > lo AND … <= hi ORDER BY time DESC, id DESC`). **Gateway and Null refuse a reverse read** (the interface default + throws `NotSupportedException`) and the node answers `BadHistoryOperationUnsupported` — the gateway streams ascending only and + rejects `end < start`, and draining the whole window to keep its tail would be exactly the unbounded read the cap prevents. +- **Gateway open upper bound.** The `DateTime.MaxValue` (UTC) bound of a start-only forward read is pinned offline through the + published client's proto `Timestamp` conversion (captured by an in-process gRPC server) and the FILETIME encoding + (`GatewayEventBoundsTests`). What a live AVEVA Historian does with a year-9999 bound is still unverified (needs the VPN). - No session on the request (in-process callers) degrades to single-shot, as for Raw. ### Modified-value history (Part 11) @@ -686,6 +706,25 @@ modification time and the Part 11 `HistoryUpdateType`. A HistoryReadRaw with `Is within one node the order is strict. - **Retention.** The trail ages out on the same **source-time** cutoff as the samples it describes: `SqlServer` purges `hist.SampleModified` per tag alongside `hist.Sample`; QuestDB puts the same TTL on `{p}sample_modified`. +- **SqlServer schema v1 (#674).** With `AutoCreateSchema=false` a database still at schema **version 1** (script `002_sample_modified.sql` not + applied) keeps historizing — live writes, raw/processed/at-time reads, alarm events and retention all work — but has no trail: modified reads + and every HistoryUpdate item answer **`BadHistoryOperationUnsupported`** with nothing sent to SQL, retention skips `SampleModified`, and the + node logs **one Warning per boot**. Apply `002_sample_modified.sql` and restart the node to enable them (the verdict is taken once per + process). `AutoCreateSchema=true` (the default) still upgrades v1 to v2 in place. See [Schema](#schema). +- **QuestDB retry duplicates (#674).** `{p}sample_modified` is append-only with **no DEDUP key** (a key would collapse genuinely distinct + modifications of one sample), and each trail row gets a fresh `modified_id`. So a HistoryUpdate a client **retries** after a timeout — when the + first attempt had in fact landed — records a second trail row for the same change (and, for Insert, the retry is `BadEntryExists`). The + sample itself is idempotent (dedup upsert); only the audit trail can show the duplicate. `SqlServer` has the same exposure only if the client + retries a committed request; there is no request id to dedupe on in either backend. +- **Cross-node ordering (QuestDB).** The per-tag lock is per process; the [Primary-only](#primary-only-674) HistoryUpdate gate (#674) is what keeps + two nodes of a pair from recording concurrent updates of one tag (and so a wrong prior value). Two **separate** pairs sharing one QuestDB can + still race; that is unsupported. +- ⚠️ **Who can read the trail (#674.4).** Modified reads are ordinary HistoryReads: with `Security:Acl:Enabled=false` (the default) **any admitted + session, including Anonymous, can read who rewrote history** (`UserName` is the principal). Turn the ACL flag on, or set + `OpcUa:AllowAnonymous=false`, if the principal names are sensitive (see [security.md](security.md)). +- **Not implemented (#674.5):** `IsDeleteModified` (deleting trail entries — `BadHistoryOperationUnsupported`); recording an **Update** (upsert) as + a split `Insert` / `Replace` — it is recorded as `Update` either way; and **`AuditHistoryUpdateEvent`** (no audit event is emitted for a history + change — the trail is the only record). ```bash otopcua-cli historyread -u opc.tcp://localhost:4840 -U apikey -P otua__ \ @@ -758,7 +797,7 @@ The bootstrapper applies every script above the recorded version, in order, unde | Object | Purpose | |---|---| -| `SchemaVersion(Version, AppliedUtc)` | Applied-script ledger. Required version: **2** (since #668). | +| `SchemaVersion(Version, AppliedUtc)` | Applied-script ledger. Required version: **1**; version **2** adds the modification trail (#668). A v1 database is served with the trail features off (#674 — see below). | | `Tag(TagId int identity, Name nvarchar(400) unique, DataType tinyint, EngineeringUnit, Description, CreatedUtc, UpdatedUtc)` | One row per historian tagname. `DataType` is a `DriverDataType`. | | `Sample(TagId, SourceTimeUtc datetime2(7), ValueKind tinyint, NumValue float, IntValue bigint, StrValue nvarchar(max), StatusCode int, ServerTimeUtc)` | **PK `(TagId, SourceTimeUtc)`** — one value per timestamp (Part 11), which makes redelivery idempotent and rules out tie clusters. `StatusCode` is the uint status stored bit-for-bit. | | `Event(EventId nvarchar(64) PK nonclustered, SourceName, AlarmId, AlarmName, AlarmTypeName, Severity, EventKind, Message, UserName, Comment, EventTimeUtc, ReceivedTimeUtc)` | Alarm history. Clustered on `(EventTimeUtc, EventId)`; extra index `(SourceName, EventTimeUtc)`. | @@ -774,11 +813,14 @@ everywhere (full 100 ns precision — never a `DataTable`, which would round to `sqlcmd -S -U -P -d -I -v Schema=hist -i docs/historian/sqlserver/001_initial.sql`, then the same with `002_sample_modified.sql` (`-v Schema=` supplies the `$(Schema)` placeholder; verified against the docker-dev SQL container). Re-running is a no-op. -⚠️ **Upgrading an existing v1 historian (#668).** A build with this change requires schema **version 2**. With `AutoCreateSchema=true` -(the default) the first historian call after the upgrade applies `002_sample_modified.sql` in place (it only creates `SampleModified`; existing -samples are untouched). With `AutoCreateSchema=false` **every historian call fails** (`missing version 2 (found 1)`, in health `LastError`) -until an operator runs `002_sample_modified.sql` by hand as above — do that before rolling the new build out. Modifications made before the -upgrade have no trail; the trail starts with the first HistoryUpdate after it. +⚠️ **Upgrading an existing v1 historian (#668, #674).** With `AutoCreateSchema=true` (the default) the first historian call after the +upgrade applies `002_sample_modified.sql` in place (it only creates `SampleModified`; existing samples are untouched). With +`AutoCreateSchema=false` a **v1 database is accepted** (#674 — between #668 and #674 it refused every historian call, continuous-historization +drains included): writes, reads, events and retention work, while modified-value reads and HistoryUpdate report +`BadHistoryOperationUnsupported` without touching SQL, retention skips `SampleModified`, and one Warning per boot names the missing script. +Run `002_sample_modified.sql` by hand as above and **restart the node** to enable them (the version is read once per process). Only a schema +below version 1 (`missing version 1 (found 0)`) still fails every call. Modifications made before the upgrade have no trail; the trail starts +with the first HistoryUpdate after it. ### Writes @@ -843,8 +885,10 @@ result never has both. ### Events -`[start, end)` by `EventTimeUtc`; **a reversed window (`end < start`) is swapped and returned ascending**. A null `sourceName` means all -sources, otherwise `SourceName = @source`. Ordered `(EventTimeUtc, EventId)` ascending. A bounded read fetches `maxEvents + 1` rows and +`[start, end)` by `EventTimeUtc`; through the five-argument read **a reversed window (`end < start`) is swapped and returned +ascending**. The direction-aware overload with `reverse: true` (what the node manager calls for a reverse read, #675) reads +`(lo, hi]` ordered `(EventTimeUtc, EventId)` **descending** — the clustered index scanned backwards. A null `sourceName` means all +sources, otherwise `SourceName = @source`. Forward is ordered `(EventTimeUtc, EventId)` ascending. A bounded read fetches `maxEvents + 1` rows and returns a non-null continuation point when the extra row exists; the node manager turns that flag into an OPC UA continuation point (see [Continuation-point paging (Events)](#continuation-point-paging-events-669)). @@ -1042,6 +1086,8 @@ equivalent TTL (`QuestDb:RetentionDays`); the Gateway backend's retention is the timeout 0) lets one purger run and the others skip that tick (Debug log). - **Observability.** Counter `otopcua.historian.retention.purged{table=Sample|SampleModified|Event}`; a failed pass logs a Warning and retries next interval — it never affects capture or reads. +- **Schema v1 (#674).** On a v1 schema (no `SampleModified`, `AutoCreateSchema=false`) the trail purge is skipped and omitted from the + pass results; samples and events purge normally. - **Not done: partition switching.** For very large installs (billions of rows) a partitioned `hist.Sample` with sliding-window `SWITCH`/drop is cheaper than row deletes; the purger is deliberately simple batched deletes. Another table is added by one more `DeleteLoopAsync` call in `SqlServerHistorianRetention.PurgeAsync` (as `hist.SampleModified` was, #668). @@ -1222,11 +1268,12 @@ Every query filters `NOT deleted`; every value goes through `StoredSample`, so t from the last timestamp ± `TimestampResolutionTicks`. - **AtTime:** previous-or-equal and next neighbour of each time by `LIMIT 1` seeks, `UNION ALL`-ed, **50 times per statement** (QuestDB 10.0.1 refuses more than 126 branches). `AtTimeEvaluator` decides each value, as on `SqlServer`. -- **Events:** `[start, end)` by `ts`, ordered `(ts, event_id)`, `maxEvents + 1` fetched to produce the continuation point. +- **Events:** `[start, end)` by `ts`, ordered `(ts, event_id)`, `maxEvents + 1` fetched to produce the continuation point; a + reverse read (#675) is `(lo, hi]` ordered `(ts, event_id)` descending. - **Processed:** one query per node; bucket index `(cast(ts AS LONG) * 10 - bucketBaseUnixTicks) / intervalTicks` (never `SAMPLE BY`, whose alignment differs), so bucket edges are identical to `SqlServer` (forward start-inclusive / end-exclusive with the partial bucket last; reverse anchored at the - later time, later-edge-inclusive, partial bucket earliest — #677; interval 0 ⇒ one bucket, 100,000-bucket cap). A second `LIMIT 1` statement fetches - the tag's earliest live sample for the start-of-data Partial bit. + later time, later-edge-inclusive, partial bucket earliest — #677; interval 0 ⇒ one bucket, 100,000-bucket cap). On a forward read of any aggregate + but Average, a second `LIMIT 1` statement (capped at the window end) fetches the tag's earliest live sample for the start-of-data Partial bit. Min/max times use `row_number()` windows with the earliest-timestamp tie-break. ### Retention (TTL) @@ -1240,7 +1287,14 @@ expired day partitions itself, tombstones included. re-read to verify; a very large `RetentionDays` that QuestDB normalises into MONTH/YEAR is re-`ALTER`ed on every boot (harmless). - ⚠️ **TTL is enforced only on write commits.** QuestDB checks for expired partitions when a write commits to the table, so **a quiet table never drops anything**: partitions older than the TTL linger until the next write lands. Do not read a stale-looking partition as a broken TTL. -- Expiry is judged against the wall clock, so a future-dated sample does not expire current data. +- **Expiry is judged against the wall clock** (`min(max timestamp, now)`), so a future-dated sample does not expire current data. + ⚠️ **Keep `cairo.ttl.use.wall.clock` at its default, `true`** (#674). It is what QuestDB documents as protecting a TTL table from a + future-dated row; with it `false` the TTL would be measured from the table's newest timestamp and one far-future row (a 2062 typo) could + drop every older partition — including `{p}sample_modified`, the record of who changed history. Evidence (QuestDB 10.0.1, 2026-10-11): the + default is `true`, and a 2066 row inserted into a 30-day-TTL table evicted nothing inside the TTL, while a partition genuinely older than + 30 days was evicted by that same commit — pinned by `TtlWallClockTests`. (The same run with the flag `false` also kept the in-TTL + partitions, so the test pins the outcome rather than isolating the flag; do not rely on that.) The server-side + `ServerHistorian:HistoryUpdateMaxFutureSkew` bound (default 1 day) is the primary guard: a HistoryUpdate cannot write a far-future row. - The TTL is visible in table metadata: `SELECT table_name, ttlValue, ttlUnit FROM tables() WHERE starts_with(table_name, 'hist_')`. ### Docker-dev recipe @@ -1281,8 +1335,8 @@ dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.Integration OPC UA **HistoryUpdate** lets a client insert, replace, update and delete stored raw history. It is available only when the selected backend provides an updater — **`SqlServer` and `QuestDb` do; `Gateway` does not**, and an updater is registered only when `ServerHistorian:Enabled=true`. -Updates go straight to the backend (not through the outbox), are **not Primary-gated** (the database is shared), and share the HistoryRead -batch limiter and `HistoryReadDeadline`. +Updates go straight to the backend (not through the outbox), are **Primary-only** (#674, [below](#primary-only-674)), and share the +HistoryRead batch limiter and `HistoryReadDeadline`. ### Operations @@ -1300,7 +1354,8 @@ caller's principal, in the [modification trail](#modified-value-history-part-11) mixed (Part 11 reports item outcomes in `OperationResults`); it is Bad only for whole-node failures. **Per-item rejections** (answered without touching SQL; other items still proceed): a null item ⇒ `BadInvalidArgument`; -`SourceTimestamp == DateTime.MinValue` ⇒ `BadInvalidTimestamp`; a value the historian cannot type (array, unsupported CLR type) ⇒ +`SourceTimestamp == DateTime.MinValue` ⇒ `BadInvalidTimestamp`; a **`SourceTimestamp` later than `now + ServerHistorian:HistoryUpdateMaxFutureSkew`** +(default 1 day, server clock; #674) ⇒ **`BadOutOfRange`** — on every backend, never sent; a value the historian cannot type (array, unsupported CLR type) ⇒ `BadTypeMismatch`; a **`NaN` / `±Infinity` `Double` ⇒ `BadInvalidArgument`** — it cannot live in a SQL `float`, and unlike the live-value writer (which stores it as `Empty`) an operator's explicit update is rejected rather than silently rewritten; and a **null value with a Good or Uncertain status ⇒ `BadInvalidArgument`** — an `Empty` sample is always Bad, so a null value is accepted only with a Bad status @@ -1323,7 +1378,19 @@ outcome, later ones `BadNoEntryExists`. **Unknown tag:** Insert / Update create `ServerHistorian:Enabled=false`); `PerformUpdateType.Remove`; `DeleteRawModifiedDetails.IsDeleteModified=true` (the [modification trail](#modified-value-history-part-11) is an audit record, aged out only by retention); and the out-of-scope detail types — **event** HistoryUpdate (`UpdateEventDetails` / `DeleteEventDetails`) and `UpdateStructureData` — which are not overridden, so the SDK base rejects them. A backend exception is `BadInternalError` for that node and a deadline is `BadTimeout`. -No `AuditHistoryUpdateEvent` is emitted. +On a `SqlServer` database still at schema v1 every item is `BadHistoryOperationUnsupported` ([#674](#modified-value-history-part-11)). +No `AuditHistoryUpdateEvent` is emitted (#674.5, not implemented). + +### Primary-only (#674) + +HistoryUpdate is served only by the node holding the redundancy **Primary** data-plane role — the same verdict +(`IDriverPrimaryRoleSource`, published by `DriverHostActor`) that gates native-alarm acknowledges (#597). On any other node every node of +the request that **passed authorization** is refused **`BadInvalidState`** (node-level code, with a message pointing at the Primary) and +nothing reaches the backend. Authorization runs first, so an unauthorized caller learns `BadUserAccessDenied`, not the node's role. +HistoryRead is **not** gated. Why: rewriting history is a single-writer operation — both backends capture the prior value for the +[modification trail](#modified-value-history-part-11) under a per-tag, **per-process** lock (QuestDB) or statement (SqlServer), so two pair +nodes updating one tag concurrently could record a wrong prior value. A single-node deployment (no redundancy manager) is unaffected. Clients +should send HistoryUpdate to the server advertising the higher `ServiceLevel`. ### The `HistoryWrite` AccessLevel bit diff --git a/docs/ServiceHosting.md b/docs/ServiceHosting.md index e5e8aa42..48b13b9b 100644 --- a/docs/ServiceHosting.md +++ b/docs/ServiceHosting.md @@ -32,13 +32,13 @@ Single-node dev: `OTOPCUA_ROLES=admin,driver` (or the `Cluster:Roles` equivalent ### Per-role configuration overlays -`Program.cs` builds a role suffix by joining the resolved `EffectiveRoles.Gossiped` roles **alphabetically** with `-` and loads `appsettings.{roleSuffix}.json` as an optional overlay on top of base `appsettings.json`. A plain role (`admin`/`driver`/`dev`) is always lower-case in `Gossiped` regardless of how the operator spelled it, so the overlay filename for those is stable (`appsettings.driver.json`); if a `cluster-{ClusterId}` role is also configured, its suffix carries into the filename **verbatim** (e.g. `Cluster:Roles: ["driver", "cluster-SITE-A"]` loads `appsettings.cluster-SITE-A-driver.json`), since `Gossiped` preserves that spelling by design (see [Configuration.md § `Roles`](Configuration.md#appsettings-sections)). Three overlays ship in `src/Server/ZB.MOM.WW.OtOpcUa.Host/`: +`Program.cs` builds a role suffix (`RoleOverlay.Suffix`) by joining the resolved `EffectiveRoles.Gossiped` **plain** roles (`admin`/`driver`/`dev`) alphabetically with `-` and loads `appsettings.{roleSuffix}.json` as an optional overlay on top of base `appsettings.json`. A plain role is always lower-case in `Gossiped` regardless of how the operator spelled it, so the overlay filename is stable (`appsettings.driver.json`). **`cluster-{ClusterId}` roles are NOT part of the filename** (#678.6): they are mesh-membership tags, so `Cluster:Roles: ["driver", "cluster-SITE-A"]` loads `appsettings.driver.json` and `["admin", "driver", "cluster-MAIN"]` loads `appsettings.admin-driver.json`. (Before #678.6 the cluster role was folded into the name, producing e.g. `appsettings.admin-cluster-MAIN-driver.json`, which never exists — so on every node with a cluster role no shipped overlay ever loaded.) Three overlays ship in `src/Server/ZB.MOM.WW.OtOpcUa.Host/`: - `appsettings.admin.json` — admin-only nodes - `appsettings.driver.json` — driver-only nodes - `appsettings.admin-driver.json` — fused single-node dev / small deployments -All three carry Serilog log-level overrides + `Security:Ldap:DevStubMode = false`. Configuration loading order (lowest to highest precedence) is: +All three carry Serilog log-level overrides + `Security:Ldap:DevStubMode = false` + `Security:Ldap:Transport = Ldaps` (audit table: [Configuration.md § Role overlays](Configuration.md#role-overlay-audit)). Configuration loading order (lowest to highest precedence) is: `appsettings.json` < `appsettings.{Environment}.json` < `appsettings.{role}.json` < environment variables < command-line args diff --git a/docs/security.md b/docs/security.md index 710e3598..87ac8d22 100644 --- a/docs/security.md +++ b/docs/security.md @@ -884,6 +884,7 @@ HistoryUpdate rewrites recorded history, so unlike HistoryRead it is gated **wha - **Flag off (default):** the session's API key is re-checked on the call (`DataPlaneAuthorizer.AuthorizeKeyUsable`, Gitea #563 — an expired or disabled key loses its activation-time roles), then the session must carry the **`HistoryUpdate` role** (`OpcUaDataPlaneRoles.HistoryUpdate`). `ApiKeyInterimRoleMapper` confers it **only from a `Cluster`-scoped grant** carrying `NodePermissions.HistoryUpdate` — narrower-scoped grants confer nothing, for the same reason as `WriteOperate` (honouring a flag without honouring its scope would widen it). A key holding only `WriteOperate` is denied `BadUserAccessDenied`. - **Flag on:** `DataPlaneAuthorizer.Authorize(identity, OpcUaOperation.HistoryUpdate, nodeId, realm)` decides per node and **replaces** the role check (never chained) — a session with no roles succeeds on the strength of a scoped `HistoryUpdate` grant, and a `WriteOperate` grant does not help. - **Availability is separate from authorization.** A node accepts HistoryUpdate only if it is historized **and** the selected historian backend has an updater when the node materialises (the `HistoryWrite` AccessLevel bit). `Backend=Gateway` has none, so it answers `BadHistoryOperationUnsupported` for every node whatever the grant. Operations, result codes and unsupported cases: [`docs/Historian.md` §HistoryUpdate](Historian.md#historyupdate). +- **Primary-only, after authorization (Gitea #674).** An authorized HistoryUpdate on a node that is not the redundancy Primary is refused `BadInvalidState` (the native-alarm-ack verdict, #597); an unauthorized one is told `BadUserAccessDenied` first, so the refusal never reveals the node's role to a caller who may not update. - **Every applied change is attributed (Part 11 modified-value history, Gitea #668).** The session's principal (the API key's `NodeAcl` principal, or the reserved `anonymous`) and key id are passed to the backend with every HistoryUpdate and recorded in the modification trail next to the prior value; a modified-value HistoryRead (`IsReadModified = true`) returns it as diff --git a/src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI/Commands/HistoryReadCommand.cs b/src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI/Commands/HistoryReadCommand.cs index 573c758f..b19e20fb 100644 --- a/src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI/Commands/HistoryReadCommand.cs +++ b/src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI/Commands/HistoryReadCommand.cs @@ -47,7 +47,7 @@ public class HistoryReadCommand : CommandBase /// /// Gets the optional aggregate name used when the operator wants processed history instead of raw samples. /// - [CommandOption("aggregate", Description = "Aggregate function: Average, Minimum, Maximum, Count, Start, End, StandardDeviation")] + [CommandOption("aggregate", Description = "Aggregate function: Average, Minimum, Maximum, Count, Start, End, StandardDeviation, TimeAverage, TimeAverage2, Total, Total2, Interpolative")] public string? Aggregate { get; init; } /// @@ -183,6 +183,12 @@ public class HistoryReadCommand : CommandBase await console.Output.WriteLineAsync(); await console.Output.WriteLineAsync($"{values.Count} values returned."); } + catch (ServiceResultException ex) when (ex.InnerException is null) + { + // e.g. HistoryReadStatus.ThrowIfRefused: a refusal is an operator-facing outcome, not a crash. + // A wrapped failure (certificate validation, identity rejection on connect) keeps its full detail. + throw new CommandException(ex.Message); + } finally { if (service != null) @@ -204,8 +210,13 @@ public class HistoryReadCommand : CommandBase "start" or "first" => AggregateType.Start, "end" or "last" => AggregateType.End, "standarddeviation" or "stddev" or "stdev" => AggregateType.StandardDeviation, + "timeaverage" or "timeavg" => AggregateType.TimeAverage, + "timeaverage2" or "timeavg2" => AggregateType.TimeAverage2, + "total" => AggregateType.Total, + "total2" => AggregateType.Total2, + "interpolative" or "interp" => AggregateType.Interpolative, _ => throw new ArgumentException( - $"Unknown aggregate: '{name}'. Supported: Average, Minimum, Maximum, Count, Start, End, StandardDeviation") + $"Unknown aggregate: '{name}'. Supported: Average, Minimum, Maximum, Count, Start, End, StandardDeviation, TimeAverage, TimeAverage2, Total, Total2, Interpolative") }; } } diff --git a/src/Client/ZB.MOM.WW.OtOpcUa.Client.Shared/Adapters/DefaultEndpointDiscovery.cs b/src/Client/ZB.MOM.WW.OtOpcUa.Client.Shared/Adapters/DefaultEndpointDiscovery.cs index 4ed0ad71..c16c020e 100644 --- a/src/Client/ZB.MOM.WW.OtOpcUa.Client.Shared/Adapters/DefaultEndpointDiscovery.cs +++ b/src/Client/ZB.MOM.WW.OtOpcUa.Client.Shared/Adapters/DefaultEndpointDiscovery.cs @@ -85,17 +85,22 @@ internal static class EndpointSelector $"No endpoint found with security mode '{requestedMode}'. Available endpoints: {available}"); } - // Rewrite endpoint URL hostname to match user-supplied hostname. Necessary - // when the OPC UA server returns a discovery URL using a different hostname - // (e.g. internal DNS name) than the one the operator routed to. + // Rewrite the endpoint URL host AND port to the user-supplied ones (as the SDK's + // CoreClientUtils.SelectEndpoint does). A server behind a port mapping or internal DNS + // advertises its own host:port (docker-dev containers all say :4840); following that would + // open the session on a different server and fail the secure-channel certificate check. + // Only when the scheme matches: a different scheme (https vs opc.tcp) is a different transport. var serverUri = new Uri(best.EndpointUrl); var requestedUri = new Uri(endpointUrl); - if (serverUri.Host != requestedUri.Host) + // A URL typed without a port (opc.tcp has no registered default, so Port is -1) keeps the advertised port. + var port = requestedUri.Port > 0 ? requestedUri.Port : serverUri.Port; + if (string.Equals(serverUri.Scheme, requestedUri.Scheme, StringComparison.OrdinalIgnoreCase) + && (serverUri.Host != requestedUri.Host || serverUri.Port != port)) { - var builder = new UriBuilder(best.EndpointUrl) { Host = requestedUri.Host }; + var builder = new UriBuilder(best.EndpointUrl) { Host = requestedUri.Host, Port = port }; best.EndpointUrl = builder.ToString(); - Logger.Debug("Rewrote endpoint host from {ServerHost} to {RequestedHost}", serverUri.Host, - requestedUri.Host); + Logger.Debug("Rewrote endpoint from {ServerHost}:{ServerPort} to {RequestedHost}:{RequestedPort}", + serverUri.Host, serverUri.Port, requestedUri.Host, port); } return best; diff --git a/src/Client/ZB.MOM.WW.OtOpcUa.Client.Shared/Helpers/AggregateTypeMapper.cs b/src/Client/ZB.MOM.WW.OtOpcUa.Client.Shared/Helpers/AggregateTypeMapper.cs index ab672344..44462199 100644 --- a/src/Client/ZB.MOM.WW.OtOpcUa.Client.Shared/Helpers/AggregateTypeMapper.cs +++ b/src/Client/ZB.MOM.WW.OtOpcUa.Client.Shared/Helpers/AggregateTypeMapper.cs @@ -24,6 +24,11 @@ public static class AggregateTypeMapper AggregateType.Start => ObjectIds.AggregateFunction_Start, AggregateType.End => ObjectIds.AggregateFunction_End, AggregateType.StandardDeviation => ObjectIds.AggregateFunction_StandardDeviationPopulation, + AggregateType.TimeAverage => ObjectIds.AggregateFunction_TimeAverage, + AggregateType.TimeAverage2 => ObjectIds.AggregateFunction_TimeAverage2, + AggregateType.Total => ObjectIds.AggregateFunction_Total, + AggregateType.Total2 => ObjectIds.AggregateFunction_Total2, + AggregateType.Interpolative => ObjectIds.AggregateFunction_Interpolative, _ => throw new ArgumentOutOfRangeException(nameof(aggregate), aggregate, "Unknown AggregateType value.") }; } diff --git a/src/Client/ZB.MOM.WW.OtOpcUa.Client.Shared/Models/AggregateType.cs b/src/Client/ZB.MOM.WW.OtOpcUa.Client.Shared/Models/AggregateType.cs index b5b292ab..b7f48ab7 100644 --- a/src/Client/ZB.MOM.WW.OtOpcUa.Client.Shared/Models/AggregateType.cs +++ b/src/Client/ZB.MOM.WW.OtOpcUa.Client.Shared/Models/AggregateType.cs @@ -24,5 +24,20 @@ public enum AggregateType End, /// Population standard deviation of values in the interval. - StandardDeviation + StandardDeviation, + + /// Time-weighted average, interpolated bounds (Part 13 TimeAverage). + TimeAverage, + + /// Time-weighted average, stepped bounds (Part 13 TimeAverage2). + TimeAverage2, + + /// Time-weighted integral over the interval (Part 13 Total). + Total, + + /// Time-weighted integral, stepped bounds (Part 13 Total2). + Total2, + + /// Interpolated value at the interval start (Part 13 Interpolative). + Interpolative } \ No newline at end of file diff --git a/src/Client/ZB.MOM.WW.OtOpcUa.Client.UI/ViewModels/HistoryViewModel.cs b/src/Client/ZB.MOM.WW.OtOpcUa.Client.UI/ViewModels/HistoryViewModel.cs index dbc82b1c..aab55811 100644 --- a/src/Client/ZB.MOM.WW.OtOpcUa.Client.UI/ViewModels/HistoryViewModel.cs +++ b/src/Client/ZB.MOM.WW.OtOpcUa.Client.UI/ViewModels/HistoryViewModel.cs @@ -53,7 +53,12 @@ public partial class HistoryViewModel : ObservableObject AggregateType.Count, AggregateType.Start, AggregateType.End, - AggregateType.StandardDeviation + AggregateType.StandardDeviation, + AggregateType.TimeAverage, + AggregateType.TimeAverage2, + AggregateType.Total, + AggregateType.Total2, + AggregateType.Interpolative ]; /// Gets a value indicating whether an aggregate read is selected. diff --git a/src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/Historian/HistoryReadContracts.cs b/src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/Historian/HistoryReadContracts.cs index 0be71792..cb98f97a 100644 --- a/src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/Historian/HistoryReadContracts.cs +++ b/src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/Historian/HistoryReadContracts.cs @@ -88,7 +88,7 @@ public sealed class HistoryReadLimitExceededException : InvalidOperationExceptio } /// -/// One row returned by — a historical +/// One row returned by — a historical /// alarm/event record, not the OPC UA live-event stream. Fields match the minimum set the /// Server needs to populate a HistoryEventFieldList for HistoryReadEvents responses. /// @@ -106,8 +106,8 @@ public sealed record HistoricalEvent( string? Message, ushort Severity); -/// Result of a call. -/// Events in chronological order by EventTimeUtc. +/// Result of a call. +/// Events ordered by EventTimeUtc in the read's direction — ascending, or descending for a reverse read (#675). /// Opaque token for the next call when more events are available; null when complete. public sealed record HistoricalEventsResult( IReadOnlyList Events, diff --git a/src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/Historian/IHistorianDataSource.cs b/src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/Historian/IHistorianDataSource.cs index a135d765..a891fd70 100644 --- a/src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/Historian/IHistorianDataSource.cs +++ b/src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/Historian/IHistorianDataSource.cs @@ -79,7 +79,7 @@ public interface IHistorianDataSource : IDisposable /// uint maxValuesPerNode because their OPC UA HistoryRead surface has no /// equivalent "use default" sentinel. /// - /// This surface declares and + /// This surface declares and /// as required members — a server-side historian owns the full read surface. /// /// The source name to filter events, or null to return events from all sources. @@ -101,6 +101,46 @@ public interface IHistorianDataSource : IDisposable int maxEvents, CancellationToken cancellationToken); + /// + /// Read historical alarm / event records in a chosen direction — the overload the server's HistoryReadEvents + /// paging calls (Gitea #675). + /// + /// + /// + /// The bounds may be given in either order; let lo/hi be the earlier/later of them. Forward + /// ( false) is exactly : + /// [lo, hi) ascending by (EventTimeUtc, EventId). Reverse is the Part 11 reverse window + /// (lo, hi] in DESCENDING (EventTimeUtc, EventId) order, so a cap keeps the LATEST + /// events — the mirror of forward, with the same tie-breaker reversed, so a + /// resumed reverse page that re-reads its boundary timestamp sees the boundary ties in the same order. + /// and the continuation-point contract are as for the forward read. + /// + /// + /// The default forwards a forward read to the five-argument overload and throws + /// for a reverse one — a backend that cannot read newest-first (Gateway, + /// Null) — which the server reports as BadHistoryOperationUnsupported rather than returning the + /// earliest events in ascending order as if they were the answer. + /// + /// + /// The source name to filter events, or null to return events from all sources. + /// One bound of the window (UTC). + /// The other bound of the window (UTC). + /// The cap; non-positive uses the backend default (see the five-argument overload). + /// true to read newest-first over (lo, hi]. + /// Cancellation. + /// The matching events in the read's direction, with a continuation point if truncated. + /// A reverse read on a backend that cannot read newest-first. + Task ReadEventsAsync( + string? sourceName, + DateTime startUtc, + DateTime endUtc, + int maxEvents, + bool reverse, + CancellationToken cancellationToken) => + reverse + ? throw new NotSupportedException("This historian backend cannot read events newest-first (reverse).") + : ReadEventsAsync(sourceName, startUtc, endUtc, maxEvents, cancellationToken); + /// /// Point-in-time health snapshot for diagnostics and dashboards. Pure /// observation; never blocks on backend I/O. diff --git a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway/GatewayHistorianDataSource.cs b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway/GatewayHistorianDataSource.cs index 556bbde2..a3e2ebb5 100644 --- a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway/GatewayHistorianDataSource.cs +++ b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway/GatewayHistorianDataSource.cs @@ -125,6 +125,21 @@ public sealed class GatewayHistorianDataSource : IHistorianDataSource, IAsyncDis } } + /// + /// + /// A reverse (newest-first) read is refused with — the server reports + /// BadHistoryOperationUnsupported (Gitea #675). The gateway's ReadEvents streams ascending only and + /// rejects end < start, so the only way to serve "the latest N" would be to drain the whole window and + /// keep its tail, an unbounded read the cap exists to prevent. Nothing is sent. + /// + public Task ReadEventsAsync( + string? sourceName, DateTime startUtc, DateTime endUtc, int maxEvents, bool reverse, + CancellationToken cancellationToken) => + reverse + ? throw new NotSupportedException( + "The HistorianGateway backend reads events oldest-first only; a reverse (newest-first) events read is not supported.") + : ReadEventsAsync(sourceName, startUtc, endUtc, maxEvents, cancellationToken); + /// public async Task ReadEventsAsync( string? sourceName, DateTime startUtc, DateTime endUtc, int maxEvents, diff --git a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbHistorianDataSource.cs b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbHistorianDataSource.cs index dc54e4d2..8c0b7f6b 100644 --- a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbHistorianDataSource.cs +++ b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbHistorianDataSource.cs @@ -361,8 +361,18 @@ public sealed class QuestDbHistorianDataSource : IHistorianDataSource /// maxEvents + 1 rows are fetched; when the extra row arrives it is dropped and the continuation point is /// the UTF-8 of for that first not-returned event. /// + public Task ReadEventsAsync( + string? sourceName, DateTime startUtc, DateTime endUtc, int maxEvents, CancellationToken cancellationToken) => + ReadEventsAsync(sourceName, startUtc, endUtc, maxEvents, reverse: false, cancellationToken); + + /// + /// + /// Forward is the five-argument read. Reverse (Gitea #675) is the (rounded) (lo, hi] ordered by + /// (ts, event_id) DESCENDING, so LIMIT keeps the latest events; truncation and the continuation point + /// work as forward. + /// public async Task ReadEventsAsync( - string? sourceName, DateTime startUtc, DateTime endUtc, int maxEvents, CancellationToken cancellationToken) + string? sourceName, DateTime startUtc, DateTime endUtc, int maxEvents, bool reverse, CancellationToken cancellationToken) { // No stored event can carry a source longer than the limit, so an over-long filter matches nothing. if (sourceName is not null && sourceName.Length > HistorianTagNames.MaxLength) @@ -376,9 +386,9 @@ public sealed class QuestDbHistorianDataSource : IHistorianDataSource var bounded = maxEvents > 0 && maxEvents < int.MaxValue; var sql = $"SELECT event_id, source, ts, received_utc, message, severity FROM {_conn.EventTable} " + - "WHERE ts >= $1 AND ts < $2" + + (reverse ? "WHERE ts > $1 AND ts <= $2" : "WHERE ts >= $1 AND ts < $2") + (sourceName is null ? "" : " AND source = $3") + - " ORDER BY ts ASC, event_id ASC" + + (reverse ? " ORDER BY ts DESC, event_id DESC" : " ORDER BY ts ASC, event_id ASC") + (bounded ? (sourceName is null ? " LIMIT $3" : " LIMIT $4") : ""); await using var conn = await _conn.OpenAsync(cancellationToken).ConfigureAwait(false); diff --git a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbSchemaBootstrapper.cs b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbSchemaBootstrapper.cs index 296f787d..f610fac5 100644 --- a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbSchemaBootstrapper.cs +++ b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbSchemaBootstrapper.cs @@ -137,7 +137,10 @@ internal sealed class QuestDbSchemaBootstrapper // The ALTER is a WAL transaction: tables() keeps the old value until it applies, so do NOT re-read to verify. await using (var alter = _conn.CreateCommand(conn, QuestDbTables.AlterTtl(table, _options.RetentionDays))) await alter.ExecuteNonQueryAsync(ct).ConfigureAwait(false); - _log.LogInformation("Applied TTL {Days} day(s) to QuestDB table {Table}.", _options.RetentionDays, table); + if (_options.RetentionDays is { } days) + _log.LogInformation("Applied TTL {Days} day(s) to QuestDB table {Table}.", days, table); + else + _log.LogInformation("Cleared TTL on QuestDB table {Table}.", table); } } } diff --git a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/HistorianStatusCodes.cs b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/HistorianStatusCodes.cs index 9fa08bcf..686b1126 100644 --- a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/HistorianStatusCodes.cs +++ b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/HistorianStatusCodes.cs @@ -27,4 +27,10 @@ public static class HistorianStatusCodes /// status, or a tag name longer than . /// public const uint BadInvalidArgument = 0x80AB0000u; + + /// + /// A modified-value read or HistoryUpdate item against a SQL Server schema still at version 1 (no Part 11 + /// modification trail, Gitea #674): refused without touching the database. + /// + public const uint BadHistoryOperationUnsupported = 0x80720000u; } diff --git a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer/SqlServerHistorianDataSource.cs b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer/SqlServerHistorianDataSource.cs index b74a45a6..8ec2308d 100644 --- a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer/SqlServerHistorianDataSource.cs +++ b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer/SqlServerHistorianDataSource.cs @@ -154,6 +154,12 @@ public sealed class SqlServerHistorianDataSource : IHistorianDataSource var bounded = maxValues > 0 && maxValues < int.MaxValue; try { + // #674: a v1 schema has no SampleModified. NotSupportedException is the interface's "no trail" signal (the + // node manager reports BadHistoryOperationUnsupported); it is not a backend failure and sends no query. + await _bootstrap.EnsureAsync(cancellationToken).ConfigureAwait(false); + if (!_bootstrap.SupportsModifiedTrail) + throw new NotSupportedException( + $"Historian schema [{_conn.Schema}] is below version {SqlServerSchemaBootstrapper.ModifiedTrailVersion}: no modified-value history."); var values = new List(); var hasMore = false; var tagId = await ResolveAsync(fullReference, cancellationToken).ConfigureAwait(false); @@ -219,7 +225,7 @@ public sealed class SqlServerHistorianDataSource : IHistorianDataSource RecordSuccess(); return new HistoryModifiedReadResult(values, hasMore); } - catch (Exception ex) + catch (Exception ex) when (ex is not NotSupportedException) { RecordFailure(ex); throw; @@ -314,8 +320,18 @@ public sealed class SqlServerHistorianDataSource : IHistorianDataSource /// of "{EventTimeUtc.Ticks}|{EventId}" of that first not-returned event (opaque to callers; only /// its presence is contractual). /// + public Task ReadEventsAsync( + string? sourceName, DateTime startUtc, DateTime endUtc, int maxEvents, CancellationToken cancellationToken) => + ReadEventsAsync(sourceName, startUtc, endUtc, maxEvents, reverse: false, cancellationToken); + + /// + /// + /// Forward is the five-argument read. Reverse (Gitea #675) is (lo, hi] ordered by + /// (EventTimeUtc, EventId) DESCENDING — the clustered index scanned backwards — so TOP keeps the latest + /// events; truncation and the continuation point work as forward. + /// public async Task ReadEventsAsync( - string? sourceName, DateTime startUtc, DateTime endUtc, int maxEvents, CancellationToken cancellationToken) + string? sourceName, DateTime startUtc, DateTime endUtc, int maxEvents, bool reverse, CancellationToken cancellationToken) { // No stored event can carry a source longer than the column (the alarm writer dead-letters it rather than // truncating), so an over-long filter matches nothing — never a truncated prefix of someone else's source. @@ -328,9 +344,10 @@ public sealed class SqlServerHistorianDataSource : IHistorianDataSource var bounded = maxEvents > 0 && maxEvents < int.MaxValue; var sql = $"SELECT {(bounded ? "TOP (@take) " : "")}EventId, SourceName, EventTimeUtc, ReceivedTimeUtc, Message, Severity " + - $"FROM {_conn.Q("Event")} WHERE EventTimeUtc >= @lo AND EventTimeUtc < @hi" + + $"FROM {_conn.Q("Event")} WHERE " + + (reverse ? "EventTimeUtc > @lo AND EventTimeUtc <= @hi" : "EventTimeUtc >= @lo AND EventTimeUtc < @hi") + (sourceName is null ? "" : " AND SourceName = @source") + - " ORDER BY EventTimeUtc ASC, EventId ASC"; + (reverse ? " ORDER BY EventTimeUtc DESC, EventId DESC" : " ORDER BY EventTimeUtc ASC, EventId ASC"); await using var conn = await _conn.OpenAsync(cancellationToken).ConfigureAwait(false); await using var cmd = _conn.CreateCommand(conn, sql); diff --git a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer/SqlServerHistorianDataUpdater.cs b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer/SqlServerHistorianDataUpdater.cs index d21d4ed2..33bb097f 100644 --- a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer/SqlServerHistorianDataUpdater.cs +++ b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer/SqlServerHistorianDataUpdater.cs @@ -27,21 +27,44 @@ namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer; /// value from deleted.* (NULL for an insert), the caller's , the /// and SYSUTCDATETIME(). One statement means the sample change and its /// audit row commit or roll back together: a failed update leaves no audit row. +/// Schema v1 (Gitea #674). Without the trail ( +/// false) a change could not be audited, so every item (and DeleteRaw's single result) reports +/// BadHistoryOperationUnsupported and nothing is sent to the database beyond the one-time schema check. /// internal sealed class SqlServerHistorianDataUpdater : IHistorianDataUpdater { private readonly SqlHistorianConnection _conn; private readonly SqlServerTagCatalog _tags; + private readonly SqlServerSchemaBootstrapper _bootstrap; /// Creates the updater. /// Connection helper. + /// Schema bootstrapper — decides whether the modification trail exists (#674). /// Tag catalog (runs the schema bootstrap itself). - public SqlServerHistorianDataUpdater(SqlHistorianConnection conn, SqlServerTagCatalog tags) + public SqlServerHistorianDataUpdater(SqlHistorianConnection conn, SqlServerSchemaBootstrapper bootstrap, + SqlServerTagCatalog tags) { _conn = conn; + _bootstrap = bootstrap; _tags = tags; } + /// #674: ensures the schema and reports whether the Part 11 trail exists (no SQL after the first success). + /// Cancellation. + /// True when the modification trail is available. + private async Task TrailAvailableAsync(CancellationToken ct) + { + await _bootstrap.EnsureAsync(ct).ConfigureAwait(false); + return _bootstrap.SupportsModifiedTrail; + } + + private static uint[] Unsupported(int count) + { + var codes = new uint[count]; + Array.Fill(codes, HistorianStatusCodes.BadHistoryOperationUnsupported); + return codes; + } + /// public async Task> UpdateDataAsync(string tag, HistoryUpdateMode mode, IReadOnlyList values, HistoryUpdateContext context, CancellationToken ct) @@ -49,6 +72,7 @@ internal sealed class SqlServerHistorianDataUpdater : IHistorianDataUpdater ArgumentNullException.ThrowIfNull(tag); ArgumentNullException.ThrowIfNull(values); ArgumentNullException.ThrowIfNull(context); + if (!await TrailAvailableAsync(ct).ConfigureAwait(false)) return Unsupported(values.Count); var plan = HistoryUpdatePlanner.Plan(tag, mode, values); if (plan.Winners.Count == 0) return HistoryUpdatePlanner.Complete(plan, mode, new Dictionary()); @@ -68,6 +92,7 @@ internal sealed class SqlServerHistorianDataUpdater : IHistorianDataUpdater { ArgumentNullException.ThrowIfNull(tag); ArgumentNullException.ThrowIfNull(context); + if (!await TrailAvailableAsync(ct).ConfigureAwait(false)) return HistorianStatusCodes.BadHistoryOperationUnsupported; if (!HistorianTagNames.IsValid(tag)) return HistorianStatusCodes.BadInvalidArgument; var tagId = await _tags.TryGetTagIdAsync(tag, ct).ConfigureAwait(false); if (tagId is null) return HistoryUpdatePlanner.DeleteRawResult(tagKnown: false, 0); @@ -95,6 +120,7 @@ internal sealed class SqlServerHistorianDataUpdater : IHistorianDataUpdater ArgumentNullException.ThrowIfNull(tag); ArgumentNullException.ThrowIfNull(timesUtc); ArgumentNullException.ThrowIfNull(context); + if (!await TrailAvailableAsync(ct).ConfigureAwait(false)) return Unsupported(timesUtc.Count); if (!HistorianTagNames.IsValid(tag)) { var refused = new uint[timesUtc.Count]; diff --git a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer/SqlServerHistorianOptions.cs b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer/SqlServerHistorianOptions.cs index 4828e4ac..94ba91b0 100644 --- a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer/SqlServerHistorianOptions.cs +++ b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer/SqlServerHistorianOptions.cs @@ -18,8 +18,10 @@ public sealed partial class SqlServerHistorianOptions /// /// true creates the schema and applies missing scripts on first use; false only verifies - /// that the required schema version (SqlServerSchemaBootstrapper.RequiredVersion, currently 2) exists and - /// fails every historian call with a clear error otherwise. + /// that the base schema version (SqlServerSchemaBootstrapper.RequiredVersion, 1) exists and fails every + /// historian call with a clear error otherwise. A verified v1 schema (no Part 11 modification trail, script 002 + /// not applied) is served with modified reads and HistoryUpdate refused BadHistoryOperationUnsupported + /// and one Warning per boot (Gitea #674). /// public bool AutoCreateSchema { get; set; } = true; diff --git a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer/SqlServerHistorianProvider.cs b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer/SqlServerHistorianProvider.cs index 8b6b5bcd..80cfe919 100644 --- a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer/SqlServerHistorianProvider.cs +++ b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer/SqlServerHistorianProvider.cs @@ -99,7 +99,7 @@ public sealed class SqlServerHistorianProvider : IHistorianBackendProvider public IHistorianDataUpdater? CreateUpdater(IServiceProvider sp) { var s = Get(sp); - return new SqlServerHistorianDataUpdater(s.Conn, s.Catalog); + return new SqlServerHistorianDataUpdater(s.Conn, s.Bootstrap, s.Catalog); } /// diff --git a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer/SqlServerHistorianRetention.cs b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer/SqlServerHistorianRetention.cs index c8bc8e00..9d1a787c 100644 --- a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer/SqlServerHistorianRetention.cs +++ b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer/SqlServerHistorianRetention.cs @@ -11,7 +11,8 @@ namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer; /// DELETE TOP(n) batches (under the lock-escalation threshold). Samples are purged per tag so each /// delete is a seek on the clustered key (TagId, SourceTimeUtc); events seek on CX_Event_Time. /// A session sp_getapplock (timeout 0) serialises purgers that share one database; a loser skips. -/// To purge another table later, add it to . +/// To purge another table later, add it to . On a v1 schema (no Part 11 trail, Gitea #674) +/// the SampleModified purge is skipped and absent from the results. /// internal sealed class SqlServerHistorianRetention : IHistorianRetention { @@ -42,6 +43,9 @@ internal sealed class SqlServerHistorianRetention : IHistorianRetention /// Test seam for . internal Action ClearPool { get; set; } = SqlConnection.ClearPool; + /// Test seam: the release statement. Tests point it at a failing statement to exercise the pool-clear fallback. + internal string ReleaseSql { get; set; } = "EXEC sp_releaseapplock @Resource = @res, @LockOwner = 'Session';"; + public async Task> PurgeAsync(CancellationToken ct) { await _bootstrap.EnsureAsync(ct).ConfigureAwait(false); @@ -69,6 +73,8 @@ internal sealed class SqlServerHistorianRetention : IHistorianRetention return []; } + // #674: a v1 schema has no SampleModified table — skip its purge rather than fail the whole pass. + var trail = _bootstrap.SupportsModifiedTrail; long samples = 0, modified = 0; var tagIds = new List(); await using (var cmd = _conn.CreateCommand(conn, $"SELECT TagId FROM {_conn.Q("Tag")}")) @@ -82,21 +88,30 @@ internal sealed class SqlServerHistorianRetention : IHistorianRetention cutoff, id, ct).ConfigureAwait(false); // The Part 11 modification trail (#668) ages out on the SAME source-time cutoff as the samples it // describes; a purge never writes to it (no trigger, no OUTPUT INTO). - modified += await DeleteLoopAsync(conn, - $"DELETE TOP (@n) FROM {_conn.Q("SampleModified")} WHERE TagId = @id AND SourceTimeUtc < @cutoff", - cutoff, id, ct).ConfigureAwait(false); + if (trail) + { + modified += await DeleteLoopAsync(conn, + $"DELETE TOP (@n) FROM {_conn.Q("SampleModified")} WHERE TagId = @id AND SourceTimeUtc < @cutoff", + cutoff, id, ct).ConfigureAwait(false); + } } var events = await DeleteLoopAsync(conn, $"DELETE TOP (@n) FROM {_conn.Q("Event")} WHERE EventTimeUtc < @cutoff", cutoff, null, ct).ConfigureAwait(false); - return - [ - new RetentionPurgeResult("Sample", samples), - new RetentionPurgeResult("SampleModified", modified), - new RetentionPurgeResult("Event", events), - ]; + return trail + ? + [ + new RetentionPurgeResult("Sample", samples), + new RetentionPurgeResult("SampleModified", modified), + new RetentionPurgeResult("Event", events), + ] + : + [ + new RetentionPurgeResult("Sample", samples), + new RetentionPurgeResult("Event", events), + ]; } finally { @@ -137,8 +152,7 @@ internal sealed class SqlServerHistorianRetention : IHistorianRetention { try { - await using var cmd = _conn.CreateCommand(conn, - "EXEC sp_releaseapplock @Resource = @res, @LockOwner = 'Session';"); + await using var cmd = _conn.CreateCommand(conn, ReleaseSql); cmd.Parameters.Add(new SqlParameter("@res", SqlDbType.NVarChar, 255) { Value = LockResource }); await cmd.ExecuteNonQueryAsync(CancellationToken.None).ConfigureAwait(false); } diff --git a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer/SqlServerSchemaBootstrapper.cs b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer/SqlServerSchemaBootstrapper.cs index c1c1a782..2fd1d9b3 100644 --- a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer/SqlServerSchemaBootstrapper.cs +++ b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer/SqlServerSchemaBootstrapper.cs @@ -15,15 +15,27 @@ namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer; /// SQL, so a missing schema (or a SQL outage) does not serialise every write, read and alarm event behind one /// semaphore and one round-trip each. /// With AutoCreateSchema=false it only verifies MAX(Version) >= . +/// Schema v1 is accepted (Gitea #674). A database at version 1 (no SampleModified) still serves +/// live writes, raw / processed / at-time reads, alarm events and retention; only the Part 11 modification trail is +/// unavailable — is false, modified reads and HistoryUpdate report +/// BadHistoryOperationUnsupported without touching SQL, and one Warning is logged per instance (= per boot: the +/// provider shares one bootstrapper). Before #674 a v1 database refused EVERY historian call, continuous-historization +/// drains included, until script 002 was applied by hand. /// internal sealed partial class SqlServerSchemaBootstrapper { /// - /// The schema version this build requires: 2 = 002_sample_modified.sql (Part 11 modified-value - /// history, Gitea #668). An existing v1 schema is upgraded in place on the next bootstrap when - /// AutoCreateSchema is on; otherwise the script must be applied by hand. + /// The schema version this build requires to serve at all: 1 = 001_initial.sql. Gitea #674 lowered it from + /// 2 — a missing modification trail disables only the features that need it (). /// - public const int RequiredVersion = 2; + public const int RequiredVersion = 1; + + /// + /// The schema version that carries the Part 11 modification trail: 2 = 002_sample_modified.sql (Gitea #668). + /// An auto-creating bootstrap always upgrades to it; with AutoCreateSchema=false the script must be applied by + /// hand (and the node restarted — the verdict is taken once per instance). + /// + public const int ModifiedTrailVersion = 2; private const string LockResource = "OtOpcUaHistorianSchema"; private const int LockTimeoutMs = 30_000; @@ -34,6 +46,13 @@ internal sealed partial class SqlServerSchemaBootstrapper private readonly SemaphoreSlim _gate = new(1, 1); private readonly BootstrapFailureCache _failure; // guarded by _gate private volatile bool _ready; + private volatile bool _supportsModifiedTrail; + + /// + /// Whether the schema carries the Part 11 modification trail (version >= ). + /// Meaningful only after has succeeded (false before). Taken once per instance. + /// + public bool SupportsModifiedTrail => _supportsModifiedTrail; /// Creates the bootstrapper. /// Connection helper. @@ -99,6 +118,13 @@ internal sealed partial class SqlServerSchemaBootstrapper throw new HistorianSchemaMissingException( $"Historian schema [{_conn.Schema}] is missing version {RequiredVersion} (found {v}) and AutoCreateSchema is false. " + "Apply the docs/historian/sqlserver/NNN_*.sql scripts above version " + v + " in order, or enable AutoCreateSchema."); + _supportsModifiedTrail = v >= ModifiedTrailVersion; + if (!_supportsModifiedTrail) + _log.LogWarning("Historian schema [{Schema}] is at version {Version}, below {TrailVersion}, and AutoCreateSchema is false: " + + "the Part 11 modification trail is unavailable, so modified-value history reads and HistoryUpdate report " + + "BadHistoryOperationUnsupported and retention skips SampleModified. Live historization, reads, events and retention " + + "are unaffected. Apply docs/historian/sqlserver/002_sample_modified.sql and restart the node to enable them.", + _conn.Schema, v, ModifiedTrailVersion); } private async Task ApplyAsync(SqlConnection conn, CancellationToken ct) diff --git a/src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/RoleOverlay.cs b/src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/RoleOverlay.cs new file mode 100644 index 00000000..b7ce2047 --- /dev/null +++ b/src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/RoleOverlay.cs @@ -0,0 +1,28 @@ +using ZB.MOM.WW.OtOpcUa.Cluster; + +namespace ZB.MOM.WW.OtOpcUa.Host.Configuration; + +/// +/// Names the per-role appsettings overlay (appsettings.{suffix}.json) for a node. +/// +/// +/// The suffix is built from the node's PLAIN roles only. cluster-{ClusterId} roles are mesh-membership +/// tags (and carry a verbatim, operator-chosen ClusterId), not configuration profiles — folding one into the +/// filename produced names like appsettings.admin-cluster-MAIN-driver.json that never exist, so on every +/// node with a cluster role the shipped overlays silently never loaded. +/// +internal static class RoleOverlay +{ + /// + /// The overlay filename suffix: the plain (non-cluster) roles, ordinal-sorted and joined with -, or + /// when the node has none. + /// + public static string? Suffix(IReadOnlyList gossipedRoles) + { + var plain = gossipedRoles + .Where(r => !string.IsNullOrWhiteSpace(r) && !RoleParser.IsClusterRole(r)) + .OrderBy(r => r, StringComparer.Ordinal) + .ToList(); + return plain.Count == 0 ? null : string.Join('-', plain); + } +} diff --git a/src/Server/ZB.MOM.WW.OtOpcUa.Host/OpcUa/OtOpcUaServerHostedService.cs b/src/Server/ZB.MOM.WW.OtOpcUa.Host/OpcUa/OtOpcUaServerHostedService.cs index aff63c4d..78827f5f 100644 --- a/src/Server/ZB.MOM.WW.OtOpcUa.Host/OpcUa/OtOpcUaServerHostedService.cs +++ b/src/Server/ZB.MOM.WW.OtOpcUa.Host/OpcUa/OtOpcUaServerHostedService.cs @@ -252,13 +252,15 @@ public sealed class OtOpcUaServerHostedService : IHostedService, IAsyncDisposabl if (_primaryRoleSource is { } primaryRoleSource) { _server.SetNativeAlarmAckPrimaryGate(() => primaryRoleSource.ShouldActAsPrimary); + // Gitea #674: HistoryUpdate is single-writer — only the Primary rewrites history (BadInvalidState elsewhere). + _server.SetHistoryUpdatePrimaryGate(() => primaryRoleSource.ShouldActAsPrimary); } else { // Every driver node registers one (AddOtOpcUaRuntime), so this is a wiring regression, not a mode: // say so rather than let a Secondary silently go back to accepting acks it will revert. _logger.LogWarning( - "OtOpcUaServerHostedService: no IDriverPrimaryRoleSource registered — native-alarm acknowledges are accepted on every node regardless of redundancy role"); + "OtOpcUaServerHostedService: no IDriverPrimaryRoleSource registered — native-alarm acknowledges and HistoryUpdate are accepted on every node regardless of redundancy role"); } // Wire the reverse-path inbound operator-write gateway: a client write to a writable equipment-tag @@ -317,6 +319,11 @@ public sealed class OtOpcUaServerHostedService : IHostedService, IAsyncDisposabl _server.NodeManager.MaxConcurrentHistoryReadBatches = _serverHistorianOptions.MaxConcurrentHistoryReadBatches; _server.NodeManager.HistoryReadDeadline = _serverHistorianOptions.HistoryReadDeadline; + // Gitea #674: HistoryUpdate refuses a SourceTimestamp beyond now + HistoryUpdateMaxFutureSkew (BadOutOfRange), + // judged against the host clock. + _server.NodeManager.HistoryUpdateMaxFutureSkew = _serverHistorianOptions.HistoryUpdateMaxFutureSkew; + _server.NodeManager.Clock = _time; + // ServiceLevel publisher needs IServerInternal — only available after Start. if (_server.CurrentInstance is { } serverInternal) { @@ -382,6 +389,7 @@ public sealed class OtOpcUaServerHostedService : IHostedService, IAsyncDisposabl // Clear the native-alarm-ack router so a late ack doesn't Tell a stopping DriverHostActor. _server?.SetNativeAlarmAckRouter(null); _server?.SetNativeAlarmAckPrimaryGate(null); + _server?.SetHistoryUpdatePrimaryGate(null); // Restore the Null historian so a late HistoryRead doesn't hit a disposed read client. _server?.SetHistorianDataSource(null); _server?.SetHistorianDataUpdater(null); diff --git a/src/Server/ZB.MOM.WW.OtOpcUa.Host/Program.cs b/src/Server/ZB.MOM.WW.OtOpcUa.Host/Program.cs index 29dd14a3..ee5e79d7 100644 --- a/src/Server/ZB.MOM.WW.OtOpcUa.Host/Program.cs +++ b/src/Server/ZB.MOM.WW.OtOpcUa.Host/Program.cs @@ -90,8 +90,9 @@ builder.WebHost.UseStaticWebAssets(); builder.Host.UseWindowsService(options => options.ServiceName = "OtOpcUaHost"); // Per-role appsettings overlay: appsettings.{role}.json (single role) or appsettings.admin-driver.json -// (both). Optional — base appsettings.json carries enough to boot if these don't exist. -var roleSuffix = roles.Count == 0 ? null : string.Join('-', roles.OrderBy(r => r, StringComparer.Ordinal)); +// (both). Optional — base appsettings.json carries enough to boot if these don't exist. Built from the +// PLAIN roles only (#678.6): a cluster-{ClusterId} role in the name pointed at a file that never exists. +var roleSuffix = RoleOverlay.Suffix(roles); if (roleSuffix is not null) { builder.Configuration.AddJsonFile($"appsettings.{roleSuffix}.json", optional: true, reloadOnChange: true); diff --git a/src/Server/ZB.MOM.WW.OtOpcUa.Host/appsettings.driver.json b/src/Server/ZB.MOM.WW.OtOpcUa.Host/appsettings.driver.json index 1db8bd01..89cf7f6f 100644 --- a/src/Server/ZB.MOM.WW.OtOpcUa.Host/appsettings.driver.json +++ b/src/Server/ZB.MOM.WW.OtOpcUa.Host/appsettings.driver.json @@ -3,7 +3,7 @@ "MinimumLevel": { "Default": "Information", "Override": { - "Opc.Ua": "Debug", + "Opc.Ua": "Information", "Akka": "Information" } } diff --git a/src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/HistoryPaging.cs b/src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/HistoryPaging.cs index 8526d254..667d16a1 100644 --- a/src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/HistoryPaging.cs +++ b/src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/HistoryPaging.cs @@ -46,6 +46,11 @@ namespace ZB.MOM.WW.OtOpcUa.OpcUaServer; /// only: the normalised window start, so a resumed page keeps the original /// window and direction (the modified read resumes by key, not by moving the window start). /// +/// +/// only (Gitea #675): the read is newest-first over +/// (EndUtc, NextStartUtc]. A resumed events read takes its direction from here, never from the resume +/// request's details. Raw keeps deriving it from the window (NextStartUtc > EndUtc). +/// internal sealed record HistoryContinuationState( string Tagname, DateTime NextStartUtc, @@ -54,7 +59,8 @@ internal sealed record HistoryContinuationState( uint NumValuesPerNode, HistoryCursorKind Kind = HistoryCursorKind.Raw, long LastModifiedId = 0, - DateTime WindowStartUtc = default); + DateTime WindowStartUtc = default, + bool Reverse = false); /// The HistoryRead kind that issued a . internal enum HistoryCursorKind diff --git a/src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/OtOpcUaNodeManager.cs b/src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/OtOpcUaNodeManager.cs index 08c15da2..841d8605 100644 --- a/src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/OtOpcUaNodeManager.cs +++ b/src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/OtOpcUaNodeManager.cs @@ -102,7 +102,7 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2 /// / . private readonly Dictionary _alarmNotifierWiring = new(); /// Phase C: event-notifier folder NodeId-identifier → the event-history source - /// name passed to . The equipment-folder NodeId + /// name passed to . The equipment-folder NodeId /// identifier IS the equipment id, which IS the sourceName, so key and value are the same string; /// the map's presence (not its value) is what makes a folder an event-history source. Populated by /// only when a real historian is wired at promotion time, @@ -247,6 +247,48 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2 /// public Func? NativeAlarmAckPrimaryGate { get; set; } + /// + /// Gitea #674 — whether this node may serve an OPC UA HistoryUpdate right now: the redundancy data-plane + /// verdict (PrimaryGatePolicy.ShouldServiceAsPrimary, as published to IDriverPrimaryRoleSource) — + /// the same source as . + /// + /// false ⇒ every node of the request that passed authorization is refused BadInvalidState + /// and nothing reaches the updater. Rewriting history is a single-writer operation: the SQL backends' + /// prior-value capture (and QuestDB's per-process per-tag lock) assume one node serialises a tag's updates, + /// so both pair nodes accepting updates concurrently could record a wrong prior value in the modification + /// trail. HistoryRead is NOT gated — reads are served from any node. + /// + /// + /// Evaluated AFTER authorization (a caller who may not update learns nothing about the node's role). It must + /// be a cheap, non-blocking read (the host passes a volatile-backed property). Null (the default) allows — the + /// single-node posture, the same default UnmanagedDriverPrimaryRoleSource answers with. + /// + /// + public Func? HistoryUpdatePrimaryGate { get; set; } + + /// + /// Gitea #674 — how far beyond 's now a HistoryUpdate (Insert / Replace / Update) item's + /// SourceTimestamp may lie. An item past now + skew is refused BadOutOfRange before it + /// reaches any backend: a far-future row (a 2062 typo) is never a legitimate history edit, and on a backend whose + /// retention is judged against the newest timestamp it could age out every older partition. Mirrors + /// ServerHistorianOptions.HistoryUpdateMaxFutureSkew; the Host sets it at StartAsync. Non-positive + /// disables the bound. Default 1 day. Live recorder writes are not bounded by this. + /// + public TimeSpan HistoryUpdateMaxFutureSkew { get; set; } = TimeSpan.FromDays(1); + + /// + /// The clock the HistoryUpdate future-skew bound () reads. The Host + /// assigns its at StartAsync; tests inject a fixed clock. Defaults to + /// ; assigning null restores it. + /// + public TimeProvider Clock + { + get => _clock; + set => _clock = value ?? TimeProvider.System; + } + + private volatile TimeProvider _clock = TimeProvider.System; + private volatile IOpcUaNodeWriteGateway _nodeWriteGateway = NullOpcUaNodeWriteGateway.Instance; /// @@ -5161,6 +5203,7 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2 var endUtc = details.EndTime; var numValuesPerNode = details.NumValuesPerNode; var boundarySkip = 0; + bool reverse; if (resumed) { @@ -5180,36 +5223,59 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2 endUtc = state.EndUtc; boundarySkip = state.BoundarySkipCount; numValuesPerNode = state.NumValuesPerNode; + // The direction is the CURSOR's (Gitea #675), never re-derived from the resume request's details: a + // reverse read resumes reverse, a forward one forward. + reverse = state.Reverse; } + else + { + // Part 11: with one bound unset, an unset EndTime (MinValue) with a StartTime means "page FORWARD from + // StartTime" — normalise it to an open upper bound, or the cursor would store MinValue and the read would + // look reversed. An unset StartTime with an EndTime and a count cap means "page BACKWARD from EndTime" + // (#675, mirroring raw): the reverse window (MinValue, EndTime]. Without a count cap an end-only read + // keeps its historical forward [MinValue, EndTime) meaning. + if (endUtc == DateTime.MinValue && startUtc != DateTime.MinValue) + endUtc = new DateTime(DateTime.MaxValue.Ticks, DateTimeKind.Utc); // Utc kind: a proto Timestamp conversion rejects Unspecified + else if (numValuesPerNode > 0 && startUtc == DateTime.MinValue && endUtc != DateTime.MinValue) + (startUtc, endUtc) = (endUtc, new DateTime(0, DateTimeKind.Utc)); - // Part 11: an unset EndTime (MinValue) with a StartTime means "page forward from StartTime". Normalise it - // to an open upper bound, or the cursor would store MinValue and a backend would swap the reversed - // resume window. A resumed read is always forward (the cursor is only ever stored for a forward read), - // so its direction comes from the cursor, not from the resume request's details. - if (!resumed && endUtc == DateTime.MinValue && startUtc != DateTime.MinValue) - endUtc = new DateTime(DateTime.MaxValue.Ticks, DateTimeKind.Utc); // Utc kind: a proto Timestamp conversion rejects Unspecified - // Reverse event reads (start > end) are not paged: the cursor below is forward-only. - var forward = resumed || startUtc <= endUtc; + // Part 11: StartTime later than EndTime is a reverse read — events newest-first over (end, start]. + reverse = startUtc > endUtc; + } // NOT under the node-manager Lock — awaiting the async source is safe here. // NumValuesPerNode==0 means "no limit — return ALL values" per OPC UA Part 4/11, so it is translated - // to UNBOUNDED (EventMaxEvents) rather than the backend's <=0 default-cap sentinel. + // to UNBOUNDED (EventMaxEvents) rather than the backend's <=0 default-cap sentinel. A resumed page re-reads its + // boundary timestamp inclusively and then drops the `boundarySkip` ties already emitted, so it asks for that + // many more rows than the cap — otherwise every resumed page would come back short by the skip. + var fetch = EventMaxEvents(numValuesPerNode); + if (numValuesPerNode > 0 && boundarySkip > 0) + fetch = (int)Math.Min(int.MaxValue - 1L, (long)fetch + boundarySkip); var sourceResult = await _historianDataSource.ReadEventsAsync( - sourceName, startUtc, endUtc, EventMaxEvents(numValuesPerNode), ct).ConfigureAwait(false); + sourceName, startUtc, endUtc, fetch, reverse, ct).ConfigureAwait(false); // Truncation is the SOURCE's word (non-null continuation point) — never inferred from the count, so a // page that happens to be exactly cap-sized but complete never emits a point. - var truncated = sourceResult.ContinuationPoint is not null && numValuesPerNode > 0 && forward; + var truncated = sourceResult.ContinuationPoint is not null && numValuesPerNode > 0; IReadOnlyList events = sourceResult.Events; if (resumed && boundarySkip > 0) { - // Drop the events at the boundary timestamp already emitted on the prior page. + // Drop the events at the boundary timestamp already emitted on the prior page. The resumed read is in + // the cursor's direction, so in both directions those ties lead the page. var i = 0; while (i < events.Count && i < boundarySkip && events[i].EventTimeUtc == startUtc) i++; if (i > 0) events = events.Skip(i).ToList(); } + // The skip over-fetch can leave more than the cap (fewer boundary ties came back than were skipped — pruned + // between pages): keep the cap, and the dropped tail means the read continues. + if (numValuesPerNode > 0 && events.Count > numValuesPerNode) + { + events = events.Take((int)Math.Min(int.MaxValue, numValuesPerNode)).ToList(); + truncated = true; + } + // The cursor to store, if any. Saved only AFTER the page is projected, so a projection throw can // never orphan a stored cursor. HistoryContinuationState? pendingCursor = null; @@ -5222,10 +5288,14 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2 // with an explicit bounded cap and page WITHIN it (same scheme as raw). var resolution = Math.Max(1L, _historianDataSource.TimestampResolutionTicks); var overfetchCap = (int)Math.Min(int.MaxValue - 1L, (long)Math.Max(1, MaxTieClusterOverfetch) + 1); - var cluster = (await _historianDataSource.ReadEventsAsync( - sourceName, startUtc, startUtc.AddTicks(resolution), overfetchCap, ct) + IReadOnlyList cluster = (await _historianDataSource.ReadEventsAsync( + sourceName, startUtc, startUtc.AddTicks(resolution), overfetchCap, reverse: false, ct) .ConfigureAwait(false)).Events; + // The over-fetch reads the cluster FORWARD (ascending EventId), but a reverse read emitted its ties — and + // counted boundarySkip — in descending order. Slice in the read's own direction, as raw does. + if (reverse) cluster = cluster.Reverse().ToList(); + if (cluster.Count > MaxTieClusterOverfetch) { _logger.LogError( @@ -5240,7 +5310,7 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2 HistoryPaging.SliceTieCluster( cluster.Count, boundarySkip, numValuesPerNode, startUtc, endUtc, out var sliceStart, out var sliceCount, out var nextStartUtc, out var nextSkip, - reverse: false, resolutionTicks: resolution); + reverse: reverse, resolutionTicks: resolution); var slice = new List(sliceCount); for (var i = sliceStart; i < sliceStart + sliceCount; i++) slice.Add(cluster[i]); @@ -5249,17 +5319,20 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2 if (nextStartUtc is { } resumeAt) { pendingCursor = new HistoryContinuationState( - sourceName ?? string.Empty, resumeAt, endUtc, nextSkip, numValuesPerNode, HistoryCursorKind.Events); + sourceName ?? string.Empty, resumeAt, endUtc, nextSkip, numValuesPerNode, HistoryCursorKind.Events, + Reverse: reverse); } } else if (truncated && events.Count > 0) { - // Cursor from the last event: resume at its time INCLUSIVE, skipping the trailing ties emitted. + // Cursor from the last event: resume at its time INCLUSIVE, skipping the trailing ties emitted. In a + // reverse read the last event is the OLDEST on the page, and the resumed (end, T] read starts there. var boundary = events[^1].EventTimeUtc; var skip = 0; for (var i = events.Count - 1; i >= 0 && events[i].EventTimeUtc == boundary; i--) skip++; pendingCursor = new HistoryContinuationState( - sourceName ?? string.Empty, boundary, endUtc, skip, numValuesPerNode, HistoryCursorKind.Events); + sourceName ?? string.Empty, boundary, endUtc, skip, numValuesPerNode, HistoryCursorKind.Events, + Reverse: reverse); } var historyEvent = ProjectEvents(events, selectClauses); @@ -5280,6 +5353,15 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2 errors[handle.Index] = StatusCodes.BadTimeout; results[handle.Index] = new SdkHistoryReadResult { StatusCode = StatusCodes.BadTimeout }; } + catch (NotSupportedException ex) + { + // The backend cannot serve this read's shape — today a reverse (newest-first) read on Gateway / Null + // (#675). A client request the backend refuses, not a backend fault: Warning, not Error. + _logger.LogWarning("OtOpcUaNodeManager: HistoryReadEvents for node {NodeId} is not supported by the historian backend: {Reason}", + handle.NodeId, ex.Message); + errors[handle.Index] = StatusCodes.BadHistoryOperationUnsupported; + results[handle.Index] = new SdkHistoryReadResult { StatusCode = StatusCodes.BadHistoryOperationUnsupported }; + } catch (Exception ex) { // One node's backend failure must not throw out of the batch — surface Bad for THIS node @@ -5373,6 +5455,8 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2 List nodesToProcess, IDictionary cache) { + // #674: one "now" per request, so every item of the batch is judged against the same bound. + var latest = LatestAcceptedSourceTime(); ServeHistoryUpdate(context, nodesToProcess, results, errors, handle => { var details = nodesToUpdate[handle.Index]; @@ -5386,7 +5470,7 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2 } var items = (IList?)details.UpdateValues ?? Array.Empty(); - return (updater, tagname, who, ct) => UpdateDataAsync(updater, tagname, mode, items, who, ct); + return (updater, tagname, who, ct) => UpdateDataAsync(updater, tagname, mode, items, latest, who, ct); }); } @@ -5448,22 +5532,40 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2 }); } + /// + /// #674: the latest SourceTimestamp a HistoryUpdate item may carry — 's now plus + /// — or null when the bound is disabled (non-positive skew). + /// Saturates at rather than overflowing for an absurd configured skew. + /// + /// The inclusive upper bound, or null for unbounded. + private DateTime? LatestAcceptedSourceTime() + { + var skew = HistoryUpdateMaxFutureSkew; + if (skew <= TimeSpan.Zero) return null; + var now = _clock.GetUtcNow().UtcDateTime; + return skew.Ticks > DateTime.MaxValue.Ticks - now.Ticks + ? new DateTime(DateTime.MaxValue.Ticks, DateTimeKind.Utc) + : now + skew; + } + /// /// Applies one node's UpdateDataDetails: items the server can refuse on its own - /// (SourceTimestamp == DateTime.MinValue ⇒ BadInvalidTimestamp; a value - /// cannot type ⇒ BadTypeMismatch) are answered locally, - /// the rest go to the updater in ONE call, and the per-item codes are stitched back in request order. + /// (SourceTimestamp == DateTime.MinValue ⇒ BadInvalidTimestamp; a SourceTimestamp later than + /// ⇒ BadOutOfRange, #674; a value + /// cannot type ⇒ BadTypeMismatch) are answered locally, the rest go to the updater in ONE call, and the + /// per-item codes are stitched back in request order. /// /// The wired updater. /// The node's historian tagname. /// The update mode. /// The request's values, in order. + /// The latest accepted source time (); null = unbounded. /// The caller, recorded in the modification trail. /// The request deadline. /// The node's result, carrying one operation result per item. private static async Task UpdateDataAsync( IHistorianDataUpdater updater, string tagname, HistoryUpdateMode mode, IList items, - HistoryUpdateContext who, CancellationToken ct) + DateTime? latest, HistoryUpdateContext who, CancellationToken ct) { var codes = new uint[items.Count]; var toSend = new List(items.Count); @@ -5479,6 +5581,11 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2 { codes[i] = StatusCodes.BadInvalidTimestamp; } + else if (latest is { } bound && item.SourceTimestamp > bound) + { + // #674: a far-future history edit is refused before any backend sees it (wire timestamps are UTC). + codes[i] = StatusCodes.BadOutOfRange; + } else if (!HistorianValue.TryFromObject(item.Value, out var value)) { codes[i] = StatusCodes.BadTypeMismatch; @@ -5529,6 +5636,21 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2 nodesToProcess = AuthorizeHistoryUpdate(context, nodesToProcess, results, errors); if (nodesToProcess.Count == 0) return; + // #674 — PRIMARY GATE, after authorization (a caller who may not update learns nothing about the node's role). + // Off-Primary every authorized node is refused BadInvalidState and nothing reaches the updater. + if (HistoryUpdatePrimaryGate is { } primary && !primary()) + { + var refusal = new ServiceResult(StatusCodes.BadInvalidState, + new LocalizedText("This server is not the redundancy Primary; update history on the Primary (the server with the higher ServiceLevel).")); + foreach (var handle in nodesToProcess) + { + errors[handle.Index] = refusal; + results[handle.Index] = new HistoryUpdateResult { StatusCode = StatusCodes.BadInvalidState }; + } + + return; + } + // Who is rewriting history — recorded on every modification-trail row (Part 11 ModificationInfo.UserName, #668). var who = HistoryUpdateCallerOf(context); @@ -5619,7 +5741,7 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2 private static int ClampToInt(uint value) => value > int.MaxValue ? int.MaxValue : (int)value; /// Map a HistoryRead-Events NumValuesPerNode request cap onto the - /// maxEvents argument, honouring the OPC UA + /// maxEvents argument, honouring the OPC UA /// Part 4/11 semantics that NumValuesPerNode == 0 means "no limit — return ALL values". /// We translate 0 to UNBOUNDED () — a very large positive cap — rather than /// the backend's maxEvents <= 0 "use the default cap" sentinel, so a client asking for the diff --git a/src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/OtOpcUaSdkServer.cs b/src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/OtOpcUaSdkServer.cs index 11eab2fb..e566b347 100644 --- a/src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/OtOpcUaSdkServer.cs +++ b/src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/OtOpcUaSdkServer.cs @@ -83,6 +83,21 @@ public sealed class OtOpcUaSdkServer : StandardServer return true; } + /// + /// Gitea #674 — wire the redundancy verdict that decides whether this node serves an OPC UA HistoryUpdate + /// (); on any other node every authorized node of a + /// HistoryUpdate is refused BadInvalidState. Passing null clears it (allow). No-op (returns + /// false) when the node manager has not been created yet (mirrors ). + /// + /// A cheap, non-blocking read of the data-plane Primary verdict; null to allow. + /// true when set on a live node manager; false when no node manager exists yet. + public bool SetHistoryUpdatePrimaryGate(Func? isPrimary) + { + if (_otOpcUaNodeManager is null) return false; + _otOpcUaNodeManager.HistoryUpdatePrimaryGate = isPrimary; + return true; + } + /// /// Wire the reverse-path gateway for inbound operator writes to writable equipment-tag nodes onto the /// created . The host calls this after start with an diff --git a/src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Historian/ServerHistorianOptions.cs b/src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Historian/ServerHistorianOptions.cs index fc6cc012..eea6046f 100644 --- a/src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Historian/ServerHistorianOptions.cs +++ b/src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Historian/ServerHistorianOptions.cs @@ -122,6 +122,16 @@ public sealed class ServerHistorianOptions /// public int MaxTimeWeightedRawRows { get; init; } = 1_000_000; + /// + /// Gitea #674: how far into the future an OPC UA HistoryUpdate (Insert / Replace / Update) item's + /// SourceTimestamp may lie, measured from the server clock. An item beyond now + skew is refused + /// BadOutOfRange by the node manager before any backend sees it — a far-future row is never a legitimate + /// history edit, and on a backend that judges retention against the newest timestamp it could age out every + /// older partition. Applies to every backend. The live recorder is not bounded. Non-positive disables the bound. + /// Default 1 day. + /// + public TimeSpan HistoryUpdateMaxFutureSkew { get; init; } = TimeSpan.FromDays(1); + /// Returns operator-facing misconfiguration warnings for an Enabled historian /// (empty when disabled or correctly configured). Pure — the registration logs each entry. /// diff --git a/tests/Client/ZB.MOM.WW.OtOpcUa.Client.CLI.Tests/Fakes/FakeOpcUaClientService.cs b/tests/Client/ZB.MOM.WW.OtOpcUa.Client.CLI.Tests/Fakes/FakeOpcUaClientService.cs index 51a1b6d6..4fb22ff6 100644 --- a/tests/Client/ZB.MOM.WW.OtOpcUa.Client.CLI.Tests/Fakes/FakeOpcUaClientService.cs +++ b/tests/Client/ZB.MOM.WW.OtOpcUa.Client.CLI.Tests/Fakes/FakeOpcUaClientService.cs @@ -108,6 +108,9 @@ public sealed class FakeOpcUaClientService : IOpcUaClientService /// public Dictionary> BrowseResultsByParent { get; } = new(); + /// Gets or sets an exception thrown by HistoryReadRawAsync (simulates a refused read). + public Exception? HistoryReadException { get; set; } + /// Gets or sets the history read result returned by HistoryReadRawAsync and HistoryReadAggregateAsync. public IReadOnlyList HistoryReadResult { get; set; } = new List { @@ -323,6 +326,7 @@ public sealed class FakeOpcUaClientService : IOpcUaClientService NodeId nodeId, DateTime startTime, DateTime endTime, int maxValues = 1000, CancellationToken ct = default) { HistoryReadRawCalls.Add((nodeId, startTime, endTime, maxValues)); + if (HistoryReadException != null) throw HistoryReadException; return Task.FromResult(HistoryReadResult); } diff --git a/tests/Client/ZB.MOM.WW.OtOpcUa.Client.CLI.Tests/HistoryReadCommandTests.cs b/tests/Client/ZB.MOM.WW.OtOpcUa.Client.CLI.Tests/HistoryReadCommandTests.cs index a5ba8af2..e0d77307 100644 --- a/tests/Client/ZB.MOM.WW.OtOpcUa.Client.CLI.Tests/HistoryReadCommandTests.cs +++ b/tests/Client/ZB.MOM.WW.OtOpcUa.Client.CLI.Tests/HistoryReadCommandTests.cs @@ -197,4 +197,53 @@ public class HistoryReadCommandTests using var console = TestConsoleHelper.CreateConsole(); await Should.ThrowAsync(async () => await command.ExecuteAsync(console)); } + + /// #677.5: the new aggregate names parse (and the aliases). + [Theory] + [InlineData("TimeAverage", AggregateType.TimeAverage)] + [InlineData("timeaverage2", AggregateType.TimeAverage2)] + [InlineData("Total", AggregateType.Total)] + [InlineData("total2", AggregateType.Total2)] + [InlineData("Interpolative", AggregateType.Interpolative)] + [InlineData("interp", AggregateType.Interpolative)] + public async Task Execute_NewAggregates_Parse(string name, AggregateType expected) + { + var fakeService = new FakeOpcUaClientService(); + var command = new HistoryReadCommand(new FakeOpcUaClientServiceFactory(fakeService)) + { + Url = "opc.tcp://localhost:4840", + NodeId = "ns=2;s=HistNode", + Aggregate = name, + }; + + using var console = TestConsoleHelper.CreateConsole(); + await command.ExecuteAsync(console); + + fakeService.HistoryReadAggregateCalls.ShouldHaveSingleItem().Aggregate.ShouldBe(expected); + } + + /// #674.6: a refused read is a one-line CommandException (non-zero exit), not a stack trace. + [Fact] + public async Task Execute_RefusedRead_ThrowsCommandException_WithOneLineMessage() + { + var fakeService = new FakeOpcUaClientService + { + HistoryReadException = new ServiceResultException(StatusCodes.BadHistoryOperationUnsupported, + "HistoryRead of ns=2;s=HistNode failed: BadHistoryOperationUnsupported (0x80B40000)."), + }; + var command = new HistoryReadCommand(new FakeOpcUaClientServiceFactory(fakeService)) + { + Url = "opc.tcp://localhost:4840", + NodeId = "ns=2;s=HistNode", + }; + + using var console = TestConsoleHelper.CreateConsole(); + var ex = await Should.ThrowAsync(async () => await command.ExecuteAsync(console)); + + ex.Message.ShouldContain("BadHistoryOperationUnsupported"); + ex.Message.ShouldNotContain("\n"); + ex.Message.ShouldNotContain(" at "); + ex.ExitCode.ShouldNotBe(0); + fakeService.DisconnectCalled.ShouldBeTrue(); + } } diff --git a/tests/Client/ZB.MOM.WW.OtOpcUa.Client.Shared.Tests/Adapters/EndpointSelectorTests.cs b/tests/Client/ZB.MOM.WW.OtOpcUa.Client.Shared.Tests/Adapters/EndpointSelectorTests.cs index 98034571..ba917d82 100644 --- a/tests/Client/ZB.MOM.WW.OtOpcUa.Client.Shared.Tests/Adapters/EndpointSelectorTests.cs +++ b/tests/Client/ZB.MOM.WW.OtOpcUa.Client.Shared.Tests/Adapters/EndpointSelectorTests.cs @@ -160,4 +160,67 @@ public class EndpointSelectorTests Should.Throw(() => EndpointSelector.SelectBest(Array.Empty(), "", MessageSecurityMode.None)); } + + /// #678.5: a server advertising its internal port must not redirect the session off the mapped port. + [Fact] + public void SelectBest_RewritesHostAndPort_WhenSchemeMatches() + { + var endpoints = new[] { Ep(MessageSecurityMode.None, SecurityPolicies.None, "opc.tcp://internal-host:4840/OtOpcUa") }; + + var best = EndpointSelector.SelectBest(endpoints, "opc.tcp://localhost:4841", MessageSecurityMode.None); + + var uri = new Uri(best.EndpointUrl); + uri.Host.ShouldBe("localhost"); + uri.Port.ShouldBe(4841); + uri.AbsolutePath.ShouldBe("/OtOpcUa"); + } + + /// A different scheme (e.g. https endpoint advertised to an opc.tcp caller) is left untouched. + [Fact] + public void SelectBest_LeavesEndpointUntouched_WhenSchemeDiffers() + { + var endpoints = new[] { Ep(MessageSecurityMode.None, SecurityPolicies.None, "https://internal-host:4843/x") }; + + var best = EndpointSelector.SelectBest(endpoints, "opc.tcp://localhost:4841", MessageSecurityMode.None); + + new Uri(best.EndpointUrl).Host.ShouldBe("internal-host"); + new Uri(best.EndpointUrl).Port.ShouldBe(4843); + } + + /// Identical host, port and scheme: nothing to rewrite. + [Fact] + public void SelectBest_Unchanged_WhenHostPortAndSchemeMatch() + { + var endpoints = new[] { Ep(MessageSecurityMode.None, SecurityPolicies.None, "opc.tcp://server:4840") }; + + var best = EndpointSelector.SelectBest(endpoints, "opc.tcp://server:4840", MessageSecurityMode.None); + + best.EndpointUrl.ShouldBe("opc.tcp://server:4840"); + } + + /// A URL typed without a port keeps the advertised port (opc.tcp has no default; Uri.Port is -1). + [Fact] + public void SelectBest_KeepsAdvertisedPort_WhenRequestedUrlHasNoPort() + { + var endpoints = new[] { Ep(MessageSecurityMode.None, SecurityPolicies.None, "opc.tcp://internal-host:4840/OtOpcUa") }; + + var best = EndpointSelector.SelectBest(endpoints, "opc.tcp://localhost", MessageSecurityMode.None); + + var uri = new Uri(best.EndpointUrl); + uri.Host.ShouldBe("localhost"); + uri.Port.ShouldBe(4840); + } + + /// An IPv6 host is rewritten with its brackets intact. + [Fact] + public void SelectBest_RewritesIpv6Host() + { + var endpoints = new[] { Ep(MessageSecurityMode.None, SecurityPolicies.None, "opc.tcp://internal-host:4840/OtOpcUa") }; + + var best = EndpointSelector.SelectBest(endpoints, "opc.tcp://[::1]:4841", MessageSecurityMode.None); + + var uri = new Uri(best.EndpointUrl); + uri.Host.ShouldBe("[::1]"); + uri.Port.ShouldBe(4841); + } } diff --git a/tests/Client/ZB.MOM.WW.OtOpcUa.Client.Shared.Tests/Helpers/AggregateTypeMapperTests.cs b/tests/Client/ZB.MOM.WW.OtOpcUa.Client.Shared.Tests/Helpers/AggregateTypeMapperTests.cs index 5a782b95..663ab473 100644 --- a/tests/Client/ZB.MOM.WW.OtOpcUa.Client.Shared.Tests/Helpers/AggregateTypeMapperTests.cs +++ b/tests/Client/ZB.MOM.WW.OtOpcUa.Client.Shared.Tests/Helpers/AggregateTypeMapperTests.cs @@ -18,6 +18,11 @@ public class AggregateTypeMapperTests [InlineData(AggregateType.Start)] [InlineData(AggregateType.End)] [InlineData(AggregateType.StandardDeviation)] + [InlineData(AggregateType.TimeAverage)] + [InlineData(AggregateType.TimeAverage2)] + [InlineData(AggregateType.Total)] + [InlineData(AggregateType.Total2)] + [InlineData(AggregateType.Interpolative)] public void ToNodeId_ReturnsNonNullForAllValues(AggregateType aggregate) { var nodeId = AggregateTypeMapper.ToNodeId(aggregate); @@ -82,4 +87,15 @@ public class AggregateTypeMapperTests Should.Throw(() => AggregateTypeMapper.ToNodeId((AggregateType)99)); } + + /// #677.5: the time-weighted Part 13 aggregates map to their ObjectIds. + [Fact] + public void ToNodeId_TimeWeightedAggregates_MapCorrectly() + { + AggregateTypeMapper.ToNodeId(AggregateType.TimeAverage).ShouldBe(ObjectIds.AggregateFunction_TimeAverage); + AggregateTypeMapper.ToNodeId(AggregateType.TimeAverage2).ShouldBe(ObjectIds.AggregateFunction_TimeAverage2); + AggregateTypeMapper.ToNodeId(AggregateType.Total).ShouldBe(ObjectIds.AggregateFunction_Total); + AggregateTypeMapper.ToNodeId(AggregateType.Total2).ShouldBe(ObjectIds.AggregateFunction_Total2); + AggregateTypeMapper.ToNodeId(AggregateType.Interpolative).ShouldBe(ObjectIds.AggregateFunction_Interpolative); + } } \ No newline at end of file diff --git a/tests/Client/ZB.MOM.WW.OtOpcUa.Client.Shared.Tests/Models/ModelConstructionTests.cs b/tests/Client/ZB.MOM.WW.OtOpcUa.Client.Shared.Tests/Models/ModelConstructionTests.cs index ed88f8da..3d3a32a4 100644 --- a/tests/Client/ZB.MOM.WW.OtOpcUa.Client.Shared.Tests/Models/ModelConstructionTests.cs +++ b/tests/Client/ZB.MOM.WW.OtOpcUa.Client.Shared.Tests/Models/ModelConstructionTests.cs @@ -136,6 +136,6 @@ public class ModelConstructionTests [Fact] public void AggregateType_Enum_HasExpectedValues() { - Enum.GetValues().Length.ShouldBe(7); + Enum.GetValues().Length.ShouldBe(12); } } \ No newline at end of file diff --git a/tests/Client/ZB.MOM.WW.OtOpcUa.Client.UI.Tests/HistoryViewModelTests.cs b/tests/Client/ZB.MOM.WW.OtOpcUa.Client.UI.Tests/HistoryViewModelTests.cs index e0b65e5e..e056f78b 100644 --- a/tests/Client/ZB.MOM.WW.OtOpcUa.Client.UI.Tests/HistoryViewModelTests.cs +++ b/tests/Client/ZB.MOM.WW.OtOpcUa.Client.UI.Tests/HistoryViewModelTests.cs @@ -141,7 +141,7 @@ public class HistoryViewModelTests public void AggregateTypes_ContainsNullForRaw() { _vm.AggregateTypes.ShouldContain((AggregateType?)null); - _vm.AggregateTypes.Count.ShouldBe(8); // null + 7 enum values + _vm.AggregateTypes.Count.ShouldBe(13); // null + 12 enum values } /// Verifies that the Clear method resets the view model state. diff --git a/tests/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions.Tests/Historian/IHistorianDataSourceContractTests.cs b/tests/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions.Tests/Historian/IHistorianDataSourceContractTests.cs index 458e6796..847c6bad 100644 --- a/tests/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions.Tests/Historian/IHistorianDataSourceContractTests.cs +++ b/tests/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions.Tests/Historian/IHistorianDataSourceContractTests.cs @@ -45,9 +45,10 @@ public sealed class IHistorianDataSourceContractTests [InlineData("ReadEventsAsync", typeof(Task))] public void ReadMethods_ReturnExpectedTaskShape(string methodName, Type expectedReturnType) { - var method = typeof(IHistorianDataSource).GetMethod(methodName); - method.ShouldNotBeNull(); - method!.ReturnType.ShouldBe(expectedReturnType); + // ReadEventsAsync is overloaded (the #675 direction-aware overload) — every overload must keep the shape. + var methods = Overloads(methodName); + methods.ShouldNotBeEmpty(); + foreach (var method in methods) method.ReturnType.ShouldBe(expectedReturnType); } /// Verifies that GetHealthSnapshot is synchronous. @@ -146,11 +147,14 @@ public sealed class IHistorianDataSourceContractTests [InlineData("ReadEventsAsync", "maxEvents", typeof(int))] public void HistoryRead_MaxParameter_TypePinned(string methodName, string parameterName, Type expectedType) { - var method = typeof(IHistorianDataSource).GetMethod(methodName); - method.ShouldNotBeNull(); - var parameter = method!.GetParameters().FirstOrDefault(p => p.Name == parameterName); - parameter.ShouldNotBeNull($"Method {methodName} should expose a '{parameterName}' parameter."); - parameter!.ParameterType.ShouldBe(expectedType); + var methods = Overloads(methodName); + methods.ShouldNotBeEmpty(); + foreach (var method in methods) + { + var parameter = method.GetParameters().FirstOrDefault(p => p.Name == parameterName); + parameter.ShouldNotBeNull($"Method {methodName} should expose a '{parameterName}' parameter."); + parameter!.ParameterType.ShouldBe(expectedType); + } } /// @@ -163,13 +167,17 @@ public sealed class IHistorianDataSourceContractTests [Fact] public void ReadEventsAsync_sentinel_and_continuation_contract_types_pinned() { - var method = typeof(IHistorianDataSource).GetMethod("ReadEventsAsync"); - method.ShouldNotBeNull(); + var methods = Overloads("ReadEventsAsync"); + methods.Count.ShouldBe(2, "the five-argument read plus the #675 direction-aware overload"); + methods.ShouldContain(m => m.GetParameters().Any(p => p.Name == "reverse" && p.ParameterType == typeof(bool))); - var maxEventsParam = method!.GetParameters().FirstOrDefault(p => p.Name == "maxEvents"); - maxEventsParam.ShouldNotBeNull("maxEvents parameter must exist for the sentinel contract"); - maxEventsParam!.ParameterType.ShouldBe(typeof(int), - "maxEvents is int (not uint) so callers can pass <=0 as a 'use backend default cap' sentinel"); + foreach (var method in methods) + { + var maxEventsParam = method.GetParameters().FirstOrDefault(p => p.Name == "maxEvents"); + maxEventsParam.ShouldNotBeNull("maxEvents parameter must exist for the sentinel contract"); + maxEventsParam!.ParameterType.ShouldBe(typeof(int), + "maxEvents is int (not uint) so callers can pass <=0 as a 'use backend default cap' sentinel"); + } // HistoricalEventsResult must carry a nullable ContinuationPoint so implementations // can signal 'more results exist' (non-null) vs 'all results returned' (null). @@ -179,4 +187,7 @@ public sealed class IHistorianDataSourceContractTests continuationProp!.PropertyType.ShouldBe(typeof(byte[]), "ContinuationPoint is byte[]? — null means no more pages, non-null means caller must page"); } + + private static List Overloads(string name) => + typeof(IHistorianDataSource).GetMethods().Where(m => m.Name == name).ToList(); } diff --git a/tests/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions.Tests/Historian/NullHistorianDataSourceTests.cs b/tests/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions.Tests/Historian/NullHistorianDataSourceTests.cs index 6ed8afb7..e9045e7b 100644 --- a/tests/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions.Tests/Historian/NullHistorianDataSourceTests.cs +++ b/tests/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions.Tests/Historian/NullHistorianDataSourceTests.cs @@ -67,6 +67,20 @@ public sealed class NullHistorianDataSourceTests result.ContinuationPoint.ShouldBeNull(); } + /// + /// #675: the direction-aware overload's default — forward is the five-argument read, reverse is refused + /// (the server answers BadHistoryOperationUnsupported). + /// + [Fact] + public async Task Direction_aware_ReadEventsAsync_forwards_forward_and_refuses_reverse() + { + IHistorianDataSource source = Source; + (await source.ReadEventsAsync(null, DateTime.UtcNow.AddHours(-1), DateTime.UtcNow, 0, reverse: false, Ct)) + .Events.ShouldBeEmpty(); + await Should.ThrowAsync(() => + source.ReadEventsAsync(null, DateTime.UtcNow, DateTime.UtcNow.AddHours(-1), 0, reverse: true, Ct)); + } + /// GetHealthSnapshot reports a fully-disabled (zeroed, disconnected, no-node) shape. [Fact] public void GetHealthSnapshot_is_disabled() diff --git a/tests/Drivers/Historian.Conformance/EventReadConformance.cs b/tests/Drivers/Historian.Conformance/EventReadConformance.cs new file mode 100644 index 00000000..8d8110ca --- /dev/null +++ b/tests/Drivers/Historian.Conformance/EventReadConformance.cs @@ -0,0 +1,106 @@ +using Shouldly; +using Xunit; +using ZB.MOM.WW.OtOpcUa.Core.Abstractions; + +namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Conformance; + +/// +/// The direction-aware event read (Gitea #675): forward is [lo, hi) ascending by (EventTimeUtc, EventId); +/// reverse is the Part 11 reverse window (lo, hi] DESCENDING by the same key, so a capped reverse read returns the +/// LATEST events and flags truncation. Paging a reverse read the way the server does — resume at the last returned +/// time inclusive and drop the boundary ties already emitted — must visit every event exactly once, newest-first, +/// through a same-timestamp tie cluster. +/// +public abstract class EventReadConformance : ConformanceBase +{ + private static readonly CancellationToken Ct = CancellationToken.None; + + /// e0@0, e1@1, then the ties t0..t4@2, then e5@3 and e6@4 (all source "s"), plus o0@2 from another source. + private async Task SeedAsync() + { + await B.SeedEventsAsync( + (At(0), "e0", "s"), (At(1), "e1", "s"), + (At(2), "t0", "s"), (At(2), "t1", "s"), (At(2), "t2", "s"), (At(2), "t3", "s"), (At(2), "t4", "s"), + (At(3), "e5", "s"), (At(4), "e6", "s"), + (At(2), "o0", "other")); + return B.CreateDataSource(); + } + + private static readonly string[] NewestFirst = ["e6", "e5", "t4", "t3", "t2", "t1", "t0", "e1", "e0"]; + + [Fact] + public async Task Reverse_read_is_newest_first_by_time_then_EventId_descending() + { + using var src = await SeedAsync(); + var r = await src.ReadEventsAsync("s", At(-1), At(10), 0, reverse: true, Ct); + r.Events.Select(e => e.EventId).ShouldBe(NewestFirst); + r.ContinuationPoint.ShouldBeNull(); + } + + [Fact] + public async Task Reverse_window_is_end_exclusive_start_inclusive_whichever_order_the_bounds_come_in() + { + using var src = await SeedAsync(); + // (At(1), At(3)]: e1 at the lower bound is excluded, e5 at the upper bound included. + (await src.ReadEventsAsync("s", At(3), At(1), 0, reverse: true, Ct)).Events.Select(e => e.EventId) + .ShouldBe(["e5", "t4", "t3", "t2", "t1", "t0"]); + (await src.ReadEventsAsync("s", At(1), At(3), 0, reverse: true, Ct)).Events.Select(e => e.EventId) + .ShouldBe(["e5", "t4", "t3", "t2", "t1", "t0"]); + // Forward is unchanged: [At(1), At(3)) ascending. + (await src.ReadEventsAsync("s", At(1), At(3), 0, reverse: false, Ct)).Events.Select(e => e.EventId) + .ShouldBe(["e1", "t0", "t1", "t2", "t3", "t4"]); + } + + [Fact] + public async Task A_capped_reverse_read_keeps_the_LATEST_events_and_flags_truncation() + { + using var src = await SeedAsync(); + var r = await src.ReadEventsAsync("s", At(10), At(-1), 3, reverse: true, Ct); + r.Events.Select(e => e.EventId).ShouldBe(["e6", "e5", "t4"]); + r.ContinuationPoint.ShouldNotBeNull(); + // An exact fit is complete. + (await src.ReadEventsAsync("s", At(10), At(-1), 9, reverse: true, Ct)).ContinuationPoint.ShouldBeNull(); + } + + [Fact] + public async Task Paging_a_reverse_read_through_a_tie_cluster_visits_every_event_once_newest_first() + { + using var src = await SeedAsync(); + foreach (var cap in new[] { 1, 2, 3, 4 }) + { + var all = new List(); + var lo = At(-1); + var from = At(10); + var skip = 0; + for (var page = 0; ; page++) + { + page.ShouldBeLessThan(40, "paging must terminate"); + var r = await src.ReadEventsAsync("s", from, lo, cap + skip, reverse: true, Ct); + var events = r.Events.ToList(); + var drop = 0; + while (drop < skip && drop < events.Count && events[drop].EventTimeUtc == from) drop++; + events = events.Skip(drop).ToList(); + // The server asks for cap + skip rows (it re-reads the boundary ties it then drops), so a page is never + // emptied by the trim while more remains. + if (r.ContinuationPoint is not null) events.Count.ShouldBe(cap); + + all.AddRange(events.Select(e => e.EventId)); + if (r.ContinuationPoint is null) break; + var boundary = events[^1].EventTimeUtc; + var ties = events.AsEnumerable().Reverse().TakeWhile(e => e.EventTimeUtc == boundary).Count(); + skip = boundary == from ? skip + ties : ties; + from = boundary; + } + + all.SequenceEqual(NewestFirst).ShouldBeTrue($"cap {cap}: {string.Join(",", all)}"); + } + } + + [Fact] + public async Task Reverse_read_with_a_null_source_spans_every_source() + { + using var src = await SeedAsync(); + var r = await src.ReadEventsAsync(null, At(2.5), At(1.5), 0, reverse: true, Ct); + r.Events.Select(e => e.EventId).ShouldBe(["t4", "t3", "t2", "t1", "t0", "o0"]); + } +} diff --git a/tests/Drivers/Historian.Conformance/IConformanceBackend.cs b/tests/Drivers/Historian.Conformance/IConformanceBackend.cs index 7e817533..afc9f69f 100644 --- a/tests/Drivers/Historian.Conformance/IConformanceBackend.cs +++ b/tests/Drivers/Historian.Conformance/IConformanceBackend.cs @@ -41,6 +41,13 @@ public interface IConformanceBackend : IAsyncDisposable /// The rows to store. Task SeedAsync(string tag, params (DateTime T, HistorianValue V, uint Status)[] samples); + /// + /// Writes historical events directly in the backend's event table (never through the alarm writer): event id, + /// source and time as given (rounded to the storage resolution), severity 500, received time = event time. + /// + /// The events to store. + Task SeedEventsAsync(params (DateTime T, string Id, string Source)[] events); + /// /// Live (non-deleted) rows for a tag, ascending by time, read directly from storage. The status is returned /// VERBATIM as stored — there is no read-side clamp (an Empty row stored non-Bad reads back non-Bad here). diff --git a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.FOCAS.IntegrationTests/Series/WireBackendTests.cs b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.FOCAS.IntegrationTests/Series/WireBackendTests.cs index 19ef82d0..6b2f619e 100644 --- a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.FOCAS.IntegrationTests/Series/WireBackendTests.cs +++ b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.FOCAS.IntegrationTests/Series/WireBackendTests.cs @@ -71,7 +71,7 @@ public sealed class WireBackendTests await driver.InitializeAsync("{}", ct); await WaitFor(() => - driver.GetDeviceState(DeviceHost) is { FixedTreeCache: not null }, TimeSpan.FromSeconds(5)); + driver.GetDeviceState(DeviceHost) is { FixedTreeCache: not null }, WaitTimeout); var state = driver.GetDeviceState(DeviceHost); state.ShouldNotBeNull(); @@ -82,7 +82,7 @@ public sealed class WireBackendTests await WaitFor(() => state.LastFixedSnapshots.ContainsKey($"{DeviceHost}/Axes/X/AbsolutePosition"), - TimeSpan.FromSeconds(3)); + WaitTimeout); state.LastFixedSnapshots[$"{DeviceHost}/Axes/X/AbsolutePosition"].ShouldBe(123456); } } @@ -136,7 +136,7 @@ public sealed class WireBackendTests await WaitFor(() => driver.GetDeviceState(DeviceHost) is { LastProgramInfo: not null }, - TimeSpan.FromSeconds(5)); + WaitTimeout); var snapshots = await driver.ReadAsync( [$"{DeviceHost}/Program/Name", @@ -194,7 +194,7 @@ public sealed class WireBackendTests { var state = driver.GetDeviceState(DeviceHost); return state is not null && state.LastTimers.Count == 4; - }, TimeSpan.FromSeconds(5)); + }, WaitTimeout); var snapshots = await driver.ReadAsync( [$"{DeviceHost}/Timers/PowerOnSeconds", @@ -265,8 +265,11 @@ public sealed class WireBackendTests { var state = driver.GetDeviceState(DeviceHost); return state?.FixedTreeCache is { Capabilities.SpindleLoad: true, Capabilities.SpindleMotorSpeed: true } - && state.LastSpindleLoads.Count >= 2; - }, TimeSpan.FromSeconds(5)); + && state.LastSpindleLoads.Count >= 2 + // Motor speeds are cached after a SECOND wire round-trip (FocasDriver spindle poll), so the loads + // arriving first does not mean the speeds have — waiting on loads alone raced under load (#678.3). + && state.LastSpindleMotorSpeeds.Count >= 2; + }, WaitTimeout); var snapshots = await driver.ReadAsync( [$"{DeviceHost}/Spindle/S1/Load", @@ -282,6 +285,10 @@ public sealed class WireBackendTests } } + /// Generous: the sim fixture answers fast, but a loaded CI host (e.g. a concurrent docker build) does not. + private static readonly TimeSpan WaitTimeout = TimeSpan.FromSeconds(15); + + /// Polls ; FAILS the test with a clear message on timeout rather than returning silently. private static async Task WaitFor(Func pred, TimeSpan timeout) { var deadline = DateTime.UtcNow + timeout; @@ -290,5 +297,7 @@ public sealed class WireBackendTests if (pred()) return; await Task.Delay(50); } + if (pred()) return; + Assert.Fail($"Condition not met within {timeout.TotalSeconds:0.#}s waiting on the FOCAS wire backend."); } } diff --git a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway.Tests/GatewayEventBoundsTests.cs b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway.Tests/GatewayEventBoundsTests.cs new file mode 100644 index 00000000..746e369d --- /dev/null +++ b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway.Tests/GatewayEventBoundsTests.cs @@ -0,0 +1,123 @@ +using System.Net; +using Google.Protobuf.WellKnownTypes; +using Grpc.Core; +using Microsoft.AspNetCore.Builder; +using Microsoft.AspNetCore.Hosting; +using Microsoft.AspNetCore.Server.Kestrel.Core; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.Logging; +using Microsoft.Extensions.Logging.Abstractions; +using Shouldly; +using Xunit; +using ZB.MOM.WW.HistorianGateway.Contracts.Grpc; +using ZB.MOM.WW.OtOpcUa.Core.Abstractions; +using ZB.MOM.WW.OtOpcUa.Runtime.Historian; + +namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway.Tests; + +/// +/// Gitea #675: the Gateway backend's event-read bounds and direction, offline. +/// +/// A reverse (newest-first) read is refused before anything is sent — the gateway streams ascending only. +/// The open upper bound a start-only forward read uses (DateTime.MaxValue, UTC kind) survives the +/// PUBLISHED client package's proto conversion — captured on the wire by an in-process gRPC server — and the +/// Windows FILETIME encoding the gateway converts it to. The live AVEVA leg (what the historian does with a +/// year-9999 bound) stays open: it needs the VPN. +/// +/// +public sealed class GatewayEventBoundsTests +{ + private static readonly DateTime UtcMax = new(DateTime.MaxValue.Ticks, DateTimeKind.Utc); + private static CancellationToken Ct => TestContext.Current.CancellationToken; + + [Fact] + public async Task A_reverse_events_read_is_refused_and_sends_nothing() + { + var fake = new FakeHistorianGatewayClient(); + IHistorianDataSource ds = new GatewayHistorianDataSource(fake, NullLogger.Instance); + + await Should.ThrowAsync(() => + ds.ReadEventsAsync("Pump1", DateTime.UtcNow, DateTime.UtcNow.AddHours(-1), 10, reverse: true, Ct)); + fake.ReadEventsCallCount.ShouldBe(0); + + // Forward through the same overload is the five-argument read, unchanged. + await ds.ReadEventsAsync("Pump1", DateTime.UtcNow.AddHours(-1), DateTime.UtcNow, 10, reverse: false, Ct); + fake.ReadEventsCallCount.ShouldBe(1); + } + + [Fact] + public void UtcMax_round_trips_through_the_proto_Timestamp_and_FILETIME_encodings() + { + var ts = Timestamp.FromDateTime(UtcMax); + ts.ToDateTime().ShouldBe(UtcMax); + ts.ToDateTime().Kind.ShouldBe(DateTimeKind.Utc); + + // The gateway hands its bounds to the historian as FILETIME (100 ns since 1601) — in range up to year 30828. + var fileTime = ts.ToDateTime().ToFileTimeUtc(); + DateTime.FromFileTimeUtc(fileTime).ShouldBe(UtcMax); + } + + [Fact] + public async Task A_start_only_bound_of_UtcMax_reaches_the_wire_intact_through_the_published_client() + { + var capture = new CapturingReadService(); + await using var server = await StartServerAsync(capture); + using var adapter = HistorianGatewayClientAdapter.Create( + new ServerHistorianOptions + { + Enabled = true, + Endpoint = $"http://127.0.0.1:{server.Port}", + ApiKey = "histgw_test_key", + UseTls = false, + }, + NullLoggerFactory.Instance); + IHistorianDataSource ds = new GatewayHistorianDataSource(adapter, NullLogger.Instance); + + var start = new DateTime(2026, 1, 1, 0, 0, 0, DateTimeKind.Utc); + var r = await ds.ReadEventsAsync("Pump1", start, UtcMax, 5, Ct); + + r.Events.ShouldBeEmpty(); + capture.Request.ShouldNotBeNull(); + capture.Request!.StartTime.ToDateTime().ShouldBe(start); + capture.Request.EndTime.ToDateTime().ShouldBe(UtcMax); + DateTime.FromFileTimeUtc(capture.Request.EndTime.ToDateTime().ToFileTimeUtc()).ShouldBe(UtcMax); + } + + private sealed class CapturingReadService : HistorianRead.HistorianReadBase + { + public ReadEventsRequest? Request { get; private set; } + + public override Task ReadEvents( + ReadEventsRequest request, IServerStreamWriter responseStream, ServerCallContext context) + { + Request = request; + return Task.CompletedTask; + } + } + + private sealed class Server(WebApplication app, int port) : IAsyncDisposable + { + public int Port { get; } = port; + + public async ValueTask DisposeAsync() + { + await app.StopAsync(); + await app.DisposeAsync(); + } + } + + private static async Task StartServerAsync(CapturingReadService service) + { + var builder = WebApplication.CreateSlimBuilder(); + builder.Logging.ClearProviders(); + builder.WebHost.ConfigureKestrel(k => + k.Listen(IPAddress.Loopback, 0, o => o.Protocols = HttpProtocols.Http2)); + builder.Services.AddGrpc(); + builder.Services.AddSingleton(service); + var app = builder.Build(); + app.MapGrpcService(); + await app.StartAsync(); + var port = new Uri(app.Urls.First()).Port; + return new Server(app, port); + } +} diff --git a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway.Tests/ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway.Tests.csproj b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway.Tests/ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway.Tests.csproj index 16e7fc33..a662c88e 100644 --- a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway.Tests/ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway.Tests.csproj +++ b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway.Tests/ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway.Tests.csproj @@ -15,12 +15,18 @@ + + all runtime; build; native; contentfiles; analyzers; buildtransitive + + + + diff --git a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/QuestDbConformance.cs b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/QuestDbConformance.cs index d0ca0bdd..a9987d1a 100644 --- a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/QuestDbConformance.cs +++ b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/QuestDbConformance.cs @@ -60,3 +60,10 @@ public sealed class QuestDbModifiedValueConformance(QuestDbFixture fx) : Modifie /// protected override Task CreateBackendAsync() => QuestDbConformanceBackend.CreateAsync(fx); } + +[Collection(QuestDbCollection.Name)] +public sealed class QuestDbEventReadConformance(QuestDbFixture fx) : EventReadConformance +{ + /// + protected override Task CreateBackendAsync() => QuestDbConformanceBackend.CreateAsync(fx); +} diff --git a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/QuestDbConformanceBackend.cs b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/QuestDbConformanceBackend.cs index 1f3be1b3..ff4d6aad 100644 --- a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/QuestDbConformanceBackend.cs +++ b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/QuestDbConformanceBackend.cs @@ -143,6 +143,27 @@ internal sealed class QuestDbConformanceBackend : IConformanceBackend await Wal.WaitAsync(Conn.SampleTable, CancellationToken.None); } + /// + public async Task SeedEventsAsync(params (DateTime T, string Id, string Source)[] events) + { + await using (var c = await Conn.OpenAsync(CancellationToken.None)) + { + foreach (var (t, id, source) in events) + { + await using var cmd = Conn.CreateCommand(c, + $"INSERT INTO {Conn.EventTable} (ts, event_id, source, severity, message, received_utc) VALUES ($1, $2, $3, $4, $5, $6)"); + QuestDbConnection.BindTimestamp(cmd, t); + QuestDbConnection.BindText(cmd, id); + QuestDbConnection.BindText(cmd, source); + cmd.Parameters.Add(new NpgsqlParameter { NpgsqlDbType = NpgsqlDbType.Integer, Value = 500 }); + QuestDbConnection.BindText(cmd, null); + QuestDbConnection.BindTimestamp(cmd, t); + await cmd.ExecuteNonQueryAsync(); + } + } + await Wal.WaitAsync(Conn.EventTable, CancellationToken.None); + } + /// Runs a statement against this table set (test-only escape hatch for hand-made rows), then waits for WAL visibility of the sample table. /// The statement; parameters are bound positionally by . /// Binds the parameters. diff --git a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/TtlWallClockTests.cs b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/TtlWallClockTests.cs new file mode 100644 index 00000000..f554da78 --- /dev/null +++ b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/TtlWallClockTests.cs @@ -0,0 +1,80 @@ +using Shouldly; +using Xunit; +using ZB.MOM.WW.OtOpcUa.Core.Abstractions; +using ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common; + +namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests; + +/// +/// Gitea #674.1 — pins the QuestDB TTL behaviour the historian's retention relies on: a table TTL is judged against +/// the wall clock (min(max timestamp, now)), so one far-future row — a 2066 typo that reached the backend +/// without going through the node manager's HistoryUpdateMaxFutureSkew bound — cannot age out every older +/// partition, including the Part 11 modification trail (the record of who changed history). +/// +/// +/// +/// Positive control. A partition genuinely older than the TTL (40 days, against a 30-day TTL) is kept until +/// the far-future commit and evicted BY it — proving TTL housekeeping ran on exactly the commit that could have +/// dropped the in-TTL partitions. Without it the "nothing evicted" assertion would also pass against a table +/// whose TTL never ran. +/// +/// +/// ⚠️ Observed on QuestDB 10.0.1 (2026-10-11): the in-TTL partitions also survived with +/// cairo.ttl.use.wall.clock=false (verified effective via SHOW PARAMETERS), so this test does not +/// isolate that flag — it pins the outcome. Keep the flag at its default (true) regardless; the server-side +/// future-skew bound is the primary guard. +/// +/// +[Collection(QuestDbCollection.Name)] +public sealed class TtlWallClockTests(QuestDbFixture fx) +{ + private static CancellationToken Ct => TestContext.Current.CancellationToken; + private static readonly HistoryUpdateContext Who = new("ttl-test", null); + + [Fact] + public async Task A_far_future_row_evicts_only_partitions_older_than_the_ttl_by_the_wall_clock() + { + fx.SkipIfUnavailable(); + var o = fx.Options(fx.UniquePrefix(), retentionDays: 30); + var c = new QuestDbConnection(o); + var wal = new QuestDbWalVisibility(c, o.VisibilityTimeout); + var tags = new QuestDbTagCatalog(c, new QuestDbSchemaBootstrapper(c, o), wal); + var updater = new QuestDbHistorianDataUpdater(c, tags, wal); + + // Whole seconds (µs-aligned), one row per DAY partition. With the newest row 10 days old, the 40-day partition is + // still inside the TTL measured from the table's max timestamp, so this commit evicts nothing. + var now = DateTime.UtcNow; + var today = new DateTime(now.Year, now.Month, now.Day, now.Hour, now.Minute, now.Second, DateTimeKind.Utc); + DateTime[] older = [today.AddDays(-40), today.AddDays(-20), today.AddDays(-10)]; + (await updater.UpdateDataAsync("ttl.tag", HistoryUpdateMode.Insert, older.Select((t, i) => V(t, i)).ToList(), Who, Ct)) + .ShouldAllBe(code => code == HistorianStatusCodes.GoodEntryInserted); + (await DaysAsync(c, c.SampleTable)).ShouldBe(older.Select(t => t.Date)); + (await DaysAsync(c, c.SampleModifiedTable)).ShouldBe(older.Select(t => t.Date)); + + // The far-future row, then one more ordinary commit (TTL housekeeping runs on commits). + (await updater.UpdateDataAsync("ttl.tag", HistoryUpdateMode.Insert, + [V(new DateTime(2066, 1, 1, 0, 0, 0, DateTimeKind.Utc), 9)], Who, Ct)).ShouldBe([HistorianStatusCodes.GoodEntryInserted]); + (await updater.UpdateDataAsync("ttl.tag", HistoryUpdateMode.Insert, [V(today, 10)], Who, Ct)) + .ShouldBe([HistorianStatusCodes.GoodEntryInserted]); + (await wal.WaitAsync(c.SampleTable, Ct)).ShouldBeTrue(); + (await wal.WaitAsync(c.SampleModifiedTable, Ct)).ShouldBeTrue(); + + // Only the genuinely expired 40-day partition is gone; the 20- and 10-day partitions survive in both tables. + DateTime[] expected = [today.AddDays(-20).Date, today.AddDays(-10).Date, today.Date, new(2066, 1, 1)]; + (await DaysAsync(c, c.SampleTable)).ShouldBe(expected); + (await DaysAsync(c, c.SampleModifiedTable)).ShouldBe(expected); + } + + private static HistorizationValue V(DateTime t, double v) => new(t, HistorianValue.FromDouble(v), HistorianStatusCodes.Good); + + /// The distinct UTC days holding rows, ascending. + private static async Task> DaysAsync(QuestDbConnection c, string table) + { + await using var conn = await c.OpenAsync(Ct); + await using var cmd = c.CreateCommand(conn, $"SELECT ts FROM {table} ORDER BY ts"); + await using var r = await cmd.ExecuteReaderAsync(Ct); + var days = new SortedSet(); + while (await r.ReadAsync(Ct)) days.Add(QuestDbConnection.ReadUtc(r, 0).Date); + return days.ToList(); + } +} diff --git a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests/StatusCodeOracleTests.cs b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests/StatusCodeOracleTests.cs index a672a621..177c7c19 100644 --- a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests/StatusCodeOracleTests.cs +++ b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests/StatusCodeOracleTests.cs @@ -14,6 +14,11 @@ public sealed class StatusCodeOracleTests ((uint)Opc.Ua.AggregateBits.Calculated).ShouldBe(HistorianInfoBits.HistorianCalculated); } + /// #674: the v1-schema refusal code is the SDK's BadHistoryOperationUnsupported. + [Fact] + public void History_operation_unsupported_matches_the_SDK() => + HistorianStatusCodes.BadHistoryOperationUnsupported.ShouldBe(Opc.Ua.StatusCodes.BadHistoryOperationUnsupported); + [Fact] public void Interpolated_bits_match_the_SDK() { diff --git a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/ModifiedHistoryAtomicityTests.cs b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/ModifiedHistoryAtomicityTests.cs index 0a3e98ea..9de80da7 100644 --- a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/ModifiedHistoryAtomicityTests.cs +++ b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/ModifiedHistoryAtomicityTests.cs @@ -54,7 +54,7 @@ public sealed class ModifiedHistoryAtomicityTests(SqlHistorianFixture fx) public async Task A_replace_whose_trail_insert_fails_changes_nothing() { var db = await ReadTestDb.CreateAsync(fx); - var u = new SqlServerHistorianDataUpdater(db.Conn, new SqlServerTagCatalog(db.Conn, db.Boot)); + var u = new SqlServerHistorianDataUpdater(db.Conn, db.Boot, new SqlServerTagCatalog(db.Conn, db.Boot)); (await u.UpdateDataAsync("t", HistoryUpdateMode.Insert, [V(T1, 1)], Who, CancellationToken.None)) .ShouldBe([HistorianStatusCodes.GoodEntryInserted]); (await CountTrailAsync(db)).ShouldBe(1); @@ -78,7 +78,7 @@ public sealed class ModifiedHistoryAtomicityTests(SqlHistorianFixture fx) public async Task Deletes_whose_trail_insert_fails_delete_nothing() { var db = await ReadTestDb.CreateAsync(fx); - var u = new SqlServerHistorianDataUpdater(db.Conn, new SqlServerTagCatalog(db.Conn, db.Boot)); + var u = new SqlServerHistorianDataUpdater(db.Conn, db.Boot, new SqlServerTagCatalog(db.Conn, db.Boot)); await u.UpdateDataAsync("t", HistoryUpdateMode.Insert, [V(T1, 1), V(T2, 2)], Who, CancellationToken.None); await BreakTheTrailAsync(db); @@ -99,7 +99,7 @@ public sealed class ModifiedHistoryAtomicityTests(SqlHistorianFixture fx) public async Task An_over_long_principal_is_truncated_never_refused() { var db = await ReadTestDb.CreateAsync(fx); - var u = new SqlServerHistorianDataUpdater(db.Conn, new SqlServerTagCatalog(db.Conn, db.Boot)); + var u = new SqlServerHistorianDataUpdater(db.Conn, db.Boot, new SqlServerTagCatalog(db.Conn, db.Boot)); var who = new HistoryUpdateContext(new string('p', 500), new string('k', 500)); (await u.UpdateDataAsync("t", HistoryUpdateMode.Insert, [V(T1, 1)], who, CancellationToken.None)) diff --git a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/RetentionTests.cs b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/RetentionTests.cs index 046869d6..35309fdb 100644 --- a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/RetentionTests.cs +++ b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/RetentionTests.cs @@ -178,6 +178,97 @@ public sealed class RetentionTests(SqlHistorianFixture fx) (await LockIsFreeAsync(s)).ShouldBeTrue(); } + [Fact] + public async Task Cancelling_mid_command_while_the_delete_is_blocked_still_releases_the_applock() + { + fx.SkipIfUnavailable(); + var (c, b, _, s) = Make(); + await SeedAsync(b, s); + + // A second session holds an exclusive table lock, so the purger's first DELETE blocks server-side. + await using var holder = new SqlConnection(fx.ConnectionString); + await holder.OpenAsync(); + await using var tx = (SqlTransaction)await holder.BeginTransactionAsync(); + await using (var cmd = holder.CreateCommand()) + { + cmd.Transaction = tx; + cmd.CommandText = $"SELECT COUNT(*) FROM [{s}].Sample WITH (TABLOCKX, HOLDLOCK)"; + await cmd.ExecuteScalarAsync(); + } + short holderSpid; + await using (var cmd = holder.CreateCommand()) + { + cmd.Transaction = tx; + cmd.CommandText = "SELECT CAST(@@SPID AS smallint)"; + holderSpid = (short)(await cmd.ExecuteScalarAsync())!; + } + + using var cts = new CancellationTokenSource(); + var purger = new SqlServerHistorianRetention(c, b, TimeSpan.FromHours(1), 30, 4000, time: new FixedTime(Now)); + var purge = purger.PurgeAsync(cts.Token); + + // Cancel only once the server reports a DELETE actually blocked behind the holder — a fixed delay could + // cancel during bootstrap/applock acquisition on a slow host and never exercise the mid-command path. + var deadline = DateTime.UtcNow.AddSeconds(15); + while (await CountAsync( + $"SELECT COUNT(*) FROM sys.dm_exec_requests WHERE blocking_session_id = {holderSpid} AND command = 'DELETE'") == 0) + { + purge.IsCompleted.ShouldBeFalse("the purge finished without ever blocking on the holder's table lock"); + DateTime.UtcNow.ShouldBeLessThan(deadline, "the purge's DELETE never blocked behind the holder"); + await Task.Delay(50); + } + cts.Cancel(); + + // SqlClient surfaces the attention as OperationCanceledException or as a SqlException. + var ex = await Should.ThrowAsync(() => purge); + (ex is OperationCanceledException or SqlException).ShouldBeTrue($"unexpected {ex.GetType().Name}: {ex.Message}"); + + await tx.RollbackAsync(); + (await LockIsFreeAsync(s)).ShouldBeTrue(); + (await CountAsync($"SELECT COUNT(*) FROM [{s}].Sample")).ShouldBe(21); + } + + [Fact] + public async Task A_failing_release_falls_back_to_clearing_the_pool_so_the_applock_is_freed() + { + fx.SkipIfUnavailable(); + var (c, b, _, s) = Make(); + await SeedAsync(b, s); + var purger = new SqlServerHistorianRetention(c, b, TimeSpan.FromHours(1), 30, 4000, time: new FixedTime(Now)) + { + ReleaseSql = "EXEC sp_this_procedure_does_not_exist;", + }; + + await purger.PurgeAsync(CancellationToken.None); + + (await LockIsFreeAsync(s)).ShouldBeTrue(); + } + + [Fact] + public async Task Falsification_control_a_failing_release_with_a_no_op_ClearPool_leaves_the_applock_held() + { + fx.SkipIfUnavailable(); + var (c, b, _, s) = Make(); + await SeedAsync(b, s); + var purger = new SqlServerHistorianRetention(c, b, TimeSpan.FromHours(1), 30, 4000, time: new FixedTime(Now)) + { + ReleaseSql = "EXEC sp_this_procedure_does_not_exist;", + ClearPool = _ => { }, + }; + + await purger.PurgeAsync(CancellationToken.None); + + try + { + // The pooled session still owns the Session-scoped lock: proves the fallback test above is not vacuous. + (await LockIsFreeAsync(s)).ShouldBeFalse(); + } + finally + { + SqlConnection.ClearAllPools(); // drop the leaked session so the lock does not outlive the test + } + } + [Fact] public async Task A_throwing_purge_releases_the_applock() { diff --git a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/SchemaBootstrapIntegrationTests.cs b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/SchemaBootstrapIntegrationTests.cs index f6a1779c..5ec131ad 100644 --- a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/SchemaBootstrapIntegrationTests.cs +++ b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/SchemaBootstrapIntegrationTests.cs @@ -32,7 +32,7 @@ public sealed class SchemaBootstrapIntegrationTests(SqlHistorianFixture fx) var s = SqlHistorianFixture.UniqueSchema(); await Boot(Opts(s)).EnsureAsync(CancellationToken.None); await Boot(Opts(s)).EnsureAsync(CancellationToken.None); // fresh instance => re-runs the guarded path - (await ScalarAsync($"SELECT COUNT(*) FROM [{s}].SchemaVersion")).ShouldBe(SqlServerSchemaBootstrapper.RequiredVersion); + (await ScalarAsync($"SELECT COUNT(*) FROM [{s}].SchemaVersion")).ShouldBe(SqlServerSchemaBootstrapper.ModifiedTrailVersion); (await ScalarAsync($"SELECT COUNT(*) FROM sys.indexes WHERE name='CX_Event_Time' AND object_id=OBJECT_ID('[{s}].Event')")).ShouldBe(1); } @@ -51,13 +51,14 @@ public sealed class SchemaBootstrapIntegrationTests(SqlHistorianFixture fx) cmd.CommandText = b; await cmd.ExecuteNonQueryAsync(); } - (await ScalarAsync($"SELECT COUNT(*) FROM [{s}].SchemaVersion")).ShouldBe(SqlServerSchemaBootstrapper.RequiredVersion); + (await ScalarAsync($"SELECT COUNT(*) FROM [{s}].SchemaVersion")).ShouldBe(SqlServerSchemaBootstrapper.ModifiedTrailVersion); } /// /// #668 upgrade path: a schema left at version 1 (only 001_initial.sql applied, as every pre-#668 install is) /// is upgraded in place by an auto-creating bootstrap — SampleModified appears, both version rows exist, and - /// existing samples are untouched. With AutoCreateSchema=false the same v1 schema is refused. + /// existing samples are untouched. With AutoCreateSchema=false the same v1 schema is accepted with the + /// modification trail switched off and left at v1 (#674). /// [Fact] public async Task A_version_1_schema_is_upgraded_to_version_2_in_place() @@ -81,11 +82,15 @@ public sealed class SchemaBootstrapIntegrationTests(SqlHistorianFixture fx) } (await ScalarAsync($"SELECT COUNT(*) FROM sys.tables WHERE name = 'SampleModified' AND schema_id = SCHEMA_ID('{s}')")).ShouldBe(0); - var refused = await Should.ThrowAsync( - () => Boot(Opts(s, auto: false)).EnsureAsync(CancellationToken.None)); - refused.Message.ShouldContain("missing version 2 (found 1)"); + // #674: verify-only accepts v1 — with the modification trail switched off — and does not upgrade it. + var verifyOnly = Boot(Opts(s, auto: false)); + await verifyOnly.EnsureAsync(CancellationToken.None); + verifyOnly.SupportsModifiedTrail.ShouldBeFalse(); + (await ScalarAsync($"SELECT MAX(Version) FROM [{s}].SchemaVersion")).ShouldBe(1); - await Boot(Opts(s)).EnsureAsync(CancellationToken.None); + var upgrading = Boot(Opts(s)); + await upgrading.EnsureAsync(CancellationToken.None); + upgrading.SupportsModifiedTrail.ShouldBeTrue(); (await ScalarAsync($"SELECT COUNT(*) FROM [{s}].SchemaVersion")).ShouldBe(2); (await ScalarAsync($"SELECT MAX(Version) FROM [{s}].SchemaVersion")).ShouldBe(2); @@ -101,7 +106,7 @@ public sealed class SchemaBootstrapIntegrationTests(SqlHistorianFixture fx) var s = SqlHistorianFixture.UniqueSchema(); var tasks = Enumerable.Range(0, 4).Select(_ => Task.Run(() => Boot(Opts(s)).EnsureAsync(CancellationToken.None))).ToArray(); await Task.WhenAll(tasks); - (await ScalarAsync($"SELECT COUNT(*) FROM [{s}].SchemaVersion")).ShouldBe(SqlServerSchemaBootstrapper.RequiredVersion); + (await ScalarAsync($"SELECT COUNT(*) FROM [{s}].SchemaVersion")).ShouldBe(SqlServerSchemaBootstrapper.ModifiedTrailVersion); } [Fact] @@ -110,7 +115,7 @@ public sealed class SchemaBootstrapIntegrationTests(SqlHistorianFixture fx) fx.SkipIfUnavailable(); var ex = await Should.ThrowAsync( () => Boot(Opts(SqlHistorianFixture.UniqueSchema(), auto: false)).EnsureAsync(CancellationToken.None)); - ex.Message.ShouldContain("missing version 2"); + ex.Message.ShouldContain("missing version 1 (found 0)"); } [Fact] diff --git a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/SchemaV1PostureTests.cs b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/SchemaV1PostureTests.cs new file mode 100644 index 00000000..85f14fa0 --- /dev/null +++ b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/SchemaV1PostureTests.cs @@ -0,0 +1,181 @@ +using Microsoft.Data.SqlClient; +using Microsoft.Extensions.Logging; +using Shouldly; +using Xunit; +using ZB.MOM.WW.OtOpcUa.Core.Abstractions; +using ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian; +using ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common; + +namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests; + +/// +/// Gitea #674: a database left at schema version 1 (no Part 11 SampleModified trail) with +/// AutoCreateSchema=false keeps historizing. Before #674 the bootstrapper required v2 and refused EVERY call — +/// live writes, raw reads, alarm events, retention — until script 002 was applied by hand. Now only the features +/// that need the trail are refused (BadHistoryOperationUnsupported, nothing sent to SQL), and the +/// bootstrapper logs ONE warning for the whole provider. Built through , the +/// production wiring, so every seam shares the one bootstrapper the way a node does. +/// +[Collection(SqlHistorianCollection.Name)] +public sealed class SchemaV1PostureTests(SqlHistorianFixture fx) +{ + private static CancellationToken Ct => TestContext.Current.CancellationToken; + private static readonly DateTime Recent = DateTime.UtcNow.AddHours(-1); + private static readonly HistoryUpdateContext Who = new("editor", "k1"); + + [Fact] + public async Task A_v1_schema_keeps_historizing_and_refuses_only_the_modified_trail_features() + { + fx.SkipIfUnavailable(); + var s = SqlHistorianFixture.UniqueSchema(); + await ApplyV1Async(s); + var old = DateTime.UtcNow.AddDays(-40); + var loggers = new CapturingLoggerFactory(); + var provider = new SqlServerHistorianProvider(new SqlServerHistorianOptions + { + ConnectionString = fx.ConnectionString, Schema = s, AutoCreateSchema = false, RetentionDays = 30, + }); + var sp = new LoggerServices(loggers); + + // Live writes (the continuous-historization drain) and raw reads work. + var writer = provider.CreateValueWriter(sp); + (await writer.WriteLiveValuesAsync("Plant.Speed", + [new HistorizationValue(old, HistorianValue.FromDouble(1), 0), new HistorizationValue(Recent, HistorianValue.FromDouble(2), 0)], + Ct)).ShouldBeTrue(); + var source = provider.CreateDataSource(sp); + var raw = await source.ReadRawAsync("Plant.Speed", Recent.AddMinutes(-1), Recent.AddMinutes(1), 10, Ct); + raw.Samples.Select(v => v.Value).ShouldBe([2d]); + + // Alarm events are written. + var alarms = provider.CreateAlarmWriter(sp); + (await alarms.WriteBatchAsync( + [new AlarmHistorianEvent("alarm-1", "Plant/Line/Eq", "HiTemp", "LimitAlarm", AlarmSeverity.High, "Activated", "m", "u", null, Recent)], + Ct)).ShouldBe([HistorianWriteOutcome.Ack]); + (await CountAsync($"SELECT COUNT(*) FROM [{s}].Event")).ShouldBe(1); + + // Modified reads: the interface's "no trail" signal (the node manager reports BadHistoryOperationUnsupported), + // not a backend failure — health stays clean. + await Should.ThrowAsync(() => + source.ReadModifiedAsync("Plant.Speed", Recent.AddDays(-1), Recent.AddDays(1), 10, null, Ct)); + source.GetHealthSnapshot().LastError.ShouldBeNull(); + + // HistoryUpdate: every item refused, and NOTHING reaches SQL — a sent MERGE/DELETE would either change the + // samples or fail on the missing SampleModified target; an Insert on an unknown tag would create its Tag row. + var updater = provider.CreateUpdater(sp)!; + var tagsBefore = await CountAsync($"SELECT COUNT(*) FROM [{s}].Tag"); + (await updater.UpdateDataAsync("Plant.Speed", HistoryUpdateMode.Update, + [new HistorizationValue(Recent, HistorianValue.FromDouble(99), 0), new HistorizationValue(Recent.AddSeconds(1), HistorianValue.FromDouble(98), 0)], + Who, Ct)).ShouldBe([HistorianStatusCodes.BadHistoryOperationUnsupported, HistorianStatusCodes.BadHistoryOperationUnsupported]); + (await updater.UpdateDataAsync("Plant.Brand.New", HistoryUpdateMode.Insert, + [new HistorizationValue(Recent, HistorianValue.FromDouble(1), 0)], Who, Ct)) + .ShouldBe([HistorianStatusCodes.BadHistoryOperationUnsupported]); + (await updater.DeleteRawAsync("Plant.Speed", Recent.AddMinutes(-1), Recent.AddMinutes(1), Who, Ct)) + .ShouldBe(HistorianStatusCodes.BadHistoryOperationUnsupported); + (await updater.DeleteAtTimeAsync("Plant.Speed", [Recent], Who, Ct)) + .ShouldBe([HistorianStatusCodes.BadHistoryOperationUnsupported]); + (await CountAsync($"SELECT COUNT(*) FROM [{s}].Tag")).ShouldBe(tagsBefore); + (await source.ReadRawAsync("Plant.Speed", Recent.AddMinutes(-1), Recent.AddMinutes(1), 10, Ct)) + .Samples.Select(v => v.Value).ShouldBe([2d]); + + // Retention purges samples and skips the trail it does not have. + var purge = await provider.CreateRetention(sp)!.PurgeAsync(Ct); + purge.Select(r => r.Table).ShouldBe(["Sample", "Event"]); + purge.Single(r => r.Table == "Sample").Rows.ShouldBe(1); + (await CountAsync($"SELECT COUNT(*) FROM [{s}].Sample")).ShouldBe(1); + + // One warning for the whole provider (= per boot), however many seams and calls touched the bootstrapper. + loggers.Warnings.Count(w => w.Contains("Part 11 modification trail is unavailable")).ShouldBe(1); + (await CountAsync($"SELECT MAX(Version) FROM [{s}].SchemaVersion")).ShouldBe(1); // verify-only: never upgraded + } + + /// + /// #674: AutoCreateSchema=true still upgrades a v1 schema to v2 — the modification trail appears and a + /// HistoryUpdate is recorded in it — and logs no v1 warning. + /// + [Fact] + public async Task Auto_create_still_upgrades_a_v1_schema_and_enables_the_trail() + { + fx.SkipIfUnavailable(); + var s = SqlHistorianFixture.UniqueSchema(); + await ApplyV1Async(s); + var loggers = new CapturingLoggerFactory(); + var provider = new SqlServerHistorianProvider(new SqlServerHistorianOptions + { + ConnectionString = fx.ConnectionString, Schema = s, AutoCreateSchema = true, + }); + var sp = new LoggerServices(loggers); + + (await provider.CreateUpdater(sp)!.UpdateDataAsync("Plant.Speed", HistoryUpdateMode.Insert, + [new HistorizationValue(Recent, HistorianValue.FromDouble(5), 0)], Who, Ct)) + .ShouldBe([HistorianStatusCodes.GoodEntryInserted]); + + (await CountAsync($"SELECT MAX(Version) FROM [{s}].SchemaVersion")).ShouldBe(SqlServerSchemaBootstrapper.ModifiedTrailVersion); + var trail = await provider.CreateDataSource(sp) + .ReadModifiedAsync("Plant.Speed", Recent.AddMinutes(-1), Recent.AddMinutes(1), 10, null, Ct); + trail.Values.ShouldHaveSingleItem().UpdateType.ShouldBe(HistoryModificationType.Insert); + loggers.Warnings.ShouldNotContain(w => w.Contains("Part 11 modification trail is unavailable")); + } + + private async Task ApplyV1Async(string schema) + { + var v1 = new StreamReader(typeof(SqlServerSchemaBootstrapper).Assembly.GetManifestResourceStream("Schema.001_initial.sql")!).ReadToEnd(); + await using var c = new SqlConnection(fx.ConnectionString); + await c.OpenAsync(Ct); + foreach (var b in SqlServerSchemaBootstrapper.SplitBatches(v1.Replace("$(Schema)", schema))) + { + await using var cmd = c.CreateCommand(); + cmd.CommandText = b; + await cmd.ExecuteNonQueryAsync(Ct); + } + } + + private async Task CountAsync(string sql) + { + await using var c = new SqlConnection(fx.ConnectionString); + await c.OpenAsync(Ct); + await using var cmd = c.CreateCommand(); + cmd.CommandText = sql; + return Convert.ToInt64(await cmd.ExecuteScalarAsync(Ct)); + } + + /// The only service the provider resolves is . + private sealed class LoggerServices(ILoggerFactory loggers) : IServiceProvider + { + public object? GetService(Type serviceType) => serviceType == typeof(ILoggerFactory) ? loggers : null; + } + + /// Captures every Warning-or-above message from every logger it creates. + private sealed class CapturingLoggerFactory : ILoggerFactory + { + private readonly List _warnings = []; + + public IReadOnlyList Warnings + { + get { lock (_warnings) return _warnings.ToList(); } + } + + public ILogger CreateLogger(string categoryName) => new Capturing(this); + + public void AddProvider(ILoggerProvider provider) + { + } + + public void Dispose() + { + } + + private sealed class Capturing(CapturingLoggerFactory owner) : ILogger + { + public IDisposable? BeginScope(TState state) where TState : notnull => null; + + public bool IsEnabled(LogLevel logLevel) => true; + + public void Log(LogLevel logLevel, EventId eventId, TState state, Exception? exception, + Func formatter) + { + if (logLevel < LogLevel.Warning) return; + lock (owner._warnings) owner._warnings.Add(formatter(state, exception)); + } + } + } +} diff --git a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/SqlServerConformance.cs b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/SqlServerConformance.cs index 790beffe..73b9862f 100644 --- a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/SqlServerConformance.cs +++ b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/SqlServerConformance.cs @@ -60,3 +60,10 @@ public sealed class SqlServerModifiedValueConformance(SqlHistorianFixture fx) : /// protected override Task CreateBackendAsync() => SqlServerConformanceBackend.CreateAsync(fx); } + +[Collection(SqlHistorianCollection.Name)] +public sealed class SqlServerEventReadConformance(SqlHistorianFixture fx) : EventReadConformance +{ + /// + protected override Task CreateBackendAsync() => SqlServerConformanceBackend.CreateAsync(fx); +} diff --git a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/SqlServerConformanceBackend.cs b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/SqlServerConformanceBackend.cs index 79b71ce4..69afa55b 100644 --- a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/SqlServerConformanceBackend.cs +++ b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/SqlServerConformanceBackend.cs @@ -34,12 +34,28 @@ internal sealed class SqlServerConformanceBackend : IConformanceBackend /// public IHistorianDataUpdater CreateUpdater() => - new SqlServerHistorianDataUpdater(_db.Conn, new SqlServerTagCatalog(_db.Conn, _db.Boot)); + new SqlServerHistorianDataUpdater(_db.Conn, _db.Boot, new SqlServerTagCatalog(_db.Conn, _db.Boot)); /// public async Task SeedAsync(string tag, params (DateTime T, HistorianValue V, uint Status)[] samples) => await _db.SeedAsync(tag, samples); + /// + public async Task SeedEventsAsync(params (DateTime T, string Id, string Source)[] events) + { + await using var c = await _db.Conn.OpenAsync(CancellationToken.None); + foreach (var (t, id, source) in events) + { + await using var cmd = _db.Conn.CreateCommand(c, + $"INSERT {_db.Conn.Q("Event")} (EventId, SourceName, AlarmId, AlarmName, AlarmTypeName, Severity, EventKind, Message, " + + "UserName, Comment, EventTimeUtc, ReceivedTimeUtc) VALUES (@id, @src, 'a', 'n', 'T', 500, 'Activated', NULL, NULL, NULL, @t, @t)"); + cmd.Parameters.AddWithValue("@id", id); + cmd.Parameters.AddWithValue("@src", source); + SqlHistorianConnection.AddDateTime2(cmd, "@t", t); + await cmd.ExecuteNonQueryAsync(); + } + } + /// public async Task> ReadBackAsync(string tag) { diff --git a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.Tests/SchemaScriptTests.cs b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.Tests/SchemaScriptTests.cs index a260fc32..3f88a628 100644 --- a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.Tests/SchemaScriptTests.cs +++ b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.Tests/SchemaScriptTests.cs @@ -32,7 +32,7 @@ public sealed class SchemaScriptTests /// /// #668: schema v2 (the Part 11 modification trail) is embedded, matches its docs copy, records version 2 (which - /// must equal ), and carries none of the constructs an + /// must equal ), and carries none of the constructs an /// OUTPUT … INTO target forbids (FOREIGN KEY / CHECK / trigger) — any of them would fail every HistoryUpdate. /// [Fact] @@ -45,7 +45,8 @@ public sealed class SchemaScriptTests text.ShouldContain("[SampleModified]"); text.ShouldContain("PRIMARY KEY CLUSTERED (TagId, SourceTimeUtc, ModifiedId)"); text.ShouldContain("VALUES (2, SYSUTCDATETIME())"); - SqlServerSchemaBootstrapper.RequiredVersion.ShouldBe(2); + SqlServerSchemaBootstrapper.ModifiedTrailVersion.ShouldBe(2); + SqlServerSchemaBootstrapper.RequiredVersion.ShouldBe(1); // #674: v1 is still served, without the trail var ddl = string.Join('\n', text.Split('\n').Where(l => !l.TrimStart().StartsWith("--", StringComparison.Ordinal))); ddl.ShouldNotContain("FOREIGN KEY", Case.Insensitive); ddl.ShouldNotContain("CHECK", Case.Insensitive); diff --git a/tests/Server/ZB.MOM.WW.OtOpcUa.Host.Tests/Configuration/RoleOverlayTests.cs b/tests/Server/ZB.MOM.WW.OtOpcUa.Host.Tests/Configuration/RoleOverlayTests.cs new file mode 100644 index 00000000..80622a4d --- /dev/null +++ b/tests/Server/ZB.MOM.WW.OtOpcUa.Host.Tests/Configuration/RoleOverlayTests.cs @@ -0,0 +1,78 @@ +using System.Text.Json; +using Shouldly; +using Xunit; +using ZB.MOM.WW.OtOpcUa.Host.Configuration; + +namespace ZB.MOM.WW.OtOpcUa.Host.Tests.Configuration; + +/// +/// #678.6 — the per-role appsettings overlay name is built from plain roles only. A cluster-* role in the +/// name produced a file that never exists, so no overlay ever loaded on a node with a cluster role. +/// +public sealed class RoleOverlayTests +{ + [Fact] + public void A_driver_site_node_loads_the_driver_overlay() + => RoleOverlay.Suffix(["driver", "cluster-SITE-A"]).ShouldBe("driver"); + + [Fact] + public void A_central_node_loads_the_admin_driver_overlay() + => RoleOverlay.Suffix(["admin", "driver", "cluster-MAIN"]).ShouldBe("admin-driver"); + + [Fact] + public void Plain_roles_are_ordinal_sorted_regardless_of_input_order() + => RoleOverlay.Suffix(["driver", "cluster-MAIN", "admin"]).ShouldBe("admin-driver"); + + [Fact] + public void Roles_without_a_cluster_role_keep_the_existing_names() + { + RoleOverlay.Suffix(["admin"]).ShouldBe("admin"); + RoleOverlay.Suffix(["driver"]).ShouldBe("driver"); + RoleOverlay.Suffix(["admin", "driver"]).ShouldBe("admin-driver"); + } + + [Fact] + public void A_cluster_role_prefix_is_matched_case_insensitively() + => RoleOverlay.Suffix(["driver", "CLUSTER-SITE-A"]).ShouldBe("driver"); + + [Fact] + public void No_plain_role_means_no_overlay() + { + RoleOverlay.Suffix([]).ShouldBeNull(); + RoleOverlay.Suffix(["cluster-SITE-A"]).ShouldBeNull(); + } + + // ---- The shipped overlays: activating them on cluster-role nodes must not change production semantics. ---- + + private static string HostDir() + { + for (var dir = new DirectoryInfo(AppContext.BaseDirectory); dir is not null; dir = dir.Parent) + { + var candidate = Path.Combine(dir.FullName, "src", "Server", "ZB.MOM.WW.OtOpcUa.Host"); + if (File.Exists(Path.Combine(candidate, "appsettings.json"))) return candidate; + } + + throw new InvalidOperationException("Host project directory not found above " + AppContext.BaseDirectory); + } + + [Theory] + [InlineData("admin")] + [InlineData("driver")] + [InlineData("admin-driver")] + public void Shipped_overlays_carry_only_the_audited_keys(string suffix) + { + using var doc = JsonDocument.Parse(File.ReadAllText(Path.Combine(HostDir(), $"appsettings.{suffix}.json"))); + var root = doc.RootElement; + + // Only Serilog + Security:Ldap, and in Security:Ldap only the two audited keys. + root.EnumerateObject().Select(p => p.Name).Order().ShouldBe(["Security", "Serilog"]); + var ldap = root.GetProperty("Security").GetProperty("Ldap"); + ldap.EnumerateObject().Select(p => p.Name).Order().ShouldBe(["DevStubMode", "Transport"]); + ldap.GetProperty("DevStubMode").GetBoolean().ShouldBeFalse(); + + // No Debug-level (or lower) category override: activating an overlay must not raise log volume on a node. + foreach (var o in root.GetProperty("Serilog").GetProperty("MinimumLevel").GetProperty("Override").EnumerateObject()) + o.Value.GetString().ShouldNotBeOneOf("Debug", "Verbose"); + root.GetProperty("Serilog").GetProperty("MinimumLevel").GetProperty("Default").GetString().ShouldBe("Information"); + } +} diff --git a/tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.IntegrationTests/EventHistoryPagingOverTheWireTests.cs b/tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.IntegrationTests/EventHistoryPagingOverTheWireTests.cs index 2ba2bd8f..c3499953 100644 --- a/tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.IntegrationTests/EventHistoryPagingOverTheWireTests.cs +++ b/tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.IntegrationTests/EventHistoryPagingOverTheWireTests.cs @@ -54,8 +54,48 @@ public sealed class EventHistoryPagingOverTheWireTests finally { await rig.DisposeAsync(); } } + [Fact] + public async Task A_capped_REVERSE_event_read_pages_newest_first_through_the_session_store_exactly_once() + { + // Gitea #675: StartTime later than EndTime. Before the fix the cap kept the EARLIEST 10, ascending, with no + // continuation point. + var ct = TestContext.Current.CancellationToken; + var rig = await StartAsync("EvtPagingWireRev", encrypted: false, ct); + try + { + var events = Enumerable.Range(0, 25) + .Select(i => new HistoricalEvent($"e{i:00}", "src", Epoch.AddSeconds(i / 2), Epoch, "m", 500)) + .ToList(); // pairs share a timestamp, so page boundaries cut through ties + rig.Nm.HistorianDataSource = new SqlLikeEventSource(events); + rig.MaterialiseNative(); + rig.Enforce(false, [Key()]); + + using var session = await rig.ConnectAsync(anonymous: true, ct); + var node = new NodeId(RawDeviceFolder, (ushort)session.NamespaceUris.GetIndex(V3NodeIds.RawNamespaceUri)); + + var collected = new List(); + var sizes = new List(); + byte[]? cp = null; + do + { + var (ids, next) = await ReadPageAsync(session, node, cp, ct, reverse: true); + collected.AddRange(ids); + sizes.Add(ids.Count); + cp = next; + sizes.Count.ShouldBeLessThan(30, "paging must terminate"); + } + while (cp is not null); + + sizes.ShouldBe([10, 10, 5]); + // Newest-first: time descending, ties by EventId descending. + collected.ShouldBe(events.OrderByDescending(e => e.EventTimeUtc) + .ThenByDescending(e => e.EventId, StringComparer.Ordinal).Select(e => e.EventId)); + } + finally { await rig.DisposeAsync(); } + } + private static async Task<(List Ids, byte[]? Cp)> ReadPageAsync( - ISession session, NodeId node, byte[]? cp, CancellationToken ct) + ISession session, NodeId node, byte[]? cp, CancellationToken ct, bool reverse = false) { var filter = new EventFilter(); filter.SelectClauses.Add(new SimpleAttributeOperand(ObjectTypeIds.BaseEventType, new QualifiedName("EventId")) @@ -64,8 +104,8 @@ public sealed class EventHistoryPagingOverTheWireTests }); var details = new ReadEventDetails { - StartTime = Epoch.AddDays(-1), - EndTime = Epoch.AddDays(1), + StartTime = reverse ? Epoch.AddDays(1) : Epoch.AddDays(-1), + EndTime = reverse ? Epoch.AddDays(-1) : Epoch.AddDays(1), NumValuesPerNode = 10, Filter = filter, }; @@ -94,6 +134,18 @@ public sealed class EventHistoryPagingOverTheWireTests return new HistoricalEventsResult(window.Take(cap).ToList(), window.Count > cap ? new byte[] { 1 } : null); } + /// The #675 overload: reverse is (lo, hi] by (time, id) descending. + public async Task ReadEventsAsync( + string? sourceName, DateTime startUtc, DateTime endUtc, int maxEvents, bool reverse, CancellationToken ct) + { + if (!reverse) return await ReadEventsAsync(sourceName, startUtc, endUtc, maxEvents, ct).ConfigureAwait(false); + await Task.Delay(1, ct).ConfigureAwait(false); + var (lo, hi) = startUtc <= endUtc ? (startUtc, endUtc) : (endUtc, startUtc); + var window = _all.Where(e => e.EventTimeUtc > lo && e.EventTimeUtc <= hi).Reverse().ToList(); + var cap = maxEvents <= 0 ? 1000 : maxEvents; + return new HistoricalEventsResult(window.Take(cap).ToList(), window.Count > cap ? new byte[] { 1 } : null); + } + public Task ReadRawAsync(string r, DateTime s, DateTime e, uint m, CancellationToken ct) => NullHistorianDataSource.Instance.ReadRawAsync(r, s, e, m, ct); public Task ReadProcessedAsync( diff --git a/tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests/History/HistoryUpdateTests.cs b/tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests/History/HistoryUpdateTests.cs index 4448d22c..5f1cb99d 100644 --- a/tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests/History/HistoryUpdateTests.cs +++ b/tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests/History/HistoryUpdateTests.cs @@ -558,6 +558,190 @@ public sealed class HistoryUpdateTests : IDisposable finally { await host.DisposeAsync(); } } + // ───────────────────────────── #674 future-skew bound ───────────────────────────── + + /// + /// #674: an item whose SourceTimestamp lies beyond now + HistoryUpdateMaxFutureSkew is refused + /// BadOutOfRange by the node manager and never reaches the backend; an item exactly AT the bound and one in + /// the past are sent. Judged against the injected clock, not the wall clock. + /// + [Fact] + public async Task A_source_time_beyond_the_future_skew_is_bad_out_of_range_and_never_sent() + { + var (host, server) = await BootAsync(); + try + { + var nm = server.NodeManager!; + var updater = new FakeUpdater(); + nm.HistorianDataUpdater = updater; + nm.Clock = new FakeClock(Start); + nm.HistoryUpdateMaxFutureSkew = TimeSpan.FromHours(1); + WireAcl(nm, enabled: false, NodePermissions.HistoryUpdate); + var nodeId = Historized(nm, AddressSpaceRealm.Raw, RawPath); + var atBound = Start.AddHours(1); + + var (results, errors) = Update(nm, KeyIdentity(OpcUaDataPlaneRoles.HistoryUpdate), nodeId, + PerformUpdateType.Update, Value(T1, 1d), Value(atBound.AddTicks(1), 2d), Value(atBound, 3d), + Value(new DateTime(2062, 1, 1, 0, 0, 0, DateTimeKind.Utc), 4d)); + + errors[0].StatusCode.Code.ShouldBe(StatusCodes.Good); + results[0].OperationResults.Select(c => c.Code).ShouldBe( + [ + StatusCodes.GoodEntryInserted, StatusCodes.BadOutOfRange, StatusCodes.GoodEntryInserted, + StatusCodes.BadOutOfRange, + ]); + updater.UpdateCalls.ShouldHaveSingleItem().Values.Select(v => v.TimestampUtc).ShouldBe([T1, atBound]); + } + finally { await host.DisposeAsync(); } + } + + /// #674: every mode is bounded, and a request whose items are ALL out of range sends nothing. + [Theory] + [InlineData(PerformUpdateType.Insert)] + [InlineData(PerformUpdateType.Replace)] + [InlineData(PerformUpdateType.Update)] + public async Task Every_update_mode_is_bounded_and_an_all_refused_request_sends_nothing(PerformUpdateType type) + { + var (host, server) = await BootAsync(); + try + { + var nm = server.NodeManager!; + var updater = new FakeUpdater(); + nm.HistorianDataUpdater = updater; + nm.Clock = new FakeClock(Start); + WireAcl(nm, enabled: false, NodePermissions.HistoryUpdate); + var nodeId = Historized(nm, AddressSpaceRealm.Raw, RawPath); + + // Default skew: one day. + var (results, _) = Update(nm, KeyIdentity(OpcUaDataPlaneRoles.HistoryUpdate), nodeId, type, + Value(Start.AddDays(1).AddSeconds(1), 1d)); + + results[0].OperationResults.Select(c => c.Code).ShouldBe([StatusCodes.BadOutOfRange]); + updater.UpdateCalls.ShouldBeEmpty(); + } + finally { await host.DisposeAsync(); } + } + + /// #674: a non-positive skew disables the bound — a far-future item reaches the backend. + [Fact] + public async Task A_non_positive_skew_disables_the_bound() + { + var (host, server) = await BootAsync(); + try + { + var nm = server.NodeManager!; + var updater = new FakeUpdater(); + nm.HistorianDataUpdater = updater; + nm.Clock = new FakeClock(Start); + nm.HistoryUpdateMaxFutureSkew = TimeSpan.Zero; + WireAcl(nm, enabled: false, NodePermissions.HistoryUpdate); + var nodeId = Historized(nm, AddressSpaceRealm.Raw, RawPath); + var farFuture = new DateTime(2062, 1, 1, 0, 0, 0, DateTimeKind.Utc); + + var (results, _) = Insert(nm, KeyIdentity(OpcUaDataPlaneRoles.HistoryUpdate), nodeId, Value(farFuture, 1d)); + + results[0].OperationResults.Select(c => c.Code).ShouldBe([StatusCodes.GoodEntryInserted]); + updater.UpdateCalls.ShouldHaveSingleItem().Values.ShouldHaveSingleItem().TimestampUtc.ShouldBe(farFuture); + } + finally { await host.DisposeAsync(); } + } + + // ───────────────────────────── #674 Primary gate ───────────────────────────── + + /// + /// #674: off-Primary, an authorized HistoryUpdate is refused BadInvalidState per node — on every arm + /// (update data, delete raw, delete at time) — and NOTHING reaches the updater. + /// + [Fact] + public async Task Off_primary_every_arm_is_bad_invalid_state_and_nothing_is_sent() + { + var (host, server) = await BootAsync(); + try + { + var nm = server.NodeManager!; + var updater = new FakeUpdater(); + nm.HistorianDataUpdater = updater; + var consulted = 0; + nm.HistoryUpdatePrimaryGate = () => { Interlocked.Increment(ref consulted); return false; }; + WireAcl(nm, enabled: false, NodePermissions.HistoryUpdate); + var nodeId = Historized(nm, AddressSpaceRealm.Raw, RawPath); + var identity = KeyIdentity(OpcUaDataPlaneRoles.HistoryUpdate); + + var (results, errors) = Insert(nm, identity, nodeId, Value(T1, 1d)); + var (rawResults, rawErrors) = Call(nm, identity, typeof(DeleteRawModifiedDetails), + new DeleteRawModifiedDetails { NodeId = nodeId, StartTime = T1, EndTime = T2 }); + var (atResults, atErrors) = Call(nm, identity, typeof(DeleteAtTimeDetails), + new DeleteAtTimeDetails { NodeId = nodeId, ReqTimes = new DateTimeCollection { T1 } }); + + foreach (var (r, e) in new[] { (results, errors), (rawResults, rawErrors), (atResults, atErrors) }) + { + e[0].StatusCode.Code.ShouldBe(StatusCodes.BadInvalidState); + r[0].StatusCode.Code.ShouldBe(StatusCodes.BadInvalidState); + } + + consulted.ShouldBe(3); + updater.UpdateCalls.ShouldBeEmpty(); + updater.DeleteRawCalls.ShouldBeEmpty(); + updater.DeleteAtTimeCalls.ShouldBeEmpty(); + } + finally { await host.DisposeAsync(); } + } + + /// + /// #674: on the Primary the gate allows and the update REACHES the updater — the allow leg that makes the deny + /// leg meaningful (a gate that refused everything, or one never consulted, fails one of the two). + /// + [Fact] + public async Task On_primary_the_update_reaches_the_updater() + { + var (host, server) = await BootAsync(); + try + { + var nm = server.NodeManager!; + var updater = new FakeUpdater(); + nm.HistorianDataUpdater = updater; + var consulted = 0; + nm.HistoryUpdatePrimaryGate = () => { Interlocked.Increment(ref consulted); return true; }; + WireAcl(nm, enabled: false, NodePermissions.HistoryUpdate); + var nodeId = Historized(nm, AddressSpaceRealm.Raw, RawPath); + + var (results, errors) = Insert(nm, KeyIdentity(OpcUaDataPlaneRoles.HistoryUpdate), nodeId, Value(T1, 1d)); + + errors[0].StatusCode.Code.ShouldBe(StatusCodes.Good); + results[0].OperationResults.Select(c => c.Code).ShouldBe([StatusCodes.GoodEntryInserted]); + updater.UpdateCalls.ShouldHaveSingleItem().Tag.ShouldBe(Tagname); + consulted.ShouldBe(1); + } + finally { await host.DisposeAsync(); } + } + + /// + /// #674: the Primary gate runs AFTER authorization — an unauthorized caller is told BadUserAccessDenied + /// (not the node's role) and the gate is never consulted. + /// + [Fact] + public async Task The_primary_gate_runs_after_authorization() + { + var (host, server) = await BootAsync(); + try + { + var nm = server.NodeManager!; + var updater = new FakeUpdater(); + nm.HistorianDataUpdater = updater; + var consulted = 0; + nm.HistoryUpdatePrimaryGate = () => { Interlocked.Increment(ref consulted); return false; }; + WireAcl(nm, enabled: false, NodePermissions.WriteOperate); + var nodeId = Historized(nm, AddressSpaceRealm.Raw, RawPath); + + var (results, _) = Insert(nm, KeyIdentity(OpcUaDataPlaneRoles.WriteOperate), nodeId, Value(T1, 1d)); + + results[0].StatusCode.Code.ShouldBe(StatusCodes.BadUserAccessDenied); + consulted.ShouldBe(0); + updater.UpdateCalls.ShouldBeEmpty(); + } + finally { await host.DisposeAsync(); } + } + // ───────────────────────────── helpers ───────────────────────────── private static DataValue Value(DateTime timestamp, object value) => diff --git a/tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests/NodeManagerHistoryReadEventsPagingTests.cs b/tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests/NodeManagerHistoryReadEventsPagingTests.cs index 6cd34534..8736fd0c 100644 --- a/tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests/NodeManagerHistoryReadEventsPagingTests.cs +++ b/tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests/NodeManagerHistoryReadEventsPagingTests.cs @@ -199,11 +199,168 @@ public sealed class NodeManagerHistoryReadEventsPagingTests : IDisposable var mixed = ReadBatch(nm, 4, null, (node, good1[0].Cp), (other, Guid.NewGuid().ToByteArray())); mixed[0].Error.StatusCode.Code.ShouldBe(StatusCodes.Good); - mixed[0].Ids.ShouldBe(src.All.Skip(4).Take(3).Select(e => e.EventId)); + // A resumed page is FULL (#675): it over-fetches by the boundary skip it then trims. + mixed[0].Ids.ShouldBe(src.All.Skip(4).Take(4).Select(e => e.EventId)); mixed[1].Error.StatusCode.Code.ShouldBe(StatusCodes.BadContinuationPointInvalid); await host.DisposeAsync(); } + // --- Gitea #675: reverse (newest-first) event paging ------------------------------------------------------ + + [Fact] + public async Task Reverse_events_page_newest_first_exactly_once_with_exact_page_counts() + { + var (host, nm, node, src) = await SetupAsync(Distinct(250)); + var pages = new List(); + var ids = DrainAsync(nm, node, max: 100, start: Late, end: Early, pageSizes: pages); + ids.ShouldBe(src.All.Reverse().Select(e => e.EventId)); + pages.ShouldBe([100, 100, 50]); + await host.DisposeAsync(); + } + + [Fact] + public async Task Reverse_first_page_is_the_LATEST_events_with_a_continuation_point() + { + // The pre-#675 shape: the cap kept the EARLIEST N, ascending, Good, and no continuation point. + var (host, nm, node, _) = await SetupAsync(Distinct(10)); + var p1 = ReadEvents(nm, node, max: 3, inboundCp: null, end: Early, start: Late); + p1.Error.StatusCode.Code.ShouldBe(StatusCodes.Good); + p1.Ids.ShouldBe(["e0009", "e0008", "e0007"]); + p1.Cp.ShouldNotBeNull("a capped reverse read has more behind it and must page"); + await host.DisposeAsync(); + } + + [Fact] + public async Task Reverse_tie_at_page_boundary_is_neither_duplicated_nor_skipped() + { + var events = new List + { + Ev("e0", 0), Ev("e1", 1), Ev("e2", 2), Ev("e3", 2), Ev("e4", 2), Ev("e5", 3), + }; + var (host, nm, node, _) = await SetupAsync(events); + // Newest-first, ties in descending EventId: cap 2 cuts the cluster at e4|e3. + DrainAsync(nm, node, max: 2, start: Late, end: Early).ShouldBe(["e5", "e4", "e3", "e2", "e1", "e0"]); + await host.DisposeAsync(); + } + + [Fact] + public async Task Reverse_tie_cluster_larger_than_the_cap_pages_within_the_cluster() + { + var events = new List { Ev("a0", 0) }; + for (var i = 0; i < 7; i++) events.Add(Ev($"t{i}", 5)); + events.Add(Ev("z0", 9)); + var (host, nm, node, _) = await SetupAsync(events); + var pages = new List(); + DrainAsync(nm, node, max: 3, start: Late, end: Early, pageSizes: pages).ShouldBe( + ["z0", "t6", "t5", "t4", "t3", "t2", "t1", "t0", "a0"]); + pages.Sum().ShouldBe(9); + await host.DisposeAsync(); + } + + [Fact] + public async Task Reverse_tie_cluster_pages_on_a_coarse_resolution_backend() + { + const long res = 10; + var events = new List(); + DateTime At(int slot) => Epoch.AddTicks(slot * res); + events.Add(new("a0", "s", At(0), Epoch, "m", 500)); + for (var i = 0; i < 7; i++) events.Add(new($"t{i}", "s", At(3), Epoch, "m", 500)); + events.Add(new("z0", "s", At(8), Epoch, "m", 500)); + var (host, nm, node, _) = await SetupAsync(events, res); + DrainAsync(nm, node, max: 3, start: Late, end: Early).ShouldBe( + ["z0", "t6", "t5", "t4", "t3", "t2", "t1", "t0", "a0"]); + await host.DisposeAsync(); + } + + [Fact] + public async Task Reverse_tie_cluster_beyond_the_overfetch_bound_fails_loudly() + { + var events = new List(); + for (var i = 0; i < 10; i++) events.Add(Ev($"t{i:00}", 5)); + var (host, nm, node, _) = await SetupAsync(events); + nm.MaxTieClusterOverfetch = 4; + var p1 = ReadEvents(nm, node, max: 3, inboundCp: null, end: Early, start: Late); + p1.Cp.ShouldNotBeNull(); + ReadEvents(nm, node, max: 3, inboundCp: p1.Cp, end: Early, start: Late).Error.StatusCode.Code + .ShouldBe(StatusCodes.BadHistoryOperationUnsupported); + await host.DisposeAsync(); + } + + [Fact] + public async Task A_reverse_cursor_resumes_reverse_whatever_the_resume_request_says() + { + // The direction lives in the cursor: a resume carrying FORWARD details still continues newest-first. + var (host, nm, node, _) = await SetupAsync(Distinct(10)); + var p1 = ReadEvents(nm, node, max: 4, inboundCp: null, end: Early, start: Late); + p1.Ids.ShouldBe(["e0009", "e0008", "e0007", "e0006"]); + var p2 = ReadEvents(nm, node, max: 4, inboundCp: p1.Cp); // default details: Early -> Late (forward) + p2.Error.StatusCode.Code.ShouldBe(StatusCodes.Good); + p2.Ids.ShouldBe(["e0005", "e0004", "e0003", "e0002"]); + await host.DisposeAsync(); + } + + [Fact] + public async Task A_forward_cursor_resumes_forward_whatever_the_resume_request_says() + { + var (host, nm, node, _) = await SetupAsync(Distinct(10)); + var p1 = ReadEvents(nm, node, max: 4, inboundCp: null); + p1.Ids.ShouldBe(["e0000", "e0001", "e0002", "e0003"]); + var p2 = ReadEvents(nm, node, max: 4, inboundCp: p1.Cp, end: Early, start: Late); + p2.Ids.ShouldBe(["e0004", "e0005", "e0006", "e0007"]); + await host.DisposeAsync(); + } + + [Fact] + public async Task Reverse_unbounded_read_returns_everything_newest_first_without_paging() + { + var (host, nm, node, src) = await SetupAsync(Distinct(30)); + var r = ReadEvents(nm, node, max: 0, inboundCp: null, end: Early, start: Late); + r.Ids.ShouldBe(src.All.Reverse().Select(e => e.EventId)); + r.Cp.ShouldBeNull(); + await host.DisposeAsync(); + } + + [Fact] + public async Task End_only_events_read_pages_backward_from_EndTime() + { + // StartTime unset + EndTime + NumValuesPerNode > 0 => backward from EndTime (Part 11; mirrors raw). The reverse + // window is (.., EndTime], so an event exactly AT EndTime is the first one returned. + var (host, nm, node, _) = await SetupAsync(Distinct(25)); + var ids = DrainAsync(nm, node, max: 10, start: DateTime.MinValue, end: Epoch.AddSeconds(20)); + ids.ShouldBe(Enumerable.Range(0, 21).Reverse().Select(i => $"e{i:0000}")); + await host.DisposeAsync(); + } + + [Fact] + public async Task A_backend_without_reverse_support_answers_BadHistoryOperationUnsupported_not_the_earliest_events() + { + var (host, nm, node, src) = await SetupAsync(Distinct(10)); + nm.HistorianDataSource = new ForwardOnlyEventSource(src); + ReadEvents(nm, node, max: 3, inboundCp: null, end: Early, start: Late).Error.StatusCode.Code + .ShouldBe(StatusCodes.BadHistoryOperationUnsupported); + // Forward reads on the same backend are unaffected. + DrainAsync(nm, node, max: 4).ShouldBe(src.All.Select(e => e.EventId)); + await host.DisposeAsync(); + } + + private static readonly DateTime Early = Epoch.AddDays(-1); + private static readonly DateTime Late = Epoch.AddDays(1); + + /// A backend that implements only the five-argument (forward) event read - Gateway / Null shaped. + private sealed class ForwardOnlyEventSource(IHistorianDataSource inner) : IHistorianDataSource + { + public Task ReadEventsAsync(string? n, DateTime s, DateTime e, int m, CancellationToken ct) => + inner.ReadEventsAsync(n, s, e, m, ct); + public Task ReadRawAsync(string r, DateTime s, DateTime e, uint m, CancellationToken ct) => + inner.ReadRawAsync(r, s, e, m, ct); + public Task ReadProcessedAsync(string r, DateTime s, DateTime e, TimeSpan i, HistoryAggregateType a, CancellationToken ct) => + inner.ReadProcessedAsync(r, s, e, i, a, ct); + public Task ReadAtTimeAsync(string r, IReadOnlyList t, CancellationToken ct) => + inner.ReadAtTimeAsync(r, t, ct); + public HistorianHealthSnapshot GetHealthSnapshot() => inner.GetHealthSnapshot(); + public void Dispose() { } + } + private sealed class RawSeriesSource : IHistorianDataSource { public Task ReadRawAsync(string r, DateTime s, DateTime e, uint m, CancellationToken ct) @@ -265,16 +422,18 @@ public sealed class NodeManagerHistoryReadEventsPagingTests : IDisposable return (started.Host, nm, nm.TryGetFolder(equipmentId)!.NodeId, src); } - private static List DrainAsync(OtOpcUaNodeManager nm, NodeId node, uint max, DateTime? end = null) + private static List DrainAsync( + OtOpcUaNodeManager nm, NodeId node, uint max, DateTime? end = null, DateTime? start = null, List? pageSizes = null) { var all = new List(); byte[]? cp = null; var pages = 0; do { - var r = ReadEvents(nm, node, max, cp, end); + var r = ReadEvents(nm, node, max, cp, end, start); r.Error.StatusCode.Code.ShouldBe(StatusCodes.Good); all.AddRange(r.Ids); + pageSizes?.Add(r.Ids.Count); cp = r.Cp; (++pages).ShouldBeLessThan(60, "paging must terminate"); } @@ -283,11 +442,15 @@ public sealed class NodeManagerHistoryReadEventsPagingTests : IDisposable } private static (List Ids, ServiceResult Error, byte[]? Cp) ReadEvents( - OtOpcUaNodeManager nm, NodeId node, uint max, byte[]? inboundCp, DateTime? end = null) => - ReadBatch(nm, max, end, (node, inboundCp))[0]; + OtOpcUaNodeManager nm, NodeId node, uint max, byte[]? inboundCp, DateTime? end = null, DateTime? start = null) => + ReadBatch(nm, max, end, start, (node, inboundCp))[0]; private static List<(List Ids, ServiceResult Error, byte[]? Cp)> ReadBatch( - OtOpcUaNodeManager nm, uint max, DateTime? end, params (NodeId Node, byte[]? Cp)[] reads) + OtOpcUaNodeManager nm, uint max, DateTime? end, params (NodeId Node, byte[]? Cp)[] reads) => + ReadBatch(nm, max, end, null, reads); + + private static List<(List Ids, ServiceResult Error, byte[]? Cp)> ReadBatch( + OtOpcUaNodeManager nm, uint max, DateTime? end, DateTime? start, params (NodeId Node, byte[]? Cp)[] reads) { var context = new OperationContext( new RequestHeader(), secureChannelContext: null, RequestType.HistoryRead, identity: null); @@ -298,7 +461,7 @@ public sealed class NodeManagerHistoryReadEventsPagingTests : IDisposable }); var details = new ReadEventDetails { - StartTime = Epoch.AddDays(-1), + StartTime = start ?? Epoch.AddDays(-1), EndTime = end ?? Epoch.AddDays(1), NumValuesPerNode = max, Filter = filter, @@ -361,6 +524,22 @@ public sealed class NodeManagerHistoryReadEventsPagingTests : IDisposable window.Take(cap).ToList(), truncated ? new byte[] { 1 } : null)); } + /// Like the SQL backends' #675 overload: reverse => (lo, hi] newest-first by (time, id) descending. + public Task ReadEventsAsync( + string? sourceName, DateTime startUtc, DateTime endUtc, int maxEvents, bool reverse, CancellationToken ct) + { + if (!reverse) return ReadEventsAsync(sourceName, startUtc, endUtc, maxEvents, ct); + Calls++; + var (lo, hi) = startUtc <= endUtc ? (startUtc, endUtc) : (endUtc, startUtc); + lo = Trunc(lo); + hi = Trunc(hi); + var window = All.Where(e => e.EventTimeUtc > lo && e.EventTimeUtc <= hi).Reverse().ToList(); + var cap = maxEvents <= 0 ? 1000 : maxEvents; + var truncated = window.Count > cap; + return Task.FromResult(new HistoricalEventsResult( + window.Take(cap).ToList(), truncated ? new byte[] { 1 } : null)); + } + public Task ReadRawAsync(string r, DateTime s, DateTime e, uint m, CancellationToken ct) => NullHistorianDataSource.Instance.ReadRawAsync(r, s, e, m, ct); public Task ReadProcessedAsync( diff --git a/tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests/NodeManagerHistoryReadModifiedTests.cs b/tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests/NodeManagerHistoryReadModifiedTests.cs index fc12cc7f..164638c1 100644 --- a/tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests/NodeManagerHistoryReadModifiedTests.cs +++ b/tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests/NodeManagerHistoryReadModifiedTests.cs @@ -203,6 +203,28 @@ public sealed class NodeManagerHistoryReadModifiedTests : IDisposable await host.DisposeAsync(); } + /// + /// #674: the HistoryUpdate Primary gate does not reach HistoryRead — a closed gate neither refuses a read nor is + /// consulted by one (reads are served from any redundancy node), for modified and raw reads alike. + /// + [Fact] + public async Task The_history_update_primary_gate_is_not_consulted_for_reads() + { + var (host, nm, node, src) = await SetupAsync([Mod(1, 1, 1, HistoryModificationType.Update, "u", Epoch)]); + var consulted = 0; + nm.HistoryUpdatePrimaryGate = () => { Interlocked.Increment(ref consulted); return false; }; + + var modified = Read(nm, node, Epoch, Epoch.AddHours(1), max: 10, cp: null); + var raw = Read(nm, node, Epoch, Epoch.AddHours(1), max: 10, cp: null, modified: false); + + modified.Error.StatusCode.Code.ShouldBe(StatusCodes.Good); + modified.Data!.DataValues.Count.ShouldBe(1); + raw.Error.StatusCode.Code.ShouldNotBe(StatusCodes.BadInvalidState); + src.Calls.ShouldBe(1); + consulted.ShouldBe(0); + await host.DisposeAsync(); + } + [Fact] public async Task A_backend_failure_is_isolated_to_its_node() { diff --git a/tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests/NodeManagerHistoryReadTests.cs b/tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests/NodeManagerHistoryReadTests.cs index b23366a3..d2b0110f 100644 --- a/tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests/NodeManagerHistoryReadTests.cs +++ b/tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests/NodeManagerHistoryReadTests.cs @@ -165,6 +165,7 @@ public sealed class NodeManagerHistoryReadTests : IDisposable StartTime = DateTime.UtcNow.AddHours(-1), EndTime = DateTime.UtcNow, NumValuesPerNode = 10, + IsReadModified = false, // SDK default is true; this test is about the raw path }; var (_, errors) = InvokeHistoryRead(server, nm, details, nodeId); diff --git a/tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/Drivers/DriverHostActorHistorizationRoleViewTests.cs b/tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/Drivers/DriverHostActorHistorizationRoleViewTests.cs index 0ca0dcac..dee3ccec 100644 --- a/tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/Drivers/DriverHostActorHistorizationRoleViewTests.cs +++ b/tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/Drivers/DriverHostActorHistorizationRoleViewTests.cs @@ -223,10 +223,30 @@ public sealed class DriverHostActorHistorizationRoleViewTests : RuntimeActorTest host.Tell(new ClusterEvent.UnreachableMember(MemberAt(Peer672))); AwaitAssert(() => hist.Published.ShouldBe([false, true]), duration: Timeout); - // And self "returning" does not close what the peer's loss opened. + // Self "returning" is ignored too, and it does not close what the peer's loss opened. Events are + // processed in order, so the peer's return (which MUST close the gate) is the only thing that may + // appear next: had the self event published anything it would sit between `true` and `false`. host.Tell(new ClusterEvent.ReachableMember(MemberAt(Self672))); + host.Tell(new ClusterEvent.ReachableMember(MemberAt(Peer672))); + AwaitAssert(() => hist.Published.ShouldBe([false, true, false]), duration: Timeout); + } + + /// + /// A host that starts while its peer is ALREADY unreachable (the InitialStateAsEvents replay + /// delivers the UnreachableMember before the first redundancy snapshot) must still treat that + /// peer as unreachable when the snapshot naming it Primary finally arrives. + /// + [Fact] + public void A_peer_unreachable_before_the_first_snapshot_still_opens_capture() + { + var hist = new RecordingHistorizationView(); + IActorRef host = SpawnHost672(hist, others: null, captureWhilePrimaryUnreachable: true); + + host.Tell(new ClusterEvent.UnreachableMember(MemberAt(Peer672))); host.Tell(Snapshot((Self672, RedundancyRole.Secondary), (Peer672, RedundancyRole.Primary))); - AwaitAssert(() => hist.Published.ShouldBe([false, true, true]), duration: Timeout); + + // Exactly one publication, and it is `true`: no spurious `false` from treating the peer as reachable. + AwaitAssert(() => hist.Published.ShouldBe([true]), duration: Timeout); } /// CaptureWhilePrimaryUnreachable = false restores the pre-#672 behaviour. diff --git a/tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/Historian/ServerHistorianOptionsTests.cs b/tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/Historian/ServerHistorianOptionsTests.cs index 4f38cc8a..767ec21f 100644 --- a/tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/Historian/ServerHistorianOptionsTests.cs +++ b/tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/Historian/ServerHistorianOptionsTests.cs @@ -1,3 +1,4 @@ +using Microsoft.Extensions.Configuration; using Xunit; using ZB.MOM.WW.OtOpcUa.Runtime.Historian; @@ -99,6 +100,29 @@ public sealed class ServerHistorianOptionsTests Assert.Contains(w, m => m.Contains("MaxTimeWeightedRawRows")); } + /// + /// #674: the HistoryUpdate future-skew bound defaults to one day and binds from configuration as a TimeSpan; + /// a non-positive value is a deliberate "disabled", not a misconfiguration, so it does not warn. + /// + [Fact] + public void HistoryUpdateMaxFutureSkew_defaults_to_one_day_binds_and_zero_is_silent() + { + Assert.Equal(TimeSpan.FromDays(1), new ServerHistorianOptions().HistoryUpdateMaxFutureSkew); + var bound = new ConfigurationBuilder() + .AddInMemoryCollection(new Dictionary + { + ["ServerHistorian:HistoryUpdateMaxFutureSkew"] = "00:05:00", + }) + .Build() + .GetSection(ServerHistorianOptions.SectionName) + .Get(); + Assert.Equal(TimeSpan.FromMinutes(5), bound!.HistoryUpdateMaxFutureSkew); + var w = new ServerHistorianOptions + { Enabled = true, Endpoint = "https://h:5222", ApiKey = "histgw_x_y", HistoryUpdateMaxFutureSkew = TimeSpan.Zero } + .Validate(); + Assert.Empty(w); + } + [Fact] public void Disabled_with_nonpositive_HistoryRead_knobs_is_silent() {